Patent · US10151588B1 · B1 · US

Determining position and orientation for aerial vehicle in GNSS-denied situations

(11) Publication number

US10151588B1

(21) Application number

15/711,492

(22) Filing date

2017-09-21

(30) Priority date

2016-09-28

(43) Publication date

2018-12-11

(45) Date of grant

2018-12-11

(51) IPC

G01C 21/16

(52) CPC

G01C Measuring distances, levels or bearings; surveying; navigation; gyroscopic instruments; photogrammetry or videogrammetry: 21/1656, 21/005, 21/165, 21/1652
B64C Aeroplanes; helicopters: 27/04
B64D Equipment for fitting in or to aircraft; flight suits; parachutes; arrangement or mounting of power plants or propulsion transmissions in aircraft: 45/00
G01S Radio direction-finding; radio navigation; determining distance or velocity by use of radio waves; locating or presence-detecting by use of the reflection or reradiation of radio waves; analogous arrangements using other waves: 17/00, 17/86, 17/89
G05D Systems for controlling or regulating non-electric variables: 1/101, 1/106
G06T Image data processing or generation, in general: 2207/10016, 2207/10024, 2207/10028, 2207/10032, 2207/30181, 2207/30244, 2207/30252, 7/246, 7/248, 7/251, 7/277, 7/337, 7/74, 7/75

(73) Assignee

Near Earth Autonomy Inc

(72) Inventors

Sanjiv Singh; Jeffrey Mishler; Michael Kaess; Garrett Hemann

(54) Title

Determining position and orientation for aerial vehicle in GNSS-denied situations

(57) Abstract

On-board, computer-based systems and methods compute continuously updated, real-time state estimates for an aerial vehicle by appropriately combining, by a suitable Kalman filter, local, relative, continuous state estimates with global, absolute, noncontinuous state estimates. The local, relative, continuous state estimates can be provided by visual odometry (VO) and/or an inertial measurement unit (IMU). The global, absolute, noncontinuous state estimates can be provided by terrain-referenced navigation, such as map-matching, and GNSS. The systems and methods can provide the real-time, continuous estimates even when reliable GNSS coordinate data is not available.

Full text

View on Google Patents

Claims (13)

An on-board state estimation system for an aerial vehicle, the system comprising: a computer database that stores a pre-loaded digital elevation model (DEM) for a ground surface over which the aerial vehicle is to fly; a sensor suite comprising one or more sensors, wherein the one or more sensors comprise: an altitude sensor for sensing an above-ground altitude of the aerial vehicle; one or more downward-pointed cameras such that the one or more cameras are pointed at the ground when the aerial vehicle is in the air; and at least one downward-pointing laser scanner such that the at least one downward-pointing laser scanner is pointed at the ground surface when the aerial vehicle is in the air; means, responsive to the sensor suite, for determining local, relative, continuous state estimates of the aerial vehicle as it moves, wherein the means for determining the local, relative, continuous state estimates comprise: a visual odometry (VO) system that is in communication with the one or more downward-pointed cameras and the altitude sensor, wherein the VO system continuously computes updated local state estimates of the aerial vehicle by comparing time-stamped ground surface images taken at different points in time, using the sensed altitude of the aerial vehicle from the altitude sensor for scaling the time-stamped ground surface images; and an inertial measurement unit (IMU) that is in communication with the one or more motion sensors and continuously measures acceleration of the aerial vehicle in three dimensions and angular velocities for roll, pitch and yaw for the aerial vehicle based on input from motion sensors that detect motion of the vehicle; means, responsive to the sensor suite, for determining global, absolute, noncontinuous state estimates of the aerial vehicle as it moves, wherein the means for determining the global, absolute, noncontinuous state estimates comprise terrain referenced navigation means; and state estimation means for determining continuously updated state estimates for the aerial vehicle as it moves without a need for GNSS coordinates by combining the local, relative, continuous state estimates with the global, absolute, noncontinuous state estimates.
The on-board state estimation system of claim 1, wherein the state estimation means comprises a Kalman filter.
The on-board state estimation system of claim 1, wherein the terrain referenced navigation means determines the global, absolute, noncontinuous state estimates of the aerial vehicle by comparing a DEM generated from data points from laser scans from the at least one downward-pointing laser scanner to the pre-loaded DEM stored in the computer database to find a sufficient match.
The on-board state estimation system of claim 3, wherein the state estimation means comprises a Kalman filter.
The on-board state estimation system of claim 4, wherein the terrain referenced navigation means uses a current state estimate of the aerial vehicle to constrain searching of the pre-loaded DEM to find a sufficient match between the pre-loaded DEM and the DEM generated from the data points from the laser scans.
The on-board state estimation system of claim 4, wherein the at least one downward-pointing laser scanner comprises at least one 2D laser scanner.
The on-board state estimation system of claim 4, wherein the at least one downward-pointing laser scanner comprises a moveable laser scanner that points downward at the ground surface for a sufficient period of time for the terrain referenced navigation means while the aerial vehicle moves.
The on-board state estimation system of claim 4, further comprising a GNSS receiver, wherein the state estimation means further uses GNSS coordinates from the GNSS receiver, when available, to determine the continuously updated state estimates for the aerial vehicle as it moves.
An aerial vehicle comprising: a sensor suite that comprises one or more sensors, wherein the one or more sensors comprise: an altitude sensor for sensing an above-ground altitude of the aerial vehicle; one or more downward-pointed cameras such that the one or more cameras are pointed at the ground when the aerial vehicle is in the air; and at least one downward-pointing laser scanner such that the at least one downward-pointing laser scanner is pointed at the ground surface when the aerial vehicle is in the air; and an inertial measurement unit (IMU) that is in communication with the one or more motion sensors and continuously measures acceleration of the aerial vehicle in three dimensions and angular velocities for roll, pitch and yaw for the aerial vehicle based on input from motion sensors that detect motion of the vehicle; and a computer system in communication with the sensor suite, wherein the computer system comprises: a computer database that stores a pre-loaded digital elevation model (DEM) for a ground surface over which the aerial vehicle is to fly; at least one processor; and at least one memory unit that stores software that is executed by at least one processor, wherein the software comprises: a visual odometry (VO) module that that continuously computes updated local state estimates of the aerial vehicle by comparing time-stamped ground surface images taken at different points in time by the one or more downward-pointed cameras, using a sensed altitude of the aerial vehicle from the altitude sensor for scaling the time-stamped ground surface images; a terrain referenced navigation module that determines global, absolute, noncontinuous state estimates of the aerial vehicle by comparing a DEM generated from data points from laser scans of the ground surface by the at least one downward-pointing laser scanner to the pre-loaded DEM stored in the computer database to find a sufficient match; and a state estimation module for determining continuously updated state estimates for the aerial vehicle as it moves without a need for GNSS coordinates by combining the local, continuous state estimates from the VO module with outputs from the IMU and with the global, absolute, noncontinuous state estimates from the terrain referenced navigation module.
The aerial vehicle of claim 9, further comprising a guidance system, wherein the continuously updated state estimates for the aerial vehicle from the state estimation module are input to the guidance system.
The aerial vehicle of claim 9, further comprising a computerized navigational system, wherein the continuously updated state estimates for the aerial vehicle from the state estimation module are input to the computerized navigational system.
The aerial vehicle of claim 11, wherein the state estimation module comprises a Kalman filter.
The aerial vehicle of claim 12, wherein the aerial vehicle comprises a rotorcraft.

Description

In many applications it is necessary to determine the position, velocity, and orientation (“pose” or “state”) in real-time of an aerial vehicle, such as an aircraft, particularly an autonomous aircraft. Global Navigation Satellite System (GNSS) coordinates (e.g., GPS coordinates in the U.S.) are often used in this process. However, sometimes GNSS signals can be lost (i.e., not received or jammed) or there can be errors in the received GNSS coordinates (including through spoofing).

In one general aspect, the present invention is directed to on-board, computer-based systems and methods that compute continuously updated, real-time state estimates for an aerial vehicle by appropriately combining, by a suitable Kalman filter or other suitable sensor fusion algorithm, local, relative, continuous state measurements with global, absolute, noncontinuous state measurements. The local, relative, continuous state measurements can be provided by visual odometry (VO) and/or an inertial measurement unit (IMU). The global, absolute, noncontinuous state estimates can be provided by terrain-referenced navigation techniques, such as map-matching, or GNSS. Moreover, the systems and methods described herein can provide the real-time, continuous state estimates even when reliable GNSS coordinate data are not available. The systems and method described herein are particularly useful for state estimation of an aircraft, such as an autonomous aircraft.

Various embodiments of the present invention are described herein by way of example in connection with the following figures, wherein:

Citations (8)

US9389298B2
US8315794B1
US8010287B1
US20140088790A1
US20140022262A1
US20160140729A1
US20160335901A1
WO2017042672A1

Record as JSON

{
  "publication_number": "US10151588B1",
  "country": "US",
  "kind": "B1",
  "title": "Determining position and orientation for aerial vehicle in GNSS-denied situations",
  "abstract": "On-board, computer-based systems and methods compute continuously updated, real-time state estimates for an aerial vehicle by appropriately combining, by a suitable Kalman filter, local, relative, continuous state estimates with global, absolute, noncontinuous state estimates. The local, relative, continuous state estimates can be provided by visual odometry (VO) and/or an inertial measurement unit (IMU). The global, absolute, noncontinuous state estimates can be provided by terrain-referenced navigation, such as map-matching, and GNSS. The systems and methods can provide the real-time, continuous estimates even when reliable GNSS coordinate data is not available.",
  "claims": [
    "1. An on-board state estimation system for an aerial vehicle, the system comprising: a computer database that stores a pre-loaded digital elevation model (DEM) for a ground surface over which the aerial vehicle is to fly; a sensor suite comprising one or more sensors, wherein the one or more sensors comprise: an altitude sensor for sensing an above-ground altitude of the aerial vehicle; one or more downward-pointed cameras such that the one or more cameras are pointed at the ground when the aerial vehicle is in the air; and at least one downward-pointing laser scanner such that the at least one downward-pointing laser scanner is pointed at the ground surface when the aerial vehicle is in the air; means, responsive to the sensor suite, for determining local, relative, continuous state estimates of the aerial vehicle as it moves, wherein the means for determining the local, relative, continuous state estimates comprise: a visual odometry (VO) system that is in communication with the one or more downward-pointed cameras and the altitude sensor, wherein the VO system continuously computes updated local state estimates of the aerial vehicle by comparing time-stamped ground surface images taken at different points in time, using the sensed altitude of the aerial vehicle from the altitude sensor for scaling the time-stamped ground surface images; and an inertial measurement unit (IMU) that is in communication with the one or more motion sensors and continuously measures acceleration of the aerial vehicle in three dimensions and angular velocities for roll, pitch and yaw for the aerial vehicle based on input from motion sensors that detect motion of the vehicle; means, responsive to the sensor suite, for determining global, absolute, noncontinuous state estimates of the aerial vehicle as it moves, wherein the means for determining the global, absolute, noncontinuous state estimates comprise terrain referenced navigation means; and state estimation means for determining continuously updated state estimates for the aerial vehicle as it moves without a need for GNSS coordinates by combining the local, relative, continuous state estimates with the global, absolute, noncontinuous state estimates.",
    "2. The on-board state estimation system of claim 1, wherein the state estimation means comprises a Kalman filter.",
    "3. The on-board state estimation system of claim 1, wherein the terrain referenced navigation means determines the global, absolute, noncontinuous state estimates of the aerial vehicle by comparing a DEM generated from data points from laser scans from the at least one downward-pointing laser scanner to the pre-loaded DEM stored in the computer database to find a sufficient match.",
    "4. The on-board state estimation system of claim 3, wherein the state estimation means comprises a Kalman filter.",
    "5. The on-board state estimation system of claim 4, wherein the terrain referenced navigation means uses a current state estimate of the aerial vehicle to constrain searching of the pre-loaded DEM to find a sufficient match between the pre-loaded DEM and the DEM generated from the data points from the laser scans.",
    "6. The on-board state estimation system of claim 4, wherein the at least one downward-pointing laser scanner comprises at least one 2D laser scanner.",
    "7. The on-board state estimation system of claim 4, wherein the at least one downward-pointing laser scanner comprises a moveable laser scanner that points downward at the ground surface for a sufficient period of time for the terrain referenced navigation means while the aerial vehicle moves.",
    "8. The on-board state estimation system of claim 4, further comprising a GNSS receiver, wherein the state estimation means further uses GNSS coordinates from the GNSS receiver, when available, to determine the continuously updated state estimates for the aerial vehicle as it moves.",
    "9. An aerial vehicle comprising: a sensor suite that comprises one or more sensors, wherein the one or more sensors comprise: an altitude sensor for sensing an above-ground altitude of the aerial vehicle; one or more downward-pointed cameras such that the one or more cameras are pointed at the ground when the aerial vehicle is in the air; and at least one downward-pointing laser scanner such that the at least one downward-pointing laser scanner is pointed at the ground surface when the aerial vehicle is in the air; and an inertial measurement unit (IMU) that is in communication with the one or more motion sensors and continuously measures acceleration of the aerial vehicle in three dimensions and angular velocities for roll, pitch and yaw for the aerial vehicle based on input from motion sensors that detect motion of the vehicle; and a computer system in communication with the sensor suite, wherein the computer system comprises: a computer database that stores a pre-loaded digital elevation model (DEM) for a ground surface over which the aerial vehicle is to fly; at least one processor; and at least one memory unit that stores software that is executed by at least one processor, wherein the software comprises: a visual odometry (VO) module that that continuously computes updated local state estimates of the aerial vehicle by comparing time-stamped ground surface images taken at different points in time by the one or more downward-pointed cameras, using a sensed altitude of the aerial vehicle from the altitude sensor for scaling the time-stamped ground surface images; a terrain referenced navigation module that determines global, absolute, noncontinuous state estimates of the aerial vehicle by comparing a DEM generated from data points from laser scans of the ground surface by the at least one downward-pointing laser scanner to the pre-loaded DEM stored in the computer database to find a sufficient match; and a state estimation module for determining continuously updated state estimates for the aerial vehicle as it moves without a need for GNSS coordinates by combining the local, continuous state estimates from the VO module with outputs from the IMU and with the global, absolute, noncontinuous state estimates from the terrain referenced navigation module.",
    "10. The aerial vehicle of claim 9, further comprising a guidance system, wherein the continuously updated state estimates for the aerial vehicle from the state estimation module are input to the guidance system.",
    "11. The aerial vehicle of claim 9, further comprising a computerized navigational system, wherein the continuously updated state estimates for the aerial vehicle from the state estimation module are input to the computerized navigational system.",
    "12. The aerial vehicle of claim 11, wherein the state estimation module comprises a Kalman filter.",
    "13. The aerial vehicle of claim 12, wherein the aerial vehicle comprises a rotorcraft."
  ],
  "description_excerpt": "In many applications it is necessary to determine the position, velocity, and orientation (“pose” or “state”) in real-time of an aerial vehicle, such as an aircraft, particularly an autonomous aircraft. Global Navigation Satellite System (GNSS) coordinates (e.g., GPS coordinates in the U.S.) are often used in this process. However, sometimes GNSS signals can be lost (i.e., not received or jammed) or there can be errors in the received GNSS coordinates (including through spoofing).\n\nIn one general aspect, the present invention is directed to on-board, computer-based systems and methods that compute continuously updated, real-time state estimates for an aerial vehicle by appropriately combining, by a suitable Kalman filter or other suitable sensor fusion algorithm, local, relative, continuous state measurements with global, absolute, noncontinuous state measurements. The local, relative, continuous state measurements can be provided by visual odometry (VO) and/or an inertial measurement unit (IMU). The global, absolute, noncontinuous state estimates can be provided by terrain-referenced navigation techniques, such as map-matching, or GNSS. Moreover, the systems and methods described herein can provide the real-time, continuous state estimates even when reliable GNSS coordinate data are not available. The systems and method described herein are particularly useful for state estimation of an aircraft, such as an autonomous aircraft.\n\nVarious embodiments of the present invention are described herein by way of example in connection with the following figures, wherein:",
  "cpc": [
    "G01C 21/1656",
    "B64C 27/04",
    "B64D 45/00",
    "G01C 21/005",
    "G01C 21/165",
    "G01C 21/1652",
    "G01S 17/00",
    "G01S 17/86",
    "G01S 17/89",
    "G05D 1/101",
    "G05D 1/106",
    "G06T 2207/10016",
    "G06T 2207/10024",
    "G06T 2207/10028",
    "G06T 2207/10032",
    "G06T 2207/30181",
    "G06T 2207/30244",
    "G06T 2207/30252",
    "G06T 7/246",
    "G06T 7/248",
    "G06T 7/251",
    "G06T 7/277",
    "G06T 7/337",
    "G06T 7/74",
    "G06T 7/75"
  ],
  "ipc": [
    "G01C 21/16"
  ],
  "assignees": [
    "Near Earth Autonomy Inc"
  ],
  "inventors": [
    "Sanjiv Singh",
    "Jeffrey Mishler",
    "Michael Kaess",
    "Garrett Hemann"
  ],
  "filing_date": "2017-09-21",
  "publication_date": "2018-12-11",
  "grant_date": "2018-12-11",
  "priority_date": "2016-09-28",
  "application_number": "US-201715711492-A",
  "family_id": "64502805",
  "cited_by_count": 33,
  "citations": [
    "US9389298B2",
    "US8315794B1",
    "US8010287B1",
    "US20140088790A1",
    "US20140022262A1",
    "US20160140729A1",
    "US20160335901A1",
    "WO2017042672A1"
  ]
}

Record 3,139 of 8,000 in Patents full text (MLC-0201). Request the full dataset.