MLchartDataset catalogue

Patent · US9689686B1 · B1 · US

Detecting of navigation data spoofing based on image data

(11) Publication number
US9689686B1
(21) Application number
14/866,719
(22) Filing date
2015-09-25
(30) Priority date
2015-09-25
(43) Publication date
2017-06-27
(45) Date of grant
2017-06-27
(51) IPC
B64C 39/02; G01C 21/20; G05D 1/00
(52) CPC
  • G01C Measuring distances, levels or bearings; surveying; navigation; gyroscopic instruments; photogrammetry or videogrammetry: 21/20
  • B64C Aeroplanes; helicopters: 39/024
  • B64U Unmanned aerial vehicles [uav]; equipment therefor: 2101/30, 2101/60, 2201/104
  • G01S Radio direction-finding; radio navigation; determining distance or velocity by use of radio waves; locating or presence-detecting by use of the reflection or reradiation of radio waves; analogous arrangements using other waves: 13/86, 19/00, 19/215, 19/485
  • G05D Systems for controlling or regulating non-electric variables: 1/0022, 1/0038
  • G06F Electric digital data processing: 21/64, 2221/2111
  • H04K Secret communication; jamming of communication: 2203/22, 3/22, 3/65, 3/90
  • H04L Transmission of digital information, e.g. telegraphic communication: 63/0869, 63/123
  • H04W Wireless communication networks: 12/06, 12/10, 12/63, 4/02
(73) Assignee
Amazon Technologies Inc
(72) Inventors
Scott Gerard Carmack; Narasimha Rao Lakkakula; Nima Sharifi Mehr
(54) Title
Detecting of navigation data spoofing based on image data
(57) Abstract

Techniques for determining whether data associated with an autonomous navigation of an unmanned vehicle may be trusted. For example, navigation-related data may be provided from a source external to the unmanned vehicle. Image data associated with the autonomous navigation may be generated. The navigation-related data and the image data may be compared to determine whether the navigation data may be trusted or not. If untrusted, the autonomous navigation may be directed independently of the navigation data.

Full text
View on Google Patents

Claims (20)

  1. A computer-implemented method, comprising: receiving, by a computer system of an unmanned aerial vehicle (UAV), global positioning system (GPS) data, the computer system configured to direct a flight of the UAV to deliver an item from a source to a destination along a predefined route based at least in part on the GPS data; requesting, by the computer system from an imaging device of the UAV, an image associated with a location of the UAV, the UAV being at the location based at least in part on the GPS data; accessing, by the computing system, a known image associated with a known location along the predefined route; determining, by the computer system, a discrepancy between the known image and the image; determining, by the computer system, that the GPS data is spoofed based at least in part on the discrepancy between the known image and the image; and directing, by the computer system, a portion of the flight independently of the GPS data that is spoofed.
  2. The computer-implemented method of claim 1, wherein determining the discrepancy comprises: transmitting, by the computer system, the image to a central station, the central station configured to: select the known image from available images based at least in part on checkpoints along the predefined route of the UAV, compare the image to the known image, and determine that the discrepancy between the image and the known image based at least in part on the comparison, and receiving the discrepancy from the central station.
  3. The computer-implemented method of claim 1, wherein determining that the GPS data is spoofed comprises: detecting, by the computer system, that the discrepancy exceeds a threshold, wherein the threshold is based at least in part on one or more of: the camera, the item, historical deliveries of the UAV, or historical deliveries of another UAV.
  4. The computer-implemented method of claim 1, wherein directing the portion of the flight independently of the GPS data comprises: receiving sensor data of a navigation sensor of the UAV; and directing the portion of the flight based at least in part on the sensor data.
  5. One or more non-transitory computer readable media comprising instructions that, when executed with a computer system, cause the computer system to at least: access navigation data of a source external to an unmanned autonomous vehicle (UAV), the navigation data associated with directing an autonomous navigation of the UAV along a predefined route; access image data of an optical sensor installed on the UAV; access known image data associated with a known location along the predefined route; determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and direct the autonomous navigation of the UAV independently of the navigation data based at least in part on the navigation data being untrusted.
  6. The one or more non-transitory computer readable media of claim 5, wherein the UAV is configured to deliver an item available from an electronic marketplace, wherein the source external to the UAV comprises a global positioning system (GPS), and wherein the navigation data comprises GPS data.
  7. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at the UAV.
  8. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at a central station and is configured to communicate with the UAV over a network, wherein accessing the image data comprises receiving an image of a location of the UAV, wherein determining that the navigation data is untrusted comprises: detecting an object in the image; matching the object to a known object from a map, the map indicating a particular location of the known object; determining the discrepancy based on a comparison of the known location along the predefined route of the UAV and the particular location of the known object; and determining that the discrepancy exceeds a threshold.
  9. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image at a particular resolution of a location of the UAV, wherein the UAV is at the location based at least in part on the navigation data.
  10. The one or more non-transitory computer readable media of claim 9, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request, based at least in part on determining that the navigation data is untrusted, a second image at a higher resolution from an imaging device comprising the optical sensor; and confirm that the navigation data is untrusted based at least in part on the second image.
  11. The one or more non-transitory computer readable media of claim 5, wherein the instructions, when executed with the computer system, further cause the computer system to at least, upon determining that the navigation data is untrusted: change a location of the UAV; request an image of the location from an imaging device comprising the optical sensor; and confirm that the navigation data is untrusted based at least in part on the image.
  12. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request an imaging device comprising the optical sensor to generate the image upon a detection that the UAV is at the location based at least in part on the navigation data.
  13. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request an imaging device comprising the optical sensor to generate images of locations of the UAV at a dynamic frequency, wherein the dynamic frequency increases upon determining that the navigation data is untrusted.
  14. A system associated with an unmanned vehicle, the system comprising: one or more processors; and one or more non-transitory computer readable media comprising instructions that, when executed with the one or more processors, cause the system to at least: access navigation data of a source external to the unmanned vehicle, the navigation data associated with directing an autonomous navigation of the unmanned vehicle along a predefined route; access image data of an optical sensor of the unmanned vehicle, the image data associated with a location of the unmanned vehicle based at least in part on the autonomous navigation; access known image data associated with a known location along the predefined route; determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and direct the autonomous navigation of the unmanned vehicle independently of the navigation data that is untrusted and based at least in part on a corrective action.
  15. The system of claim 14, wherein the image data forms an image at a resolution, wherein the instructions, when executed with the one or more processors, further cause the system to at least: detect that the unmanned vehicle is at the location based at least in part on the navigation data, and request an imaging device comprising the optical sensor to generate the image at the resolution upon the detecting, and wherein the corrective action comprises: requesting the imaging device to generate an additional image at a higher resolution upon determining that the navigation data is untrusted; and confirming that the navigation data is untrusted based at least in part on the additional image.
  16. The system of claim 14, wherein the corrective action comprises: reporting that the navigation data is untrusted to a central station; and facilitating a remote control over operations of the unmanned vehicle from the central station.
  17. The system of claim 14, wherein determining that the navigation data is untrusted comprises: accessing a known image from a data store, the known image corresponding to a checkpoint along a predefined route of the unmanned vehicle; comparing the image data of the optical sensor to second image data of the known image; and determining the discrepancy based at least in part on comparing the image data and the second image data.
  18. The system of claim 14, wherein the optical sensor comprises an infrared sensor, wherein the image data forms an infrared image of the location, and wherein determination that the navigation data is untrusted comprises: accessing a known infrared image from a data store based at least in part on the navigation data; and comparing the infrared image to the known infrared image to detect the discrepancy.
  19. The system of claim 14, wherein the navigation data is available to the unmanned vehicle from a first source, wherein the corrective action comprises transmitting the image data and an indication that the navigation data is untrusted to a central station, wherein the central station is configured to: request second image data of the location from a second unmanned vehicle and a second indication from the second unmanned vehicle whether second navigation data associated with the location is trusted; compare the image data to the second image data based at least in part on the second navigation data being trusted, wherein the second navigation data is available to the second unmanned vehicle from a second source different from the first source; identifying the first source of the navigation data based at least in part on comparing the image data and the second image data.
  20. The system of claim 14, wherein determining that the navigation data is untrusted comprises: detecting an object based at least in part on the image data; determining a known location associated with the object; and determining the discrepancy between a predefined route of the unmanned vehicle and the known location.

Description

Unmanned vehicles may provide different operations, some of which may be autonomous and support various applications. For example, an unmanned aerial vehicle (UAV) may be used as an autonomous delivery vehicle associated with an electronic marketplace. The electronic marketplace may offer items and selectable delivery methods. Based on a selected delivery method, the UAV may be deployed to deliver a purchased item. The UAV may be configured to autonomously perform various delivery-related operations, such as autonomously flying between a source and a destination.

The unmanned vehicles may rely on externally and internally generated data to perform some of the operations. Referring back to the UAV example, to autonomously fly between the source and the destination, the UAV may use global positioning system (GPS) data, or some other navigation data, received from a satellite-based system. In certain situations, proper execution of autonomous operations may depend on using trusted data. For instance, the UAV may properly deliver the purchased item to the destination if the GPS data is reliable.

Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:

FIG. 1 illustrates an example environment for autonomous operations of an unmanned vehicle, according to embodiments;

FIG. 2 illustrates an example environment for autonomous operations of multiple unmanned vehicles, according to embodiments;

FIG. 3 illustrates an example unmanned vehicle configuration, according to embodiments;

Citations (20)

  • US6771214B2
  • US7409290B2
  • US7546187B2
  • US8315794B1
  • WO2012021547A2
  • US20130002477A1
  • US20150061043A1
  • US9254363B2
  • US20150234053A1
  • US20140222248A1
  • US20140172194A1
  • US20150203213A1
  • US20150046017A1
  • US20150127239A1
  • US20150260526A1
  • US20150283701A1
  • US20150292887A1
  • US9359074B2
  • US20160210525A1
  • WO2016154949A1
Record as JSON 
{
  "publication_number": "US9689686B1",
  "country": "US",
  "kind": "B1",
  "title": "Detecting of navigation data spoofing based on image data",
  "abstract": "Techniques for determining whether data associated with an autonomous navigation of an unmanned vehicle may be trusted. For example, navigation-related data may be provided from a source external to the unmanned vehicle. Image data associated with the autonomous navigation may be generated. The navigation-related data and the image data may be compared to determine whether the navigation data may be trusted or not. If untrusted, the autonomous navigation may be directed independently of the navigation data.",
  "claims": [
    "1. A computer-implemented method, comprising: receiving, by a computer system of an unmanned aerial vehicle (UAV), global positioning system (GPS) data, the computer system configured to direct a flight of the UAV to deliver an item from a source to a destination along a predefined route based at least in part on the GPS data; requesting, by the computer system from an imaging device of the UAV, an image associated with a location of the UAV, the UAV being at the location based at least in part on the GPS data; accessing, by the computing system, a known image associated with a known location along the predefined route; determining, by the computer system, a discrepancy between the known image and the image; determining, by the computer system, that the GPS data is spoofed based at least in part on the discrepancy between the known image and the image; and directing, by the computer system, a portion of the flight independently of the GPS data that is spoofed.",
    "2. The computer-implemented method of claim 1, wherein determining the discrepancy comprises: transmitting, by the computer system, the image to a central station, the central station configured to: select the known image from available images based at least in part on checkpoints along the predefined route of the UAV, compare the image to the known image, and determine that the discrepancy between the image and the known image based at least in part on the comparison, and receiving the discrepancy from the central station.",
    "3. The computer-implemented method of claim 1, wherein determining that the GPS data is spoofed comprises: detecting, by the computer system, that the discrepancy exceeds a threshold, wherein the threshold is based at least in part on one or more of: the camera, the item, historical deliveries of the UAV, or historical deliveries of another UAV.",
    "4. The computer-implemented method of claim 1, wherein directing the portion of the flight independently of the GPS data comprises: receiving sensor data of a navigation sensor of the UAV; and directing the portion of the flight based at least in part on the sensor data.",
    "5. One or more non-transitory computer readable media comprising instructions that, when executed with a computer system, cause the computer system to at least: access navigation data of a source external to an unmanned autonomous vehicle (UAV), the navigation data associated with directing an autonomous navigation of the UAV along a predefined route; access image data of an optical sensor installed on the UAV; access known image data associated with a known location along the predefined route; determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and direct the autonomous navigation of the UAV independently of the navigation data based at least in part on the navigation data being untrusted.",
    "6. The one or more non-transitory computer readable media of claim 5, wherein the UAV is configured to deliver an item available from an electronic marketplace, wherein the source external to the UAV comprises a global positioning system (GPS), and wherein the navigation data comprises GPS data.",
    "7. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at the UAV.",
    "8. The one or more non-transitory computer readable media of claim 5, wherein the computer system is installed at a central station and is configured to communicate with the UAV over a network, wherein accessing the image data comprises receiving an image of a location of the UAV, wherein determining that the navigation data is untrusted comprises: detecting an object in the image; matching the object to a known object from a map, the map indicating a particular location of the known object; determining the discrepancy based on a comparison of the known location along the predefined route of the UAV and the particular location of the known object; and determining that the discrepancy exceeds a threshold.",
    "9. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image at a particular resolution of a location of the UAV, wherein the UAV is at the location based at least in part on the navigation data.",
    "10. The one or more non-transitory computer readable media of claim 9, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request, based at least in part on determining that the navigation data is untrusted, a second image at a higher resolution from an imaging device comprising the optical sensor; and confirm that the navigation data is untrusted based at least in part on the second image.",
    "11. The one or more non-transitory computer readable media of claim 5, wherein the instructions, when executed with the computer system, further cause the computer system to at least, upon determining that the navigation data is untrusted: change a location of the UAV; request an image of the location from an imaging device comprising the optical sensor; and confirm that the navigation data is untrusted based at least in part on the image.",
    "12. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request an imaging device comprising the optical sensor to generate the image upon a detection that the UAV is at the location based at least in part on the navigation data.",
    "13. The one or more non-transitory computer readable media of claim 5, wherein the image data forms an image of a location of the UAV, wherein the instructions, when executed with the computer system, further cause the computer system to at least: request an imaging device comprising the optical sensor to generate images of locations of the UAV at a dynamic frequency, wherein the dynamic frequency increases upon determining that the navigation data is untrusted.",
    "14. A system associated with an unmanned vehicle, the system comprising: one or more processors; and one or more non-transitory computer readable media comprising instructions that, when executed with the one or more processors, cause the system to at least: access navigation data of a source external to the unmanned vehicle, the navigation data associated with directing an autonomous navigation of the unmanned vehicle along a predefined route; access image data of an optical sensor of the unmanned vehicle, the image data associated with a location of the unmanned vehicle based at least in part on the autonomous navigation; access known image data associated with a known location along the predefined route; determine that the navigation data is untrusted based at least in part on a discrepancy between the known image data and the image data; and direct the autonomous navigation of the unmanned vehicle independently of the navigation data that is untrusted and based at least in part on a corrective action.",
    "15. The system of claim 14, wherein the image data forms an image at a resolution, wherein the instructions, when executed with the one or more processors, further cause the system to at least: detect that the unmanned vehicle is at the location based at least in part on the navigation data, and request an imaging device comprising the optical sensor to generate the image at the resolution upon the detecting, and wherein the corrective action comprises: requesting the imaging device to generate an additional image at a higher resolution upon determining that the navigation data is untrusted; and confirming that the navigation data is untrusted based at least in part on the additional image.",
    "16. The system of claim 14, wherein the corrective action comprises: reporting that the navigation data is untrusted to a central station; and facilitating a remote control over operations of the unmanned vehicle from the central station.",
    "17. The system of claim 14, wherein determining that the navigation data is untrusted comprises: accessing a known image from a data store, the known image corresponding to a checkpoint along a predefined route of the unmanned vehicle; comparing the image data of the optical sensor to second image data of the known image; and determining the discrepancy based at least in part on comparing the image data and the second image data.",
    "18. The system of claim 14, wherein the optical sensor comprises an infrared sensor, wherein the image data forms an infrared image of the location, and wherein determination that the navigation data is untrusted comprises: accessing a known infrared image from a data store based at least in part on the navigation data; and comparing the infrared image to the known infrared image to detect the discrepancy.",
    "19. The system of claim 14, wherein the navigation data is available to the unmanned vehicle from a first source, wherein the corrective action comprises transmitting the image data and an indication that the navigation data is untrusted to a central station, wherein the central station is configured to: request second image data of the location from a second unmanned vehicle and a second indication from the second unmanned vehicle whether second navigation data associated with the location is trusted; compare the image data to the second image data based at least in part on the second navigation data being trusted, wherein the second navigation data is available to the second unmanned vehicle from a second source different from the first source; identifying the first source of the navigation data based at least in part on comparing the image data and the second image data.",
    "20. The system of claim 14, wherein determining that the navigation data is untrusted comprises: detecting an object based at least in part on the image data; determining a known location associated with the object; and determining the discrepancy between a predefined route of the unmanned vehicle and the known location."
  ],
  "description_excerpt": "Unmanned vehicles may provide different operations, some of which may be autonomous and support various applications. For example, an unmanned aerial vehicle (UAV) may be used as an autonomous delivery vehicle associated with an electronic marketplace. The electronic marketplace may offer items and selectable delivery methods. Based on a selected delivery method, the UAV may be deployed to deliver a purchased item. The UAV may be configured to autonomously perform various delivery-related operations, such as autonomously flying between a source and a destination.\n\nThe unmanned vehicles may rely on externally and internally generated data to perform some of the operations. Referring back to the UAV example, to autonomously fly between the source and the destination, the UAV may use global positioning system (GPS) data, or some other navigation data, received from a satellite-based system. In certain situations, proper execution of autonomous operations may depend on using trusted data. For instance, the UAV may properly deliver the purchased item to the destination if the GPS data is reliable.\n\nVarious embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:\n\nFIG. 1 illustrates an example environment for autonomous operations of an unmanned vehicle, according to embodiments;\n\nFIG. 2 illustrates an example environment for autonomous operations of multiple unmanned vehicles, according to embodiments;\n\nFIG. 3 illustrates an example unmanned vehicle configuration, according to embodiments;",
  "cpc": [
    "G01C 21/20",
    "B64C 39/024",
    "B64U 2101/30",
    "B64U 2101/60",
    "B64U 2201/104",
    "G01S 13/86",
    "G01S 19/00",
    "G01S 19/215",
    "G01S 19/485",
    "G05D 1/0022",
    "G05D 1/0038",
    "G06F 21/64",
    "G06F 2221/2111",
    "H04K 2203/22",
    "H04K 3/22",
    "H04K 3/65",
    "H04K 3/90",
    "H04L 63/0869",
    "H04L 63/123",
    "H04W 12/06",
    "H04W 12/10",
    "H04W 12/63",
    "H04W 4/02"
  ],
  "ipc": [
    "B64C 39/02",
    "G01C 21/20",
    "G05D 1/00"
  ],
  "assignees": [
    "Amazon Technologies Inc"
  ],
  "inventors": [
    "Scott Gerard Carmack",
    "Narasimha Rao Lakkakula",
    "Nima Sharifi Mehr"
  ],
  "filing_date": "2015-09-25",
  "publication_date": "2017-06-27",
  "grant_date": "2017-06-27",
  "priority_date": "2015-09-25",
  "application_number": "US-201514866719-A",
  "family_id": "59070153",
  "cited_by_count": 43,
  "citations": [
    "US6771214B2",
    "US7409290B2",
    "US7546187B2",
    "US8315794B1",
    "WO2012021547A2",
    "US20130002477A1",
    "US20150061043A1",
    "US9254363B2",
    "US20150234053A1",
    "US20140222248A1",
    "US20140172194A1",
    "US20150203213A1",
    "US20150046017A1",
    "US20150127239A1",
    "US20150260526A1",
    "US20150283701A1",
    "US20150292887A1",
    "US9359074B2",
    "US20160210525A1",
    "WO2016154949A1"
  ]
}

Record 4,052 of 8,000 in Patents full text (MLC-0201). Request the full dataset.