Automatic feature extraction from imagery

{
  "publication_number": "US11657072B2",
  "country": "US",
  "kind": "B2",
  "title": "Automatic feature extraction from imagery",
  "abstract": "An apparatus, or corresponding method, for building or updating a map database is described. In one example, the apparatus includes an image correlation module, a training device, and a learned model or neural network. The image correlation module is configured to correlate a first aerial image and terrestrial sensor data collected at a terrestrial vehicle based on at least one control point from the terrestrial data. The learned model training device is configured to define a learned model based using at least one control point from the terrestrial sensor data as ground truth for analysis of the first aerial image. The learned model inference module is configured to receive a second aerial image and apply the learned model on the second aerial image for identification of mapping information for the map data.",
  "claims": [
    "1. A method for creating or updating a map database, the method further comprising: receiving at least a first aerial image for a training geographic area; receiving terrestrial sensor data collected at a terrestrial vehicle; identifying at least one control point of at least one object detected in the terrestrial sensor data; correlating points of the at least one object represented in at least the first aerial image with the at least one control point of the at least one object detected in the terrestrial sensor data; orthorectifying the at least one object represented in at least the first aerial image using the correlated points; training a neural network using the at least one control point from the terrestrial sensor data collected at the terrestrial vehicle and the orthorectified at least one object represented in at least the first aerial image; wherein the training of the neural network includes performing iterative operations of the neural network based on data from a low altitude aerial vehicle and a high altitude aerial vehicle; receiving additional aerial images as input to the trained neural network and identifying additional objects in the additional aerial images using the trained neural network; and updating map information in a map database to include the identified additional objects; wherein the identified additional object in the map information is at least one of a sign, a pole, a lane line, a road boundary or a barrier.",
    "2. The method of claim 1, wherein the at least first aerial image includes a plurality of images for the training geographic area and the neural network is defined based on the plurality of images.",
    "3. The method of claim 1, wherein orthorectifying the at least one object further comprises: identifying a characteristic of an aerial vehicle; and calculating at least one resultant image feature that reduces distortion from the characteristic of the aerial vehicle.",
    "4. The method of claim 1, wherein the map information includes a navigation object and corresponding location.",
    "5. The method of claim 1, wherein the neural network includes a semantic segmentation model or a polyline model.",
    "6. The method of claim 5, wherein the semantic segmentation model outputs pixel values for the map information.",
    "7. The method of claim 5, wherein the polyline model outputs coordinates for at least one polyline for the map information.",
    "8. The method of claim 1, further comprising: limiting the training geographic area using at least one control point from the terrestrial sensor data as ground truth.",
    "9. The method of claim 1, further comprising: generating a navigation instruction from the map information of the map database.",
    "10. The method of claim 1, further comprising: generating a driving command from the map information of the map database.",
    "11. The method of claim 1, further comprising: generating a model including the map information of the map database.",
    "12. The method of claim 1, further comprising: receiving at least an additional image for the training geographic area; orthorectifying at least the additional image using the at least one control point from the terrestrial sensor data, wherein the orthorectifying is repeated until images for the training geographic area are orthorectified.",
    "13. A non-transitory computer readable medium including instructions that when performed by a processor are configured to: receive at least a first aerial image for a training geographic area; receive terrestrial sensor data collected at a terrestrial vehicle; identify at least one control point of at least one object detected in the terrestrial sensor data; correlate points of the at least one object represented in at least the first aerial image with the at least one control point of the at least one object detected in the terrestrial sensor data; orthorectify the at least one object represented in at least the first aerial image using the correlated points; train a neural network using: at least one control point from the terrestrial sensor data collected at the terrestrial vehicle and the orthorectified at least one object represented in the at least the first aerial image; wherein the training of the neural network includes performing iterative operations of the neural network based on data from a low altitude aerial vehicle and a high altitude aerial vehicle; receive additional aerial images as input to the trained neural network and identifying additional objects in the additional aerial images using the trained neural network; and update map information in a map database to include the identified additional objects; wherein the identified additional object in the map information is at least one of a sign, a pole, a lane line, a road boundary or a barrier.",
    "14. The computer readable medium of claim 13, wherein orthorectifying the at least one object further comprises: identifying a characteristic of an aerial vehicle; and calculating at least one resultant image feature that reduces distortion from the characteristic of the aerial vehicle.",
    "15. The computer readable medium of claim 13, wherein the neural network includes a semantic segmentation model configured to output pixel values for the map information or a polyline model configured to output coordinates for at least one polyline for the map information.",
    "16. A method for creating or updating a map database, the method further comprising: receiving at least a first aerial image for a training geographic area; receiving terrestrial sensor data collected at a terrestrial vehicle; identifying at least one control point of at least one object detected in the terrestrial sensor data; correlating points of the at least one object represented in at least the first aerial image with the at least one control point of the at least one object detected in the terrestrial sensor data; orthorectifying the at least one object represented in at least the first aerial image using the correlated points; training a neural network using: at least one control point from the terrestrial sensor data collected at a terrestrial vehicle and the orthorectified at least one object represented in at least the first aerial image, wherein the training of the neural network includes performing iterative operations of the neural network based on data from a low altitude aerial vehicle and a high altitude aerial vehicle; receiving additional aerial images as input to the trained neural network and identifying additional objects in the additional aerial images using the trained neural network; updating map information in a map database to include road attributes associated with the identified additional objects."
  ],
  "cpc": [
    "G06F 16/29",
    "G01S 13/865",
    "G01S 13/89",
    "G01S 15/931",
    "G01S 17/86",
    "G01S 17/89",
    "G01S 17/931",
    "G05D 1/0274",
    "G05D 2201/0213",
    "G06F 16/23",
    "G06N 20/00",
    "G06N 3/0464",
    "G06N 3/08",
    "G06N 3/09",
    "G06T 7/32",
    "G06V 10/82",
    "G06V 20/13",
    "G06V 20/176",
    "G06V 20/582",
    "G06V 20/588"
  ],
  "assignees": [
    "HERE GLOBAL BV"
  ],
  "filing_date": "2019-05-16",
  "publication_date": "2023-05-23",
  "grant_date": "2023-05-23",
  "priority_date": "2019-05-16",
  "application_number": "US-201916413873-A",
  "family_id": "70738375",
  "citations": [
    "US2004167709A1",
    "US2010208937A1",
    "US2011282578A1",
    "US2016005145A1",
    "US2016341554A1",
    "US2018012371A1",
    "US2018218214A1",
    "US2018372493A1",
    "US8847982B2",
    "WO2010068186A1",
    "WO2018104563A2",
    "WO2018104563A3"
  ]
}