Localization of individual plants based on high-elevation imagery

{
  "publication_number": "US12112501B2",
  "country": "US",
  "kind": "B2",
  "title": "Localization of individual plants based on high-elevation imagery",
  "abstract": "Implementations are described herein for localizing individual plants using high-elevation images at multiple different resolutions. A first set of high-elevation images that capture the plurality of plants at a first resolution may be analyzed to classify a set of pixels as invariant anchor points. High-elevation images of the first set may be aligned with each other based on the invariant anchor points that are common among at least some of the first set of high-elevation images. A mapping may be generated between pixels of the aligned high-elevation images of the first set and spatially-corresponding pixels of a second set of higher-resolution high-elevation images. Based at least in part on the mapping, individual plant(s) of the plurality of plants may be localized within one or more of the second set of high-elevation images for performance of one or more agricultural tasks.",
  "claims": [
    "1. A method for localizing one or more individual plants of a plurality of plants, the method implemented using one or more processors and comprising: analyzing a first set of high-elevation images that capture the plurality of plants at a first resolution; based on the analyzing, classifying a set of pixels of the first set of high-elevation images as depicting invariant anchor points, wherein the invariant anchor points include one or more visual features that are known to be movement-invariant; aligning high-elevation images of the first set based on one or more of the invariant anchor points that are common among at least some of the first set of high-elevation images; generating a mapping, based on the invariant anchor points, between pixels of the aligned high-elevation images of the first set and spatially-corresponding pixels of a second set of high-elevation images, wherein the second set of high-elevation images capture the plurality of plants at a second resolution that is greater than the first resolution; and based at least in part on the mapping, localizing one or more individual plants of the plurality of plants within one or more of the second set of high-elevation images for performance of one or more agricultural tasks.",
    "2. The method of claim 1, further including downsampling the second set of high-elevation digital images to generate the first set of high-elevation digital images.",
    "3. The method of claim 1, wherein the first set of high-elevation images are captured at a first elevation and the second set of high-elevation images are captured at a second elevation that is less than the first elevation.",
    "4. The method of claim 1, wherein the localizing includes assigning position coordinates to the one or more individual plants based on position coordinates generated by an airborne vehicle that acquired the first set or the second set of high-elevation images.",
    "5. The method of claim 1, wherein the localizing includes mapping the one or more individual plants to one or more rows of a plurality of rows in which the plurality of plants are arranged.",
    "6. The method of claim 1, further including deploying one or more agricultural robots to the one or more localized individual plants to perform one or more of the agricultural tasks.",
    "7. The method of claim 1, wherein the first set or the second set of high-elevation images is acquired by an unmanned aerial vehicle (UAV).",
    "8. The method of claim 1, wherein the classifying includes processing the first set of high-elevation images based on one or more machine learning models that are trained to recognize one or more objects known to be movement-invariant.",
    "9. The method of claim 8, wherein the classifying includes processing the first set of high-elevation images based on one or more machine learning models that are trained to recognize lodged plants among the plurality of plants.",
    "10. The method of claim 8, wherein the classifying includes detecting agricultural equipment in spatial proximity with the plurality of plants.",
    "11. The method of claim 8, wherein the classifying includes detecting one or more water features or roads in spatial proximity with the plurality of plants.",
    "12. A system for localizing one or more individual plants of a plurality of plants, the system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to: analyze a first set of high-elevation images that capture the plurality of plants at a first resolution; based on the analysis, classify a set of pixels of the first set of high-elevation images as depicting invariant anchor points, wherein the invariant anchor points include one or more visual features that are known to be movement-invariant; align high-elevation images of the first set based on one or more of the invariant anchor points that are common among at least some of the first set of high-elevation images; generate a mapping, based on the invariant anchor points, between pixels of the aligned high-elevation images of the first set and spatially-corresponding pixels of a second set of high-elevation images, wherein the second set of high-elevation images capture the plurality of plants at a second resolution that is greater than the first resolution; and based at least in part on the mapping, localize one or more individual plants of the plurality of plants within one or more of the second set of high-elevation images for performance of one or more agricultural tasks.",
    "13. The system of claim 12, further including instructions to downsample the second set of high-elevation digital images to generate the first set of high-elevation digital images.",
    "14. The system of claim 12, wherein the first set of high-elevation images are captured at a first elevation and the second set of high-elevation images are captured at a second elevation that is less than the first elevation.",
    "15. The system of claim 12, wherein the instructions to localize include instructions to assign position coordinates to the one or more individual plants based on position coordinates generated by an airborne vehicle that acquired the first set or the second set of high-elevation images.",
    "16. The system of claim 12, wherein the instructions to localize include instructions to map the one or more individual plants to one or more rows of a plurality of rows in which the plurality of plants are arranged.",
    "17. The system of claim 12, further including instructions to deploy one or more agricultural robots to the one or more localized individual plants to perform one or more of the agricultural tasks.",
    "18. The system of claim 12, wherein the first set or the second set of high-elevation images is acquired by an unmanned aerial vehicle (UAV).",
    "19. A non-transitory computer-readable medium for localizing one or more individual plants of a plurality of plants, the medium comprising instructions that, in response to execution of the instructions by a processor, cause the processor to: analyze a first set of high-elevation images that capture the plurality of plants at a first resolution; based on the analysis, classify a set of pixels of the first set of high-elevation images as depicting invariant anchor points, wherein the invariant anchor points include one or more visual features that are known to be movement-invariant; align high-elevation images of the first set based on one or more of the invariant anchor points that are common among at least some of the first set of high-elevation images; generate a mapping, based on the invariant anchor points, between pixels of the aligned high-elevation images of the first set and spatially-corresponding pixels of a second set of high-elevation images, wherein the second set of high-elevation images capture the plurality of plants at a second resolution that is greater than the first resolution; and based at least in part on the mapping, localize one or more individual plants of the plurality of plants within one or more of the second set of high-elevation images for performance of one or more agricultural tasks.",
    "20. The non-transitory computer-readable medium of claim 19, further including instructions to downsample the second set of high-elevation digital images to generate the first set of high-elevation digital images."
  ],
  "description_excerpt": "With large scale agriculture, crops typically are observed, measured, and/or interacted with in a relatively coarse manner. For example, data gathered from sparse sampling may be used to extrapolate crop yields, disease diagnoses, and/or pest presence/population for entire plots of plants. This can lead to less-than-ideal agricultural practices such as over/under application of fertilizer or other chemicals, over/under remediation of weeds and/or pests, and so forth. These agricultural practices may yield less than optical crop yields because healthy plants may be destroyed or damaged, unhealthy plants may be inadequately remediated, etc. “Precision agriculture” refers to techniques for observing, measuring, and/or interacting with (e.g., harvesting, applying chemicals, pruning, etc.) crops in a highly targeted and granular manner, including at the level of individual, localized plants. Precision agriculture may improve crop yields and increase agricultural efficiency and/or land use overall. As agricultural robots become increasingly available and capable, precision agriculture has become more feasible, technologically and economically, with localization of individual plants being a key feature. However, existing plant localization techniques suffer from various shortcomings, such as being computationally expensive, error-prone, and/or too time-consuming.\n\nImplementations are described herein for localizing individual plants by processing high-elevation images with multiple different resolutions.",
  "cpc": [
    "G06T 7/74",
    "B25J 11/00",
    "G06F 18/24",
    "G06T 2207/10032",
    "G06T 2207/20016",
    "G06T 2207/20081",
    "G06T 2207/30188",
    "G06T 3/40",
    "G06T 3/4038",
    "G06T 7/33",
    "G06V 10/24",
    "G06V 10/82",
    "G06V 20/182",
    "G06V 20/188"
  ],
  "ipc": [
    "B25J 11/00",
    "G06F 18/24",
    "G06T 3/40",
    "G06T 7/33",
    "G06T 7/73",
    "G06V 20/10"
  ],
  "assignees": [
    "DEERE & CO"
  ],
  "inventors": [
    "YUAN ZHIQIANG",
    "YANG JIE"
  ],
  "filing_date": "2021-06-22",
  "publication_date": "2024-10-08",
  "grant_date": "2024-10-08",
  "priority_date": "2021-06-22",
  "application_number": "US-202117354147-A",
  "family_id": "84490564",
  "citations": [
    "US10402942B2",
    "US10614305B2",
    "US2017061052A1",
    "US2017358106A1",
    "US2019205610A1",
    "US2020126232A1",
    "US2020294620A1",
    "US2020401883A1"
  ]
}