Image data processing system, unmanned aerial vehicle, image data processing method, and non-transitory computer-readable recording medium

{
  "publication_number": "US12106447B2",
  "country": "US",
  "kind": "B2",
  "title": "Image data processing system, unmanned aerial vehicle, image data processing method, and non-transitory computer-readable recording medium",
  "abstract": "An image data processing system is provided including: a discharge portion for discharging a content in a container onto a subject to form a sprayed symbol; a camera for capturing an image of a region including the sprayed symbol on the subject; and an image superimposition processing portion for performing an image superimposition process on the image using the sprayed symbol as a reference point. The camera may include a visible light camera for a visible light image and an infrared camera for capturing a thermal image of a subject.",
  "claims": [
    "1. An image data processing system comprising: a discharge portion for discharging a content in a container onto a subject to form a sprayed symbol; a camera for capturing an image of a region including the sprayed symbol on the subject; and an image superimposition processing portion for performing an image superimposition process on the image using the sprayed symbol as a reference point, wherein the camera includes a visible light camera for a visible light image and an infrared camera for capturing a thermal image of the subject, the discharge portion is configured to use the content to form, on a surface of the subject, a different temperature area having a temperature that is different from those of other parts of the subject, the infrared camera is configured to capture the thermal image of a region including the different temperature area, and the image superimposition processing portion is configured to, in an image superimposition process on the thermal image and the visible light image, use the different temperature area and the sprayed symbol as reference points in the image superimposition process.",
    "2. The mage data processing system according to claim 1, wherein the content includes a vaporizable component and the different temperature area is formed by vaporization heat generated when the content discharged onto the subject evaporates.",
    "3. The image data processing system according to claim 1, further comprising a connectivity evaluation portion for evaluating a difficulty of the image superimposition process based on a continuity of a superimposed part of the image in the image superimposition process, wherein if the connectivity evaluation portion evaluates that the image superimposition process on an image where the subject is captured is difficult, the discharge portion forms the sprayed symbol.",
    "4. The image data processing system according to claim 3, wherein the connectivity evaluation portion is configured to perform a process to extract feature points in an image captured by the camera, and evaluate a difficulty of the image superimposition process based on a quantity of the obtained feature points.",
    "5. The image data processing system according to claim 3, wherein the connectivity evaluation portion is configured to sense whether there is a periodic repeated pattern, and evaluate the difficulty based on a sensing result.",
    "6. The image data processing system according to claim 3, wherein the connectivity evaluation portion is configured to attempt the image superimposition process on the image and evaluate the difficulty based on the attempt result.",
    "7. The image data processing system according to claim 3, wherein the image superimposition process includes a connecting process on a plurality of images.",
    "8. The image data processing system according to claim 1, wherein the image superimposition process includes a synthetic process on a plurality of images.",
    "9. The image data processing system according to claim 1, further comprising a light for shooting in a dark.",
    "10. The image data processing system according to claim 1, further comprising a temperature adjustment apparatus for adjusting a temperature of the content to a temperature that is different from a surface temperature of the subject.",
    "11. The image data processing system according to claim 1, wherein the content is colored.",
    "12. The image data processing system according to claim 1, wherein the container is an aerosol container.",
    "13. An unmanned aerial vehicle that incorporates the image data processing system according to claim 1.",
    "14. The image data processing system according to claim 1, further comprising a connectivity evaluation portion for evaluating a difficulty of the image superimposition process based on a continuity of a superimposed part of the image in the image superimposition process, wherein if the connectivity evaluation portion evaluates that the image superimposition process on an image where the subject 1s captured 1s difficult, the discharge portion forms the sprayed symbol.",
    "15. An image data processing method comprising: discharging a content from a container incorporated in an unmanned aerial vehicle onto a subject to form a sprayed symbol; capturing a plurality of images related to a region of the subject including the sprayed symbol; and processing image data of the plurality of images using the sprayed symbol as a reference point for an image superimposition process, wherein the capturing includes a visible light camera for a visible light image and an infrared camera for capturing a thermal image of the subject, the discharging is configured to use the content to form, on a surface of the subject, a different temperature area having a temperature that is different from those of other parts of the subject, the infrared camera is configured to capture the thermal image of a region including the different temperature area, and the image superimposition process is configured to, in an image superimposition process on the thermal image and the visible light image, use the different temperature area and the sprayed symbol as reference points in the image superimposition process.",
    "16. The image data processing method according to claim 15, comprising: capturing a surface profile of the subject; and determining whether to discharge the content onto the subject based on the captured surface profile.",
    "17. The image data processing method according to claim 16, wherein the determining whether to discharge the content onto the subject includes, in a case where the surface profile does not have unevenness deeper than a predetermined first threshold value, determining that the surface profile is a smooth surface and then determining to discharge the content.",
    "18. The image data processing method according to claim 16, wherein the determining whether to discharge the content onto the subject includes, in a case where the surface profile has a scratch or unevenness deeper than a predetermined second threshold value, determining to discharge the content onto the subject, and capturing the plurality of images includes capturing the subject at a predetermined first distance, and capturing the subject at a second distance, which is farther than the first distance.",
    "19. A non-transitory computer-readable recording medium that records a program that, when executed by a computer, causes the computer to perform steps of: discharging a content from a container incorporated in an unmanned aerial vehicle onto a subject to form a sprayed symbol; capturing a plurality of images related to a region of the subject including the sprayed symbol; and processing image data of the plurality of images using the sprayed symbol as a reference point for an image superimposition process, wherein the capturing includes a visible light camera for a visible light image and an infrared camera for capturing a thermal image of the subject, the discharging is configured to use the content to form, on a surface of the subject, a different temperature area having a temperature that is different from those of other parts of the subject, the infrared camera is configured to capture the thermal image of a region including the different temperature area, and the image superimposition process is configured to, in an image superimposition process on the thermal image and the visible light image, use the different temperature area and the sprayed symbol as reference points in the image superimposition process."
  ],
  "description_excerpt": "The present invention relates to an image data processing system, an unmanned aerial vehicle, an image data processing method, and a non-transitory computer-readable recording medium.\n\nConventionally, there has been no known example in which a sprayed symbol from a discharge apparatus of an unmanned aerial vehicle is used as a mark for image synthesis during an image superimposition process (see, for example, Patent documents 1 and 2).\n\nThere is provided an image data processing system that performs an appropriate image superimposition process on a subject that has few features to be used as a reference point for an image synthesis.\n\nIn order to solve the above-described problem, a first aspect of the present invention provides an image data processing system including a discharge apparatus that discharges the content within a container onto a subject to form a sprayed symbol, a camera that captures an image of a region including the sprayed symbol of the subject, and an image superimposition processing portion that performs an image superimposition process on the image using the sprayed symbol as a reference point.\n\nThe camera may have a visible light camera for a visible light image and an infrared camera that captures a thermal image of the subject. The discharge apparatus may use the content to form a different temperature area on the surface of the subject that has a temperature different from other parts of the subject. The infrared camera may capture a thermal image of a region including the different temperature area.",
  "cpc": [
    "G06T 3/4038",
    "B64D 1/18",
    "B64U 10/14",
    "B64U 20/87",
    "B64U 2201/10",
    "B64U 30/21",
    "G01J 5/48",
    "G01N 1/28",
    "G01N 25/72",
    "G06T 1/00",
    "G06T 2207/10032",
    "G06T 2207/10048",
    "G06T 2207/30136",
    "G06T 2207/30252",
    "G06T 7/0004",
    "G06T 7/33",
    "G06T 7/73",
    "H04N 23/23",
    "H04N 23/60",
    "H04N 23/951",
    "H04N 5/33"
  ],
  "ipc": [
    "B64C 39/02",
    "B64D 47/08",
    "G06T 3/40",
    "B64D 1/18",
    "G06T 7/00",
    "H04N 23/23",
    "G06T 3/4038"
  ],
  "assignees": [
    "Toyo Seikan Co Ltd"
  ],
  "inventors": [
    "Atsushi Kominami",
    "Souji Araki"
  ],
  "filing_date": "2020-07-21",
  "publication_date": "2024-10-01",
  "grant_date": "2024-10-01",
  "priority_date": "2019-07-23",
  "application_number": "US-202017627814-A",
  "family_id": "74193165",
  "cited_by_count": 2,
  "citations": [
    "JPH01191212A",
    "JPH06178965A",
    "WO1997037336A1",
    "US20030081827A1",
    "JP2008045888A",
    "JP5795850B2",
    "JP2014164363A",
    "US20140240451A1",
    "CN105960800A",
    "US20170006234A1",
    "WO2016098146A1",
    "KR101785202B1",
    "US20210018910A1",
    "US20190206013A1",
    "CN109417598A",
    "WO2018096577A1",
    "US20190199898A1",
    "JP2019118043A",
    "US20190205699A1",
    "CN108197563A",
    "US20220212796A1",
    "US20220225584A1",
    "US11297820B1",
    "US11921493B2"
  ]
}