Processing images to remove precipitation

{
  "publication_number": "US10182225B1",
  "country": "US",
  "kind": "B1",
  "title": "Processing images to remove precipitation",
  "abstract": "Images captured by a camera system can be processed to detect precipitation in one or more of the images, and to generate a reconstructed image(s) without the precipitation, or with a reduced amount of the precipitation. Detection of precipitation can be based on a difference between a first feature in a first image and a second feature in a second image that corresponds to the first feature, where the first and second images were captured by different cameras at different times. A determination as to whether precipitation is present in the first image and the second image can be based at least in part on a disparity between the first feature and the second feature.",
  "claims": [
    "1. An image processing system to remove precipitation from images captured by cameras mounted on an unmanned aerial vehicle (UAV) comprising: a first camera of a stereo pair of cameras mounted on the UAV, the first camera being configured to capture multiple first image frames at a first frame rate; a second camera of the stereo pair of cameras configured to capture multiple second image frames at a second frame rate, wherein the first camera and the second camera do not capture image frames simultaneously, and wherein the second camera is spaced a known distance from the first camera; and a processor configured to: determine, in a first stage of multiple stages, a presence of the precipitation in the images using a noise pattern based detection scheme; confirm, in a second stage of the multiple stages, the presence of the precipitation determined in the first stage using a stereo disparity based detection scheme that comprises: aligning a second image frame of the multiple second image frames with a first image frame of the multiple first image frames based at least in part on the known distance at which the second camera is spaced from the first camera; identifying multiple first features in the first image frame that are indicative of the precipitation; identifying multiple second features in the second image frame that correspond to the multiple first features; determining a difference between the multiple first features and the multiple second features; and determining that the first image frame and the second image frame include the precipitation based at least in part on the difference; and generate, using image data associated with the first image frame or the second image frame, a reconstructed image frame without depiction of the precipitation, or with depiction of a lesser amount of the precipitation than an amount of the precipitation depicted in the first image frame or the second image frame.",
    "2. The image processing system of claim 1, wherein determining, in the first stage, the presence of the precipitation in the images using the noise pattern based detection scheme comprises: transforming spatial domain data of at least one of the first image frame or the second image frame into frequency domain data; identifying a portion of the frequency domain data that is within a predetermined frequency range associated with the presence of the precipitation in the first image frame or the second image frame; and determining that the first image frame or the second image frame includes the precipitation based at least in part on identifying the portion of the frequency domain data that is within the predetermined frequency range.",
    "3. The image processing system of claim 1, wherein the processor is further configured to classify the precipitation at a severity level among multiple possible severity levels based at least in part on visibility of an object in the first image frame or the second image frame, and wherein generating the reconstructed image frame is based at least in part on classifying the precipitation at the severity level.",
    "4. The image processing system of claim 1, wherein the processor is further configured to generate the reconstructed image frame by: identifying a first set of pixels in the first image frame that include the precipitation; identifying a second set of pixels in the second image frame corresponding to the first set of pixels that do not include the precipitation; and replacing the first set of pixels in the first image frame with the second set of pixels from the second image frame to generate the reconstructed image frame.",
    "5. A system comprising: a first image capture device; a second image capture device spaced a known distance from the first image capture device; and a processor configured to: determine, in a first stage of multiple stages, a presence of precipitation in images captured by the first image capture device and the second image capture device using a frame difference based detection scheme that compares patterns of features in multiple ones of the images; confirm, in a second stage of the multiple stages, the presence of the precipitation determined in the first stage by: aligning a first image captured by the first image capture device with a second image captured by the second image capture device based at least in part on the known distance at which the second image capture device is spaced from the first image capture device, the first image having been captured at a first time, and the second image having been captured at a second time that is different from the first time; identifying a first feature in the first image; identifying a second feature in the second image that corresponds to the first feature; and determining that the precipitation is present in the first image and the second image based at least in part on a difference between the first feature and the second feature; and generate a reconstructed image without the precipitation or with an amount of the precipitation that is less than an original amount of the precipitation in the first image or the second image.",
    "6. The system of claim 5, the processor being further configured to determine, in a third stage of the multiple stages, the presence of the precipitation by: transforming spatial domain data of at least one of the first image or the second image into frequency domain data; identifying a portion of the frequency domain data that is within a predetermined frequency range associated with the presence of the precipitation in the first image or the second image; and determining that the precipitation is present in the first image or the second image based on identifying the portion of the frequency domain data that is within the predetermined frequency range.",
    "7. The system of claim 5, wherein determining, in the first stage, the presence of the precipitation in the images using the frame difference based detection scheme comprises: identifying a first pattern of first features in the first image; identifying a second pattern of second features in a third image captured at a third time by the first image capture device, the third time being different from the first time; determining a difference between the first pattern and the second pattern; and determining that the precipitation is present in the first image based at least in part on the difference between the first pattern and the second pattern.",
    "8. The system of claim 5, the processor being further configured to classify the precipitation at a severity level among multiple possible severity levels based at least in part on visibility of an object in the first image or the second image, and wherein generating the reconstructed image is based at least in part on classifying the precipitation at the severity level.",
    "9. The system of claim 5, wherein determining, in the first stage, the presence of the precipitation in the images using the frame difference based detection scheme comprises: identifying a first pattern of first features in the first image; identifying a second pattern of second features in the second image; determining a difference between the first pattern and the second pattern; and determining that the precipitation is present in the first image and the second image based at least in part on the difference between the first pattern and the second pattern.",
    "10. The system of claim 5, wherein the processor is further configured to generate the reconstructed image by: identifying a first set of pixels in the first image that include the precipitation; identifying a second set of pixels in the second image corresponding to the first set of pixels that do not include the precipitation; and replacing the first set of pixels in the first image with the second set of pixels from the second image to generate the reconstructed image.",
    "11. The system of claim 6, wherein the processor is further configured to generate the reconstructed image by: filtering out the portion of the frequency domain data to obtain filtered frequency domain data; and transforming the filtered frequency domain data into new spatial domain data of the reconstructed image.",
    "12. The system of claim 5, further comprising a communications connection to receive weather report information over a computer network, the processor being further configured to: determine, prior to determining the presence of the precipitation in the first stage and based at least in part on the weather report information received via the communications connection, a probability of precipitation being present in an environment of the system, wherein the processor is configured to generate the reconstructed image in response to determining that the probability of precipitation being present in the environment is greater than a threshold probability.",
    "13. The system of claim 5, further comprising a sensor to detect a level of a weather related parameter in an environment of the system, the sensor comprising at least one of a humidity sensor, a barometer, a temperature sensor, or a wind speed sensor, the processor being further configured to: determine, prior to determining the presence of the precipitation in the first stage and based on the level of the weather related parameter detected by the sensor, a probability of precipitation being present in the environment of the system, wherein the processor is configured to generate the reconstructed image in response to determining that the probability of precipitation being present in the environment is greater than a threshold probability.",
    "14. The system of claim 5, the processor being further configured to: generate a three-dimensional (3D) point cloud based on the images captured by the first image capture device and the second image capture device over a time period; measure a level of degradation of the 3D point cloud over the time period; and determine, prior to determining the presence of the precipitation in the first stage and based on the level of degradation of the 3D point cloud, a probability of precipitation being present in an environment of the system, wherein the processor is configured to generate the reconstructed image in response to determining that the probability of precipitation being present in the environment is greater than a threshold probability.",
    "15. The system of claim 5, wherein the first image capture device and the second image capture device are mounted on a moving vehicle.",
    "16. The system of claim 15, wherein the moving vehicle comprises an unmanned aerial vehicle (UAV).",
    "17. A method comprising: receiving, by a processor, multiple images of a scene, the multiple images captured by a stereo pair of cameras onboard an unmanned aerial vehicle (UAV; determining, in a first stage of multiple stages, a presence of precipitation in the scene using a noise pattern based detection scheme; confirming, in a second stage of the multiple stages, the presence of the precipitation determined in the first stage by: aligning, by the processor, a first image of the multiple images with a second image of the multiple images based at least in part on a known distance between a first camera and a second camera of the stereo pair of cameras, the first image having been captured by the first camera at a first time, and the second image having been captured by the second camera at a second time that is different from the first time; identifying, by the processor, a first feature in the first image; identifying, by the processor, a second feature in the second image that corresponds to the first feature in the first image; and determining the presence of the precipitation in the scene based at least in part on a difference between the first feature and the second feature; and generating, by the processor, a reconstructed image of the scene, the reconstructed image showing the scene without the precipitation, or the scene with a lesser amount of the precipitation than an amount of the precipitation in at least one image of the multiple images.",
    "18. The method of claim 17, further comprising: generating a three-dimensional (3D) point cloud based on the multiple images captured by the first camera and the second camera over a time period; measuring a level of degradation of the 3D point cloud over the time period; and determining, prior to determining the presence of the precipitation in the first stage and based on the level of degradation of the 3D point cloud, a probability of precipitation being present in an environment of the UAV, wherein the reconstructed image is generated in response to determining that the probability of precipitation being present in the environment is greater than a threshold probability.",
    "19. The method of claim 17, wherein: identifying the first feature comprises identifying the first feature as a first background feature in the first image; determining the second feature comprises identifying the second feature as a second background feature in the second image; and determining the presence of the precipitation in the scene based at least in part on the difference comprises determining a disparity between the first feature and the second feature.",
    "20. The method of claim 17, wherein determining, in the first stage, the presence of the precipitation in the scene using the noise pattern based detection scheme comprises: transforming spatial domain data of at least one of the first image or the second image into frequency domain data; identifying a portion of the frequency domain data that is within a predetermined frequency range associated with the presence of the precipitation in the first image or the second image; and determining the presence of the precipitation in the scene based at least in part on identifying the portion of the frequency domain data that is within the predetermined frequency range."
  ],
  "description_excerpt": "Unmanned aerial vehicles (UAVs) are used by hobbyists, some commercial entities, and various militaries. Many UAVs include an integrated camera system to capture imagery of the UAV's environment. These captured images can be used for various purposes. As one example, the images can be used to allow the UAV to fly autonomously to a destination without a human operator controlling the UAV in any respect. For instance, the UAV can capture images of its environment during flight, and utilize the captured images for navigational purposes (e.g., determining its location, trajectory, etc.), safety determinations (e.g., object detection and avoidance), or to build up knowledge of the UAV's environment (e.g., building a three-dimensional (3D) model of the UAV's environment).\n\nWhen it is raining or snowing in the UAV's environment, however, it may be much more difficult for the UAV to use images for autonomous flight purposes because the rain or snow may partially occlude objects of interest (e.g., buildings, trees, etc.) in the captured images. Furthermore, in the process of building a 3D model of the UAV's environment from a collection of images, rain or snow that is present in the environment will be included in such 3D models by virtue of rain and snow being real objects that are detectable in image data. Over time, this “noise” caused by the rain or snow is compounded in the 3D model of the environment, thereby degrading the accuracy of the 3D model. This, in turn, makes it difficult for the UAV to discern its true environment during flight.",
  "cpc": [
    "H04N 13/239",
    "G06F 18/22",
    "G06K 9/52",
    "G06K 9/6215",
    "G06K 9/6267",
    "G06T 2207/10021",
    "G06T 2207/10028",
    "G06T 2207/10032",
    "G06T 2207/20056",
    "G06T 2207/20076",
    "G06T 2207/30168",
    "G06T 5/50",
    "G06T 5/77",
    "G06T 7/0002",
    "G06T 7/002",
    "G06T 7/60",
    "G06V 10/30",
    "G06V 10/761",
    "G06V 20/17",
    "G06V 20/176",
    "H04N 2013/0081",
    "H04N 23/45",
    "H04N 5/2252"
  ],
  "ipc": [
    "G06T 7/00",
    "G06T 7/60",
    "G06V 10/30",
    "H04N 13/239",
    "H04N 5/225"
  ],
  "assignees": [
    "Amazon Technologies Inc"
  ],
  "inventors": [
    "Chengwu Cui",
    "Barry James O'Brien",
    "Joshua John Watson"
  ],
  "filing_date": "2016-08-31",
  "publication_date": "2019-01-15",
  "grant_date": "2019-01-15",
  "priority_date": "2016-08-31",
  "application_number": "US-201615253602-A",
  "family_id": "64953916",
  "cited_by_count": 42,
  "citations": [
    "US20070053671A1",
    "US20090252430A1",
    "US20130004065A1",
    "US9180756B2",
    "US20140214255A1",
    "US20140286566A1",
    "US20140363045A1",
    "US20160328832A1",
    "US20170061217A1"
  ]
}