High dynamic range imaging for event detection and inventory management

{
  "publication_number": "US10863105B1",
  "country": "US",
  "kind": "B1",
  "title": "High dynamic range imaging for event detection and inventory management",
  "abstract": "Described are systems and methods for generating high dynamic range (“HDR”) images based on image data obtained from different image sensors for use in detecting events and monitoring inventory within a materials handling facility. The different image sensors may be aligned and calibrated and the image data from the sensors may be generated at approximately the same time but at different exposures. The image data may then be preprocessed, matched, aligned, and blended to produce an HDR image that does not include overexposed regions or underexposed regions.",
  "claims": [
    "1. A system, comprising: a first image sensor oriented toward an inventory location within a materials handling facility; a second image sensor horizontally aligned with the first image sensor, at a first distance from the first image sensor and oriented toward the inventory location; one or more processors; and a memory storing program instructions that when executed by the one or more processors cause the one or more processors to at least: receive, from the first image sensor, first image data obtained at a first time and corresponding to a first image of the inventory location obtained at a first exposure, wherein a first region of the first image is overexposed and a first plurality of pixels within the first region do not include data representative of the inventory location; receive, from the second image sensor, second image data obtained at the first time, and corresponding to a second image of the inventory location obtained at a second exposure that is different than the first exposure; preprocess the first image data and the second image data, by at least: perspective warping a first channel of the first image data and a second channel of the second image data according to at least one rectification homography to produce a rectified first channel and a rectified second channel that correlates rows of pixels of the first image data with rows of pixels of the second image data; determine, based at least in part on the rectified first channel or the rectified second channel, a first boundary of pixels of the first image data surrounding the first region; alter the second image data to form altered image data that includes a second region that is overexposed and corresponds to the first region of the first image; determine, based at least in part on the rectified first channel or the rectified second channel, a second boundary of pixels of the second image data surrounding the second region; form third image data that includes pixel values of a first plurality of pixels of the first image data that are outside the first boundary and pixel values of a second plurality of pixels of the second image data that are inside the second boundary; and blend at least a portion of the pixel values of the third image data with a third plurality of pixels of the second image data.",
    "2. The system of claim 1, further comprising: a third image sensor oriented toward the inventory location; and wherein the program instructions further cause the one or more processors to at least: receive, from the third image sensor, fourth image data obtained at the first time, and corresponding to a third image of the inventory location obtained at a third exposure that is different than the first exposure and the second exposure; determine a third boundary of pixels of the fourth image data surrounding a third region of pixels within the fourth image data; and form fifth image data that includes pixel values of a third plurality of pixels of the fourth image data that is inside the third boundary and pixel values of the third image data.",
    "3. The system of claim 1, wherein the program instructions further cause the one or more processors to at least: generate a first binary representation of a first luminance channel of the first image data; generate a second binary representation of a second luminance channel of the second image data; and wherein the first region is determined at least in part from the first binary representation.",
    "4. The system of claim 3, wherein the program instructions further cause the one or more processors to at least: transform the first binary representation into a first compressed representation; and wherein the first region is determined at least in part from the first compressed representation.",
    "5. A computer implemented method, comprising: under control of one or more computing systems configured with executable instructions, receiving, from a first image sensor, first image data representative of a first image obtained at a first exposure, wherein a first region of the first image is overexposed; receiving, from a second image sensor, second image data representative of a second image obtained at a second exposure that is different than the first exposure; preprocessing the first image data and the second image data, wherein preprocessing includes: perspective warping a first channel of the first image data and a second channel of the second image data according to at least one rectification homography to produce a rectified first channel and a rectified second channel that correlates rows of pixels of the first image data with rows of pixels of the second image data; determining, based at least in part on one or more of the rectified first channel or the rectified second channel, a first boundary of pixels of the first image data surrounding the first region; determining, based at least in part on one or more of the rectified first channel or the rectified second channel, a second boundary of pixels of the second image data surrounding a second region within the second image data that corresponds to the first region; forming third image data that includes: pixel values of a first plurality of pixels of the first image data that are outside the first boundary of pixels of the first image data, and pixel values of a second plurality of pixels of the second image data that are inside the second boundary of pixels of the second image data; and blending pixel values of at least a portion of the first plurality of pixels with pixel values of at least a portion of the second image data.",
    "6. The computer implemented method of claim 5, further comprising: aligning at least a portion of the first boundary of pixels of the first image data with at least a portion of the second boundary of pixels of the second image data to determine an aligned relationship between the first image data and the second image data.",
    "7. The computer implemented method of claim 5, wherein the first image is taken at a first time and the second image is taken at a second time that is different than the first time.",
    "8. The computer implemented method of claim 5, wherein determining a second boundary includes: altering the second image data to cause pixels of the second image data to become overexposed and form a second region having a similar shape and a similar size as the first region.",
    "9. The computer implemented method of claim 5, wherein the first exposure is based on one or more of a shutter speed corresponding to the first image sensor, an aperture corresponding to the first image sensor, or a sensitivity of the first image sensor.",
    "10. The computer implemented method of claim 5, wherein determining a first boundary includes: generating a first binary representation of the first channel such that each pixel of the first image data has a corresponding binary value; transforming the first binary representation into a first compressed representation; and determining a first boundary within the first compressed representation that includes a first plurality of pixels having a first binary value that are adjacent to pixels having a second binary value.",
    "11. The computer implemented method of claim 10, wherein determining a second boundary includes: generating a second binary representation of the second channel such that each pixel of the first image data has a corresponding binary representation; and determining a second boundary within the second binary representation that includes a second plurality of pixels having the first binary value that are adjacent to pixels having the second binary value.",
    "12. The computer implemented method of claim 11, further comprising: matching at least a portion of the first plurality of pixels with at least a portion of the second plurality of pixels; and updating a disparity map indicating a disparity between the first image data and the second image data, wherein the disparity map indicates a horizontal offset between pixels of the first image data and pixels of the second image data.",
    "13. The computer implemented method of claim 5, further comprising: aligning the first image data and the second image data, based at least in part on the first boundary and the second boundary so that pixels of the first boundary of pixels are aligned with pixels of the second boundary of pixels.",
    "14. The computer implemented method of claim 5, wherein forming the third image data includes: extracting, from the second image data, at least one channel of the second image data corresponding to pixels within the second boundary; extracting, from the first image data, at least one channel of the first image data corresponding to pixels outside the first boundary; and forming third image data that includes the at least one channel extracted from the second image data and the at least one channel extracted from the first image data.",
    "15. The computer implemented method of claim 5, wherein the first image data corresponds to a YUV color space.",
    "16. A system, comprising: a first image sensor having a first field of view that includes an object; a second image sensor having a second field of view that includes the object; one or more processors; and a memory storing program instructions that when executed by the one or more processors cause the at least one processor to at least: receive, from the first image sensor, first image data having a first exposure, wherein the first image data includes a first plurality of pixels, each of the first plurality of pixels having a corresponding first luminance channel, first color channel, and second color channel, wherein at least some of the first plurality of pixels are overexposed; receive, from the second image sensor, second image data having a second exposure that is different than the first exposure, wherein the second image data includes a second plurality of pixels, each of the second plurality of pixels having a corresponding second luminance channel, third color channel, and fourth color channel; preprocess the first image data and the second image data, by at least: perspective warping the first luminance channel and the second luminance channel according to at least one rectification homography to produce a rectified first luminance channel and a rectified second luminance channel that correlates rows of pixels of the first image data with rows of pixels of the second image data; determine, based at least in part on the rectified first luminance channel or the rectified second luminance channel, a first region that includes a third plurality of pixels of the first image data that are overexposed; determine a second region that includes a fourth plurality of pixels of the second image data that corresponds to the third plurality of pixels; match the first plurality of pixels and the second plurality of pixels, based at least in part on the first region and the second region to determine a disparity between the first image data and the second image data; align the first plurality of pixels and the second plurality of pixels, based at least in part on the disparity; and form combined image data that includes pixel values of the fourth plurality of pixels of the second image data that are within the second region and pixel values of a seventh plurality of pixels of the first image data that are outside the first region, wherein each pixel of the combined image data includes pixel values having a corresponding third luminance channel, fifth color channel, and sixth color channel.",
    "17. The system of claim 16, wherein the program instructions further cause the one or more processors to at least: blend at least one of the third luminance channel, the fifth color channel or the sixth color channel with a corresponding first luminance channel, first color channel, or second color channel.",
    "18. The system of claim 16, further comprising: a third image sensor having a third field of view that includes the object; and wherein the program instructions further cause the one or more processors to at least: receive, from the third image sensor, fourth image data having a third exposure that is different than the first exposure and the second exposure; align an eighth plurality of pixels of the fourth image data with pixels of the first image data; and wherein the combined image data further includes pixel values corresponding to at least a portion of the eighth plurality of pixels of the fourth image data.",
    "19. The system of claim 16, wherein the first image sensor and the second image sensor are horizontally aligned.",
    "20. The system of claim 16, wherein the disparity is based at least in part on a distance between the first image sensor and the second image sensor."
  ],
  "description_excerpt": "Today, imaging devices such as digital cameras are frequently used for conducting surveillance or monitoring operations. For example, digital cameras are often used to monitor the arrivals or departures of goods or the performance of services in materials handling facilities such as warehouses, fulfillment centers, retail establishments or other like facilities, as well as the travels of persons or objects in locations such as airports, stadiums or other dense environments, or the flow of traffic on one or more sidewalks, roadways or highways. Digital cameras are commonplace in financial settings such as banks or casinos, where money changes hands in large amounts or at high rates of speed.\n\nWhen conducting surveillance or monitoring operations, digital cameras may be aligned and configured to capture imaging data such as still or moving images of actions or events occurring within their respective fields of view. Information regarding the captured imaging data or the observed actions or events may be subjected to further analysis to identify aspects, elements or features of the content expressed therein. In performing such operations, digital cameras may be utilized alone or in groups and objects may be tracked as they move between fields of views of different cameras.\n\nIn environments with varying lighting conditions, such as materials handling facilities, transportation centers, financial institutions or like structures in which different light sources are present in different areas, it may be difficult to determine events in regions of images that become overexposed by excessive light from the different light sources.",
  "cpc": [
    "G06Q 10/087",
    "G06F 18/22",
    "G06F 18/251",
    "G06K 9/4661",
    "G06K 9/6201",
    "G06T 2207/10024",
    "G06T 2207/10144",
    "G06T 2207/20208",
    "G06T 2207/20221",
    "G06T 2207/30232",
    "G06T 5/50",
    "G06T 5/90",
    "G06T 7/337",
    "G06V 10/60",
    "G06V 10/758",
    "G06V 10/803",
    "G06V 20/52",
    "H04N 17/002",
    "H04N 23/45",
    "H04N 23/57",
    "H04N 23/741",
    "H04N 23/90",
    "H04N 5/2258",
    "H04N 5/2355",
    "H04N 5/44504"
  ],
  "ipc": [
    "G06K 9/46",
    "G06K 9/62",
    "G06Q 10/08",
    "G06T 7/33",
    "H04N 5/225",
    "H04N 5/235",
    "H04N 5/445"
  ],
  "assignees": [
    "Amazon Technologies Inc"
  ],
  "inventors": [
    "Eliezer Rosengaus",
    "Abhinav Kashyap",
    "Lorenzo Sorgi"
  ],
  "filing_date": "2017-06-27",
  "publication_date": "2020-12-08",
  "grant_date": "2020-12-08",
  "priority_date": "2017-06-27",
  "application_number": "US-201715635031-A",
  "family_id": "73653707",
  "cited_by_count": 13,
  "citations": [
    "US20030002712A1",
    "US20040181467A1",
    "US20080109114A1",
    "US8688598B1",
    "US7225980B2",
    "US8750636B2",
    "US20070047833A1",
    "US20080043108A1",
    "US8175925B1",
    "US20080055087A1",
    "US20080077511A1",
    "US8189855B2",
    "US20110011936A1",
    "US7949568B2",
    "US8630924B2",
    "US8009864B2",
    "US20090121017A1",
    "US8423431B1",
    "US20090245573A1",
    "US8977073B2",
    "US8896624B2",
    "US20100259626A1",
    "US8228400B2",
    "US8570396B2",
    "US9055231B2",
    "US20110176024A1",
    "US8699821B2",
    "US20120210229A1",
    "US9019402B2",
    "US20120284132A1",
    "US20130076898A1",
    "US8965120B2",
    "US9384539B2",
    "US20130253700A1",
    "US20130329004A1",
    "US20140176731A1",
    "US20140176730A1",
    "US9349166B2",
    "US20150073907A1",
    "US20140198226A1",
    "US20140279294A1",
    "US9117134B1",
    "US20140307960A1",
    "US20140347521A1",
    "US9131201B1",
    "US20140362223A1",
    "US20150019391A1",
    "US9473747B2",
    "US20150043811A1",
    "US20150116525A1",
    "US20150170389A1",
    "US20170018062A1",
    "US20150348242A1",
    "US20150350509A1",
    "US20150350547A1",
    "US9483704B2",
    "US20160267695A1",
    "US9600741B1",
    "US20180082408A1",
    "US20160371634A1",
    "US20170069060A1",
    "US20170163966A1",
    "US9762893B2",
    "US10477185B2",
    "US10187628B2",
    "US20170318243A1",
    "US20170359534A1",
    "US10129485B2",
    "US20180241929A1",
    "US20180035058A1",
    "US20180060804A1",
    "US20180060803A1",
    "US9916644B1",
    "US20180160051A1",
    "US10165194B1",
    "US20180374233A1",
    "US20190147598A1",
    "US10498963B1",
    "US10706587B1"
  ]
}