Method and system for generating a range image using sparse depth data

{
  "publication_number": "US10706505B2",
  "country": "US",
  "kind": "B2",
  "title": "Method and system for generating a range image using sparse depth data",
  "abstract": "A system and method for generating a range image using sparse depth data is disclosed. The method includes receiving, by a controller, image data of a scene. The image data includes a first set of pixels. The method also includes receiving, by the controller, a sparse depth data of the scene. The sparse depth data includes a second set of pixels, and the number of the second set of pixels is less than the number of first set of pixels. The method also includes combining the image data and the sparse depth data into a combined data. The method also includes generating a range image using the combined data.",
  "claims": [
    "1. A method, the method comprising: receiving, by a controller, image data of a scene, wherein the image data comprises a first set of pixels comprising a first order of magnitude; receiving, by the controller, a sparse depth data of the scene, wherein the sparse depth data comprises a second set of pixels comprising a second order of magnitude, wherein the second order of magnitude is less than the first order of magnitude by at least one order of magnitude; performing a feature extraction and regression process on the image data to generate a features vector, wherein said feature extraction and regression process is performed by a trained neural network, wherein said trained neural network is trained to reduce a depth error between a generated depth data and an actual depth data; performing a first normalization on the sparse depth data to generate a depth vector, wherein the first normalization comprises modifying a range of values of the sparse depth data to match a range of values of the features vector; combining the features vector and the depth vector into a combined vector; and generating a range image using the combined vector.",
    "2. The method of claim 1, wherein the second set of pixels corresponds to a fixed number of pixels arranged at fixed pixel locations.",
    "3. The method of claim 1, wherein the receiving the image data comprises receiving the image data from a monocular camera.",
    "4. The method of claim 1, further comprising performing a second normalization on the combined vector, wherein the second normalization comprises modifying the combined vector into a specific vector length.",
    "5. The method of claim 1, wherein said trained neural network is trained to preserve a consistent ordering of depth of pixels.",
    "6. The method of claim 1, wherein the features vector reflects identified spatial relationships between different identifiable features.",
    "7. The method of claim 1 wherein the feature extraction process identifies spatial relationships between different identifiable features within the scene.",
    "8. The method of claim 1 wherein the second order of magnitude is less than the first order of magnitude by at least two orders of magnitude.",
    "9. A system within a vehicle, comprising: a camera; a range sensor; an electronic controller configured to: receive, from the camera, image data of a scene, wherein the image data comprises a first set of pixels comprising a first order of magnitude; receive, from the range sensor, a sparse depth data of the scene, wherein the sparse depth data comprises a second set of pixels comprising a second order of magnitude, wherein the second order of magnitude is less than the first order of magnitude by at least one order of magnitude; perform a feature extraction and regression process on the image data to generate a features vector, wherein said feature extraction and regression process is performed by a trained neural network, wherein said trained neural network is trained to reduce a depth error between a generated depth data and an actual depth data; perform a first normalization on the sparse depth data to generate a depth vector, wherein the first normalization comprises modifying a range of values of the sparse depth data to match a range of values of the features vector; combine the features vector and the depth vector into a combined vector; and generate a range image using the combined vector.",
    "10. The system of claim 9, wherein the second set of pixels corresponds to a fixed number of pixels arranged at fixed pixel locations.",
    "11. The system of claim 9, wherein the receiving the image data comprises receiving the image data from a monocular camera.",
    "12. The system of claim 9, wherein the controller is further configured to perform a second normalization on the combined vector, wherein the second normalization comprises modifying the combined vector into a specific vector length.",
    "13. The system of claim 9, wherein said trained neural network is trained to preserve a consistent ordering of depth of pixels.",
    "14. The system of claim 9, wherein the features vector reflects identified spatial relationships between different identifiable features.",
    "15. The system of claim 9 wherein the feature extraction process identifies spatial relationships between different identifiable features within the scene.",
    "16. The system of claim 9 wherein the second order of magnitude is less than the first order of magnitude by at least two orders of magnitude."
  ],
  "description_excerpt": "The subject embodiments relate to generating a range image using sparse depth data. Specifically, one or more embodiments can be directed to generating a high-resolution range image by using at least one camera and at least one range sensor. One or more embodiments can generate the high-resolution range image by combining image data (that is captured by the at least one camera) with sparse depth data (that is captured by the at least one range sensor), for example.\n\nA range image is a two-dimensional image where distances between a specific point (i.e., a location where a range sensor is positioned) and points within a scene of the two-dimensional image are reflected by the two-dimensional image. With certain range images, the pixels that make up the range images can include values that correspond to the distances between the specific point and the points within the captured scene.\n\nIn one exemplary embodiment, a method includes receiving, by a controller, image data of a scene. The image data includes a first set of pixels. The method also includes receiving, by the controller, a sparse depth data of the scene. The sparse depth data includes a second set of pixels, and the number of the second set of pixels is less than the number of first set of pixels. The method also includes combining the image data and the sparse depth data into a combined data. The method also includes generating a range image using the combined data.\n\nIn another exemplary embodiment, the method also includes performing a feature extraction process on the image data to generate a features vector.",
  "cpc": [
    "G01S 17/86",
    "G01S 17/89",
    "G06K 9/42",
    "G06K 9/4604",
    "G06T 2207/10024",
    "G06T 2207/10028",
    "G06T 2207/10044",
    "G06T 2207/20084",
    "G06T 2207/20212",
    "G06T 2207/30252",
    "G06T 3/4053",
    "G06T 5/50",
    "G06T 5/60",
    "G06T 5/73",
    "G06T 7/50",
    "G06T 7/521",
    "G06T 7/55"
  ],
  "ipc": [
    "G01S 17/86",
    "G01S 17/89",
    "G06K 9/42",
    "G06K 9/46",
    "G06T 3/40",
    "G06K 9/00",
    "G06K 9/62",
    "G06T 15/00",
    "G06T 7/50"
  ],
  "assignees": [
    "GM Global Technology Operations LLC"
  ],
  "inventors": [
    "Wei Tong",
    "Shuqing Zeng",
    "Upali P. Mudalige"
  ],
  "filing_date": "2018-01-24",
  "publication_date": "2020-07-07",
  "grant_date": "2020-07-07",
  "priority_date": "2018-01-24",
  "application_number": "US-201815878937-A",
  "family_id": "67145406",
  "cited_by_count": 4,
  "citations": [
    "US6269153B1",
    "US20050105827A1",
    "US20050196035A1",
    "US20050286767A1",
    "US20090196491A1",
    "US7672504B2",
    "US20100172567A1",
    "US20090123070A1",
    "US20100309292A1",
    "US8340402B2",
    "US8643701B2",
    "US20110211749A1",
    "US20120182392A1",
    "US20110299736A1",
    "US20120039525A1",
    "US20140002611A1",
    "US20130230235A1",
    "US20120250978A1",
    "US20120163701A1",
    "US20120294527A1",
    "US20130010070A1",
    "US20140254874A1",
    "US20130236089A1",
    "US20130129192A1",
    "US20130129193A1",
    "US20130301906A1",
    "US9030470B2",
    "US20150235351A1",
    "US20140153816A1",
    "US20140169701A1",
    "US20150206318A1",
    "US20140368615A1",
    "US20150015569A1",
    "US20160231866A1",
    "US20150248764A1",
    "US20150279016A1",
    "US20160134858A1",
    "US20160198147A1",
    "US20180059225A1",
    "US20170180639A1",
    "US20170200273A1",
    "US20180211403A1",
    "US20180211128A1"
  ]
}