Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications

{
  "publication_number": "US11282287B2",
  "country": "US",
  "kind": "B2",
  "title": "Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications",
  "abstract": "The disclosed subject matter is directed to employing machine learning models configured to predict 3D data from 2D images using deep learning techniques to derive 3D data for the 2D images. In some embodiments, a method is provided that comprises receiving, by a system comprising a processor, a panoramic image, and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data.",
  "claims": [
    "1. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the convolutional layers minimize or eliminate edge effects associated with deriving the three-dimensional data based on wrapping around the panoramic image as projected on the two-dimensional plane.",
    "2. The method of claim 1, wherein the receiving comprises receiving the panoramic image as projected on the two-dimensional plane.",
    "3. The method of claim 1, wherein the receiving comprises receiving the panoramic image as a spherical or cylindrical panoramic image, and wherein the method further comprises: projecting, by the system, the spherical or cylindrical panoramic image on the two-dimensional plane prior to the employing the 3D-from-2D convolutional neural network model to derive the three-dimensional data.",
    "4. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: receiving a panoramic image; and employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the convolutional layers minimize or eliminate edge effects associated with deriving the three-dimensional data based on wrapping around the panoramic image as projected on the two-dimensional plane.",
    "5. The non-transitory machine-readable storage medium of claim 4, wherein the receiving comprises receiving the panoramic image as projected on the two-dimensional plane.",
    "6. The non-transitory machine-readable storage medium of claim 4, wherein the receiving comprises receiving the panoramic image as a spherical or cylindrical panoramic image, the operations further comprising: projecting the spherical or cylindrical panoramic image on the two-dimensional plane prior to the employing the 3D-from-2D convolutional neural network model to derive the three-dimensional data.",
    "7. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (4D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the 3D-from-2D convolution neural network was previously trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels.",
    "8. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein the 3D-from-2D convolution neural network was previously trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels, wherein the weighted values were decreased as the angular area of the respective pixels decreased.",
    "9. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the panoramic image processed by the preceding layer in association with deriving depth data for the panoramic image, resulting in generation of a re-projected version of the panoramic image for each of the downstream convolutional layers.",
    "10. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the panoramic image processed by the preceding layer in association with deriving depth data for the panoramic image, resulting in generation of a re-projected version of the panoramic image for each of the downstream convolutional layers and wherein the downstream convolutional layers are further configured to employ input data from the preceding layer by extracting the input data from the re-projected version of the panoramic image.",
    "11. A method, comprising: receiving, by a system comprising a processor, a panoramic image; and employing, by the system, a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the panoramic image processed by the preceding layer in association with deriving depth data for the panoramic image, resulting in generation of a re-projected version of the panoramic image for each of the downstream convolutional layers and wherein the downstream convolutional layers are further configured to employ input data from the preceding layer by extracting the input data from the re-projected version of the panoramic image, wherein the input data is exacted from the re-projected version of the panoramic image based on locations in the portion of the of the panoramic image that correspond to a defined angular receptive field based the re- proj ected version of the panoramic image.",
    "12. A method comprising: receiving, by a system operatively coupled to a processor, a request for depth data associated with a region of an environment depicted in a panoramic image; based on the receiving, deriving, by the system, depth data for an entirety of the panoramic image using a neural network model configured to derive depth data from a single two-dimensional image, wherein the neural network model comprises a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model that employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data; extracting, by the system, a portion of the depth data corresponding to the region of the environment; and providing, by the system, the portion of the depth data to an entity associated with the request.",
    "13. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: receiving a panoramic image; and employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data and wherein the 3D-from-2D convolutional neural network model was previously trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels.",
    "14. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: receiving a panoramic image; and employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data and wherein the 3D-from-2D convolutional neural network model was previously trained based on weighted values applied to respective pixels of projected panoramic images in association with deriving depth data for the respective pixels, wherein the weighted values varied based on an angular area of the respective pixels, wherein the weighted values were decreased as the angular area of the respective pixels decreased.",
    "15. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: receiving a panoramic image; and employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the panoramic image processed by the preceding layer in association with deriving depth data for the panoramic image, resulting in generation of a re-projected version of the panoramic image for each of the downstream convolutional layers.",
    "16. A non-transitory machine-readable storage medium comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: receiving a panoramic image; and employing a three-dimensional data from two-dimensional data (3D-from-2D) convolutional neural network model to derive three-dimensional data from the panoramic image, wherein the 3D-from-2D convolutional neural network model employs convolutional layers that wrap around the panoramic image as projected on a two-dimensional plane to facilitate deriving the three-dimensional data, wherein downstream convolutional layers of the convolutional layers that follow a preceding layer are configured to re-project a portion of the panoramic image processed by the preceding layer in association with deriving depth data for the panoramic image, resulting in generation of a re-projected version of the panoramic image for each of the downstream convolutional layers and wherein the downstream convolutional layers are further configured to employ input data from the preceding layer by extracting the input data from the re-projected version of the panoramic image."
  ],
  "description_excerpt": "This application generally relates to techniques for employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications.\n\nInteractive, first- person 3D immersive environments are becoming increasingly popular. In these environments, a user is able to navigate through a virtual space. Examples of these environments include first person video games and tools for visualizing 3D models of terrain. Aerial navigation tools allow users to virtually explore urban areas in three dimensions from an aerial point of view. Panoramic navigation tools (e.g. street views) allow users to view multiple 360-degree (360°) panoramas of an environment and to navigate between these multiple panoramas with a visually blended interpolation.\n\nSuch interactive 3D immersive environments can be generated from real-world environments based on photorealistic 2D images captured from the environment with 3D depth information for the respective 2D images. While methods for capturing 3D depth for 2D imagery have existed for over a decade, such methods are traditionally expensive and require complex 3D capture hardware, such as a light detection and ranging (LiDAR) devices, laser rangefinder devices, time-of-flight sensor devices, structured light sensor devices, lightfield-cameras, and the like. In addition, current alignment software remains limited in its capabilities and ease of use. For example, existing alignment methods, such as the Iterative Closest Point algorithm (ICP), require users to manually input an initial rough alignment.",
  "cpc": [
    "G06T 19/20",
    "G06T 17/00",
    "G06T 19/006",
    "G06T 2207/10016",
    "G06T 2207/10024",
    "G06T 2207/10052",
    "G06T 2207/20084",
    "G06T 2210/04",
    "G06T 7/521",
    "G06T 7/579",
    "G06T 7/593",
    "H04N 13/10",
    "H04N 13/106",
    "H04N 13/156",
    "H04N 13/204",
    "H04N 13/246",
    "H04N 13/271",
    "H04N 2013/0081",
    "H04N 2213/001"
  ],
  "ipc": [
    "G06T 19/20",
    "H04N 13/10",
    "H04N 13/106",
    "H04N 13/156",
    "H04N 13/204",
    "H04N 13/246",
    "G06T 17/00",
    "G06T 19/00",
    "G06T 7/521",
    "G06T 7/579",
    "G06T 7/593",
    "H04N 13/271"
  ],
  "assignees": [
    "Matterport Inc"
  ],
  "inventors": [
    "David Alan Gausebeck"
  ],
  "filing_date": "2018-09-25",
  "publication_date": "2022-03-22",
  "grant_date": "2022-03-22",
  "priority_date": "2012-02-24",
  "application_number": "US-201816141630-A",
  "family_id": "65023065",
  "cited_by_count": 32,
  "citations": [
    "US7054478B2",
    "US6396496B1",
    "US20030227615A1",
    "US20030231788A1",
    "US20040023612A1",
    "US20050257748A1",
    "US7697749B2",
    "US20060244746A1",
    "US20130063550A1",
    "WO2007102663A1",
    "US20080170123A1",
    "US20080310757A1",
    "US20110018973A1",
    "US20100097444A1",
    "US20100194860A1",
    "US8973029B2",
    "US20110102763A1",
    "US20110157696A1",
    "US20110211042A1",
    "US10860100B2",
    "US9001227B2",
    "US8957919B2",
    "US20120185094A1",
    "US20120105574A1",
    "US20120162366A1",
    "US20120223937A1",
    "US20180139431A1",
    "US20160005179A1",
    "US20150288877A1",
    "US20160249039A1",
    "US20160291136A1",
    "US9609307B1",
    "US20180144555A1",
    "US10154244B2",
    "US20170278289A1",
    "US20180253869A1",
    "US20180276841A1",
    "US20190259170A1"
  ]
}