Augmented reality overlay

{
  "publication_number": "US11989835B2",
  "country": "US",
  "kind": "B2",
  "title": "Augmented reality overlay",
  "abstract": "A computing device configured to display a virtual representation of an environment of a robot includes a display device, a memory, and a processor coupled to the memory. The processor is configured to receive data from the one or more sensors of the robot with respect to an object within an environment of the robot. The processor is also configured to display a virtual representation of the object within a virtual mapping of the environment based on the data received from the one or more sensors. The processor is further configured to receive input data selecting the virtual representation of the object. The processor is also further configured to send instructions to the robot to act in response to the received input data.",
  "claims": [
    "1. A computing device configured to display a virtual representation of an environment of a robot on a display device, the computing device comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to: receive data from a visual detection sensor and a spatial detection sensor of an other robot that was previously in the environment of the robot with respect to an object within the environment; generate point cloud data to denote coordinates of the object based on the data received from the other robot that was previously in the environment of the robot; generate, for display on the display device, a virtual representation of the object by visually imposing the data received from the spatial detection sensor upon the data received from the visual detection sensor, wherein the virtual representation of the object for display comprises a plurality of point cloud indicators generated based on the point cloud data; receive input data selecting the virtual representation of the object; output instructions for the robot to interact with the object within the environment of the robot based upon the input data selecting the virtual representation of the object; and display a navigation path of the robot generated based on mapping data of the environment received from the other robot.",
    "2. The computing device of claim 1, wherein the processor is further configured to: receive data from a user to specify or update a label of the object; associate the specified or updated label with the object; and output on the display device the object and the specified or updated label.",
    "3. The computing device of claim 1, wherein the processor is further configured to output to a different robot the data corresponding to the virtual representation of the object.",
    "4. The computing device of claim 3, wherein the processor is further configured to: receive output from the different robot comprising an updated virtual representation of the object; and output the updated virtual representation to the other robot.",
    "5. The computing device of claim 1, wherein the processor is further configured to: assign a label to the object; and display the label assigned to the object within the virtual representation of the environment of a robot.",
    "6. The computing device of claim 1, wherein the virtual representation of the environment comprises the point cloud overlaid upon a camera representation of the environment of the robot.",
    "7. The computing device of claim 6, further comprising an additional sensor utilized for at least one of object recognition, object labeling, pose estimation, a depth view, and a skeleton view.",
    "8. The computing device of claim 1, wherein the processor is further configured to: receive the mapping data from the other robot; determine an action that the robot will perform; and output, within the virtual representation of the environment of the robot, an indication of the action that the robot will perform.",
    "9. The computing device of claim 1, wherein the navigation path is updated in response to detection of an obstacle detected by the robot.",
    "10. The computing device of claim 1, wherein the processor is further configured to: receive data from a visual detection sensor and a spatial detection sensor of the robot that is presently in the environment with respect to the object with in the environment; and update the virtual representation of the object based on the data received from the robot presently in the environment.",
    "11. A system for object interaction in an environment of a robot, comprising: the robot comprising a plurality of sensors, wherein: a first sensor is configured for visual detection; and a second sensor is configured for spatial detection; a display device configured to display a virtual representation of the object within a virtual mapping of the environment based on the data received from at least one of the plurality of sensors; and a computing device configured to: receive data from the plurality of sensors of an other robot that was previously in the environment of the robot with respect to an object within the environment; generate point cloud data to denote coordinates of the object based on the data received from the other robot that was previously in the environment of the robot; generate, for display on the display device, the virtual representation of the object by visually imposing spatial data received from the second sensor upon visual data received from the first sensor, wherein the virtual representation of the object for display comprises a plurality of point cloud indicators generated based on the point cloud data; receive input data selecting the virtual representation of the object; output instructions for the robot to interact with the object within the environment of the robot based upon the input data selecting the virtual representation of the object; and display a navigation path of the robot generated based on mapping data of the environment received from the other robot.",
    "12. The system of claim 11, wherein: the robot is further configured to obtain data from the second sensor as a second point cloud; and the display device is configured to output the second point cloud representation of the environment of the robot.",
    "13. The system of claim 12, wherein the display device is further configured to generate an augmented reality view on the display device by utilizing the second point cloud representation to overlay second point cloud data and a label for an object displayed on the display device.",
    "14. The system of claim 11, wherein the display device is further configured to receive: data from the robot pertaining to an object in the environment of the robot; and data from the computing device pertaining to a different object in the environment of the robot.",
    "15. The system of claim 11, wherein the robot further comprises an on-board mapping system utilizing at least one of the plurality of sensors for obstacle avoidance.",
    "16. The system of claim 11, wherein: the robot further comprises an on-board navigation system and a motor encoder; the first sensor comprises a camera configured to provide image data for object recognition of the object; and the second sensor comprises a lidar sensor configured to generate second point cloud data of the environment of the robot.",
    "17. The system of claim 16, wherein: the computing device is configured to output the second point cloud data of the environment of the robot to the display device; the display device comprises a camera; and the display device is configured to receive the second point cloud data to overlay over a camera view provided by a camera.",
    "18. A method for object interaction in an environment of a robot, comprising: receiving data from one of a visual detection sensor and a spatial detection sensor of an other robot that was previously in the environment of the robot with respect to an object within the environment; generating point cloud data to denote coordinates of the object based on the data received from the other robot that was previously in the environment of the robot; generating, for display on a display device, a virtual representation of the object by visually imposing the data received from the spatial detection sensor upon the data received from the visual detection sensor, wherein the virtual representation of the object for display comprises a plurality of point cloud indicators generated based on the point cloud data; receiving input data selecting the virtual representation of the object; outputting instructions for the robot to interact with the object within the environment of the robot based upon the input data selecting the virtual representation of the object; and displaying a navigation path of the robot generated based on mapping data of the environment received from the other robot.",
    "19. The method of claim 18 further comprising: receiving image data pertaining to a component of the robot; outputting configuration data pertaining to the component of the robot; receiving input data to modify the configuration data; and outputting the input data to modify how an operation is performed by the component of the robot.",
    "20. The method of claim 18 further comprising: receiving image data from a camera, wherein the image data comprises a representation of a client device; determining a pose of the client device based upon a plurality of emitters on the client device; and projecting a line in a virtual mapping of the environment extending from the client device in a direction of the pose of the client device.",
    "21. The method of claim 20 further comprising instructing the robot to bring the object to the client device."
  ],
  "description_excerpt": "The present specification generally relates to augmented reality and, more particularly, to presenting an augmented reality view to users based upon data gathered from autonomous or semi-autonomous robots.\n\nUsers of camera-enabled devices sometimes use the camera to observe their surroundings. Recently, augmented reality has provided users with onscreen information about objects that appear in the view of the camera. However, the available data about objects viewed in augmented reality is limited to data from a fixed database. This only provides the user with a static augmented reality view that is not dynamic.\n\nAccordingly, a need exists for presenting an augmented reality to users utilizing data obtained by the perception of robots.\n\nA computing device configured to display a virtual representation of an environment of a robot includes a display device, a memory, and a processor coupled to the memory. The processor is configured to receive data from the one or more sensors of the robot with respect to an object within an environment of the robot. The processor is also configured to display a virtual representation of the object within a virtual mapping of the environment based on the data received from the one or more sensors. The processor is further configured to receive input data selecting the virtual representation of the object. The processor is also further configured to send instructions to the robot to act in response to the received input data.\n\nIn another embodiment, a system for object interaction in an environment of a robot includes the robot, a computing device, and a display device.",
  "cpc": [
    "G06T 19/006",
    "B25J 9/1697",
    "G05D 1/0038",
    "G05D 1/224",
    "G06F 3/04815",
    "G06V 20/10",
    "G06V 20/20",
    "G06V 20/64",
    "G06V 2201/12"
  ],
  "ipc": [
    "B25J 9/16",
    "G05D 1/00",
    "G06F 3/04815",
    "G06T 19/00",
    "G06V 20/10",
    "G06V 20/20",
    "G06V 20/64"
  ],
  "assignees": [
    "Toyota Research Institute Inc"
  ],
  "inventors": [
    "Matthew Amacker",
    "Arshan Poursohi",
    "Allison Thackston"
  ],
  "filing_date": "2018-07-18",
  "publication_date": "2024-05-21",
  "grant_date": "2024-05-21",
  "priority_date": "2017-09-26",
  "application_number": "US-201816038248-A",
  "family_id": "65808342",
  "cited_by_count": 0,
  "citations": [
    "US6144375A",
    "US20090015702A1",
    "US8755936B2",
    "US8396597B2",
    "US20140157156A1",
    "US20140064607A1",
    "US20150069248A1",
    "US9265187B2",
    "US20150298315A1",
    "US9283674B2",
    "US20150217449A1",
    "US20150341552A1",
    "US9457468B1",
    "US20160019458A1",
    "US9540171B2",
    "US20170116781A1",
    "US20170185085A1",
    "US11017610B2",
    "US20180130224A1",
    "US20180080774A1",
    "CN106500700A"
  ]
}