Device, method and program for improving cooperation between tele-existence and head-mounted display

{
  "publication_number": "US12001018B2",
  "country": "US",
  "kind": "B2",
  "title": "Device, method and program for improving cooperation between tele-existence and head-mounted display",
  "abstract": "A control device generates a first image in which, together with an object in the real world appearing in an image captured by a camera of a robot, a virtual object appears and which presents an exterior appearance of the virtual object while the position of the camera of the robot is set as a view-point. An HMD displays the first image generated by the control device. In a case in which the position of the HMD in the height direction changes, the control device generates a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the HMD in the height direction. The HMD displays the second image in place of the first image.",
  "claims": [
    "1. An information processing device comprising: an acquisition unit that acquires an image in which an object in a real world captured by an imaging device appears; an image generation unit that generates a first image in which a virtual object appears together with the object in the real world appearing in the image acquired by the acquisition unit and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point; a display control unit that causes a head-mounted display to display the first image generated by the image generation unit; and a detection unit that detects a change in a position of the head-mounted display in a height direction, wherein, in a case in which the position of the head-mounted display in the height direction changes, the image generation unit generates a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction, and the display control unit causes the head-mounted display to display the second image in place of the first image.",
    "2. The information processing device according to claim 1, wherein the image generation unit generates, in a case in which the position of the head-mounted display in the height direction changes from down to up, a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a view-point at a position higher than the view-point in the first image.",
    "3. The information processing device according to claim 1, wherein the imaging device is mounted to the robot, and the robot is configured such that the position of the imaging device in the height direction does not change beyond a predetermined range.",
    "4. An information processing method executed by a computer, comprising: acquiring an image in which an object in a real world captured by an imaging device appears; generating a first image in which a virtual object appears together with the object in the real world appearing in the image acquired in the acquisition and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point; causing a head-mounted display to display the first image generated by the generation; detecting a change in a position of the head-mounted display in a height direction; generating, in a case in which the position of the head-mounted display in the height direction changes, a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction; and causing the head-mounted display to display the second image in place of the first image.",
    "5. A non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to perform information processing method by carrying out actions, comprising: acquiring an image in which an object in a real world captured by an imaging device appears; generating a first image in which a virtual object appears together with the object in the real world appearing in the image acquired by the acquisition and which presents an exterior appearance of the virtual object while a position of the imaging device is set as a view-point; causing a head-mounted display to display the first image generated by the generation; and detecting a change in a position of the head-mounted display in a height direction, wherein, in a case in which the position of the head-mounted display in the height direction changes, the generating includes generating a second image from which the object in the real world is deleted and which presents the exterior appearance of the virtual object viewed from a new view-point according to the change in the position of the head-mounted display in the height direction, and the causing includes causing the head-mounted display to display the second image in place of the first image."
  ],
  "description_excerpt": "The present invention relates to a data processing technology and particularly relates to an information processing device, an information processing method, and a computer program.\n\nIn recent years, there has emerged a technology called tele-existence which uses a robot positioned at a remote location, as an alter ego of a user himself or herself. The robot which is present at the remote location transmits image data and sound data on the periphery, and the image data and the sound data are reproduced on the user side, thereby enabling the user to experience such a sense of presence that the user is present at the location of the robot.\n\nMoreover, a head-mounted display (HMD) is used in various fields. It is possible to increase a sense of immersion into the video world by implementing a head tracking function in the HMD, to thereby update a display screen operationally in association with a posture of the head portion of the user.\n\nThe present inventors focused on a possibility of cooperation between the tele-existence and the head-mounted display and has developed a technology for improving user experience by the cooperation between the tele-existence and the head-mounted display.\n\nThe present invention has been made based on recognition of the above-described problem by the present inventors and has one object to provide a technology for improving user experience achieved by cooperation between the tele-existence and the head-mounted display.",
  "cpc": [
    "G02B 27/0093",
    "B25J 13/00",
    "B25J 19/023",
    "B25J 5/007",
    "G02B 2027/0187",
    "G02B 27/017",
    "G02B 27/0179",
    "G05D 1/0246",
    "G05D 1/2245",
    "G05D 1/249",
    "G05D 2105/32",
    "G05D 2109/10",
    "G06F 3/011",
    "G06F 3/012",
    "G06T 19/006",
    "H04N 5/66",
    "H04N 7/18"
  ],
  "ipc": [
    "G02B 27/00",
    "G02B 27/01",
    "G05D 1/00",
    "G06F 3/01",
    "G06T 19/00"
  ],
  "assignees": [
    "Sony Interactive Entertainment Inc",
    "Sony Group Corp"
  ],
  "inventors": [
    "Yasushi Okumura",
    "Yukari Konishi",
    "Fumihiko Iida",
    "Ryuichi Suzuki",
    "Kotaro IMAMURA"
  ],
  "filing_date": "2021-12-24",
  "publication_date": "2024-06-04",
  "grant_date": "2024-06-04",
  "priority_date": "2021-01-05",
  "application_number": "US-202118259093-A",
  "family_id": "82357725",
  "cited_by_count": 0,
  "citations": [
    "WO2012029576A1",
    "US20130194305A1",
    "JP2014072576A",
    "JP2018007180A",
    "WO2019176035A1"
  ]
}