Remote operation apparatus for communication with a user

{
  "publication_number": "US12136248B2",
  "country": "US",
  "kind": "B2",
  "title": "Remote operation apparatus for communication with a user",
  "abstract": "In a robot (3) in a remote location, an action scene of the robot (3) is determined based on a feature amount derived from its position and motion detection data and video data, and a video parameter or an imaging mode corresponding to the determined action scene is selected. Then, a process of adjusting the selected video parameter for the video data or a process of setting the selected imaging mode to the camera is performed, and the processed video data is transmitted to the information processing apparatus (2) on the user side via the network (4) and displayed on the HMD (1).",
  "claims": [
    "1. A remote operation apparatus capable of communicating with a user apparatus connected to a display unit via a network, the remote operation apparatus comprising: a controller comprising a hardware processor configured to: acquire, from a camera, video data of a target captured by the camera; acquire, from a sensor, state detection data representing at least one of a position or an operation of the remote operation apparatus detected by the sensor; determine an action scene of the remote operation apparatus based on the acquired video data and the acquired state detection data; select a video parameter to be adjusted in accordance with the determined action scene; adjust a selected video parameter for the video data; and transmit, to the user apparatus via the network, the video data whose video parameter has been adjusted, wherein the controller is configured to determine, based on the video data and the state detection data, whether the action scene is a first scene in which a target mainly in a stationary state is captured by the camera, the controller is configured to select a spatial resolution as the video parameter or the imaging mode in a case where the action scene is determined to be the first scene, and the controller is configured to perform a process of adjusting the selected spatial resolution for the video data or a process of setting an imaging mode for obtaining the selected spatial resolution to the camera.",
    "2. The remote operation apparatus according to claim 1, wherein the controller is configured to further determine whether a color tone of the target is mainly monochrome based on the video data, in a case where the action scene is determined to be the first scene, the controller is configured to further select luminance as the video parameter, in a case where the color tone of the target is determined to be mainly monochrome, and the controller is configured to perform a process of further adjusting the selected luminance for the video data or a process of setting an imaging mode for obtaining the selected luminance to the camera.",
    "3. The remote operation apparatus according to claim 1, wherein the controller is configured to further determine whether a color tone of the target is mainly color based on the video data, in a case where the action scene is determined to be the first scene, the controller is configured to further select a color tone as the video parameter or the imaging mode, in a case where the color tone of the target is determined to be mainly color, and the controller is configured to perform a process of further adjusting the selected color tone for the video data or a process of setting an imaging mode for obtaining the selected color tone to the camera.",
    "4. A non-transitory computer-readable medium recording a program for causing the hardware processor included in the remote operation apparatus according to claim 1 to execute processing to be performed by the controller included in the remote operation apparatus.",
    "5. A remote operation apparatus capable of communicating with a user apparatus connected to a display unit via a network, the remote operation apparatus comprising: a controller comprising a hardware processor configured to: acquire, from a camera, video data of a target captured by the camera; acquire, from a sensor, state detection data representing at least one of a position or an operation of the remote operation apparatus detected by the sensor; determine an action scene of the remote operation apparatus based on the acquired video data and the acquired state detection data; select an imaging mode corresponding to the determined action scene; set the selected imaging mode to the camera; and transmit, to the user apparatus via the network, the video data captured by the camera using the set imaging mode, wherein the controller is configured to determine, based on the video data and the state detection data, whether the action scene is a first scene in which a target mainly in a stationary state is captured by the camera, the controller is configured to select a spatial resolution as the video parameter or the imaging mode in a case where the action scene is determined to be the first scene, and the controller is configured to perform a process of adjusting the selected spatial resolution for the video data or a process of setting an imaging mode for obtaining the selected spatial resolution to the camera.",
    "6. The remote operation apparatus according to claim 5, wherein the controller is configured to further determine whether a color tone of the target is mainly monochrome based on the video data, in a case where the action scene is determined to be the first scene, the controller is configured to further select luminance as the video parameter, in a case where the color tone of the target is determined to be mainly monochrome, and the controller is configured to perform a process of further adjusting the selected luminance for the video data or a process of setting an imaging mode for obtaining the selected luminance to the camera.",
    "7. The remote operation apparatus according to claim 5, wherein the controller is configured to further determine whether a color tone of the target is mainly color based on the video data, in a case where the action scene is determined to be the first scene, the controller is configured to further select a color tone as the video parameter or the imaging mode, in a case where the color tone of the target is determined to be mainly color, and the controller is configured to perform a process of further adjusting the selected color tone for the video data or a process of setting an imaging mode for obtaining the selected color tone to the camera.",
    "8. A remote operation apparatus capable of communicating with a user apparatus connected to a display unit via a network, the remote operation apparatus comprising: a controller comprising a hardware processor configured to: acquire, from a camera, video data of a target captured by the camera; acquire, from a sensor, state detection data representing at least one of a position or an operation of the remote operation apparatus detected by the sensor; determine an action scene of the remote operation apparatus based on the acquired video data and the acquired state detection data; select a video parameter to be adjusted in accordance with the determined action scene; adjust a selected video parameter for the video data; and transmit, to the user apparatus via the network, the video data whose video parameter has been adjusted, wherein the controller is configured to determine, based on the video data and the state detection data, whether the action scene is a second scene in which a target mainly in a moving state is captured by the camera, the controller is configured to select a time resolution as the video parameter or the imaging mode in a case where the action scene is determined to be the second scene, and the controller is configured to perform a process of adjusting the selected time resolution for the video data or a process of setting an imaging mode for obtaining the selected spatial resolution to the camera.",
    "9. The remote operation apparatus according to claim 8, wherein the controller is configured to determine whether the target is outdoors based on the video data, in a case where the action scene is determined to be the second scene, the controller is configured to further select a viewing angle as the video parameter or the imaging mode, in a case where the target is determined to be outdoors, and the controller is configured to perform a process of further adjusting the selected viewing angle for the video data or a process of setting an imaging mode for obtaining the selected viewing angle to the camera.",
    "10. The remote operation apparatus according to claim 8, wherein the controller is configured to determine whether the target is indoors based on the video data, in a case where the action scene is determined to be the second scene, the controller is configured to further select a stereoscopic effect as the video parameter or the imaging mode, in a case where the target is determined to be indoors, and the controller is configured to perform a process of further adjusting the selected stereoscopic effect for the video data or a process of setting an imaging mode for obtaining the selected stereoscopic effect to the camera.",
    "11. A remote operation apparatus capable of communicating with a user apparatus connected to a display unit via a network, the remote operation apparatus comprising: a controller comprising a hardware processor configured to: acquire, from a camera, video data of a target captured by the camera; acquire, from a sensor, state detection data representing at least one of a position or an operation of the remote operation apparatus detected by the sensor; determine an action scene of the remote operation apparatus based on the acquired video data and the acquired state detection data; select an imaging mode corresponding to the determined action scene; set the selected imaging mode to the camera; and transmit, to the user apparatus via the network, the video data captured by the camera using the set imaging mode, wherein the controller is configured to determine, based on the video data and the state detection data, whether the action scene is a second scene in which a target mainly in a moving state is captured by the camera, the controller is configured to select a time resolution as the video parameter or the imaging mode in a case where the action scene is determined to be the second scene, and the controller is configured to perform a process of adjusting the selected time resolution for the video data or a process of setting an imaging mode for obtaining the selected spatial resolution to the camera.",
    "12. The remote operation apparatus according to claim 6, wherein the controller is configured to determine whether the target is outdoors based on the video data, in a case where the action scene is determined to be the second scene, the controller is configured to further select a viewing angle as the video parameter or the imaging mode, in a case where the target is determined to be outdoors, and the controller is configured to perform a process of further adjusting the selected viewing angle for the video data or a process of setting an imaging mode for obtaining the selected viewing angle to the camera.",
    "13. The remote operation apparatus according to claim 11, wherein the controller is configured to determine whether the target is indoors based on the video data, in a case where the action scene is determined to be the second scene, the controller is configured to further select a stereoscopic effect as the video parameter or the imaging mode, in a case where the target is determined to be indoors, and the controller is configured to perform a process of further adjusting the selected stereoscopic effect for the video data or a process of setting an imaging mode for obtaining the selected stereoscopic effect to the camera."
  ],
  "description_excerpt": "Embodiments described herein relate generally to a remote operation apparatus capable of communicating with, for example, a user apparatus via a network, and a program used in the remote operation apparatus.\n\nA system has been developed in which a user remotely controls an operation of a robot located in a remote location, while watching a video image transmitted from the robot via a network. In a system of this type, for example, a head mounted display (HMD) (hereinafter also referred to as a headset) is put on, for example, the head of a user. A user moves his/her body or operates a controller while watching a video image captured by a robot located in a remote location through a headset, thereby remotely controlling the motion of the robot.\n\nFurthermore, in a system of this type, for example, an augmented reality (AR) technique or a virtual reality (VR) technique is used to generate an AR video image or a VR video image based on a video image transmitted from a robot and to display the generated video image on a headset, so that a user can obtain a high sense of immersion.\n\nHowever, the conventional system has the following problems to be solved. That is, in a case where a captured video image is transmitted via a network, a simple system in which a compression condition of the video image is fixed is affected by a communication environment. In particular, in a wireless environment, when a user moves to an environment in which radio waves are weak, a communication band that allows transmission within a certain period of time becomes narrow.",
  "cpc": [
    "G06V 10/56",
    "B25J 11/008",
    "G06F 3/14",
    "G06F 3/147",
    "G06T 2207/10024",
    "G06T 7/70",
    "G06V 10/60",
    "G06V 20/36",
    "G06V 20/38",
    "G09G 2320/0613",
    "G09G 2320/106",
    "G09G 2350/00",
    "G09G 2354/00",
    "G09G 2360/10",
    "G09G 2370/02",
    "G09G 2370/04",
    "G09G 2370/16",
    "G09G 2380/14",
    "G09G 2380/16",
    "G09G 3/003",
    "G09G 5/028",
    "H04N 13/106",
    "H04N 13/344",
    "H04N 13/398",
    "H04N 7/188"
  ],
  "ipc": [
    "G06F 3/14",
    "G06T 7/70",
    "G06V 10/56",
    "G06V 10/60",
    "G06V 20/00",
    "H04N 13/106"
  ],
  "assignees": [
    "NTT Communications Corp"
  ],
  "inventors": [
    "Toshiya NAKAKURA"
  ],
  "filing_date": "2022-11-10",
  "publication_date": "2024-11-05",
  "grant_date": "2024-11-05",
  "priority_date": "2020-05-14",
  "application_number": "US-202218054237-A",
  "family_id": "74529660",
  "cited_by_count": 0,
  "citations": [
    "US6611242B1",
    "US20050267826A1",
    "JP2008211405A",
    "JP2016072686A",
    "US20170195549A1",
    "US20200007746A1",
    "US20190068749A1",
    "US20200084424A1",
    "US20220224865A1",
    "CN107357316A",
    "JP2019106628A",
    "US20200319463A1",
    "US10969591B2",
    "US20190212752A1",
    "JP2019198041A"
  ]
}