Spatially and user aware second screen projection from a companion robot or device

{
  "publication_number": "US10155166B1",
  "country": "US",
  "kind": "B1",
  "title": "Spatially and user aware second screen projection from a companion robot or device",
  "abstract": "A system is provided, including the following: a computing device that executes a video game and renders a primary video feed of the video game to a display device, the primary video feed providing a first view into a virtual space; a robot, including, a camera that captures images of a user, a projector, and, a controller that processes the images of the user to identify a gaze direction of the user; wherein when the gaze direction of the user changes from a first gaze direction that is directed towards the display device, to a second gaze direction that is directed away from the display device, the computing device generates a secondary video feed providing a second view into the virtual space; wherein the controller of the robot activates the projector to project the secondary video feed onto the projection surface in the local environment.",
  "claims": [
    "1. A system, comprising: a computing device that executes a video game and renders a primary video feed of the video game to a display device, the primary video feed providing a first view into a virtual space that is defined by the executing video game; a robot, including, a camera that captures images of a user in a local environment in which the robot is disposed, a projector, and, a controller that processes the images of the user to identify a gaze direction of the user, and that communicates the gaze direction of the user to the computing device; wherein when the gaze direction of the user changes from a first gaze direction that is directed towards the display device, to a second gaze direction that is directed away from the display device and towards a projection surface in the local environment, the computing device generates a secondary video feed of the video game and transmits the secondary video feed of the video game to the robot, the secondary video feed providing a second view into the virtual space that is changed from the first view as determined by the change in the gaze direction of the user; wherein the controller of the robot activates the projector to project the secondary video feed onto the projection surface in the local environment.",
    "2. The system of claim 1, wherein the first view into the virtual space is a forward view defined from a perspective of a virtual object of the video game that is controlled by the user; wherein the second view into the virtual space is a side view defined from the perspective of the virtual object.",
    "3. The system of claim 1, wherein the change from the first to the second view into the virtual space includes an angular rotation from the first to the second view; wherein the change from the first to the second gaze direction includes an angular rotation from the first to the second gaze direction; wherein an amount of the angular rotation from the first to the second view is determined by an amount of the angular rotation from the first to the second gaze direction.",
    "4. The system of claim 1, wherein a display surface of the display device is oriented along a first plane; wherein the projection surface is oriented along a second plane that is not substantially parallel with the first plane.",
    "5. The system of claim 1, wherein the robot further includes at least one actuator that controls a direction of the camera, to enable the camera to capture the images of the user.",
    "6. The system of claim 1, wherein the robot further includes at least one actuator that controls a direction of the projector, to enable the projector to project the secondary video feed onto the projection surface.",
    "7. The system of claim 1, wherein the robot further includes at least one actuator that controls movement of the robot to different locations within the local environment, to enable the camera to capture the images of the user and/or to enable the projector to project the secondary video feed onto the projection surface.",
    "8. A method, comprising: using a robot to scan a local environment to identify a surface for projection of video thereon; determining a spatial relationship between a display device in the local environment and the identified surface; using the robot to track a gaze direction of a user in the local environment; when the gaze direction of the user is directed towards the display device, then rendering on the display device a primary view of a virtual space; when the gaze direction of the user is directed towards the identified surface, then using the robot to project a secondary view of the virtual space onto the identified surface, wherein a spatial relationship between the primary view and the secondary view in the virtual space is determined by the spatial relationship between the display device and the identified surface.",
    "9. The method of claim 8, wherein scanning the local environment to identify the surface for projection includes capturing images of the local environment by the robot, and analyzing the captured images of the local environment to identify a substantially flat surface having a predefined minimum size.",
    "10. The method of claim 8, wherein determining the spatial relationship between the display device and the identified surface includes determining locations and/or orientations of the display device and the identified surface in the local environment.",
    "11. The method of claim 10, wherein determining the spatial relationship between the display device and the identified surface includes determining an angle formed by an intersection between a first plane along which the display device is oriented and a second plane along which the identified surface is oriented.",
    "12. The method of claim 8, wherein using the robot to scan the local environment includes moving the robot to different locations within the local environment.",
    "13. The method of claim 8, wherein the primary view of the virtual space is defined by a first view frustum defined from a virtual viewpoint and having a first direction in the virtual space; wherein the secondary view of the virtual space is defined by a second view frustum defined from the virtual viewpoint and having a second direction in the virtual space.",
    "14. A non-transitory computer readable medium having program instructions embodied thereon, that when executed by a processer, cause said processor to perform a method including the following operations: using a robot to scan a local environment to identify a surface for projection of video thereon; determining a spatial relationship between a display device in the local environment and the identified surface; using the robot to track a gaze direction of a user in the local environment; when the gaze direction of the user is directed towards the display device, then rendering on the display device a primary view of a virtual space; when the gaze direction of the user is directed towards the identified surface, then using the robot to project a secondary view of the virtual space onto the identified surface, wherein a spatial relationship between the primary view and the secondary view in the virtual space is determined by the spatial relationship between the display device and the identified surface.",
    "15. The non-transitory computer readable medium of claim 14, wherein scanning the local environment to identify the surface for projection includes capturing images of the local environment by the robot, and analyzing the captured images of the local environment to identify a substantially flat surface having a predefined minimum size.",
    "16. The non-transitory computer readable medium of claim 14, wherein determining the spatial relationship between the display device and the identified surface includes determining locations and/or orientations of the display device and the identified surface in the local environment.",
    "17. The non-transitory computer readable medium of claim 16, wherein determining the spatial relationship between the display device and the identified surface includes determining an angle formed by an intersection between a first plane along which the display device is oriented and a second plane along which the identified surface is oriented.",
    "18. The non-transitory computer readable medium of claim 14, wherein using the robot to scan the local environment includes moving the robot to different locations within the local environment.",
    "19. The non-transitory computer readable medium of claim 14, wherein the primary view of the virtual space is defined by a first view frustum defined from a virtual viewpoint and having a first direction in the virtual space; wherein the secondary view of the virtual space is defined by a second view frustum defined from the virtual viewpoint and having a second direction in the virtual space."
  ],
  "description_excerpt": "The present disclosure relates to spatially and user aware second screen projection from a companion robot or device, and related methods, apparatus, and systems.\n\nThe video game industry has seen many changes over the years. As computing power has expanded, developers of video games have likewise created game software that takes advantage of these increases in computing power. To this end, video game developers have been coding games that incorporate sophisticated operations and mathematics to produce very detailed and engaging gaming experiences.\n\nExample gaming platforms include the Sony Playstation®, Sony Playstation2® (PS2), Sony Playstation3® (PS3), and Sony Playstation4® (PS4), each of which is sold in the form of a game console. As is well known, the game console is designed to connect to a display (typically a television) and enable user interaction through handheld controllers. The game console is designed with specialized processing hardware, including a CPU, a graphics synthesizer for processing intensive graphics operations, a vector unit for performing geometry transformations, and other glue hardware, firmware, and software. The game console may be further designed with an optical disc reader for receiving game discs for local play through the game console. Online gaming is also possible, where a user can interactively play against or with other users over the Internet. As game complexity continues to intrigue players, game and hardware manufacturers have continued to innovate to enable additional interactivity and computer programs.",
  "cpc": [
    "A63F 13/655",
    "A63F 13/213",
    "A63F 13/25",
    "A63F 13/26",
    "A63F 13/428",
    "A63F 13/65",
    "A63F 13/98",
    "G06F 3/013",
    "G09G 2354/00"
  ],
  "ipc": [
    "A63F 13/25",
    "A63F 13/655",
    "A63F 9/00",
    "G06F 3/01"
  ],
  "assignees": [
    "Sony Interactive Entertainment Inc"
  ],
  "inventors": [
    "Michael Taylor",
    "Jeffrey Roger Stafford"
  ],
  "filing_date": "2017-09-08",
  "publication_date": "2018-12-18",
  "grant_date": "2018-12-18",
  "priority_date": "2017-09-08",
  "application_number": "US-201715700005-A",
  "family_id": "63722767",
  "cited_by_count": 25,
  "citations": [
    "US20120035934A1",
    "US20120272179A1",
    "US20140328505A1",
    "US20140380230A1",
    "US20150281640A1",
    "US20160378179A1",
    "US20180004286A1",
    "US9711114B1"
  ]
}