VR body tracking without external sensors

{
  "publication_number": "US10777006B2",
  "country": "US",
  "kind": "B2",
  "title": "VR body tracking without external sensors",
  "abstract": "Plural individual sensor assemblies are engaged with respective parts of a person's body. Each assembly may include accelerometers, magnetometers, and gyroscopes. Sensor data is fused together to get the orientation at each body location. To simplify, the body is assumed to consist of rigid bars of known length connected with ball joints so that once the relative orientations of all bars are given by the respective assemblies, body pose can be computed. Then the body pose is translated as a virtual body into a virtual world either by a ray cast method that anchors a foot of the virtual body to the ground assuming infinite gravity and infinite friction and then translating the other body parts to make the ground contact point fixed, or by implementing an approximate dynamics physics engine on the virtual body. The technique may be used in VR location-based entertainment and for motion capture.",
  "claims": [
    "1. A method comprising: engaging plural motion sensor assemblies (MSA) to respective body parts, the MSA outputting pose information related to the respective body parts; providing the pose information to a display apparatus for presenting an image of the body parts having respective poses according to the pose information and rendering the image using ray cast in which a foot of the image is constrained to remain on virtual ground in virtual space.",
    "2. The method of claim 1, comprising rendering the image using ray cast in which the foot of the image is constrained to remain on virtual ground with infinite gravity and infinite friction.",
    "3. The method of claim 1, comprising rendering the image using a modeled dynamics physics engine.",
    "4. The method of claim 1, comprising providing the pose information to a robot to cause the robot to move according to the pose information.",
    "5. The method of claim 4, comprising implementing machine learning on the robot to obey movement commands by generating pose sequences in such a way that robot does not tip over.",
    "6. The method of claim 1, comprising: processing the pose information using a machine learning-based robot controller; and moving a robot according to an output of the controller.",
    "7. An assembly comprising: at least one apparatus; plural motion sensor assemblies (MSA) outputting pose information related to poses of respective real-world body parts; and at least one processor associated with the apparatus and configured with instructions for: receiving the pose information; and rendering an image using ray cast in which a foot of the image is constrained to remain on virtual ground in virtual space.",
    "8. The assembly of claim 7, wherein the processor is configured with instructions for: rendering the image using a modeled dynamics involving physics engine.",
    "9. The assembly of claim 7, wherein three respective MSA are attached to each respective arm of the body parts, two MSA are attached to each respective leg of the body parts, and at least one MSA is attached to a torso of the body parts.",
    "10. An assembly comprising: at least one virtual reality (VR) apparatus; plural motion sensor assemblies (MSA) outputting pose information related to poses of respective real-world body parts; at least one transmitter sending the pose information to the VR apparatus; and at least one processor associated with the VR apparatus and configured with instructions for: receiving the pose information; presenting on the VR apparatus at least one image in a VR space that moves in the VR space according to the pose information; rendering the image using ray cast in which a foot of the image is constrained to remain on virtual ground in the virtual space.",
    "11. An assembly comprising: at least one virtual reality (VR) apparatus; plural motion sensor assemblies (MSA) outputting pose information related to poses of respective real-world body parts; at least one transmitter sending the pose information to the VR apparatus; and at least one processor associated with the VR apparatus and configured with instructions for: receiving the pose information; presenting on the VR apparatus at least one image in a VR space that moves in the VR space according to the pose information; and rendering the image using ray cast in which a foot of the image is constrained to remain on virtual ground with infinite gravity and infinite friction.",
    "12. An assembly comprising: at least one apparatus; plural motion sensor assemblies (MSA) outputting pose information related to poses of respective real-world body parts; and at least one processor associated with the apparatus and configured with instructions for: receiving the pose information; and presenting on the apparatus at least one image using ray cast technique in which a foot of the image remains on virtual ground in virtual space.",
    "13. An assembly comprising: at least one virtual reality (VR) apparatus; and at least one processor associated with the VR apparatus and configured with instructions for: receiving pose information; and presenting at least one image in a VR space that moves in the VR space according to the pose information using a ray cast technique in which a portion of the image remains on virtual ground in the virtual space.",
    "14. An assembly comprising: at least one virtual reality (VR) apparatus; at least one processor associated with the VR apparatus and configured with instructions for: receiving pose information; presenting on the VR apparatus at least one image in a VR space that moves in the VR space according to the pose information using a technique in which a portion of the image is constrained to remain on virtual ground with infinite gravity and infinite friction."
  ],
  "description_excerpt": "The application relates to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements.\n\nKnowing the “pose” (location and orientation) of various objects can be useful in many computer applications. As but one example, computer games such as virtual reality (VR) or augmented reality (AR) games are sometimes designed to receive, as input, pose information from a VR/AR headset worn by a player, or pose information of a hand-held device such as a computer game handset.\n\nCurrent positioning solutions sometimes rely on visual tracking of objects with a video camera or laser beam to track the pose of objects of interest. These technologies require a sensor device to be within line of sight of the object for light to be able to travel towards device without meeting obstacles.\n\nAs understood herein, the line of sight between the light sensor and the object of interest may be blocked. As also understood herein, it would be advantageous to capture motion of a person for a variety of applications without having to use a camera.\n\nAccordingly, a method includes engaging plural motion sensor assemblies (MSA) to respective body parts, the MSA outputting pose information related to the respective body parts. The MSA may require no external power and may wirelessly communicate information to devices spaced from the person to which they are attached. The method also includes providing the pose information to a display apparatus for presenting an image of the body parts having respective poses according to the pose information.",
  "cpc": [
    "G06T 19/006",
    "A63F 13/211",
    "A63F 13/212",
    "A63F 13/213",
    "A63F 13/217",
    "A63F 13/235",
    "A63F 13/428",
    "A63F 13/56",
    "A63F 2300/66",
    "B25J 9/163",
    "G05B 11/32",
    "G05B 2219/40116",
    "G06F 3/011",
    "G06F 3/012",
    "G06F 3/014",
    "G06F 3/017",
    "G06N 20/00",
    "G06T 11/60",
    "G06T 13/40",
    "G06T 2207/30196"
  ],
  "ipc": [
    "B25J 9/16",
    "G05B 11/32",
    "G06N 20/00",
    "G06T 11/60",
    "G06T 13/40",
    "G06T 19/00"
  ],
  "assignees": [
    "Sony Interactive Entertainment Inc"
  ],
  "inventors": [
    "Sergey Bashkirov",
    "Takeo Matsukawa"
  ],
  "filing_date": "2017-10-23",
  "publication_date": "2020-09-15",
  "grant_date": "2020-09-15",
  "priority_date": "2017-10-23",
  "application_number": "US-201715791029-A",
  "family_id": "66171083",
  "cited_by_count": 1,
  "citations": [
    "US20010024512A1",
    "US20050197198A1",
    "US6806879B2",
    "US20050071306A1",
    "US20100201500A1",
    "US20090175540A1",
    "US20100250001A1",
    "US10210382B2",
    "US20100302257A1",
    "US20160184639A1",
    "US20120290131A1",
    "US8948447B2",
    "US20140258192A1",
    "US20140149903A1",
    "US9135516B2",
    "US20160140750A1",
    "US20150217449A1",
    "US20160046023A1",
    "US20160059412A1",
    "US20170200304A1",
    "US20170192496A1",
    "US20170178382A1",
    "US20170228922A1",
    "US20180045963A1",
    "US20180225858A1",
    "CN106896796A",
    "US20190116322A1",
    "US20190122436A1"
  ]
}