System and Method for Reconstructing a VR Avatar With Full Body Pose

{
  "publication_number": "US2022374070A1",
  "country": "US",
  "kind": "A1",
  "title": "System and Method for Reconstructing a VR Avatar With Full Body Pose",
  "abstract": "A system for producing a shared virtual reality having a communication network. The system has a plurality of inside-out tracked HMDs that are worn by participants to view the shared virtual reality in communication with the network, right and left hand controllers that are held by the participant and right and left trackers that are worn on the ankles or feet of the participant. The system has a server computer in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality. A method for producing a shared virtual reality of participants.",
  "claims": [
    "1. A system for reconstructing a VR avatar with full body pose from a participant comprising: an inside-out tracked HMD that is to be worn by the participant; a left-hand controller to be held by a left hand of the participant that produces position data of the left hand of the participant; a right-hand controller to be held by a right hand of the participant that produces position data of the right hand of the participant; a right tracker worn on a right foot or right ankle of the participant; a left tracker worn on a left foot or left ankle of the participant; a communication network in communication with the left-hand controller and the right-hand controller and the right tracker and the left tracker and the motion capture suit and the motion capture unit; and a server computer in communication with the network that constructs a full body pose of the participant using real time data from only the right-hand controller and left-hand controller and right tracker and left tracker and HMD and from previously stored data about motion of the participant. 2. The system of claim 1 wherein the right tracker has a 6° of freedom IMU to produce tracked data from the right foot and a wireless transmitter which transmits the tracked data to the server computer through the communications network, the 6° of freedom are 3° of orientation and 3° of acceleration in an X, Y, Z coordinate system, and the left tracker has an imu. 3. The system of claim 2 wherein the right tracker has a microprocessor in communication with the wireless transmitter and the IMU, the microprocessor processes the tracked data and sends the tracked data that has been processed to the server computer through the wireless transmitter. 4. The system of claim 3 wherein the right tracker has a foot pressure tracking insole that is worn underneath the right foot, the foot pressure tracking insole produces pressure data which is provided to the microprocessor, and the left tracker has a foot pressure tracking insole that is worn underneath the left foot. 5. The system of claim 4 wherein the right tracker includes a proximity sensor which measures time-varying distance between the right foot and a floor upon which the participant is walking. 6. The system of claim 5 wherein the server computer receives input that includes position and orientation of each of the HMD, the left-hand controller and the right hand controller, gyroscopic orientation and accelerometer data from the IMU of the right tracker and left tracker, and six pressure values of the right foot and the left foot from the foot pressure tracking insole of the right tracker and the left tracker. 7. The system of claim 6 wherein the server computer receives a total number of input scalar values of 42, where six scalar values are for the position and orientation of the HMD and for each of the right-hand controller and the left-hand controller, six from each of the IMU of the right tracker and left tracker, and six for each of the foot pressure tracking insole of the right tracker and the left tracker, the server computer produces a body pose output of the participant from the 42 input scalar values. 8. The system of claim 7 wherein the server computer produces a total number of 48 output scalar values, where six scalar values for the position and orientation of each of the head, hands, feet, upper torso and pelvis, +2 scalar values for each of a right shoulder and left shoulder, +1 scalar value for each of the right elbow, left elbow, right knee and left knee of the participant, the computer produces a body pose output of the participant from the 48 scalar values. 9. The system of claim 8 wherein the body pose output of the participant includes position and orientation of the right hand and left hand, right foot and left foot, upper torso, pelvis, head, forward and upward rotation of each shoulder about the upper torso and inward to outward turning of elbows and knees of the participant. 10. The system of claim 9 including a motion capture suit to be worn by the participant; and a motion capture unit which captures motion of the participant from the motion capture suit worn by the participant, the server computer in communication with the network that constructs the full body pose of the participant using real time data from only the right-hand controller and left-hand controller and right tracker and left tracker and HMD and from previously stored data from the motion capture unit. 11. A system for producing a shared virtual reality comprising: a communication network; a plurality of inside-out tracked HMDs that are worn by participants to view the shared virtual reality in communication with the network; and a server computer in communication with the network which produces a full body pose of at least one of participant of the participants and transmits the full body pose via the network to all other participants HMD, so that each participant can see the full body pose in each participant's view into the shared virtual reality. 12. The system of claim 11 wherein the server produces the full body pose over time. 13. A method for producing a shared virtual reality of participants comprising the steps of: sending data from an inside-out tracked HMD, hand controllers and feet or ankle trackers worn or held by each participant to a server computer; reconstructing a full body pose for each participant from each participant's data with the server computer; and sending the full body pose of all participants wirelessly to all HMDs, so that each participant sees the full body pose of all participants in the shared VR displayed in each participant's HMD."
  ],
  "description_excerpt": "The present invention is related to reconstructing VR avatars with full body pose from inside-out tracked HMD. (As used herein, references to the “present invention” or “invention” relate to exemplary embodiments and not necessarily to every embodiment encompassed by the appended claims.) More specifically, the present invention is related to reconstructing VR avatars with full body pose from inside-out tracked HMD using hand controllers and foot or ankle trackers.\n\nThis section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present invention. The following discussion is intended to provide information to facilitate a better understanding of the present invention. Accordingly, it should be understood that statements in the following discussion are to be read in this light, and not as admissions of prior art.\n\nUsing a Virtual Reality head mounted display (HMD) which is capable of inside-out tracking, such as the Oculus Quest, it is possible to accurately track six degrees of freedom (6DOF) for a user's head and also for each of two handheld controllers - specifically, the xyz position as well as the three axes of orientation for each of the head, the left hand and the right hand. Because the Quest uses inside-out tracking, it has the desirable quality that it requires no external markers or other optical apparatus in the surrounding space.\n\nIn a VR application shared by two or more users, it is desirable for each user to see all of the other users within the shared experience, represented as visible avatars.",
  "cpc": [
    "G06F 3/011",
    "G06F 2203/04802",
    "G06F 3/0346",
    "G06F 3/04817",
    "G06F 3/0484",
    "G06F 3/04845",
    "G06F 3/1454",
    "G06F 3/147"
  ],
  "ipc": [
    "G06F 3/01",
    "G06F 3/0346",
    "G06F 3/04817",
    "G06F 3/0484",
    "G06F 3/04845",
    "G06F 3/14"
  ],
  "assignees": [
    "New York University NYU",
    "Tactonic Tech LLC"
  ],
  "inventors": [
    "Kenneth Perlin",
    "Benjamin Ahlbrand",
    "Charles Hendee"
  ],
  "filing_date": "2022-08-04",
  "publication_date": "2022-11-24",
  "priority_date": "2019-09-12",
  "application_number": "US-202217881590-A",
  "family_id": "74869514",
  "cited_by_count": 5,
  "citations": [
    "US5592401A",
    "US5930741A",
    "US6148280A",
    "US5913727A",
    "US20030030397A1",
    "US20030120183A1",
    "US6741911B2",
    "US20070003915A1",
    "US7542040B2",
    "US8868373B2",
    "US20110202306A1",
    "US20110009241A1",
    "US9067097B2",
    "US9827478B1",
    "US8979665B1",
    "US20120021833A1",
    "US20110306398A1",
    "US20120116548A1",
    "US20130278494A1",
    "US11052288B1",
    "US9443521B1",
    "US20140276242A1",
    "US9381426B1",
    "US20170178272A1",
    "US9990689B2",
    "US20180196505A1",
    "US20180322681A1",
    "US10403050B1",
    "US20180350136A1",
    "US20210233304A1",
    "US20190354201A1",
    "US20200004349A1",
    "US10761618B2",
    "US20200380784A1",
    "US20210149190A1"
  ]
}