Hand tracking using an ultrasound sensor on a head-mounted display

{
  "publication_number": "US10572024B1",
  "country": "US",
  "kind": "B1",
  "title": "Hand tracking using an ultrasound sensor on a head-mounted display",
  "abstract": "A head-mounted display (HMD) tracks a user's hand positions, orientations, and gestures using an ultrasound sensor coupled to the HMD. The ultrasound sensor emits ultrasound signals that reflect off the hands of the user, even if a hand of the user is obstructed by the other hand. The ultrasound sensor identifies features used to train a machine learning model based on detecting reflected ultrasound signals. For example, one of the features is the time delay between consecutive reflected ultrasound signals detected by the ultrasound sensor. The machine learning model learns to determine poses and gestures of the user's hands. The HMD optionally includes a camera that generates image data of the user's hands. The image data can also be used to train the machine learning model. The HMD may perform a calibration process to avoid detecting other objects and surfaces such as a wall next to the user.",
  "claims": [
    "1. A head-mounted display (HMD) comprising: a body configured to secure to a user's face; an ultrasound transmitter coupled to the body and facing away from the user's face, the ultrasound transmitter configured to emit an ultrasound signal towards a region in front of the HMD; an ultrasound receiver coupled to the body and configured to generate ultrasound data corresponding to a reflected version of the emitted ultrasound signal received at the ultrasound receiver; a camera coupled to the body and facing away from the user's face to capture images in the region in front of the HMD; and a processor coupled to the ultrasound receiver and the camera, the processor configured to: determine that a portion of a first hand of the user is obstructed by a second hand of the user in a field of view of the camera based on the captured images; and responsive to determining that the portion of the first hand is obstructed by the second hand, determine a pose or a gesture of both of the user's hands including the obstructed portion of the first hand by using a machine learning model trained using feature vectors of previous ultrasound data and a corresponding hand pose or hand gesture.",
    "2. The HMD of claim 1, wherein the ultrasound transmitter, the ultrasound receiver, and the camera each face the user's hands.",
    "3. The HMD of claim 1, wherein at least one of the feature vectors includes a plurality of mappings of an attribute of the previous ultrasound data to at least one hand pose or hand gesture.",
    "4. The HMD of claim 3, wherein the attribute indicates a time delay between a first reflected ultrasound signal received at the ultrasound receiver and a second reflected ultrasound signal received at the ultrasound receiver subsequent to the first reflected ultrasound signal.",
    "5. The HMD of claim 1, wherein the processor is further configured to retrieve calibration data based on previous ultrasound data, wherein the pose or the gesture of the user's hands is generated by taking into account the calibration data.",
    "6. The HMD of claim 5, wherein the calibration data indicates at least a mapping of an ultrasound signal time of flight value to an object in front of the user other than the first hand of the user.",
    "7. The HMD of claim 1, wherein the pose or the gesture of the user's hands is indicated in at least two dimensions.",
    "8. The HMD of claim 1, wherein the ultrasound transmitter generates the emitted ultrasound signal based on a windowed cardinal sine function, and wherein the processor is further configured to modify the ultrasound data using pulse compression to reduce a noise level of the ultrasound data for processing to determine the pose or the gesture of the user's hands.",
    "9. A head-mounted display (HMD) comprising: an ultrasound sensor; and a processor comprising: an ultrasound interface configured to receive ultrasound data from the ultrasound sensor; a camera interface configured to receive image data from a camera; a machine learning engine communicatively coupled to the ultrasound interface and the camera interface, the machine learning engine trained using feature vectors of previous ultrasound data and a corresponding hand pose or hand gesture and configured to: determine that a portion of a first hand of a user is obstructed by a second hand of the user in a field of view of the camera based on the captured images; and responsive to determining that the portion of the first hand is obstructed by the second hand, determine a pose or a gesture of both of the user's hands including the obstructed portion of the first hand.",
    "10. The HMD of claim 9, wherein at least one of the feature vectors includes a plurality of mappings of an attribute of the previous ultrasound data to a hand pose or a hand gesture, at least one of the attributes indicating a time delay between a first reflected ultrasound signal received at the ultrasound sensor and a second reflected ultrasound signal received at the ultrasound sensor subsequent to the first reflected ultrasound signal.",
    "11. A method comprising: capturing images in a region in front of a head-mounted display (HMD) worn by a user to generate image data; emitting an ultrasound signal towards the region in front of the HMD; responsive to emitting the ultrasound signal, detecting a reflected version of the emitted ultrasound signal at an ultrasound sensor coupled to the HMD; generating ultrasound data by digitally processing the detected ultrasound signal; determining that a portion of a first hand of the user is obstructed by a second hand of the user in a field of view of the HMD based on the captured images; and responsive to determining that the portion of the first hand is obstructed by the second hand, determining a pose or a gesture of both of the user's hands including the obstructed portion of the first hand by using a machine learning model trained using feature vectors of previous ultrasound data and a corresponding hand pose or hand gesture.",
    "12. The method of claim 11, wherein the ultrasound sensor includes at least one ultrasound transmitter and at least one ultrasound receiver each facing the user's hands.",
    "13. The method of claim 11, wherein at least one of the feature vectors includes a plurality of mappings of an attribute of the previous ultrasound data to at least one hand pose or hand gesture.",
    "14. The method of claim 13, wherein the attribute indicates a time delay between a first reflected ultrasound signal received at the ultrasound sensor and a second reflected ultrasound signal received at the ultrasound sensor subsequent to the first reflected ultrasound signal.",
    "15. The method of claim 11, further comprising retrieving calibration data based on previous ultrasound data, wherein the pose or the gesture of the user's hands is determined by taking into account the calibration data.",
    "16. The method of claim 15, wherein the calibration data indicates at least a mapping of an ultrasound signal time of flight value to an object in front of the user other than the user's hands.",
    "17. The method of claim 11, wherein the pose or the gesture of the user's hands is indicated in at least two dimensions.",
    "18. The method of claim 11, wherein the emitted ultrasound signal is generated based on a windowed cardinal sine function, and wherein the method further comprises modifying the ultrasound data using pulse compression to reduce a noise level of the ultrasound data for determining the pose or the gesture of both of the user's hands."
  ],
  "description_excerpt": "The present disclosure relates to a hand tracking system and method, for example, a hand tracking system and method for determining hand poses and gestures of an obstructed hand using an ultrasound sensor on a head-mounted display.\n\nIt is known that motion tracking systems can use image processing to determine the position and gestures of a person. Existing systems can use different types of cameras (for example, structured light scanners, RGB cameras, and depth cameras) to capture images and video of a person. However, the person must be in the field of view of the camera. For instance, if the person's right hand is obstructed by the person's left hand, then the camera cannot capture images or video of the right hand. Accordingly, image based motion tracking systems are unable to determine the position of obstructed body parts.\n\nAn ultrasound sensor can determine the distance between the ultrasound sensor and another object or surface. The ultrasound sensor emits an ultrasound signal, i.e., a sound wave between 20 kiloHertz and 200 megaHertz. The ultrasound signal reflects off of the object or surface, and the ultrasound sensor detects the reflected ultrasound signal. The ultrasound sensor determines the distance based on the known speed of sound (approximately 340 meters per second) and the time of flight of the reflected ultrasound signal.\n\nEmbodiments relate to a system for determining hand poses and gestures of a user wearing a head-mounted display (HMD).",
  "cpc": [
    "G06F 3/017",
    "G01N 29/12",
    "G01S 15/104",
    "G01S 15/582",
    "G01S 15/66",
    "G01S 15/86",
    "G01S 15/88",
    "G01S 7/52004",
    "G02B 2027/0138",
    "G02B 27/0149",
    "G06F 3/011",
    "G06F 3/014",
    "G06N 20/00",
    "G06N 20/10",
    "G06N 20/20",
    "G06N 5/01",
    "G06N 7/01"
  ],
  "ipc": [
    "G01N 29/12",
    "G02B 27/01",
    "G06F 3/01",
    "G06N 20/00"
  ],
  "assignees": [
    "Facebook Technologies LLC"
  ],
  "inventors": [
    "Elliot Saba",
    "Robert Y. Wang",
    "Christopher David Twigg",
    "Ravish Mehra"
  ],
  "filing_date": "2017-08-03",
  "publication_date": "2020-02-25",
  "grant_date": "2020-02-25",
  "priority_date": "2016-09-28",
  "application_number": "US-201715668418-A",
  "family_id": "69590927",
  "cited_by_count": 29,
  "citations": [
    "US20140045594A1",
    "JP2010218365A",
    "US20120243374A1",
    "US20120001875A1",
    "US20120280900A1",
    "US20120309532A1",
    "US20130229508A1",
    "US20140243614A1",
    "US10037474B2",
    "US20150029092A1",
    "US9921657B2",
    "US9778749B2",
    "US20170329004A1",
    "US20170060254A1",
    "US20160274732A1",
    "US20170262045A1",
    "CN106203380A",
    "US20180188353A1"
  ]
}