Calibration systems and methods for depth-based interfaces with disparate fields of view

{
  "publication_number": "US10437342B2",
  "country": "US",
  "kind": "B2",
  "title": "Calibration systems and methods for depth-based interfaces with disparate fields of view",
  "abstract": "Various of the disclosed embodiments provide Human Computer Interfaces (HCI) that incorporate depth sensors at multiple positions and orientations. The depth sensors may be used in conjunction with a display screen to permit users to interact dynamically with the system, e.g., via gestures. Calibration methods for orienting depth values between sensors are also presented. The calibration methods may generate both rotation and translation transformations that can be used to determine the location of a depth value acquired in one sensor from the perspective of another sensor. The calibration process may itself include visual feedback to direct a user assisting with the calibration. In some embodiments, floor estimation techniques may be used alone or in conjunction with the calibration process to facilitate data processing and gesture identification.",
  "claims": [
    "1. A display structure for gesture interactions comprising: a display screen; a plurality of sensors, at least two of the plurality of sensors configured to acquire depth information and at least two of the plurality of sensors configured to acquire pixel image information in a region before the display screen; a computer system configured to perform a method comprising: capturing a first pixel image at a primary sensor of at least a portion of the primary sensor's field of view; capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor's field of view; causing the display screen to display the first pixel image; causing the display screen to display the second pixel image; capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.",
    "2. The display structure of claim 1, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.",
    "3. The display structure of claim 1, wherein the calibration object is a planar board.",
    "4. The display structure of claim 1, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap.",
    "5. The display structure of claim 1, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.",
    "6. The display structure of claim 5, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.",
    "7. A computer-implemented method for calibrating a plurality of sensors, comprising: capturing a first pixel image at a primary sensor of at least a portion of the primary sensor's field of view; capturing a second pixel image at a first secondary sensor of at least a portion of the first secondary sensor's field of view; causing a first display screen to display the first pixel image; causing a second display screen to display the second pixel image; capturing a first set of depth values associated with a calibration object at the primary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.",
    "8. The computer-implemented method of claim 7, wherein the first display and the second display are the same display.",
    "9. The computer-implemented method of claim 7, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.",
    "10. The computer-implemented method of claim 7, wherein the calibration object is a planar board.",
    "11. The computer-implemented method of claim 7, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap.",
    "12. The computer-implemented method of claim 7, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.",
    "13. The computer-implemented method of claim 12, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.",
    "14. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method comprising: capturing a first set of depth values associated with a calibration object at a primary sensor when the calibration object is in both the primary sensor's field of view and a first secondary sensor's field of view; capturing a second set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the primary sensor's field of view and the first secondary sensor's field of view; determining a first normal associated with a plane of the calibration object for the first set of depth values; determining a first on-plane point associated with the plane of the calibration object for the first set of depth values; determining a second normal associated with the plane of the calibration object for the second set of depth values; determining a second on-plane point associated with the plane of the calibration object for the second set of depth values; generating a first system of linear equations, in part, using a linear equation derived from the first normal and the second normal; solving for components of a rotation transform using the first system of linear equations; generating a second system of linear equations, in part, using a linear equation derived from the first on-plane point and the second on-plane point; and solving for components of a translation transform using the second system of linear equations, wherein the rotation transform and translation transform move a point from the perspective of the first secondary sensor to the perspective of the primary sensor.",
    "15. The non-transitory computer-readable medium of claim 14, wherein the first on-plane point is an average of the depth values associated with the plane visible to the primary sensor.",
    "16. The non-transitory computer-readable medium of claim 14, wherein the calibration object is a planar board.",
    "17. The non-transitory computer-readable medium of claim 14, wherein the primary sensor's field of view and the first secondary sensor's field of view do not overlap.",
    "18. The non-transitory computer-readable medium of claim 14, the method further comprising: capturing a third set of depth values associated with the calibration object at a second secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; capturing a fourth set of depth values associated with the calibration object at the first secondary sensor when the calibration object is in both the second secondary sensor's field of view and the first secondary sensor's field of view, but not within the primary sensor's field of view; determining a third normal associated with a plane of the calibration object for the third set of depth values; determining a third on-plane point associated with the plane of the calibration object for the third set of depth values; determining a fourth normal associated with a plane of the calibration object for the fourth set of depth values; and determining a fourth on-plane point associated with the plane of the calibration object for the fourth set of depth values, wherein generating the first system of linear equations further comprises inserting a linear equation derived from the third normal and the fourth normal, and wherein generating the second system of linear equations further comprises inserting a linear equation derived from the third on-plane point and the fourth on-plane point.",
    "19. The non-transitory computer-readable medium of claim 18, wherein at least two of the plurality of sensors are vertically offset relative to one another by a bracket.",
    "20. The display structure of claim 1, wherein the display screen comprises multiple screens."
  ],
  "description_excerpt": "Human-computer interaction (HCI) systems are becoming increasingly prevalent in our society. With this increasing prevalence has come an evolution in the nature of such interactions. Punch cards have been surpassed by keyboards, which were themselves complemented by mice, which are themselves now complemented by touch screen displays, etc. Various machine vision approaches may even now facilitate visual, rather than the mechanical, user feedback. Machine vision allows computers to interpret images from their environment to, e.g., recognize users' faces and gestures. Some machine vision systems rely upon grayscale or RGB images of their surroundings to infer user behavior. Some machine vision systems may also use depth-based sensors, or rely exclusively upon depth-based sensors, to recognize user behavior (e.g., the Microsoft Kinect™, Intel RealSense™, Apple PrimeSense™, Structure Sensor™, Velodyne HDL-32E LiDAR™, Orbbec Astra™, etc.).\n\nWhile depth-based approaches to HCI remove certain problems common to optical systems (e.g., problematic lighting, shadows, user discoloration, etc.) depth-based approaches to HCI may also introduce their own obstacles and complexities. Many depth-based systems may be located within a house, office, shopping center or other environment having dynamic and static qualities. Creating devices and observation platforms that process and interpret data from these environments to extract meaningful data remains quite challenging. Particularly, there is a need to integrate design conditions with mechanical constraints and processing capabilities to achieve a successful user experience.",
  "cpc": [
    "G06F 3/017",
    "G06F 2203/0382",
    "G06F 3/0304",
    "G06T 2207/10028",
    "G06T 2207/20",
    "G06T 7/50",
    "G06T 7/80"
  ],
  "ipc": [
    "G06T 7/00",
    "G06F 3/01",
    "G06F 3/03",
    "G06T 7/50",
    "G06T 7/80"
  ],
  "assignees": [
    "Youspace Inc"
  ],
  "inventors": [
    "Ralph Brunner"
  ],
  "filing_date": "2016-12-05",
  "publication_date": "2019-10-08",
  "grant_date": "2019-10-08",
  "priority_date": "2016-12-05",
  "application_number": "US-201615369799-A",
  "family_id": "62243145",
  "cited_by_count": 2,
  "citations": [
    "US3992629A",
    "US4408678A",
    "US5534917A",
    "US6109460A",
    "US6353428B1",
    "US6130663A",
    "US7227526B2",
    "US6518565B1",
    "US7058204B2",
    "US7340077B2",
    "US20100146455A1",
    "US7379563B2",
    "US7379566B2",
    "US7948481B2",
    "US20140225977A1",
    "US20140226900A1",
    "US20080151092A1",
    "US20110199294A1",
    "US20140226000A1",
    "US20160127710A1",
    "US9075441B2",
    "US20130241833A1",
    "US20120157207A1",
    "US20150145860A1",
    "US20100199228A1",
    "US8553939B2",
    "US8610665B2",
    "US20110090147A1",
    "US20110122048A1",
    "US20110288964A1",
    "US9298346B2",
    "US9081419B2",
    "US20140043232A1",
    "US8788973B2",
    "US20120314031A1",
    "US9459758B2",
    "US8830302B2",
    "US9164589B2",
    "US20130278499A1",
    "US9292767B2",
    "US20130179162A1",
    "US20130251192A1",
    "US20130278501A1",
    "US9448636B2",
    "US9383895B1",
    "US8842906B2",
    "US20130336524A1",
    "US20140043435A1",
    "WO2014100250A2",
    "US20140241570A1",
    "US9389779B2",
    "US9429833B1",
    "US20150277700A1",
    "US9323338B2",
    "DE102013211335A1",
    "US20150091790A1",
    "US20150109202A1",
    "US9317112B2",
    "US20160328604A1",
    "USD727995S1",
    "US9684928B2",
    "US20150199816A1",
    "US20150261184A1",
    "US9438891B2",
    "WO2015195652A1",
    "US9424490B2",
    "US20160048726A1",
    "US20160260260A1",
    "USD753754S1",
    "US20160209655A1",
    "US20160288330A1",
    "KR20160116903A",
    "US20160292521A1",
    "US20160289042A1",
    "US20180181362A1",
    "WO2017095948A1",
    "US20170161592A1",
    "US20170161911A1",
    "US20170161607A1",
    "US20170160751A1",
    "US20170161591A1",
    "US20170161555A1",
    "US20180150713A1",
    "US20180300591A1"
  ]
}