Systems and methods for using hover information to predict touch locations and reduce or eliminate touchdown latency

{
  "publication_number": "US10592050B2",
  "country": "US",
  "kind": "B2",
  "title": "Systems and methods for using hover information to predict touch locations and reduce or eliminate touchdown latency",
  "abstract": "A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.",
  "claims": [
    "1. A method for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event in advance of an actual touch event, the method comprising: storing a model of user interaction with a touch surface; sensing at least one current user input via the touch sensing system; creating in the electronic device data representative of the sensed at least one current user input, the data reflecting hover information; applying the model of user interaction to the data representative of the at least one current user input to create data reflecting a prediction of a future user input event; and, prior to occurrence of the predicted user input event, providing to a touch response system an approach direction of a finger or object; wherein the providing step is performed prior to a time t−(L−1), where t is a predicted time that the future user input event will occur, L is an estimate of actual latency in the device, and i is a time required for perceived zero latency.",
    "2. The method for using current user input according to claim 1, further comprising a step of providing to the touch response system a velocity of the finger or object.",
    "3. The method for using current user input according to claim 1, further comprising: prior to occurrence of the predicted user input event, providing to the touch response system data reflecting a likelihood that the predicted future user input event will occur.",
    "4. The method for using current user input according to claim 1, further comprising: prior to occurrence of the predicted user input event, providing to the touch response system a parameter of a finger or object's motion.",
    "5. The method for using current user input according to claim 4, wherein the parameter is speed.",
    "6. The method for using current user input according to claim 4, wherein the parameter is acceleration.",
    "7. The method for using current user input according to claim 4, wherein the parameter is a vector.",
    "8. The method for using current user input according to claim 1, further comprising: prior to occurrence of the predicted user input event, providing to the touch response system an approach direction of a finger or object.",
    "9. The method for using current user input according to claim 1, wherein the hover information comprises information derived from tracking a finger or object as it approaches the touch surface.",
    "10. The method for using current user input according to claim 1, wherein the providing step is performed during a window that is calculated to be prior to the time t−(L−i).",
    "11. The method for using current user input according to claim 1, wherein the providing step is performed at a time that provides the touch response system with a predetermined prediction horizon prior to occurrence of the predicted user input event.",
    "12. The method for using current user input according to claim 11, wherein the prediction horizon is 128-200 ms.",
    "13. The method for using current user input according to claim 1, wherein the current user input is a drop-down suggestive of a tapping event.",
    "14. The method for using current user input according to claim 13, wherein the providing step is performed at a time that is calculated to provide the touch response system time to deliver a response to the predicted future user input event less than 24 milliseconds after occurrence of the tapping event.",
    "15. The method for using current user input according to claim 1, wherein the current user input is a drop-down suggestive of a motion event.",
    "16. The method for using current user input according to claim 15, wherein the providing step is performed at a time that is computed to provide the touch response system time to deliver a response to the predicted future user input event less than 6 milliseconds after occurrence of the motion event.",
    "17. The method for using current user input according to claim 16, wherein the step of providing comprises providing position of the object or finger.",
    "18. The method for using current user input according to claim 16, wherein the motion event is a dragging event.",
    "19. The method for using current user input according to claim 1, wherein the providing step is performed at a time that is computed to provide the touch response system time to deliver a response to the predicted future user input event less than a predetermined time period after occurrence of the actual user input event.",
    "20. The method for using current user input according to claim 1, wherein the steps of creating data representative of current user input, applying the model of user interaction to the data, and providing to the touch response system are performed by a processor associated with the touch sensor of the electronic device.",
    "21. The method for using current user input according to claim 1, wherein the model is stored in the electronic device as a table.",
    "22. The method for using current user input according to claim 1, wherein the model comprises a model of a liftoff phase.",
    "23. The method for using current user input according to claim 1, wherein the model comprises a model of a correction phase.",
    "24. The method for using current user input according to claim 1, wherein the model comprises a model of a drop-down phase.",
    "25. The method for using current user input according to claim 1, wherein the providing step comprises sequentially providing, prior to occurrence of the actual user input event, a plurality of predicted locations at which the predicted future user input event will occur, whereby the predicted location of the predicted future user input event is updated.",
    "26. The method for using current user input according to claim 1, wherein the providing step comprises sequentially providing, prior to occurrence of the actual user input event, a plurality of predicted times at which the predicted future user input event will occur, whereby the predicted time at which the predicted future user input event will occur is updated.",
    "27. The method for using current user input according to claim 20, wherein the providing step comprises sequentially providing, prior to occurrence of the actual user input event, a plurality of confidence levels associated with the prediction of the user input event, whereby confidence level associated with the prediction is updated.",
    "28. The method for using current user input according to claim 1, wherein the providing step further comprises providing, prior to occurrence of the actual user input event, data indicating that the predicted future user input event is an unintended input.",
    "29. The method for using current user input according to claim 1, wherein the providing step further comprises providing data indicating that prediction of the future user input event is canceled.",
    "30. The method for using current user input according to claim 1, wherein the data representative of current user input comprises an approach trajectory of a finger or object.",
    "31. A method for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event in advance of an actual touch event, the method comprising: storing a model of user interaction with a touch surface; sensing at least one current user input via the touch sensing system; creating in the electronic device data representative of the sensed at least one current user input, the data reflecting hover information; applying the model of user interaction to the data representative of the at least one current user input to create data reflecting a prediction of a future user input event; prior to occurrence of the predicted user input event, providing to a touch response system an approach direction of a finger or object; wherein the step of applying the model of user input to the data representative of current user input comprises fitting a shape to an approach trajectory of a finger or object; and wherein the step of fitting the shape to the approach trajectory comprises constraining the shape such that an angle of intersection with a display of the electronic device is orthogonal.",
    "32. The method for using current user input according to claim 31, wherein the shape comprises a shape selected from the set consisting of: a parabola, a line, or a circle.",
    "33. The method for using current user input according to claim 31, wherein the step of fitting the shape to the approach trajectory comprises fitting a plane to a drop-down trajectory.",
    "34. The method for using current user input according to claim 31, wherein the step of fitting the shape to the approach trajectory comprises constraining a parabola such that a position of the finger or object at the time of the fit is on the parabola.",
    "35. The method for using current user input according to claim 31, wherein the step of fitting the shape to the approach trajectory comprises constraining a parabola such that an angle of movement at the time of the fit is made a tangent to the parabola.",
    "36. A method for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event in advance of an actual touch event, the method comprising: storing a model of user interaction with a touch surface; sensing at least one current user input via the touch sensing system; creating in the electronic device data representative of the sensed at least one current user input, the data reflecting hover information; applying the model of user interaction to the data representative of the at least one current user input to create data reflecting a prediction of a future user input event; prior to occurrence of the predicted user input event, providing to a touch response system an approach direction of a finger or object; and wherein the step of applying the model of user input to the data representative of current user input comprises fitting a parabola to an approach trajectory of a finger or object and predicting a touch point by determining an intersection of the fitted parabola with a display of the electronic device.",
    "37. The method for using current user input according to claim 1, wherein the providing step further comprises providing a range of locations of the predicted future touch event.",
    "38. The method for using current user input according to claim 1, wherein the providing step further comprises providing a range of times at which the predicted future touch event is predicted to occur.",
    "39. The method for using current user input according to claim 1, wherein the predicted future user input event is a contact with the touch surface.",
    "40. The method for using current user input according to claim 1, wherein the predicted future user input event is a hover in proximity to the touch surface.",
    "41. The method for using current user input according to claim 1, wherein the touch sensing system utilizes a plurality of sensors types.",
    "42. The method for using current user input according to claim 1, wherein the touch sensing system comprises a contact sensing system.",
    "43. The method for using current user input according to claim 1, wherein the touch response system is an operating system.",
    "44. The method for using current user input according to claim 1, wherein the touch response system is an application running in an operating system."
  ],
  "description_excerpt": "This application is a divisional of U.S. patent application Ser. No. 14/859,185 filed Sep. 18, 2015 entitled “Systems and Methods for Using Hover Information to Predict Touch Locations and Reduce or Eliminate Touchdown Latency,” which claims priority to U.S. Provisional Patent Application No. 62/052,323 filed Sep. 18, 2014 entitled “Systems And Methods For Using Hover Information To Predict Touch Locations And Reduce Or Eliminate Touchdown Latency,” the entire disclosure of each of which is incorporated herein by reference. This application relates to the subject matter of U.S. patent application Ser. No. 14/490,363 filed Sep. 18, 2014, the entire disclosure of which is incorporated herein by reference.\n\nThis application includes material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office files or records, but otherwise reserves all copyright rights whatsoever. This application relates to fast multi-touch sensors such as those disclosed in U.S. patent application Ser. No. 14/490,363 filed Sep. 18, 2014 entitled “Systems And Methods For Providing Response To User Input Using Information About State Changes And Predicting Future User Input,” U.S. patent application Ser. No. 13/841,436 filed Mar. 15, 2013 entitled “Low-Latency Touch Sensitive Device,” U.S. Patent Application No. 61/798,948 filed Mar. 15, 2013 entitled “Fast Multi-Touch Stylus,” U.S. Patent Application No. 61/799,035 filed Mar. 15, 2013 entitled “Fast Multi-Touch Sensor With User-Identification Techniques,” U.S.",
  "cpc": [
    "G06F 3/0418",
    "G06F 2203/04101",
    "G06F 3/0416",
    "G06F 3/04883",
    "G06N 5/04",
    "G06N 7/00"
  ],
  "ipc": [
    "G06F 3/041",
    "G06F 3/0488",
    "G06N 5/04",
    "G06N 7/00"
  ],
  "assignees": [
    "Tactual Labs Co"
  ],
  "inventors": [
    "Clifton Forlines",
    "Ricardo Jorge Jota Costa",
    "Daniel Wigdor",
    "Karan Singh",
    "Haijun Xia"
  ],
  "filing_date": "2018-06-14",
  "publication_date": "2020-03-17",
  "grant_date": "2020-03-17",
  "priority_date": "2014-09-18",
  "application_number": "US-201816008332-A",
  "family_id": "55533938",
  "cited_by_count": 3,
  "citations": [
    "US20110175832A1",
    "US20120169646A1",
    "US20140198062A1",
    "US20130181908A1",
    "US20130222329A1",
    "US20140143692A1",
    "US20140240242A1"
  ]
}