Stabilization of captured images for a robot

{
  "publication_number": "US9749535B1",
  "country": "US",
  "kind": "B1",
  "title": "Stabilization of captured images for a robot",
  "abstract": "An example method for stabilizing images involves obtaining an image of an environment from a perspective of the robot. The method also involves receiving one or more depth measurements. The depth measurements correspond to locations within the image. The method further involves determining first boundaries around a particular location within a section of the image. The particular location is approximately centered within the first boundaries. Additionally, the method involves determining an extent of deviation of a particular position of the image capture device from a reference position. The particular position is a position of the image capture device at a point in time at which the image was captured. Further, the method involves determining second boundaries by shifting the first boundaries by an amount proportional to the determined extent of deviation. The method additionally involves providing, to a display device, a portion of the image defined by the second boundaries.",
  "claims": [
    "1. A method comprising: receiving, by a processor and from a depth sensor coupled to a robot, one or more depth measurements, wherein the depth measurements correspond to locations within an image that was captured by an image capture device coupled to the robot, wherein the image capture device was in a particular position when the image was captured, and wherein the depth measurements are indicative of distances between the image capture device and surfaces of an environment depicted in the image; determining, by the processor, first boundaries around a location within the image, wherein the location was measured by a particular depth measurement of the one or more depth measurements; determining, by the processor, a deviation of the particular position of the image capture device from a reference position; determining, by the processor, second boundaries by shifting the first boundaries by an amount proportional to the deviation; and providing, to a display device, a portion of the image enclosed within the second boundaries.",
    "2. The method of claim 1, wherein providing the portion of the image to the display device causes the display device to display the portion of the image.",
    "3. The method of claim 1, wherein the location represents an object closest to the image capture device.",
    "4. The method of claim 1, wherein the location represents an object that is a pre-determined distance from the image capture device.",
    "5. The method of claim 1, wherein determining the deviation of the particular position from the reference position comprises: determining the reference position as a position of the image capture device when the robot is in a particular posture; determining, based on one or more measurements of joint angles of the robot, the particular position; and determining the deviation by comparing the particular position to the reference position.",
    "6. The method of claim 1, wherein determining the deviation of the particular position from the reference position comprises: determining the reference position as being situated at a predetermined height and at a point between two legs of the robot when the robot is in a standing posture; determining, based on one or more measurements from an inertial sensor coupled to the robot, the particular position of the image capture device; and determining the deviation by comparing the particular position to the reference position.",
    "7. The method of claim 6, wherein the predetermined height is a height of the image capture device with respect to a ground surface when the robot is in a standing posture on the ground surface.",
    "8. The method of claim 6, further comprising: determining a position of a foot of the robot that was most recently placed on a ground surface; and determining the point as a predetermined distance from the position of the foot.",
    "9. The method of claim 1, wherein the image represents a first field-of-view of the environment, and the method further comprising: receiving, from an orientation sensor coupled to a head mounted display, an orientation measurement indicative of an angular position of the head mounted display; and determining third boundaries within the image that define the section of the image based on the orientation measurement, wherein the third boundaries represent the section of the image having a second field-of-view, and wherein the second field-of-view is contained within the first field-of-view.",
    "10. The method of claim 1, wherein the first boundaries define a first area of a first predetermined size.",
    "11. The method of claim 10, wherein the second boundaries define a second area of a second predetermined size, and wherein the first predetermined size and the second predetermined size are equal.",
    "12. The method of claim 1, wherein the deviation includes a vertical displacement and a horizontal displacement, and wherein determining the second boundaries further comprises: horizontally shifting the first boundaries by an amount proportional to the horizontal displacement; and vertically shifting the first boundaries by an amount proportional to the vertical displacement.",
    "13. The method of claim 1, wherein the depth sensor includes a stereoscopic camera.",
    "14. The method of claim 1, wherein the depth sensor includes a LIDAR device.",
    "15. The method of claim 1, wherein the image capture device includes a wide-angle lens, wherein the captured image is distorted, and the method further comprising: transforming the portion of the image to remove the distortion.",
    "16. The method of claim 1, wherein the display device is part of a teleoperation device that is configured to remotely control the robot.",
    "17. A robot comprising: a depth sensor; an image capture device; a processor; memory; and program instructions, stored in the memory, that upon execution by the processor cause the robot to perform operations comprising: receiving, from the depth sensor, one or more depth measurements, wherein the depth measurements correspond to locations within an image that was captured by the image capture device, wherein the image capture device was in a particular position when the image was captured, and wherein the depth measurements are indicative of distances between the image capture device and surfaces of an environment depicted in the image; determining first boundaries around a location within the image, wherein the location was measured by a particular depth measurement of the one or more depth measurements; determining a deviation of the particular position of the image capture device from a reference position; determining second boundaries by shifting the first boundaries by an amount proportional to the deviation; and providing, to a display device, a portion of the image enclosed within the second boundaries.",
    "18. The robot of claim 17, wherein the location represents an object closest to the image capture device.",
    "19. The robot of claim 17, wherein the location represents an object that is a pre-determined distance from the image capture device.",
    "20. An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a processor of a robot, cause the robot to perform operations comprising: receiving, from a depth sensor coupled to the robot, one or more depth measurements, wherein the depth measurements correspond to locations within an image that was captured by an image capture device coupled to the robot, wherein the image capture device was in a particular position when the image was captured, and wherein the depth measurements are indicative of distances between the image capture device and surfaces of an environment depicted in the image; determining first boundaries around a location within the image, wherein the location was measured by a particular depth measurement of the one or more depth measurements; determining a deviation of the particular position of the image capture device from a reference position; determining second boundaries by shifting the first boundaries by an amount proportional to the deviation; and providing, to a display device, a portion of the image enclosed within the second boundaries."
  ],
  "cpc": [
    "G06T 7/30",
    "G01B 11/14",
    "G01S 17/08",
    "G06T 2207/10012",
    "G06T 2207/10016",
    "G06T 2207/10028",
    "G06T 2207/20201",
    "G06T 2207/30244",
    "G06T 5/002",
    "G06T 5/70",
    "G06T 7/593",
    "G06T 7/74",
    "H04N 13/0007",
    "H04N 13/0203",
    "H04N 13/0296",
    "H04N 13/106",
    "H04N 13/204",
    "H04N 13/296",
    "H04N 23/6812",
    "H04N 23/683",
    "H04N 5/23267",
    "Y10S 901/01",
    "Y10S 901/47"
  ],
  "assignees": [
    "X DEV LLC"
  ],
  "filing_date": "2016-12-23",
  "publication_date": "2017-08-29",
  "grant_date": "2017-08-29",
  "priority_date": "2015-08-10",
  "application_number": "US-201615389847-A",
  "family_id": "57908827",
  "citations": [
    "US2002018070A1",
    "US2004057613A1",
    "US2004101162A1",
    "US2004104935A1",
    "US2004193323A1",
    "US2008246848A1",
    "US2011244958A1",
    "US2013044227A1",
    "US2013114886A1",
    "US2013242122A1",
    "US2014371912A1",
    "US6985619B1",
    "US7327112B1",
    "US7398136B2",
    "US8134604B2",
    "US8508651B2"
  ]
}