System and method for motion compensation in images

{
  "publication_number": "US10397479B2",
  "country": "US",
  "kind": "B2",
  "title": "System and method for motion compensation in images",
  "abstract": "A method of compensating for camera motion during capture of the image in a rolling shutter camera, the method comprising: receiving an image of a scene captured by a camera with a rolling shutter; extracting line segments in said image; estimating the movement of the camera during the capturing of the image from the received image; and producing an image compensated for the movement during capture of the image, wherein the scene is approximated by a horizontal plane and two vertical planes that intersect at a line at infinity and estimating the movement of the camera during the capture of the image comprises assuming that the extracted line segments are vertical and lie on the vertical planes.",
  "claims": [
    "1. A method of compensating for camera motion during capture of an image in a rolling shutter camera, the method comprising: receiving an image of a scene captured by a camera with a rolling shutter; extracting line segments in said image; estimating movement of the camera during the capturing of the image from the received image; and producing an image compensated for the movement during the capture of the image, wherein the scene is approximated by a horizontal plane and two vertical planes that intersect at a line at infinity and estimating the movement of the camera during the capture of the image comprises assuming that the extracted line segments are vertical and lie on the vertical planes.",
    "2. A method according to claim 1, wherein determining whether a line segment lies on the vertical planes comprises determining whether a normal to the line segment is perpendicular to a vertical direction.",
    "3. A method according to claim 1, wherein estimating the movement of the camera comprises representing pixels in a received two dimensional image in three dimensions using homogeneous coordinates.",
    "4. A method according to claim 1, wherein estimating the movement of the camera comprises assuming a linear relationship between an inverse scene depth in the image captured by the rolling shutter camera and the inverse scene depth in the image which has been compensated for the motion of the camera, the scene depth for a 3D point P in the scene being a forward distance of the 3D point P from the camera.",
    "5. A method according to claim 1, wherein estimating the movement of the camera comprises: relating a position of a pixel in the image captured using the rolling shutter camera to a position of the image which has been compensated for the motion of the camera, wherein the motion of the camera during capture of the image is modeled using Ackermann motion.",
    "6. A method according to claim 1, wherein estimating the movement of the camera comprises: relating a position of a pixel in the image captured using the rolling shutter camera to a position of an the image which has been compensated for the motion of the camera, wherein the motion of the camera during capture of the image is modeled using purely rotational motion.",
    "7. A method according to claim 1, wherein estimating the movement of the camera comprises: relating a position of a pixel in the image captured using the rolling shutter camera to a position of the image which has been compensated for the motion of the camera, wherein the motion of the camera during capture of the image is modeled using purely translational motion.",
    "8. A method according to claim 1, wherein a vertical position of the camera is determined via a sensor in addition to the camera that captures the image.",
    "9. A method according to claim 1, wherein a camera position is rotated with respect to vertical and estimating for the movement of the camera during the capturing of the image comprises correcting for rotation of the camera from the vertical.",
    "10. A method according to claim 1, further comprising refining a model using a robust estimator, wherein model parameters of the model describe the movement of the camera using image capture and depth of the scene.",
    "11. A method according to claim 10, wherein using the robust estimator comprises: setting the model parameters as best model parameters of a best model; evaluating a fit of the best model; extracting further line segments from said image and calculating new model parameters to produce a new model; evaluating a fit of the new model; updating the best model parameters with the new model parameters if the fit of the new model is better than the fit of the best model; and continually repeating the process of selecting further line segments to produce a new model, evaluating the fit of the new model and updating the best model parameters with the new model parameters if the fit of the new model is better than the fit of the best model until a stopping criteria is reached.",
    "12. A method according to claim 11, wherein the fit is evaluated by calculating a number of inliers.",
    "13. A method according to claim 1, the movement of the camera during the capturing of the image is estimated using only image data from a single received image.",
    "14. A method according to claim 5, wherein estimating the movement of the camera during the capture of the image comprises: fitting three of the extracted segments to one of said vertical planes to obtain an estimate of camera motion parameters and a parameter related to a depth of the scene; determining a further parameter related to the depth of the scene by assuming the normal of a further line segment extracted from the image when expressed in motion compensated coordinates is normal to a vertical of the scene.",
    "15. A method according to claim 1, the method further comprising comparing the extracted segments with at least one threshold to determine prior to determining whether said line segments lies on at least one of the said two vertical planes.",
    "16. A system for compensating for camera motion during capture of an image in a rolling shutter camera, the system comprising: a processor, said processor comprising: an input for receiving an image of a scene captured by a camera with a rolling shutter; and an output for outputting image data compensated for movement of the camera during the capture of the image, wherein the processor is adapted to: extract line segments from the received image estimate the movement of the camera during the capturing of the image from the received image; and produce image data compensated for the movement of the camera during the capture of the image wherein the scene is approximated by a horizontal plane and two vertical planes that intersect at a line at infinity and estimating the movement of the camera during the capture of the image comprises assuming that the extracted line segments are vertical and lie on vertical planes.",
    "17. A system according to claim 16, wherein said camera is mounted on a vehicle.",
    "18. An autonomous driving system provided on a vehicle, said system comprising an image input from the system of claim 16.",
    "19. An autonomous driving system according to claim 18, adapted to recognise objects from image data that has been compensated for motion of the vehicle during the capture of the image.",
    "20. A carrier medium comprising computer readable code configured to cause a computer to perform the method of claim 1."
  ],
  "description_excerpt": "Embodiments described herein generally relate to the field of computer vision.\n\nAutomotive driver-less vehicles have sparked a lot of vision research and unlocked the demands for the real time solutions for a number of unsolved problems in computer vision. While a commercial budget camera is an attractive choice of these vehicles, a significant distortion could be observed in the captured images. These cameras are generally built upon CMOS sensors, which possess a prevalent mechanism widely known as rolling shutter (RS). In contrast to global shutter (GS) camera which capture the scene in a row-wise fashion from top to bottom with a constant inter-row delay. The RS imaging acquires apparent camera motion for different rows and violates the properties of the perspective camera model. This causes noticeable and prominent distortions.\n\nFIG. 1(a) is a schematic of a vehicle undergoing rotation used to describe Ackermann motion;\n\nFIG. 1(b) is a schematic of a vehicle undergoing translation used to describe Ackermann motion;\n\nFIG. 2(a) is a schematic showing a vehicle and the approximation the scene observed by a camera on the vehicle;\n\nFIG. 2(b) is a is a schematic showing a vehicle and the approximation the scene observed by a camera on the vehicle where the scene is not symmetric;\n\nFIG. 3 is a flow diagram showing a method in accordance with an embodiment;\n\nFIG. 4 is a flow diagram showing in more detail the steps of deriving the model parameters in the method of FIG. 3;\n\nFIG. 5(a) is a plot of the measured angular velocity against the actual angular velocity for a vehicle exhibiting just angular velocity;",
  "cpc": [
    "H04N 23/6811",
    "G05D 1/0088",
    "G05D 1/0251",
    "G05D 2201/0213",
    "G06T 2207/30244",
    "G06T 7/12",
    "G06T 7/13",
    "G06T 7/20",
    "G06T 7/251",
    "G06T 7/70",
    "H04N 23/6812",
    "H04N 23/683",
    "H04N 25/531",
    "H04N 5/23254",
    "H04N 5/23258",
    "H04N 5/23267"
  ],
  "ipc": [
    "G05D 1/00",
    "G05D 1/02",
    "G06T 7/246",
    "H04N 5/232"
  ],
  "assignees": [
    "Toshiba Corp"
  ],
  "inventors": [
    "Pulak Purkait",
    "Christopher Zach"
  ],
  "filing_date": "2018-02-13",
  "publication_date": "2019-08-27",
  "grant_date": "2019-08-27",
  "priority_date": "2017-08-29",
  "application_number": "US-201815895320-A",
  "family_id": "60037147",
  "cited_by_count": 3,
  "citations": [
    "JPH07302325A",
    "EP2091020A1",
    "US20150086080A1",
    "US20160037032A1"
  ]
}