Systems and methods for rolling shutter correction

{
  "publication_number": "US10771699B2",
  "country": "US",
  "kind": "B2",
  "title": "Systems and methods for rolling shutter correction",
  "abstract": "An image processing method includes obtaining an image frame through an imaging device over a period of time. The image frame includes a plurality of groups of pixels that are exposed to light at different time points within the period of time. The method further includes obtaining attitude information of the imaging device during the period of time, deriving positional state of an individual group of pixels in the plurality of groups of pixels based on the attitude information of the imaging device, and processing the image frame using the positional state.",
  "claims": [
    "1. An image processing method comprising: obtaining, through an imaging device, an image frame over a period of time, wherein the image frame comprises a plurality of groups of pixels that are exposed to light at different time points within the period of time; obtaining attitude information of the imaging device during the period of time, wherein the attitude information of the imaging device is measured by a motion capture system located remotely from the imaging device, the motion capture system including at least one of a vision sensor, a barometer, an ultrasonic based navigation system, an indoor positioning system, or a lidar; deriving positional state of an individual group of pixels in the plurality of groups of pixels based on the attitude information of the imaging device; and processing the image frame using the positional state.",
    "2. The method of claim 1, wherein the image frame comprises an image of a target or a portion thereof, and the image frame is obtained while the target and the imaging device are moving relative to each other.",
    "3. The method of claim 1, wherein the positional state of the individual group of pixels is derived from the attitude information of the imaging device when exposing the individual group of pixels at one of the time points.",
    "4. The method of claim 1, wherein: obtaining the attitude information of the imaging device comprises: obtaining, from one or more sensors operably coupled to the imaging device, a first set of positional data of the imaging device at a first one of the time points; obtaining, from the one or more sensors, a second set of positional data of the imaging device at a second one of the time points; the individual group of pixels are exposed at one of the time points between the first one of the time points and the second one of the time points; and deriving the positional state of the individual group of pixels comprises: calculating an individual set of positional data of the imaging device at the one of the time points based on the first set of positional data and the second set of positional data according to a simulation model; and deriving the positional state of the individual group of pixels based on the individual set of positional data.",
    "5. The method of claim 1, wherein an inertial measurement unit (IMU) is rigidly coupled to the imaging device and configured to measure the attitude information of the imaging device.",
    "6. The method of claim 1, wherein processing the image frame comprises correcting the individual group of pixels using position data from the positional state of the individual group of pixels.",
    "7. The method of claim 6, wherein the individual group of pixels are compensated with a direction and a magnitude derived from the position data, and the direction is opposite to a translational shifting direction of the imaging device determined from the position data.",
    "8. The method of claim 6, wherein the translational shifting direction of the imaging device is relative to another group of pixels within the frame.",
    "9. The method of claim 1, wherein processing the image frame comprises correcting the individual group of pixels using attitude data of the positional state of the individual group of pixels.",
    "10. The method of claim 9, wherein the individual group of pixels are corrected using a transformation matrix determined from the attitude data.",
    "11. The method of claim 1, wherein processing the image frame comprises correcting the individual group of pixels using position data from the positional state of the individual group of pixels in conjunction with attitude data of the positional state of the individual group of pixels.",
    "12. The method of claim 11, wherein the individual group of pixels are corrected using a transformation matrix determined from position data and attitude data corresponding to another group of pixels and the position data and the attitude data corresponding to the individual group of pixels.",
    "13. The method of claim 1, wherein the imaging device is operably coupled to a movable object.",
    "14. The method of claim 13, wherein the movable object is stationary relative to the imaging device, and the attitude information of the imaging device is obtained from one or more sensors configured to measure attitude information of the movable object.",
    "15. The method of claim 13, wherein: the movable object permits relative movement between the imaging device and the movable object; and obtaining the attitude information of the imaging device comprises compensating attitude information of the movable object based upon the relative movement between the movable object and the imaging device.",
    "16. The method of claim 15, wherein: a first inertial measurement unit (IMU) is coupled to the movable object and configured to measure the attitude information of the movable object; the imaging device is operably coupled to a second IMU; and the relative movement between the movable object and the imaging device is determined by data retrieved from the first IMU and data retrieved from the second IMU.",
    "17. The method of claim 13, wherein the imaging device is mounted on a stabilization system supported by the movable object, and the stabilization system permits the imaging device to rotate along one or more axes relative to the movable object.",
    "18. The method of claim 17, wherein the attitude information of the imaging device is obtained by compensating attitude information of the movable object along at least one of the one or more rotational axes.",
    "19. An image processing apparatus comprising: one or more processors; and a non-transitory computer readable medium storing program instructions that, when executed by the one or more processors, cause the one or more processors to individually or collectively: obtain, through an imaging device, an image frame over a period of time, wherein the image frame comprises a plurality of groups of pixels that are exposed to light at different time points within the period of time; obtain attitude information of the imaging device during the period of time, wherein the attitude information of the imaging device is measured by a motion capture system located remotely from the imaging device, the motion capture system including at least one of a vision sensor, a barometer, an ultrasonic based navigation system, an indoor positioning system, or a lidar; derive positional state of an individual group of pixels in the plurality of groups of pixels based on the attitude information of the imaging device; and process the image frame using the positional state.",
    "20. An image processing method comprising: obtaining, through an imaging device operably coupled to a movable object, an image frame over a period of time, wherein the image frame comprises a plurality of groups of pixels that are exposed to light at different time points within the period of time; obtaining attitude information of the imaging device during the period of time, wherein obtaining the attitude information of the imaging device includes compensating attitude information of the movable object based upon a relative movement between the movable object and the imaging device; deriving positional state of an individual group of pixels in the plurality of groups of pixels based on the attitude information of the imaging device; and processing the image frame using the positional state."
  ],
  "description_excerpt": "Most digital cameras today can enable video capture. Examples of consumer products include regular cameras, portable hand-held electronic devices, and other movable devices. Some of these consumer devices may use complementary metal oxide semi-conductor (CMOS)-based camera sensors. Most CMOS sensors use rolling shutter (RS) mechanism, as opposed to using a global shutter (GS), e.g., charge-coupled device (CCD)-based camera sensors. In a RS camera, detector rows are read and reset sequentially. Each row of the CMOS sensor is exposed during a slightly different time window. Since pixels are acquired at different points in time, motion of either the camera or the imaged object may cause geometrical distortion in the captured images. The geometric distortion may be exaggerated when the RS camera is coupled to a movable object, such as an unmanned aerial vehicle (UAV). In some instances, motion of the movable object may cause the resulting image to be tilted or skewed at an angle.\n\nIn some cases, video quality can be improved using mechanical systems. For example, mechanical image stabilization (MIS) systems can be used to actuate the camera lenses or CMOS image sensor, to compensate for small pan and tilt rotational motions. MIS systems can stabilize images substantially in real-time, and do not require significant computation (e.g., image processing) by the camera. However, MIS systems may be unable to compensate for rapid high frequency motions, such as those caused by vibrations from a vehicle engine. Moreover, MIS systems are generally not suitable for most consumer-based digital cameras due to their costs and form factor.",
  "cpc": [
    "H04N 23/683",
    "H04N 23/6812",
    "H04N 23/689",
    "H04N 25/531",
    "H04N 5/23258",
    "H04N 5/23267",
    "H04N 5/2329",
    "H04N 5/3532"
  ],
  "ipc": [
    "H04N 5/232",
    "H04N 5/353"
  ],
  "assignees": [
    "SZ DJI Technology Co Ltd"
  ],
  "inventors": [
    "You Zhou",
    "Cong Zhao",
    "Jie Qian",
    "Peiliang LI"
  ],
  "filing_date": "2018-11-08",
  "publication_date": "2020-09-08",
  "grant_date": "2020-09-08",
  "priority_date": "2016-05-20",
  "application_number": "US-201816184557-A",
  "family_id": "60324662",
  "cited_by_count": 2,
  "citations": [
    "US20120249784A1",
    "WO2013127338A1",
    "WO2015106462A1",
    "CN105031935A",
    "CN105516583A",
    "US9826202B2",
    "CN105519094A",
    "CN105049706A"
  ]
}