Patent · US10498966B2 · B2 · US

Rolling shutter correction for images captured by a camera mounted on a moving vehicle

(11) Publication number

US10498966B2

(21) Application number

16/162,224

(22) Filing date

2018-10-16

(30) Priority date

2017-10-19

(43) Publication date

2019-12-03

(45) Date of grant

2019-12-03

(52) CPC

G01S Radio direction-finding; radio navigation; determining distance or velocity by use of radio waves; locating or presence-detecting by use of the reflection or reradiation of radio waves; analogous arrangements using other waves: 7/497, 17/42, 17/86, 17/87, 17/89, 17/931, 7/4817, 7/4972
G01C Measuring distances, levels or bearings; surveying; navigation; gyroscopic instruments; photogrammetry or videogrammetry: 21/1652, 21/3602, 25/00
G05D Systems for controlling or regulating non-electric variables: 1/0231, 1/0248, 1/0287
G06T Image data processing or generation, in general: 2207/10028, 2207/10048, 2207/20092, 2207/20221, 2207/30241, 2207/30242, 2207/30252, 7/13, 7/33, 7/55, 7/80
G06V Image or video recognition or understanding: 20/56
H04N Pictorial communication, e.g. television: 13/106, 23/54, 23/60, 23/689, 23/90, 5/04, 5/2329

(73) Assignee

DEEPMAP INC

(54) Title

Rolling shutter correction for images captured by a camera mounted on a moving vehicle

(57) Abstract

A system performs rolling shutter correction to transform data captured by sensors of a vehicle, for example, cameras or lidar mounted on a vehicle, for example, an autonomous vehicle. The images captured by a rolling shutter camera mounted on a moving vehicle show rolling shutter distortion. The rolling shutter correction transforms the data representing points of scenes to perform rolling shutter compensation, i.e., compensation for the rolling shutter distortion. The rolling shutter compensation ensures that data representing as three dimensional points, for example, data captured by lidar is consistent with data represented in images captured by a rolling shutter camera. The system performs rolling shutter compensation by estimating a distance travelled by the vehicle between the time that the camera captured a point and the time that the image scan was completed and translating the 3 D points by the estimated distance.

Full text

View on Google Patents

Claims (20)

A computer implemented method for performing rolling shutter correction of data captured by sensors of a vehicle, comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.
The method of claim 1, further comprising: project the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.
The method of claim 2, wherein the user interface is displayed by a toolkit for development of a high definition map.
The method of claim 3, wherein the vehicle is an autonomous vehicle, further comprising: sending signals to the controls of the autonomous vehicle based on the high definition map.
The method of claim 1, wherein image is associated with a reference point, wherein determining the estimate of a distance travelled by the vehicle comprises: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.
The method of claim 1, wherein the plurality of 3D points are determined using a lidar scan.
The method of claim 1, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.
The method of claim 1, further comprising, mapping a shape displayed in the image to 3D space, comprising: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.
A non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.
The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to perform the steps comprising: projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.
The non-transitory computer readable medium of claim 10, wherein the user interface is displayed by a toolkit for development of a high definition map.
The non-transitory computer readable medium of claim 11, wherein the vehicle is an autonomous vehicle, wherein the stored instructions further cause the processor to perform the steps comprising: sending signals to the controls of the autonomous vehicle based on the high definition map.
The non-transitory computer readable medium of claim 9, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.
The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using a lidar scan.
The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.
The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.
A computer system comprising: a processor; and a non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.
The computer system of claim 17, wherein the stored instructions further cause the processor to perform the steps comprising: projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.
The computer system of claim 17, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.
The computer system of claim 17, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.

Citations (9)

US2007154202A1
US2008195261A1
US2009201361A1
US2011134254A1
US2015054955A1
US2015073711A1
US2015181198A1
US2015234045A1
US2016253566A1

Record as JSON

{
  "publication_number": "US10498966B2",
  "country": "US",
  "kind": "B2",
  "title": "Rolling shutter correction for images captured by a camera mounted on a moving vehicle",
  "abstract": "A system performs rolling shutter correction to transform data captured by sensors of a vehicle, for example, cameras or lidar mounted on a vehicle, for example, an autonomous vehicle. The images captured by a rolling shutter camera mounted on a moving vehicle show rolling shutter distortion. The rolling shutter correction transforms the data representing points of scenes to perform rolling shutter compensation, i.e., compensation for the rolling shutter distortion. The rolling shutter compensation ensures that data representing as three dimensional points, for example, data captured by lidar is consistent with data represented in images captured by a rolling shutter camera. The system performs rolling shutter compensation by estimating a distance travelled by the vehicle between the time that the camera captured a point and the time that the image scan was completed and translating the 3 D points by the estimated distance.",
  "claims": [
    "1. A computer implemented method for performing rolling shutter correction of data captured by sensors of a vehicle, comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.",
    "2. The method of claim 1, further comprising: project the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.",
    "3. The method of claim 2, wherein the user interface is displayed by a toolkit for development of a high definition map.",
    "4. The method of claim 3, wherein the vehicle is an autonomous vehicle, further comprising: sending signals to the controls of the autonomous vehicle based on the high definition map.",
    "5. The method of claim 1, wherein image is associated with a reference point, wherein determining the estimate of a distance travelled by the vehicle comprises: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.",
    "6. The method of claim 1, wherein the plurality of 3D points are determined using a lidar scan.",
    "7. The method of claim 1, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.",
    "8. The method of claim 1, further comprising, mapping a shape displayed in the image to 3D space, comprising: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.",
    "9. A non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.",
    "10. The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to perform the steps comprising: projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.",
    "11. The non-transitory computer readable medium of claim 10, wherein the user interface is displayed by a toolkit for development of a high definition map.",
    "12. The non-transitory computer readable medium of claim 11, wherein the vehicle is an autonomous vehicle, wherein the stored instructions further cause the processor to perform the steps comprising: sending signals to the controls of the autonomous vehicle based on the high definition map.",
    "13. The non-transitory computer readable medium of claim 9, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.",
    "14. The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using a lidar scan.",
    "15. The non-transitory computer readable medium of claim 9, wherein the plurality of 3D points are determined using sensor data captured by one or more vehicles previously travelling along the path of the moving vehicle.",
    "16. The non-transitory computer readable medium of claim 9, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane.",
    "17. A computer system comprising: a processor; and a non-transitory computer readable storage medium storing instructions for performing rolling shutter correction of data captured by sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps comprising: receiving an image captured by a rolling shutter camera mounted on a vehicle, the rolling shutter camera capturing the image via an image scan, the image comprising a plurality of scan-lines, the image associated with a distortion axis, wherein each scan-line of the image maps to a distinct point along the distortion axis; identifying a plurality of three-dimensional (3D) data points corresponding to the scene captured by the image, each 3D data point having a location in a 3D space; for each of the plurality of 3D points: projecting the 3D point to the image coordinates to obtain a projected point; determining based on the position of the projected point, an estimate of a distance travelled by the vehicle between the time of capture of the 3D point by the rolling shutter camera and the time of completion of the image scan; translating the 3D point to location along a direction of movement of the vehicle by the estimate of the distance; storing the translated 3D points for displaying in conjunction with the image captured by the rolling shutter camera.",
    "18. The computer system of claim 17, wherein the stored instructions further cause the processor to perform the steps comprising: projecting the plurality of translated 3D points on the image to obtain an overlapping image wherein the projection of the 3D points is aligned with the pixels of the image; and configuring a user interface for displaying the overlapping image.",
    "19. The computer system of claim 17, wherein image is associated with a reference point, wherein instructions for determining the estimate of a distance travelled by the vehicle comprise instructions: determining a distance of the projected point from the reference point along the distortion axis; and determining an estimate of a distance travelled by the vehicle between the time that the 3D point was captured by the rolling shutter camera and the time that the image scan was completed, the estimate of the distance travelled determined based on the distance of the projected point from the reference point along the distortion axis.",
    "20. The computer system of claim 17, wherein the stored instructions further cause the processor to map a shape displayed in the image to 3D space by performing the steps of: projecting a set of the translated 3D points to the image; selecting a subset of the set of translated 3D points, the subset comprising translated 3D points whose projection falls within the shape displayed in the image; fitting a plane through the selected subset of translated 3D points; and determining a depth of a 2D point within the shape displayed in the image by projecting the 2D point to the fitted plane."
  ],
  "cpc": [
    "G01S 7/497",
    "G01C 21/1652",
    "G01C 21/3602",
    "G01C 25/00",
    "G01S 17/42",
    "G01S 17/86",
    "G01S 17/87",
    "G01S 17/89",
    "G01S 17/931",
    "G01S 7/4817",
    "G01S 7/4972",
    "G05D 1/0231",
    "G05D 1/0248",
    "G05D 1/0287",
    "G06T 2207/10028",
    "G06T 2207/10048",
    "G06T 2207/20092",
    "G06T 2207/20221",
    "G06T 2207/30241",
    "G06T 2207/30242",
    "G06T 2207/30252",
    "G06T 7/13",
    "G06T 7/33",
    "G06T 7/55",
    "G06T 7/80",
    "G06V 20/56",
    "H04N 13/106",
    "H04N 23/54",
    "H04N 23/60",
    "H04N 23/689",
    "H04N 23/90",
    "H04N 5/04",
    "H04N 5/2329"
  ],
  "assignees": [
    "DEEPMAP INC"
  ],
  "filing_date": "2018-10-16",
  "publication_date": "2019-12-03",
  "grant_date": "2019-12-03",
  "priority_date": "2017-10-19",
  "application_number": "US-201816162224-A",
  "family_id": "66169248",
  "citations": [
    "US2007154202A1",
    "US2008195261A1",
    "US2009201361A1",
    "US2011134254A1",
    "US2015054955A1",
    "US2015073711A1",
    "US2015181198A1",
    "US2015234045A1",
    "US2016253566A1"
  ]
}

Record 971 of 5,000 in Patents full text (MLC-0201). Request the full dataset.