Stereo auto-calibration from structure-from-motion

{
  "publication_number": "US9729858B2",
  "country": "US",
  "kind": "B2",
  "title": "Stereo auto-calibration from structure-from-motion",
  "abstract": "Auto-calibration of stereo cameras installable behind the windshield of a host vehicle and oriented to view the environment through the windshield. Multiple first image points are located of one of the first images captured from the first camera at a first time and matched with first image points of at least one other of the first images captured from the first camera at a second time to produce pairs of corresponding first image points respectively in the first images captured at the different times. World coordinates are computed from the corresponding first image points. Second image points in the second images captured from the second camera are matched to at least a portion of the first image points. The world coordinates as determined from the first camera are used, to solve for camera parameters of the second camera from the matching second image points of the second camera.",
  "claims": [
    "1. A computer implemented method for auto-calibration of a camera of a host vehicle, the method comprising the following operations performed by one or more processors: receiving, from a first camera, a first image and a second image, the first and second images including an environment of the host vehicle; receiving, from a second camera, a third image including the environment of the host vehicle; identifying a first image point corresponding to a feature in the first image and a second image point corresponding to the feature in the second image; determining a three-dimensional coordinate of the feature based on a motion parameter of the host vehicle and a comparison of the first image point and the second image point; determining a third image point corresponding to the feature in the third image; and determining at least one calibration parameter of the second camera based on the three-dimensional coordinate of the feature and the third image point.",
    "2. The method of claim 1, further comprising: identifying a fourth image point corresponding to a second feature in a fourth image received from the first camera and a fifth image point corresponding to the second feature in a fifth image received from the second camera; and determining a three dimensional coordinate of the second feature based on the calibration parameter of the second camera and a calculated stereo disparity between the fourth image and the fifth image.",
    "3. The method of claim 1, wherein the feature in the first and second image includes an object in the environment of the host vehicle.",
    "4. The method of claim 1, wherein at least one of the first and second cameras faces a windshield of the host vehicle.",
    "5. The method of claim 1, wherein the first and second cameras are displaced from each other relative to a forward or rearward placement on the host vehicle.",
    "6. The method of claim 1, wherein the first and second cameras are displaced from each other relative to a left side and a right side placement on the host vehicle.",
    "7. The method of claim 1, wherein the first and second cameras are displaced from each other relative to a vertical placement on the host vehicle.",
    "8. The method of claim 1, wherein the calibration parameter is representative of a distortion of a lens of the second camera.",
    "9. The method of claim 1, wherein the motion parameter of the host vehicle is indicative of ego motion of the host vehicle.",
    "10. The method of claim 1, wherein the first and second cameras include a rolling shutter, and the method further comprises: adjusting the three-dimensional coordinate of the feature based on a time when the third image was captured.",
    "11. The method of claim 10, wherein the adjustment of the three-dimensional coordinate is further based on a motion of the host vehicle.",
    "12. A system for auto-calibration of a camera of a host vehicle, the system comprising: at least one processor programmed to: receive, from a first camera, a first image and a second image, the first and second images including an environment of the host vehicle; receive, from a second camera, a third image including the environment of the host vehicle; identify a first image point corresponding to a feature in the first image and a second image point corresponding to the feature in the second image; determine a three-dimensional coordinate of the feature based on a motion parameter of the host vehicle and a comparison of the first image point and the second image point; determine a third image point corresponding to the feature in the third image; and determine at least one calibration parameter of the second camera based on the three-dimensional coordinate of the feature and the third image point.",
    "13. The system of claim 12, wherein the processor is further programmed to: identify a fourth image point corresponding to a second feature in a fourth image received from the first camera and a fifth image point corresponding to the second feature in a fifth image received from the second camera; and determine a three dimensional coordinate of the second feature based on the calibration parameter of the second camera and a calculated stereo disparity between the fourth image and the fifth image.",
    "14. The system of claim 12, wherein the feature in the first and second image includes an object in the environment of the host vehicle.",
    "15. The system of claim 12, wherein at least one of the first and second cameras faces a windshield of the host vehicle.",
    "16. The system of claim 12, wherein the first and second cameras are displaced from each other relative to a forward or rearward placement on the host vehicle.",
    "17. The system of claim 12, wherein the first and second cameras are displaced from each other relative to a left side and a right side placement on the host vehicle.",
    "18. The system of claim 12, wherein the first and second cameras are displaced from each other relative to a vertical placement on the host vehicle.",
    "19. The system of claim 12, wherein the calibration parameter is representative of a distortion of a lens of the second camera and the motion parameter of the host vehicle is indicative of ego motion of the host vehicle.",
    "20. The system of claim 12, wherein the first and second cameras include a rolling shutter, and wherein the processor is further programmed to: adjust the three-dimensional coordinate of the feature based on a time when the third image was captured.",
    "21. A non-transitory computer-readable storage medium that comprises instructions that are executable by at least one processor to cause the at least one processor to perform a method for auto-calibration of a camera of a host vehicle, the method comprising: receiving, from a first camera, a first image and a second image, the first and second images including an environment of the host vehicle; receiving, from a second camera, a third image including the environment of the host vehicle; identifying a first image point corresponding to a feature in the first image and a second image point corresponding to the feature in the second image; determining a three-dimensional coordinate of the feature based on a motion parameter of the host vehicle and a comparison of the first image point and the second image point; determining a third image point corresponding to the feature in the third image; and determining at least one calibration parameter of the second camera based on the three-dimensional coordinate of the feature and the third image point."
  ],
  "description_excerpt": "1. Technical Field\n\nThe present disclosure relates to a method for calibration stereo cameras and in particular for use inside a vehicle as part of a driver assistance system.\n\n2. Description of Related Art\n\nStereo vision is the process of recovering depth from camera images by comparing two or more views of the same scene. Binocular stereo uses two images, taken with cameras that are separated by a horizontal distance known as the “baseline”. Calibrating the stereo camera system allows computation of three-dimensional world points in actual units, e.g. millimeters, relative to the cameras based on the image coordinates.\n\nCalibration of a stereo camera system involves the estimation of extrinsic parameters which describe translation and rotation of the second camera relative to the first camera and intrinsic parameters of each camera. Intrinsic parameters include focal lengths, principal points and other parameters which describe camera image distortion. Image distortion means that image points are displaced from the position predicted by an ideal pinhole projection model. The most common form of distortion is radial distortion, which is inherent in all single-element lenses. Under radial distortion, e.g. pincushion distortion and/or barrel distortion, image points are displaced in a radial direction from the image center.\n\nDifferent sources of information can be used to obtain camera calibration. One approach (sometimes called “off-line” calibration) is to use a known target where the three-dimensional world coordinates (or locations in three-dimensional space) of respective multiple points are known.",
  "cpc": [
    "H04N 13/246",
    "B60R 1/00",
    "B60R 11/04",
    "B60R 2011/0026",
    "B60R 2300/105",
    "B60R 2300/402",
    "G06F 18/22",
    "G06K 9/00791",
    "G06K 9/6201",
    "G06T 2207/30252",
    "G06T 7/579",
    "G06T 7/85",
    "G06V 20/56",
    "H04N 13/0239",
    "H04N 13/0246",
    "H04N 13/0271",
    "H04N 13/239",
    "H04N 13/271",
    "H04N 2013/0081",
    "H04N 25/531",
    "H04N 5/3532"
  ],
  "ipc": [
    "G06T 7/00",
    "B60R 11/04",
    "G06K 9/00",
    "G06K 9/62",
    "H04N 13/239",
    "H04N 5/353",
    "B60R 1/00",
    "G06T 7/579",
    "G06T 7/80"
  ],
  "assignees": [
    "Mobileye Vision Technologies Ltd"
  ],
  "inventors": [
    "Harel Livyatan",
    "Oded Berberian"
  ],
  "filing_date": "2016-09-29",
  "publication_date": "2017-08-08",
  "grant_date": "2017-08-08",
  "priority_date": "2013-11-26",
  "application_number": "US-201615280117-A",
  "family_id": "53182319",
  "cited_by_count": 10,
  "citations": [
    "US6118475A",
    "US9053562B1",
    "US20150294505A1",
    "US20160054452A1",
    "US20160070102A1",
    "US20160073098A1"
  ]
}