Markerless Geometric Registration Of Multiple Projectors On Extruded Surfaces Using An Uncalibrated Camera

{
  "publication_number": "US2012098937A1",
  "country": "US",
  "kind": "A1",
  "title": "Markerless Geometric Registration Of Multiple Projectors On Extruded Surfaces Using An Uncalibrated Camera",
  "abstract": "A method for registering multiple projectors on a vertically extruded three dimensional display surface with a known aspect ratio includes recovering both the camera parameters and the three dimensional shape of the surface from a single image of the display surface from an uncalibrated camera, capturing images from the projectors to relate the projector coordinates with the display surface points, and segmenting parts of the image for each projector to register the projectors to create a seamlessly wall-paper projection on the display surface using a representation between the projector coordinates with display surface points using a rational Bezier patch. A method for performing a deterministic geometric auto-calibration to find intrinsic and extrinsic parameters of each projector is included.",
  "claims": [
    "1. A method for geometrically registering multiple projectors on a extruded three dimensional display surface with a known aspect ratio comprising: recovering the camera parameters from a single image of the display surface from an uncalibrated camera; recovering three dimensional shape of the surface from a single image of the display surface from an uncalibrated camera and the recovered camera parameters; capturing images from the projectors to relate the projector coordinates with the display surface points; and segmenting parts of the image for each projector to register the projectors to create a seamless projection on the display surface. 2. The method of claim 1 where recovering the camera parameters from a single image of the display surface from an uncalibrated camera comprises using a single image of the display from the uncalibrated camera and the known aspect ratio of the display surface to recover the camera properties (intrinsic and extrinsic parameter matrices) using a nonlinear optimization. 3. The method of claim 1 where recovering three dimensional shape of the surface from a single image of the display surface comprises ray casting using the recovered camera parameters through the image of the flanking top curves to recover the top curve in three dimensions and extruding the top curve down to recover the shape of the entire display. 4. The method of claim 3 where the extruded surface is a smooth vertically extruded surface where capturing images from the projectors to relate the projector coordinates with the display surface points comprises capturing images from the projectors to relate the projector coordinates with the display surface points using the recovered the three dimensional shape of the display to generate a two dimensional display parameterization. 5. The method of claim 4 where capturing images from the projectors to relate the projector coordinates with the display surface points using the recovered the three dimensional shape of the display to generate a two dimensional display parameterization comprises generating an arc length parameterization of the smooth path surface s, and generating a linear parametrization of the vertical line t, to together provide a two dimensional parametrization of the display (s,t). 6. The method of claim 1 where segmenting parts of the image for each projector to register the projectors to create a seamless projection on the display surface comprises finding correspondences from the projector (x,y) to the display (s, t), and approximating a mapping from the samples by fitting a rational Bezier to these correspondences; and then setting an image parameterization (s i, t i) to be identical to the two dimensional display parameterization to define the part of the image to be projected by each projector so that the resulting display is wall papered and seamless. 7. The method of claim 1 where segmenting parts of the image for each projector to register the projector to create a seamless projection on the display surface comprises segmenting parts of the image for each projector to register the projectors from an arbitrary viewpoint instead of wall papering on a smooth extruded surface by rendering a virtual image of a three dimensional image from a virtual arbitrary viewpoint in the virtual image coordinates (s i, t i); and mapping the (s i,t i) to the display coordinate (s,t) via a projective texture from the virtual arbitrary viewpoint. 8. The method of claim 2 where using a single image of the display from the uncalibrated camera and the known aspect ratio of the display surface to recover the camera properties (intrinsic and extrinsic parameter matrices) using a nonlinear optimization comprises performing a corner-based optimization and an curve-based optimization. 9. The method of claim 8 where a corner-based optimization and a curve-based optimization comprises of a first phase of non-linear corner based optimization that minimizes the reprojection error of the four corners of the display in their captured image, followed by a curve based optimization that minimizes reprojection error between the captured shape of the top and bottom flanking path curves. 10. The method of claim 9 further comprising initializing the optimization of previous claim by initializing the angle of rotations about the X, Y and Z axes to make R zero and T approximately at the center of a planar rectangle with the four corners of the display surface at a depth of approximately the same order of magnitude as the size of the display surface. 11. The method of claim 1 where the extruded surface is a smooth or piecewise planar vertically extruded surface formed by smooth and piecewise linear path curve respectively, where segmenting parts of the image for each projector comprises a deterministic geometric auto-calibration to find intrinsic and extrinsic parameters of each projector; and a mapping that relates the projector coordinates (x,y) to the three dimensional display coordinates directly, where the intrinsic parameters comprise focal length and vertical offset and where the extrinsic parameters comprise the position and orientation of the projector. 12. The method of claim 11 where geometric auto-calibration is independently performed for each projector and comprises projecting a pattern from each projector which pattern includes a top and bottom curve, capturing an image of this pattern by the camera, and then using a deterministic algorithm using this captured image, the recovered camera parameters and the three dimensional display shape to estimate the intrinsic and extrinsic parameter of each projector. 13. The method of claim 12 further comprising finding a view frustum of each projector using constraints due to the known shape of the display and camera parameters. 14. The method of claim 13 using constraints due to the known shape of the display and camera parameters comprises using the planarity of the 3D curve formed by the projected line on the extruded surface, and/or distance of the image plane with respect to image size. 15. The method of claim 11 where mapping between the projector coordinates and the three dimensional display coordinates allows a registration which is correct from an arbitrary view point by rendering a virtual image of the three dimensional scene from the virtual arbitrary viewpoint in the virtual image coordinates (s i, t i), mapping the (s i,t i) to the three dimensional display coordinate via a projective texture from the virtual viewpoint, and mapping the three dimensional display coordinates back to the projector coordinates (x,y) using the recovered intrinsic and extrinsic properties. 16. The method of claim 11 where the extruded surface includes vertically extruded surfaces with corners. 17. The method of claim 12 where the steps of projecting a pattern from each projector, capturing an image of this pattern by the camera, and then using a deterministic algorithm using this captured image, the recovered camera parameters and the three dimensional display shape to estimate the intrinsic and extrinsic parameter of each projector further comprises changing projector pose or orientation quickly reperforming each of the steps. 18. The method of claim 1 further comprising providing overlapping projectors on the corner of a CAVE. 19. The method of claim 1 further comprising performing each of the steps automatically using a single uncalibrated camera. 20. The method of claim 1 further comprising performing each of the steps in real-time using special purpose embedded hardware or more general hardware in the machines driving the projectors making it suitable for real-time head-tracked users in a virtual reality environment."
  ],
  "cpc": [
    "G03B 21/14",
    "G03B 21/13",
    "G03B 37/04",
    "H04N 9/3147",
    "H04N 9/3185",
    "H04N 9/3194",
    "H04N 9/3197"
  ],
  "assignees": [
    "SAJADI BEHZAD",
    "MAJUMDER ADITI"
  ],
  "filing_date": "2010-04-28",
  "publication_date": "2012-04-26",
  "priority_date": "2009-04-28",
  "application_number": "US-201013318092-A",
  "family_id": "43050767",
  "citations": [
    "US2010266220A1",
    "US6456339B1",
    "US6510244B2",
    "US6733138B2",
    "US6811264B2",
    "US7010080B2",
    "US7019713B2",
    "US7663640B2",
    "US7740361B2",
    "US7893393B2",
    "US7942530B2",
    "US7967451B2",
    "US8023726B2",
    "US8147073B2",
    "US8237873B2",
    "US8328365B2",
    "US8355601B2"
  ]
}