Industrial vehicle with feature-based localization and navigation

{
  "publication_number": "US9880009B2",
  "country": "US",
  "kind": "B2",
  "title": "Industrial vehicle with feature-based localization and navigation",
  "abstract": "An industrial vehicle is provided comprising a drive mechanism, a steering mechanism, a vehicle controller, a camera, and a navigation module. The camera is communicatively coupled to the navigation module, the vehicle controller is responsive to commands from the navigation module, and the drive mechanism and the steering mechanism are responsive to commands from the vehicle controller. The camera is configured to capture an input image of a warehouse ceiling comprising elongated skylights characterized by different rates of image intensity change along longitudinal and transverse axial directions, and ceiling lights characterized by a circularly symmetric rate of image intensity change. The navigation module is configured to distinguish between the ceiling lights and the skylights and send commands to the vehicle controller for localization, or to navigate the industrial vehicle through the warehouse based upon valid ceiling light identification, valid skylight identification, or both.",
  "claims": [
    "1. An industrial vehicle comprising a drive mechanism, a steering mechanism, a vehicle controller, a camera, and a navigation module, wherein: the camera is communicatively coupled to the navigation module; the vehicle controller is responsive to commands from the navigation module; the drive mechanism and the steering mechanism are responsive to commands from the vehicle controller; the camera is configured to capture an input image of a warehouse ceiling comprising elongated skylights characterized by different rates of image intensity change along longitudinal and transverse axial directions, and ceiling lights characterized by a circularly symmetric rate of image intensity change; and the navigation module is configured to distinguish between the ceiling lights and the skylights by executing machine readable instructions to create a Gaussian scale space pyramid from the input image of the warehouse ceiling, wherein the Gaussian scale space pyramid comprises a plurality of scale space images, calculate a determinant of Hessian response for each image within the Gaussian scale space pyramid, build a determinant of Hessian response pyramid of the same size and structure as the Gaussian scale space pyramid, calculate a trace of Hessian response for each image within the Gaussian scale space pyramid, build a trace of Hessian response pyramid of the same size and structure as the Gaussian scale space pyramid, utilize the determinant of Hessian response pyramid to identify ceiling light candidates in the input image of the warehouse ceiling, utilize the trace of Hessian response pyramid to identify skylight candidates in the input image of the warehouse ceiling, subject ceiling light candidates to ceiling light candidate feature processing to identify valid ceiling lights in the warehouse, subject skylight candidates to skylight candidate feature processing to identify valid skylights in the warehouse, and send commands to the vehicle controller to navigate the industrial vehicle through the warehouse based upon valid ceiling light identification, valid skylight identification, or both.",
    "2. The industrial vehicle of claim 1 wherein the Gaussian scale space pyramid is created by running a cascading series of image smoothing operations applied to the input image of the warehouse ceiling.",
    "3. The industrial vehicle of claim 2 wherein the Gaussian scale space pyramid is approximated by convolution with binomial filter kernels.",
    "4. The industrial vehicle of claim 1 wherein the Gaussian scale space pyramid is created by supplementing a cascading series of image smoothing operations with subsampling operations.",
    "5. The industrial vehicle of claim 4 wherein the supplemental subsampling operations are implemented conditionally as a function of available navigation module computing power.",
    "6. The industrial vehicle of claim 1 wherein the determinant of Hessian response and trace of Hessian response are calculated based on each scale space image being convolved with second-order partial derivative filter kernels.",
    "7. The industrial vehicle of claim 1 wherein the determinant of Hessian response is calculated by subtraction of a mixed second-order partial derivative term.",
    "8. The industrial vehicle of claim 1 wherein the determinant of Hessian response suppresses responses to objects characterized by different rates of image intensity change along longitudinal and transverse axial directions.",
    "9. The industrial vehicle of claim 1 wherein the determinant response is utilized for multiscale non-maximum suppression wherein local maxima are located within a window comprising scale and spatial dimensions.",
    "10. The industrial vehicle of claim 9 wherein an absolute minimum threshold is applied to prevent excessive noisy false positive determinant responses.",
    "11. The industrial vehicle of claim 1 wherein the determinant response is utilized in a filtering keypoints function for removing candidate points that do not likely correspond to ceiling lights.",
    "12. The industrial vehicle of claim 11 wherein the filtering keypoints function utilizes other filters on candidate points to check against empirically set thresholds.",
    "13. The industrial vehicle of claim 12 wherein the other filters comprise scales at which keypoints are detected, a spatial location of a keypoint, a magnitude of the determinant of Hessian response at the keypoint location, a machine-learning classifier, or an average trace of Hessian response values of a surrounding area.",
    "14. The industrial vehicle of claim 1 wherein the determinant response is utilized in a refining keypoints function for refining spatial position and scale of ceiling light candidate keypoints.",
    "15. The industrial vehicle of claim 1 wherein the trace response is utilized in a sum large-scale trace of Hessian response images function for summing a selection of trace of Hessian response images from a trace of Hessian response pyramid into an integrated trace of Hessian response image.",
    "16. The industrial vehicle of claim 15 wherein skylight regions are searched for within the trace of Hessian response pyramid.",
    "17. The industrial vehicle of claim 1 wherein the trace response is further processed by a threshold integrated trace of Hessian response function, wherein the trace of Hessian response is smoothed and a threshold is applied.",
    "18. The industrial vehicle of claim 17 wherein the threshold is fixed.",
    "19. The industrial vehicle of claim 17 wherein the threshold is not fixed.",
    "20. The industrial vehicle of claim 1 wherein the trace of Hessian response is utilized in a connected component filtering function for extracting and filtering connected components based on a binary thresholded integrated trace of Hessian response image.",
    "21. The industrial vehicle of claim 20 wherein the connected component filtering function filters for size and aspect ratio to select substantially rectangular regions.",
    "22. The industrial vehicle of claim 1 wherein the ceiling lights are circularly symmetric round lights and the skylights are substantially elongated.",
    "23. An industrial vehicle comprising a drive mechanism, a steering mechanism, a vehicle controller, a camera, and a navigation module, wherein: the camera is communicatively coupled to the navigation module; the vehicle controller is responsive to commands from the navigation module; the drive mechanism and the steering mechanism are responsive to commands from the vehicle controller; the camera is configured to capture an input image of a warehouse ceiling comprising elongated skylights characterized by different rates of image intensity change along longitudinal and transverse axial directions, and ceiling lights characterized by a circularly symmetric rate of image intensity change; and the navigation module is configured to distinguish between the ceiling lights and the skylights by executing machine readable instructions to create a Gaussian scale space pyramid from the input image of the warehouse ceiling, wherein the Gaussian scale space pyramid comprises a plurality of scale space images, calculate a determinant of Hessian response for each image within the Gaussian scale space pyramid, build a determinant of Hessian response pyramid of the same size and structure as the Gaussian scale space pyramid, calculate a trace of Hessian response for each image within the Gaussian scale space pyramid, build a trace of Hessian response pyramid of the same size and structure as the Gaussian scale space pyramid, utilize the determinant of Hessian response pyramid to identify ceiling light candidates in the input image of the warehouse ceiling, utilize the trace of Hessian response pyramid to identify skylight candidates in the input image of the warehouse ceiling, subject ceiling light candidates to ceiling light candidate feature processing to identify valid ceiling lights in the warehouse, subject skylight candidates to skylight candidate feature processing to identify valid skylights in the warehouse, and localize the industrial vehicle in the warehouse based upon valid ceiling light identification, valid skylight identification, or both."
  ],
  "description_excerpt": "Field\n\nThe present disclosure relates to systems and methods for providing illumination-invariant feature detection and, more specifically, to systems and methods for providing illumination-invariant functions for feature detection.\n\nTechnical Background\n\nIn order to move items about an industrial environment, workers often utilize industrial vehicles, including for example, forklift trucks, hand and motor driven pallet trucks, and/or other materials handling vehicles. The industrial vehicles can be configured as an automated guided vehicle or a manually guided vehicle that navigates through the environment. In order to facilitate automated guidance, or any type of vehicle navigation, the industrial vehicle may be adapted for localization within the environment. That is, the industrial vehicle can be adapted with sensors and processors for localization, i.e., determining the location and, optionally, the pose of the industrial vehicle within the environment. The sensors can be configured to detect objects in the environment and the localization can be dependent upon features extracted from such detected objects. Systems of this nature are described in, for example, US PG Pub. Nos. 2016/0090281 and 2016/0011595.\n\nIn accordance with one embodiment of the present disclosure, an industrial vehicle is provided comprising a drive mechanism, a steering mechanism, a vehicle controller, a camera, and a navigation module.",
  "cpc": [
    "G01C 21/206",
    "G01C 21/34",
    "G05D 1/0246",
    "G05D 2111/10",
    "G05D 2201/0216",
    "G06K 9/00791",
    "G06K 9/6232",
    "G06K 9/78",
    "G06T 2207/10016",
    "G06T 2207/20016",
    "G06T 2207/30241",
    "G06T 2207/30244",
    "G06T 2207/30252",
    "G06T 5/002",
    "G06T 5/20",
    "G06T 5/70",
    "G06T 7/73",
    "G06V 10/462",
    "G06V 10/52",
    "G06V 20/10",
    "G06V 20/56"
  ],
  "ipc": [
    "G06T 7/00",
    "G01C 21/20",
    "G05D 1/02",
    "G06T 5/00",
    "G06T 5/20",
    "G06V 10/52",
    "G06T 7/73"
  ],
  "assignees": [
    "Crown Equipment Corp"
  ],
  "inventors": [
    "Mark Bell"
  ],
  "filing_date": "2016-09-02",
  "publication_date": "2018-01-30",
  "grant_date": "2018-01-30",
  "priority_date": "2015-09-04",
  "application_number": "US-201615255785-A",
  "family_id": "58188480",
  "cited_by_count": 50,
  "citations": [
    "EP0366350A2",
    "US5586620A",
    "US6459955B1",
    "US20100232643A1",
    "US20140235267A1",
    "US20140195076A1",
    "US9087384B2",
    "US20150094900A1",
    "US20160011595A1",
    "US20150379704A1",
    "US9349181B2",
    "US20160090281A1",
    "US20160353099A1"
  ]
}