Method and apparatus for optimization of a monocular visual-inertial localization system

{
  "publication_number": "US11521332B1",
  "country": "US",
  "kind": "B1",
  "title": "Method and apparatus for optimization of a monocular visual-inertial localization system",
  "abstract": "The method and device disclosed herein presents a method that includes capturing, by an optical sensor disposed on a device moving in an environment, a plurality of optical data at respective locations within a portion of the environment; capturing, by a wheel encoder disposed on the device, a set of encoder data corresponding to the plurality of optical data at the respective locations; determining a first relative motion based on the plurality of optical data; determining a corresponding second relative motion based on the set of encoder data. In accordance with determining that a difference between the first relative motion and the corresponding second relative motion is larger than a first threshold: increasing a counter indicating a slip event of the wheel encoder. The slip event corresponds to a wheel of the device advancing and the corresponding second relative motion being below a second threshold.",
  "claims": [
    "1. A method, comprising: capturing, by an optical sensor disposed on a device moving in an environment, a plurality of optical data at respective locations within a portion of the environment; capturing, by a wheel encoder disposed on the device, a set of encoder data corresponding to the plurality of optical data at the respective locations; determining a first relative motion based on the plurality of optical data; determining a corresponding second relative motion based on the set of encoder data; in accordance with determining that a difference between the first relative motion and the corresponding second relative motion is larger than a first threshold: increasing a counter indicating a slip event of the wheel encoder, wherein the slip event corresponds to a wheel of the device advancing and the corresponding second relative motion being below a second threshold.",
    "2. The method of claim 1, wherein a state of the device is set to a first state when the counter is above a second threshold, and the method includes: excluding the plurality of optical data from further processing when the state of the device is set to the first state.",
    "3. The method of claim 1, wherein the optical sensor comprises a camera and the optical data comprises image frames captured by the camera.",
    "4. The method of claim 3, further comprising determining whether a jump in a position of the device occurs between two adjacent frames of the image frames captured by the camera.",
    "5. The method of claim 4, further comprising: in accordance with a determination that the jump occurs: capturing an additional image frame by the camera until two adjacent frames of captured image frames do not exhibit the jump in the position of the device, prior to determining the first relative motion.",
    "6. The method of claim 3, further comprising: determining whether two adjacent frames of the image frames captured by the camera qualifies as valid measurements prior to calculating the first relative motion.",
    "7. The method of claim 6, further comprising: obtaining wheel encoder readings between the two adjacent frames of the image frames captured by the camera; determining a motion mode based on the wheel encoder readings; determining a feasible motion range according to the motion mode; in accordance with a determination that the first relative motion between the two adjacent frames is outside the feasible motion range: disqualifying the two adjacent frames of the image frames from being valid measurements; and capturing an additional image frame by the camera until two adjacent frames of captured image frames contain relative motion that is within the feasible motion range.",
    "8. The method of claim 7, wherein the motion mode includes one or more of a forward mode, a clockwise mode, a backward mode, and a counter-clockwise mode.",
    "9. The method of claim 1, wherein the optical sensor comprises an optical tracking sensor and determining the first relative motion comprises integrating measurements captured by the optical tracking sensor.",
    "10. The method of claim 1, further includes capturing, by a camera disposed on the device, a plurality of image frames corresponding to the sequence of optical data and the set of encode data recorded at the respective locations within the portion of the environment.",
    "11. An electronic device, comprising: one or more processing units; memory; and a plurality of programs stored in the memory that, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising: capturing, by an optical sensor disposed on a device moving in an environment, a plurality of optical data at respective locations within a portion of the environment; capturing, by a wheel encoder disposed on the device, a set of encoder data corresponding to the plurality of optical data at the respective locations; determining a first relative motion based on the plurality of optical data; determining a corresponding second relative motion based on the set of encoder data; in accordance with determining that a difference between the first relative motion and the corresponding second relative motion is larger than a first threshold: increasing a counter indicating a slip event of the wheel encoder, wherein the slip event corresponds to a wheel of the device advancing and the corresponding second relative motion being below a second threshold.",
    "12. The electronic device of claim 11, wherein the optical sensor comprises a camera and the optical data comprises image frames captured by the camera.",
    "13. The electronic device of claim 12, wherein the plurality of programs causes the one or more processing units to perform operations further comprising: determining whether a jump in a position of the device occurs between two adjacent frames of the image frames captured by the camera.",
    "14. The electronic device of claim 13, wherein the plurality of programs causes the one or more processing units to perform operations further comprising: obtaining wheel encoder readings between the two adjacent frames of the image frames captured by the camera; determining a motion mode based on the wheel encoder readings; determining a feasible motion range according to the motion mode; in accordance with a determination that the first relative motion between the two adjacent frames is outside the feasible motion range: disqualifying the two adjacent frames of the image frames from being valid measurements; and capturing an additional image frame by the camera until two adjacent frames of captured image frames contain relative motion that is within the feasible motion range.",
    "15. The electronic device of claim 14, wherein the motion mode includes one or more of a forward mode, a clockwise mode, a backward mode, and a counter-clockwise mode.",
    "16. A non-transitory computer readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, wherein the plurality of programs, when executed by the one or more processing units, cause the processing units to perform operations comprising: capturing, by an optical sensor disposed on a device moving in an environment, a plurality of optical data at respective locations within a portion of the environment; capturing, by a wheel encoder disposed on the device, a set of encoder data corresponding to the plurality of optical data at the respective locations; determining a first relative motion based on the plurality of optical data; determining a corresponding second relative motion based on the set of encoder data; in accordance with determining that a difference between the first relative motion and the corresponding second relative motion is larger than a first threshold: increasing a counter indicating a slip event of the wheel encoder, wherein the slip event corresponds to a wheel of the device advancing and the corresponding second relative motion being below a second threshold.",
    "17. The non-transitory computer readable storage medium of claim 16, wherein the optical sensor comprises a camera and the optical data comprises image frames captured by the camera.",
    "18. The non-transitory computer readable storage medium of claim 17, wherein the plurality of programs causes the one or more processing units to perform operations further comprising: determining whether two adjacent frames of the image frames captured by the camera qualifies as valid measurements prior to calculating the first relative motion.",
    "19. The non-transitory computer readable storage medium of claim 18, wherein the plurality of programs causes the one or more processing units to perform operations further comprising: obtaining wheel encoder readings between the two adjacent frames of the image frames captured by the camera; determining a motion mode based on the wheel encoder readings; determining a feasible motion range according to the motion mode; in accordance with a determination that the first relative motion between the two adjacent frames is outside the feasible motion range: disqualifying the two adjacent frames of the image frames from being valid measurements; and capturing an additional image frame by the camera until two adjacent frames of captured image frames contain relative motion that is within the feasible motion range.",
    "20. The non-transitory computer readable storage medium of claim 18, wherein the motion mode includes one or more of a forward mode, a clockwise mode, a backward mode, and a counter-clockwise mode."
  ],
  "description_excerpt": "The present disclosure generally relates to the technology of simultaneous localization and mapping (SLAM) in an environment, and in particular, to systems and methods for characterizing physical environments and localizing a mobile robot with respect to its environment using image data.\n\nLocalization, place recognition, and environment understanding allow a mobile robot to become a fully autonomous or semi-autonomous system in an environment. Simultaneous localization and mapping (SLAM) is a method that builds a map of an environment and simultaneously estimates the pose of a mobile robot (e.g., using the estimated pose of its cameras) in the environment. SLAM algorithms allow the mobile robot to map out unknown environments and localize itself in the environment to carry out tasks such as path planning and obstacle avoidance.\n\nMonocular camera based localization technology extracts information from captured consecutive frames of the surrounding environment, such as features (points and lines) or raw pixel values to solve the relative pose (e.g., orientation and translation) between those frames by solving a 3D geometry problem using, for example, epipolar geometry or perspective-n-point. Since a single RGB camera cannot measure depth of the scenes (e.g., measuring a distance of an object captured in a camera frame directly), so the distance from associated features to the camera centers in two related frames is unknown when solving using epipolar geometry.",
  "cpc": [
    "G06T 7/74",
    "G06T 2207/10016",
    "G06T 2207/30244",
    "G06T 7/20",
    "G06T 7/248"
  ],
  "ipc": [
    "G06T 7/246",
    "G06T 7/73"
  ],
  "assignees": [
    "Midea Group Co Ltd"
  ],
  "inventors": [
    "Yi Chen",
    "Ke Huang",
    "Wei Xi"
  ],
  "filing_date": "2021-06-29",
  "publication_date": "2022-12-06",
  "grant_date": "2022-12-06",
  "priority_date": "2021-06-29",
  "application_number": "US-202117362846-A",
  "family_id": "84325021",
  "cited_by_count": 5,
  "citations": [
    "US20110054686A1",
    "US20170371329A1",
    "US20180143645A1",
    "US20180364731A1",
    "CN107643186A",
    "CN108638053A",
    "CN109414145A",
    "US20200042010A1",
    "CN112740274A",
    "CN110946511A",
    "US20210310962A1",
    "CN111707261A",
    "CN111578937A",
    "CN112649016A"
  ]
}