Visual-inertial positional awareness for autonomous and non-autonomous device

{
  "publication_number": "US10390003B1",
  "country": "US",
  "kind": "B1",
  "title": "Visual-inertial positional awareness for autonomous and non-autonomous device",
  "abstract": "The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.",
  "claims": [
    "1. A system including: a mobile platform controllable by a host, the mobile platform having disposed thereon: a visual sensor comprising at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration; a global positioning system (GPS) receiver; and a visual inertial control unit, including a processor and a coupled memory storing instructions for guiding the mobile platform, which instructions when executed by the processor perform: receiving images from the at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from the multi-axis inertial measuring unit (IMU) to provide an intermediate propagated pose to an extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a third propagated pose from the global positioning system (GPS) receiver; preparing an updated propagated pose from the intermediate and third propagated poses; and using the updated propagated pose to guide the mobile platform.",
    "2. The system of claim 1, further comprising instructions that when executed by the processor perform: detecting a failure condition of the multi-axis inertial measuring unit (IMU); and responsive to the failure condition of the multi-axis inertial measuring unit (IMU) detected, determining the intermediate propagated pose using the first propagated pose from the images.",
    "3. The system of claim 1, further comprising instructions that when executed by the processor perform: detecting a failure condition of at least one of the at least 4 cameras; whenever a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and whenever all cameras fail resulting in no image information, responsively using data from the multi-axis inertial measuring unit (IMU) to determine the intermediate propagated pose.",
    "4. The system of claim 1, further comprising instructions that when executed by the processor perform: detecting a failure condition of the global positioning system (GPS) receiver; and responsive to the failure condition, providing the intermediate propagated pose as the updated propagated pose.",
    "5. The system of claim 1, wherein the global positioning system (GPS) receiver provides position, velocity and covariance information in North-east down (NED) format and further comprising instructions that when executed by the processor perform: converting the intermediate propagated pose into North-east down (NED) format; and wherein the extended Kalman filter (EKF) propagator produces propagated poses in North-east down (NED) format.",
    "6. The system of claim 1, wherein the global positioning system (GPS) receiver provides doppler, pseudo ranges and covariance information in Earth-centered, Earth-fixed (ECEF) format and further comprising instructions that when executed by the processor perform: converting the intermediate propagated pose into Earth-centered, Earth-fixed (ECEF) format; and wherein the extended Kalman filter (EKF) propagator produces propagated poses in Earth-centered, Earth-fixed (ECEF) format.",
    "7. The system of claim 1, wherein the global positioning system (GPS) receiver provides position and covariance information in North-east down (NED) format and further comprising instructions that when executed by the processor perform: converting the intermediate propagated pose into North-east down (NED) format; and wherein the extended Kalman filter (EKF) propagator produces propagated poses in North-east down (NED) format.",
    "8. The system of claim 1, wherein the multi-axis inertial measuring unit (IMU) propagates a position every 5 ms, and wherein an error is accumulated over time; and every 100 ms, an updated information is received from the global positioning system (GPS) receiver; and wherein the updated information is used to correct the error.",
    "9. A non-transitory computer readable medium storing instructions for guiding a mobile platform, which instructions, when executed by a processor perform: receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from a multi-axis inertial measuring unit (IMU) to provide an intermediate propagated pose to an extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a third propagated pose from a global positioning system (GPS) receiver; preparing an updated propagated pose from the intermediate and third propagated poses; and using the updated propagated pose to guide the mobile platform.",
    "10. A system including: a mobile platform controllable by a host, the mobile platform having disposed thereon: a visual sensor comprising at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration; an odometry sensor coupled with at least one wheel of the mobile platform; and a visual inertial control unit, including a processor and a coupled memory storing instructions for guiding the mobile platform, which instructions when executed by the processor perform: receiving images from the at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from the multi-axis inertial measuring unit (IMU) to provide an intermediate propagated pose to an extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a third propagated pose from the odometry sensor coupled with at least one wheel of the mobile platform; preparing an updated propagated pose from the intermediate and third propagated poses; and using the updated propagated pose to guide the mobile platform.",
    "11. The system of claim 10, further comprising instructions that when executed by the processor perform: detecting a failure condition of the multi-axis inertial measuring unit (IMU); and responsive to the failure condition of the multi-axis inertial measuring unit (IMU) detected, determining the intermediate propagated pose using the first propagated pose from the images.",
    "12. The system of claim 10, further comprising instructions that when executed by the processor perform: detecting a failure condition of at least one of the at least 4 cameras; whenever a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and whenever all cameras fail resulting in no image information, responsively using data from the multi-axis inertial measuring unit (IMU) to determine the intermediate propagated pose.",
    "13. The system of claim 10, further comprising instructions that when executed by the processor perform: detecting a failure condition of the odometry sensor; and responsive to the failure condition, providing the intermediate propagated pose as the updated propagated pose.",
    "14. A non-transitory computer readable medium storing instructions for guiding a mobile platform, which instructions, when executed by a processor perform: receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from a multi-axis inertial measuring unit (IMU) to provide an intermediate propagated pose to an extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a third propagated pose from an odometry sensor coupled with at least one wheel of the mobile platform; preparing an updated propagated pose from the intermediate and third propagated poses; and using the updated propagated pose to guide the mobile platform.",
    "15. A system including: a mobile platform controllable by a host, the mobile platform having disposed thereon: a visual sensor comprising at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration; a global positioning system (GPS) receiver; an odometry sensor coupled to at least one wheel; and a visual inertial control unit, including a processor and a coupled memory storing instructions for guiding the mobile platform, which instructions when executed by the processor perform: receiving images from the at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from the multi-axis inertial measuring unit (IMU) to provide a first intermediate propagated pose to an extended Kalman filter (EKF) propagator; fusing the first intermediate propagated pose from the images and the multi-axis inertial measuring unit (IMU) with a third propagated pose from an odometry sensor coupled to at least one wheel, to provide a second intermediate propagated pose to the extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a fourth propagated pose from the global positioning system (GPS) receiver; preparing an updated propagated pose from the second intermediate and fourth propagated poses; and using the updated propagated pose to guide the mobile platform.",
    "16. The system of claim 15, further comprising instructions that when executed by the processor perform: detecting a failure condition of the multi-axis inertial measuring unit (IMU); and responsive to the failure condition of the multi-axis inertial measuring unit (IMU) detected, determining the first intermediate propagated pose using the first propagated pose from the images.",
    "17. The system of claim 15, further comprising instructions that when executed by the processor perform: detecting a failure condition of at least one of the at least 4 cameras; whenever a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and whenever all cameras fail resulting in no image information, responsively using data from the multi-axis inertial measuring unit (IMU) to determine the first intermediate propagated pose.",
    "18. The system of claim 15, further comprising instructions that when executed by the processor perform: detecting a failure condition of the odometry sensor; and responsive to the failure condition, providing the first intermediate propagated pose as the second intermediate propagated pose.",
    "19. The system of claim 15, further comprising instructions that when executed by the processor perform: detecting a failure condition of the global positioning system (GPS) receiver; and responsive to the failure condition, providing the second intermediate propagated pose as the updated propagated pose.",
    "20. A non-transitory computer readable medium storing instructions for guiding a mobile platform, which instructions, when executed by a processor perform: receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform; determining a 360-degrees depth map from the frames; estimating a propagated pose from the 360-degrees depth map; fusing a first propagated pose from the 360-degrees depth map with a second propagated pose from an multi-axis inertial measuring unit (IMU) to provide a first intermediate propagated pose to an extended Kalman filter (EKF) propagator; fusing the first intermediate propagated pose from the images and the multi-axis inertial measuring unit (IMU) with a third propagated pose from an odometry sensor coupled to at least one wheel, to provide a second intermediate propagated pose to the extended Kalman filter (EKF) propagator; receiving at the extended Kalman filter (EKF) propagator a fourth propagated pose from a global positioning system (GPS) receiver; preparing an updated propagated pose from the second intermediate and fourth propagated poses; and using the updated propagated pose to guide the mobile platform."
  ],
  "description_excerpt": "This application is a Continuation of U.S. application Ser. No. 15/925,289, filed Mar. 19, 2018, entitled “VISUAL-INERTIAL POSITIONAL AWARENESS FOR AUTONOMOUS AND NON-AUTONOMOUS DEVICE” (PERC 1010-1), which is a continuation-in-part of U.S. application Ser. No. 15/250,419, filed Aug. 29, 2016 entitled “VISUAL-INERTIAL POSITIONAL AWARENESS FOR AUTONOMOUS AND NON-AUTONOMOUS DEVICE” (PERC 1000-1). The non-provisional application is hereby incorporated by reference for all purposes.\n\nThe technology disclosed generally relates to detecting location and positioning of a mobile device, and more particularly relates to application of visual processing and inertial sensor data to positioning and guidance technologies.\n\nThe subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology.\n\nAutonomous robots have long been the stuff of science fiction fantasy. One technical challenge in realizing the truly autonomous robot is the need for the robot to be able to identify where they are, where they have been and plan where they are going.",
  "cpc": [
    "H04N 13/25",
    "G01C 21/12",
    "G01C 21/1656",
    "G01C 21/206",
    "G01P 15/18",
    "G05D 1/0248",
    "G06T 2207/10021",
    "G06T 2207/10028",
    "G06T 2207/30244",
    "G06T 5/002",
    "G06T 5/70",
    "G06T 7/246",
    "G06T 7/248",
    "G06T 7/277",
    "G06T 7/73",
    "H04N 13/239",
    "H04N 13/257",
    "H04N 13/296"
  ],
  "ipc": [
    "G01C 21/12",
    "G01P 15/18",
    "G06T 5/00",
    "G06T 7/20",
    "G06T 7/246",
    "H04N 13/25",
    "H04N 13/257"
  ],
  "assignees": [
    "Perceptln Shenzhen Ltd"
  ],
  "inventors": [
    "Shaoshan Liu",
    "Zhe Zhang",
    "Grace Tsai"
  ],
  "filing_date": "2018-03-29",
  "publication_date": "2019-08-20",
  "grant_date": "2019-08-20",
  "priority_date": "2016-08-29",
  "application_number": "US-201815940292-A",
  "family_id": "67620675",
  "cited_by_count": 63,
  "citations": [
    "US9076212B2",
    "US20080249732A1",
    "US20110044543A1",
    "US8774517B1",
    "US20090234499A1",
    "US20100045701A1",
    "US20100121601A1",
    "US8824802B2",
    "US20100220173A1",
    "US20120201469A1",
    "WO2012040644A1",
    "US8655094B2",
    "US9280576B2",
    "US8565958B1",
    "US8825391B1",
    "US20170089948A1",
    "US9378431B2",
    "US8787700B1",
    "US20130282208A1",
    "US20130335554A1",
    "US20140369557A1",
    "US20150012209A1",
    "US20150071524A1",
    "US20150219767A1",
    "US20160327653A1",
    "US20150268058A1",
    "US20150369609A1",
    "US9026941B1",
    "US9058563B1",
    "US20160209217A1",
    "US9836653B2",
    "US20170206418A1",
    "US20170357873A1",
    "US20170010109A1",
    "US20160364835A1",
    "US9607428B2",
    "US20170277197A1",
    "US9965689B2",
    "US20180035606A1",
    "US10043076B1",
    "US10032276B1",
    "US20180158197A1",
    "US20180188032A1",
    "US20180224286A1",
    "CN206932676U",
    "CN206932902U",
    "CN207070638U",
    "CN207070621U",
    "CN207151236U",
    "CN108010271A",
    "CN206932653U",
    "CN207070613U",
    "CN206932609U",
    "CN206932645U",
    "CN206935560U",
    "CN206932680U",
    "CN206932646U",
    "CN207443493U",
    "CN206932647U",
    "CN107137026A",
    "CN107291080A",
    "CN206946068U",
    "CN107153247A",
    "CN107241441A",
    "CN107235013A",
    "CN107451611A",
    "CN207360243U",
    "CN207328169U",
    "CN207328170U",
    "CN107323301A",
    "CN107462892A",
    "CN207070630U",
    "CN207152927U",
    "CN207321871U",
    "CN207070619U",
    "CN207070709U",
    "CN207070612U",
    "CN207070629U",
    "CN207070610U",
    "CN207070652U",
    "CN207354913U",
    "CN207154149U",
    "CN207070703U",
    "CN207322208U",
    "CN207321889U",
    "CN207151465U",
    "CN207070641U",
    "CN207443447U",
    "CN207070710U",
    "CN207070639U",
    "CN207321872U",
    "CN207322217U",
    "CN207160626U",
    "CN207328818U",
    "CN207159970U",
    "CN207159810U",
    "CN207155841U",
    "CN207154238U",
    "CN207159840U",
    "CN207158940U",
    "CN207155840U",
    "CN207159812U",
    "CN207157464U",
    "CN207159724U",
    "CN207159811U",
    "CN207155071U",
    "CN207073092U",
    "CN207074269U",
    "CN207074202U",
    "CN207071933U",
    "CN207336762U",
    "CN207328819U",
    "CN207074560U",
    "CN107329478A",
    "CN207367052U",
    "CN207164772U",
    "CN107273881A",
    "CN107562660A",
    "CN107444179A",
    "CN207155775U",
    "CN207369157U",
    "CN207356422U",
    "CN207155818U",
    "CN207367336U",
    "CN207356420U",
    "CN207155774U",
    "CN207356394U",
    "CN207155819U",
    "CN207356393U",
    "CN207359050U",
    "CN207155773U",
    "CN207356421U",
    "CN207164589U",
    "CN207155776U",
    "CN207155817U",
    "CN107958285A",
    "CN107976999A",
    "CN107958451A",
    "CN207356963U"
  ]
}