Mobile robot device for correcting position by fusing image sensor and plurality of geomagnetic sensors, and control method

{
  "publication_number": "US12001218B2",
  "country": "US",
  "kind": "B2",
  "title": "Mobile robot device for correcting position by fusing image sensor and plurality of geomagnetic sensors, and control method",
  "abstract": "Provided are a mobile robot device and a control method thereof. The mobile robot device comprises: a driving unit; an image sensor; a plurality of geomagnetic sensors; a memory for storing at least one instruction; and a processor for executing at least one instruction, wherein the processor may obtain, while the mobile robot device moves by means of the driving unit, a plurality of image data through the image sensor and obtain sensing data through the plurality of geomagnetic sensors, extract a feature point from the plurality of image data and obtain key nodes on the basis of the feature point, obtain a node sequence on the basis of the sensing data, generate a graph structure that estimates a position of the mobile robot device on the basis of the key nodes and the node sequence, and correct the graph structure based on the mobile failing in position recognition.",
  "claims": [
    "1. A mobile robot device, comprising: a driving unit; an image sensor; a plurality of geomagnetic sensors; a memory storing at least one instruction; and a processor configured to execute the at least one instruction, wherein the processor is configured to: obtain, while the mobile robot device moves through the driving unit, a plurality of image data through the image sensor and obtain sensing data through the plurality of geomagnetic sensors, extract a feature point from the plurality of image data and obtain key nodes based on the feature point, obtain a node sequence based on the sensing data, generate a graph structure estimating a position of the mobile robot device based on the key nodes and the node sequence, recognize a position of the mobile robot device based on the generated graph structure; based on failing in position recognition of the mobile robot device, stop generating the graph structure that estimates the position based on the key nodes and the node sequence; obtain new image data from the image sensor; obtain new sensing data through the plurality of geomagnetic sensors; obtain a new key node corresponding to the new image data and a new node sequence corresponding to the new sensing data; and correct the graph structure based on the new sensing data and the new node sequence.",
    "2. The mobile robot device of claim 1, wherein the processor is configured to extract a plurality of feature points by using an Oriented FAST and Rotated BRIEF (ORB) algorithm in the plurality of image data.",
    "3. The mobile robot device of claim 1, wherein the processor is configured to: form a submap by accumulating the feature points; and obtain the key nodes by performing a Random Sample Consensus (RANSAC) algorithm on the submap and matching with the most proximate node.",
    "4. The mobile robot device of claim 1, wherein the processor is configured to: obtain a sensing data group by grouping the obtained sensing data; perform matching by comparing a magnetic field value of the sensing data group with magnetic field values of the previously stored sensing data group; and obtain a node bundle comprised of a pair of graph structures based on the matched sensing data group.",
    "5. The mobile robot device of claim 1, wherein the processor is configured to: identify whether or not a newly obtained node bundle is present between a previously stored node bundle; based on the newly obtained node bundle being identified as present between the previously stored node bundle, extract only the newly obtained node bundle present between the previously obtained node bundle; and select only the extracted newly obtained node bundle as the node sequence.",
    "6. The mobile robot device of claim 5, wherein the processor is configured to: update the previously obtained node bundle with a smaller distance difference with the extracted newly obtained node bundle; search for a node bundle with the smallest distance difference from among the extracted newly obtained node bundle and the updated previously obtained node bundle; and locate a position of the node sequence by using the node bundle with smallest distance difference.",
    "7. The mobile robot device of claim 6, wherein the processor is configured to update to a previously obtained node bundle with a smaller distance difference by using a Gaussian process.",
    "8. A control method of a mobile robot device, comprising: while the mobile robot device is moving, obtaining a plurality of image data through an image sensor and obtaining sensing data through a plurality of geomagnetic sensors; extracting a feature point from the plurality of image data and obtaining key nodes based on the feature point; obtaining a node sequence based on the sensing data; generating a graph structure which estimates a position of the mobile robot device based on the key nodes and the node sequence; recognizing a position of the mobile robot device based on the generated graph structure; based on failing in position recognition of the mobile robot device, stop generating the graph structure that estimates the position based on the key nodes and the node sequence; obtaining new image data from the image sensor; obtaining new sensing data through the plurality of geomagnetic sensors; obtaining a new key node corresponding to the new image data and a new node sequence corresponding to the new sensing data; and correcting the graph structure based on the new sensing data and the new node sequence.",
    "9. The method of claim 8, wherein the obtaining the key nodes comprises extracting a plurality of feature points by using an Oriented FAST and Rotated BRIEF (ORB) algorithm in the plurality of image data.",
    "10. The method of claim 8, wherein the obtaining the key nodes comprises: forming a submap by accumulating the feature points; and performing a Random Sample Consensus (RANSAC) algorithm on the submap and obtaining the key node by matching with a most proximate node.",
    "11. The method of claim 8, wherein the obtaining the node sequence comprises: obtaining a sensing data group by grouping the obtained sensing data; matching by comparing a magnetic field value of the sensing data group with magnetic field values of a previously stored sensing data group; and obtaining a node bundle comprised of a pair of graph structures based on the matched sensing data group.",
    "12. The method of claim 11, wherein the obtaining the node sequence comprises: identifying whether or not a newly obtained node bundle is present between a previously stored node bundle; based on identifying that the newly obtained node bundle is present between the previously stored node bundle, extracting only the newly obtained node bundle that is present between the previously obtained node bundle; and selecting only the extracted newly obtained node bundle as the node sequence."
  ],
  "description_excerpt": "The disclosure relates to a mobile robot device which corrects position by combining an image sensor and a plurality of geomagnetic sensors and produces a map and a control method thereof. More specifically, the disclosure relates to a device which estimates and corrects a current position of a mobile robot by analyzing and simultaneously matching data obtained through each sensor and a control method thereof.\n\nRobots have been enhancing work efficiency by performing jobs difficult for humans to approach in a variety of industries. In industries related to manufacturing, a robot arm form fixed at a specific position and performing repetitive work has been utilized, but research on and demand for mobile robot devices capable of relatively free movement and performing various jobs are gradually increasing. However, the mobile robot devices, in order to move, need to identify which location its current position is at and at the same time construct a route, that is a map, by recognizing its surrounding environment. A technology that achieves the above two process simultaneously is referred to as a Simultaneous Localization And Mapping (SLAM) technique. The above-described SLAM technique has recently undergone development into various fields, and is receiving special attention as a technique essential for effectively driving robot cleaners, autonomous vehicles, or the like. Further, as a type of the above-described SLAM, a graph-based SLAM which represents a position of a robot and odometry as a node and edge (or, constraint) is widely being used.",
  "cpc": [
    "G05D 1/0246",
    "B25J 11/00",
    "B25J 9/161",
    "B25J 9/162",
    "B25J 9/1664",
    "B25J 9/1692",
    "B25J 9/1697",
    "G05D 1/0261",
    "G05D 1/0272",
    "G05D 1/249",
    "G06F 16/9024",
    "G06F 18/251",
    "G06V 10/803",
    "G06V 20/10"
  ],
  "ipc": [
    "B25J 9/16",
    "G05D 1/00",
    "G06F 16/901",
    "G06V 10/80",
    "G06V 20/10"
  ],
  "assignees": [
    "SAMSUNG ELECTRONICS CO LTD",
    "KOREA ADVANCED INST SCI & TECH"
  ],
  "inventors": [
    "HONG SOONHYUK",
    "MYEONG HYEON",
    "KIM HYONGJIN",
    "SONG SEUNGWON",
    "HYUN JIEUM"
  ],
  "filing_date": "2019-06-21",
  "publication_date": "2024-06-04",
  "grant_date": "2024-06-04",
  "priority_date": "2018-06-22",
  "application_number": "US-201917055415-A",
  "family_id": "69369426",
  "citations": [
    "CN103278170A",
    "CN103278170B",
    "CN103674015A",
    "EP3252714A1",
    "EP3367199B1",
    "JP2012064131A",
    "JP2017045447A",
    "KR100801261B1",
    "KR20140108821A",
    "KR20140110586A",
    "KR20160147179A",
    "KR20160150504A",
    "KR20170004556A",
    "KR20170061373A",
    "US10347001B2",
    "US10852729B2",
    "US10949798B2",
    "US2011125323A1",
    "US2011182476A1",
    "US2013166137A1",
    "US2014244094A1",
    "US2016062361A1",
    "US2016377688A1",
    "US2017003131A1",
    "US2017024877A1",
    "US2018079085A1",
    "US7725253B2",
    "US9164509B2",
    "US9224043B2"
  ]
}