System and method for fast object detection in robot picking

{
  "publication_number": "US11312020B2",
  "country": "US",
  "kind": "B2",
  "title": "System and method for fast object detection in robot picking",
  "abstract": "A method and system for monitoring an e-commerce platform. The system includes a computing device and a visual sensor. The computing device includes a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to: extract image keypoints from an image of the object captured by the visual sensor; retrieve a template of the object, where the template includes template keypoints of at least one template side surface of the object; pick two template keypoints from the template side surface and determine two image keypoints respectively matching the two picked template keypoints; build a bounding box of the object based on the two determined image keypoints; and refine the bounding box.",
  "claims": [
    "1. A system for determining a pose of a target object from a plurality of objects, the system comprising a computing device and a visual sensor in communication with the computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to: extract first keypoints from an image, wherein the image is captured by the visual sensor on a region of interest, the region of interest comprises the plurality of objects, and each of the first keypoints is defined with its location and feature descriptor; retrieve a template of the target object, wherein the template is obtained by recording image features of each of a plurality of side surfaces of the target object, the template comprises a plurality of template side surfaces corresponding to the plurality of side surfaces of the target object respectively, each of the template side surfaces is characterized with a plurality of second keypoints, and each of the second keypoints is defined with its location and feature descriptor; match the first keypoints to the second keypoints in each of the template side surfaces; select one of the template side surfaces having a greatest number of matches between the first keypoints and the second keypoints in the one of the template side surfaces; pick two second keypoints from the selected template side surface and determine two first keypoints respectively matching the two picked second keypoints; build a bounding box based on the two determined first keypoints, wherein the bounding box represents an image side surface corresponding to the selected template side surface of the template; determine inlier keypoints, wherein the inlier keypoints are located inside the bounding box, and have matched second keypoints in the selected template side surface; refine the bounding box using the inlier keypoints, wherein the pose of the target object is determined based on the refined bounding box.",
    "2. The system of claim 1, wherein the computer executable code is configured to define a mask to the objects in the image before extracting first keypoints.",
    "3. The system of claim 1, wherein the computer executable code is configured to pick two second keypoints from the selected template side surface iteratively for a pre-determined number of times.",
    "4. The system of claim 3, wherein each of the template side surface is selected for a percentage of the pre-determined number of times, and the percentage is determined by a number of matched keypoints in the selected template side surface divided by a total number of matched keypoints in all the template side surfaces.",
    "5. The system of claim 1, wherein the two picked second keypoints are at least 5 pixels apart, and each of the two picked second keypoints has a match with one of the first keypoints.",
    "6. The system of claim 1, wherein the two second keypoints are selected randomly from the selected template side surface; wherein the computer executable code is configured to perform the step of picking two second keypoints, determining two matched first keypoints, building the bounding box, determining inlier keypoints, and refining the bounding box iteratively; and wherein each iteration results in a refined bounding box, each of the refined bounding boxes is characterized with a confidence value based on a number of inlier keypoints and a difference before and after the refinement, and the bounding box having a highest confidence value is used to determine the pose of the target object.",
    "7. The system of claim 6, wherein when the difference is defined as scale change between the bounding box and the side surface of the template, when a length of each side edge of the bounding box divided by a length of a corresponding side edge of the side surface of the template is out of a range of 0.8-1.25, the bounding box before the refinement is kept as the refined bounding box, and the confidence value of the refined bounding box is reduced.",
    "8. The system of claim 1, wherein the pose of the target object is provided to a robotic device for manipulating the target object.",
    "9. A system for detecting an object, the system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to: provide first keypoints from an image of the object; retrieve a template of the object, wherein the template is obtained by recording image features of each of a plurality of side surfaces of the target object, the template comprises a plurality of template side surfaces corresponding to the plurality of side surfaces of the target object respectively, at least one template side surface of the object has a plurality of second keypoints; select one of the template side surfaces having a greatest number of matched second keypoints and first keypoints; pick two second keypoints from the selected template side surface and determine two first keypoints respectively matching the two picked second keypoints; build a bounding box of the object based on the two determined first keypoints; and refine the bounding box to detect the object.",
    "10. The system of claim 9, wherein each of the first keypoints and the second keypoints is characterized by a location and descriptor.",
    "11. The system of claim 9, wherein the two picked second keypoints are picked randomly and the computer executable code is configured to perform the step of picking iteratively for a pre-determined times.",
    "12. The system of claim 11, wherein the two picked second keypoints has a distance greater than a pre-determined distance, and each of the two picked second keypoints has a match with one of the first keypoints.",
    "13. The system of claim 9, wherein the first keypoints in the bounding box having match with the second keypoints in the template side surface are defined as inlier keypoints, and the bounding box is refined using the inlier keypoints.",
    "14. A method for detecting an object, the method comprising: providing, by a computing device, first keypoints from an image of the object; retrieving, by the computing device, a template of the object, wherein the template is obtained by recording image features of each of a plurality of side surfaces of the target object, the template comprises a plurality of template side surfaces corresponding to the plurality of side surfaces of the target object respectively, at least one template side surface has a plurality of second keypoints; selecting, by the computing device, one of the template side surfaces having a greatest number of matched second keypoints and first keypoints; picking, by the computing device, two second keypoints from the selected template side surface and determining two first keypoints respectively matching the two picked second keypoints; building a bounding box of the object based on the two determined first keypoints; and refining the bounding box to detect the object.",
    "15. The method of claim 14, wherein each of the first keypoints and the second keypoints is characterized by a location and descriptor.",
    "16. The method of claim 14, wherein the two picked second keypoints are picked randomly, as long as the two picked second keypoints has a distance greater than a pre-determined distance and each of the two picked second keypoints has a match with one of the first keypoints.",
    "17. The method of claim 14, wherein the first keypoints in the bounding box having match with the second keypoints in the template side surface are defined as inlier keypoints, and the bounding box is refined using the inlier keypoints."
  ],
  "description_excerpt": "The present disclosure relates generally to operation of robot picking, and more particularly to system and methods for fast object detection during robot picking of items from a robot bin.\n\nThe background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.\n\nRobot bin picking is an important application in warehouse automation. In a robot bin picking system, a computer vision system is needed to first locate the items having stock keeping unit (SKU) in the transport bin, and then send the location and pose of each SKU item to the robot controller, so that the robot arm can be guided to perform picking actions from the bin. However, it is a challenge to localize the SKU items fast and accurately.\n\nTherefore, an unaddressed need exists in the art to address the aforementioned deficiencies and inadequacies.\n\nIn certain aspects, the present disclosure relates to a system for determining a pose of a target object from a plurality of objects. In certain embodiments, the system includes a computing device and a visual sensor in communication with the computing device. The computing device has a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to:",
  "cpc": [
    "G06T 7/74",
    "B25J 9/163",
    "B25J 9/1697",
    "G06T 2207/30164",
    "G06T 7/13",
    "G06T 7/62",
    "G06T 7/73",
    "G06V 10/255",
    "G06V 10/462",
    "G06V 30/248"
  ],
  "ipc": [
    "B25J 9/16",
    "G06T 7/62",
    "G06V 10/20"
  ],
  "assignees": [
    "Beijing Jingdong Shangke Information Technology Co Ltd",
    "JD com American Technologies Corp"
  ],
  "inventors": [
    "Xiang Li",
    "Peng Chang",
    "Hui Cheng",
    "Zhengmao Liu"
  ],
  "filing_date": "2018-11-20",
  "publication_date": "2022-04-26",
  "grant_date": "2022-04-26",
  "priority_date": "2018-11-20",
  "application_number": "US-201816197021-A",
  "family_id": "70726151",
  "cited_by_count": 0,
  "citations": [
    "US6493465B2",
    "US7406213B2",
    "US9418442B2",
    "US9746855B2",
    "US9317771B2",
    "US9842399B2",
    "US10805546B2",
    "US20190197727A1"
  ]
}