Patent · US10565548B2 · B2 · US

Planogram assisted inventory system and method

(11) Publication number

US10565548B2

(21) Application number

15/471,813

(22) Filing date

2017-03-28

(30) Priority date

2016-03-29

(43) Publication date

2020-02-18

(45) Date of grant

2020-02-18

(51) IPC

B25J 19/02; G05B 19/048; G06K 9/00; G06K 9/46; G06K 9/62; G06N 20/00; G06Q 10/08; H04N 5/232

(52) CPC

G06Q Information and communication technology [ICT] specially adapted for administrative, commercial, financial, managerial or supervisory purposes; systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not otherwise provided for: 10/087, 10/067
B25J Manipulators; chambers provided with manipulation devices: 19/022
G01S Radio direction-finding; radio navigation; determining distance or velocity by use of radio waves; locating or presence-detecting by use of the reflection or reradiation of radio waves; analogous arrangements using other waves: 17/88
G05B Control or regulating systems in general; functional elements of such systems; monitoring or testing arrangements for such systems or elements: 19/042, 19/048, 2219/39046
G05D Systems for controlling or regulating non-electric variables: 1/0246
G06F Electric digital data processing: 18/24, 18/2413
G06K Graphical data reading; presentation of data; record carriers; handling record carriers: 9/00664, 9/4642, 9/6202, 9/6267, 9/627
G06N Computing arrangements based on specific computational models: 20/00, 20/10, 3/008, 3/045, 3/0464, 3/048, 3/09
G06V Image or video recognition or understanding: 10/245, 10/255, 10/454, 10/751, 20/10, 20/64, 2201/06, 30/1448, 30/1473, 30/18086, 30/19013, 30/414
H04N Pictorial communication, e.g. television: 23/698, 5/23238

(73) Assignee

Bossa Nova Robotics IP Inc

(72) Inventors

Sarjoun Skaff; Jonathan Davis Taylor; Stephen Vincent Williams; Simant Dube

(54) Title

Planogram assisted inventory system and method

(57) Abstract

A manually assisted robot inventory monitoring method provides for detecting and reading shelf labels using an autonomous robot. Bounding boxes around possible products in a panoramic image can be taken with at least one camera associated with the autonomous robot. Products in the bounding boxes are automatically identified, with those that are not being later manually identified.

Full text

View on Google Patents

Claims (17)

A method, comprising: providing a planogram with identified products; creating a panoramic image spanning a shelving unit with possible products being surrounded by bounding boxes, including: accessing a first plurality of images of the shelving unit captured from a vertical arrangement of a plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; accessing a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitching together the second plurality of images into second vertically stitched images; horizontally stitching together the first vertically stitched images and the second vertically stitched images into the panoramic image; and deriving bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; detecting and reading shelving unit shelf labels to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and identifying the possible products.
The method of claim 1, further comprising segmenting the first plurality of images to extract product images and identifiers from the first plurality of images.
The method of claim 1, further comprising template matching possible products in the bounding boxes without product segmentation.
The method of claim 1, further comprising using vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.
The method of claim 1, further comprising using a deep learning method to build classifiers that identify possible products as products and identify the gaps.
The method of claim 1, further comprising instructing an autonomous robot to move in the horizontal dimension beside the shelving unit, with the autonomous robot acting as a movable base capable of both autonomously detecting and reading shelf labels and creating the panoramic image.
The method of claim 1, further comprising instructing an autonomous robot to move in the horizontal dimension beside the shelving unit, with the autonomous robot acting as a movable base to capture a depth map of the shelving unit and of products positioned on the shelving unit using 3D cameras or structure from motion and multiple cameras.
The method of claim 1, further comprising using manual input to identify possible products designated in the bounding boxes.
The method of claim 1, further comprising identifying stock depletion level for a product in a bounding box.
An inventory system comprising: a planogram with identified products; an autonomous robot including a vertical arrangement of a plurality of cameras, the autonomous robot able to capture a panoramic image spanning a shelving unit, with possible products being surrounded by bounding boxes, including: accessing a first plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; accessing a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitching together the second plurality of images into second vertically stitched images; horizontally stitching together the first vertically stitched images and the second vertically stitched images into the panoramic image; and deriving bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; a shelf label detector and reader attached to the autonomous robot and able to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and a product classifier to use information from the planogram and the possible products in the bounding boxes to identify products in the panoramic image.
The inventory system of claim 10, further comprising an image segmentation system able to extract product images and identifiers and associate them with the planogram.
The inventory system of claim 10, further comprising a template matching system to match possible products in the bounding boxes without product segmentation.
The inventory system of claim 10, further comprising a system for vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.
The inventory system of claim 10, further comprising a deep learning system to build classifiers that identify products.
A computer system, the computer system comprising: a processor; system memory coupled to the processor and storing instructions configured to cause the processor to: provide a planogram with identified products; create a panoramic image spanning a shelving unit with possible products being surrounded by bounding boxes, including: access a first plurality of images of the shelving unit captured from a vertical arrangement of a plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; access a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitch together the second plurality of images into second vertically stitched images; horizontally stitch together the first vertically stitched images and the second vertically stitched images into the panoramic image; and derive bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; detect and read shelving unit shelf labels to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and identify the possible products with a product classifier.
The computer system of claim 15, further comprising instructions configured to use manual input to identify possible products designated in the bounding boxes.
The computer system of claim 15, further comprising instructions configured to identify stock depletion level for a product in a bounding box.

Description

The present disclosure relates generally to retail or warehouse product inventory systems that use a planogram. The planogram can be updated using an autonomous robot with an image capture system and onboard processing to provide near real time product tracking.

Retail stores or warehouses can have thousands of distinct products that are often sold, removed, added, or repositioned. Even with frequent restocking schedules, products assumed to be in stock may be out of stock, decreasing both sales and customer satisfaction. Point of sales data can be used to roughly estimate product availability, but does not help with identifying misplaced, stolen, or damaged products, all of which can reduce product availability. Manually monitoring product inventory and tracking product position is possible, but can be expensive and time consuming.

A low cost, accurate, and scalable camera system for product or other inventory monitoring can include a movable base. Multiple cameras supported by the movable base are directable toward shelves or other systems for holding products or inventory. A processing module is connected to the multiple cameras and able to construct from the camera derived images an updateable map of product or inventory position.

In some embodiments, the described camera system for inventory monitoring can be used for detecting shelf labels; optionally comparing shelf labels to a depth map; defining a product bounding box; associating the bounding box to a shelf label to build a training data set; and using the training data set to train a product classifier.

Citations (46)

US20080077511A1
US20090059270A1
US20090063307A1
US8189855B2
US20090060259A1
US20090094140A1
US20100065634A1
US20090192921A1
US9076042B2
US8508527B2
US20120323620A1
US20130182114A1
US20150162048A1
US20140003655A1
US20140003727A1
US20150248592A1
US9097800B1
US20150117788A1
US20150052027A1
US9015072B2
US9373057B1
US20150139536A1
US20150193909A1
US20160309082A1
US20150363758A1
US20170032311A1
US20160027159A1
US20160119540A1
US20160171429A1
US20160180533A1
US20170193434A1
US20190215420A1
US9984451B2
US20170178310A1
US20170178372A1
US20170193324A1
US20170286901A1
US20170286805A1
US20180005176A1
US20180108134A1
US20190019293A1
US20180218494A1
US20180260772A1
US20190034864A1
US20190156275A1
US20190057588A1

Record as JSON

{
  "publication_number": "US10565548B2",
  "country": "US",
  "kind": "B2",
  "title": "Planogram assisted inventory system and method",
  "abstract": "A manually assisted robot inventory monitoring method provides for detecting and reading shelf labels using an autonomous robot. Bounding boxes around possible products in a panoramic image can be taken with at least one camera associated with the autonomous robot. Products in the bounding boxes are automatically identified, with those that are not being later manually identified.",
  "claims": [
    "1. A method, comprising: providing a planogram with identified products; creating a panoramic image spanning a shelving unit with possible products being surrounded by bounding boxes, including: accessing a first plurality of images of the shelving unit captured from a vertical arrangement of a plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; accessing a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitching together the second plurality of images into second vertically stitched images; horizontally stitching together the first vertically stitched images and the second vertically stitched images into the panoramic image; and deriving bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; detecting and reading shelving unit shelf labels to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and identifying the possible products.",
    "2. The method of claim 1, further comprising segmenting the first plurality of images to extract product images and identifiers from the first plurality of images.",
    "3. The method of claim 1, further comprising template matching possible products in the bounding boxes without product segmentation.",
    "4. The method of claim 1, further comprising using vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.",
    "5. The method of claim 1, further comprising using a deep learning method to build classifiers that identify possible products as products and identify the gaps.",
    "6. The method of claim 1, further comprising instructing an autonomous robot to move in the horizontal dimension beside the shelving unit, with the autonomous robot acting as a movable base capable of both autonomously detecting and reading shelf labels and creating the panoramic image.",
    "7. The method of claim 1, further comprising instructing an autonomous robot to move in the horizontal dimension beside the shelving unit, with the autonomous robot acting as a movable base to capture a depth map of the shelving unit and of products positioned on the shelving unit using 3D cameras or structure from motion and multiple cameras.",
    "8. The method of claim 1, further comprising using manual input to identify possible products designated in the bounding boxes.",
    "9. The method of claim 1, further comprising identifying stock depletion level for a product in a bounding box.",
    "10. An inventory system comprising: a planogram with identified products; an autonomous robot including a vertical arrangement of a plurality of cameras, the autonomous robot able to capture a panoramic image spanning a shelving unit, with possible products being surrounded by bounding boxes, including: accessing a first plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; accessing a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitching together the second plurality of images into second vertically stitched images; horizontally stitching together the first vertically stitched images and the second vertically stitched images into the panoramic image; and deriving bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; a shelf label detector and reader attached to the autonomous robot and able to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and a product classifier to use information from the planogram and the possible products in the bounding boxes to identify products in the panoramic image.",
    "11. The inventory system of claim 10, further comprising an image segmentation system able to extract product images and identifiers and associate them with the planogram.",
    "12. The inventory system of claim 10, further comprising a template matching system to match possible products in the bounding boxes without product segmentation.",
    "13. The inventory system of claim 10, further comprising a system for vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.",
    "14. The inventory system of claim 10, further comprising a deep learning system to build classifiers that identify products.",
    "15. A computer system, the computer system comprising: a processor; system memory coupled to the processor and storing instructions configured to cause the processor to: provide a planogram with identified products; create a panoramic image spanning a shelving unit with possible products being surrounded by bounding boxes, including: access a first plurality of images of the shelving unit captured from a vertical arrangement of a plurality of cameras in a first position; vertically stitching together the first plurality of images into first vertically stitched images; access a second plurality of images of the shelving unit captured from the vertical arrangement of the plurality of cameras in a second position, the second position differing in a horizontal dimension relative to the first position; vertically stitch together the second plurality of images into second vertically stitched images; horizontally stitch together the first vertically stitched images and the second vertically stitched images into the panoramic image; and derive bounding boxes around the possible products within the panoramic image and deriving bounding boxes around gaps between the possible products within the panoramic image; detect and read shelving unit shelf labels to localize and associate possible products with both the planogram and the possible products and gaps in the bounding boxes; and identify the possible products with a product classifier.",
    "16. The computer system of claim 15, further comprising instructions configured to use manual input to identify possible products designated in the bounding boxes.",
    "17. The computer system of claim 15, further comprising instructions configured to identify stock depletion level for a product in a bounding box."
  ],
  "description_excerpt": "The present disclosure relates generally to retail or warehouse product inventory systems that use a planogram. The planogram can be updated using an autonomous robot with an image capture system and onboard processing to provide near real time product tracking.\n\nRetail stores or warehouses can have thousands of distinct products that are often sold, removed, added, or repositioned. Even with frequent restocking schedules, products assumed to be in stock may be out of stock, decreasing both sales and customer satisfaction. Point of sales data can be used to roughly estimate product availability, but does not help with identifying misplaced, stolen, or damaged products, all of which can reduce product availability. Manually monitoring product inventory and tracking product position is possible, but can be expensive and time consuming.\n\nA low cost, accurate, and scalable camera system for product or other inventory monitoring can include a movable base. Multiple cameras supported by the movable base are directable toward shelves or other systems for holding products or inventory. A processing module is connected to the multiple cameras and able to construct from the camera derived images an updateable map of product or inventory position.\n\nIn some embodiments, the described camera system for inventory monitoring can be used for detecting shelf labels; optionally comparing shelf labels to a depth map; defining a product bounding box; associating the bounding box to a shelf label to build a training data set; and using the training data set to train a product classifier.",
  "cpc": [
    "G06Q 10/087",
    "B25J 19/022",
    "G01S 17/88",
    "G05B 19/042",
    "G05B 19/048",
    "G05B 2219/39046",
    "G05D 1/0246",
    "G06F 18/24",
    "G06F 18/2413",
    "G06K 9/00664",
    "G06K 9/4642",
    "G06K 9/6202",
    "G06K 9/6267",
    "G06K 9/627",
    "G06N 20/00",
    "G06N 20/10",
    "G06N 3/008",
    "G06N 3/045",
    "G06N 3/0464",
    "G06N 3/048",
    "G06N 3/09",
    "G06Q 10/067",
    "G06V 10/245",
    "G06V 10/255",
    "G06V 10/454",
    "G06V 10/751",
    "G06V 20/10",
    "G06V 20/64",
    "G06V 2201/06",
    "G06V 30/1448",
    "G06V 30/1473",
    "G06V 30/18086",
    "G06V 30/19013",
    "G06V 30/414",
    "H04N 23/698",
    "H04N 5/23238"
  ],
  "ipc": [
    "B25J 19/02",
    "G05B 19/048",
    "G06K 9/00",
    "G06K 9/46",
    "G06K 9/62",
    "G06N 20/00",
    "G06Q 10/08",
    "H04N 5/232"
  ],
  "assignees": [
    "Bossa Nova Robotics IP Inc"
  ],
  "inventors": [
    "Sarjoun Skaff",
    "Jonathan Davis Taylor",
    "Stephen Vincent Williams",
    "Simant Dube"
  ],
  "filing_date": "2017-03-28",
  "publication_date": "2020-02-18",
  "grant_date": "2020-02-18",
  "priority_date": "2016-03-29",
  "application_number": "US-201715471813-A",
  "family_id": "59958856",
  "cited_by_count": 25,
  "citations": [
    "US20080077511A1",
    "US20090059270A1",
    "US20090063307A1",
    "US8189855B2",
    "US20090060259A1",
    "US20090094140A1",
    "US20100065634A1",
    "US20090192921A1",
    "US9076042B2",
    "US8508527B2",
    "US20120323620A1",
    "US20130182114A1",
    "US20150162048A1",
    "US20140003655A1",
    "US20140003727A1",
    "US20150248592A1",
    "US9097800B1",
    "US20150117788A1",
    "US20150052027A1",
    "US9015072B2",
    "US9373057B1",
    "US20150139536A1",
    "US20150193909A1",
    "US20160309082A1",
    "US20150363758A1",
    "US20170032311A1",
    "US20160027159A1",
    "US20160119540A1",
    "US20160171429A1",
    "US20160180533A1",
    "US20170193434A1",
    "US20190215420A1",
    "US9984451B2",
    "US20170178310A1",
    "US20170178372A1",
    "US20170193324A1",
    "US20170286901A1",
    "US20170286805A1",
    "US20180005176A1",
    "US20180108134A1",
    "US20190019293A1",
    "US20180218494A1",
    "US20180260772A1",
    "US20190034864A1",
    "US20190156275A1",
    "US20190057588A1"
  ]
}

Record 2,320 of 8,000 in Patents full text (MLC-0201). Request the full dataset.