Patent · US11151632B2 · B2 · US
Systems and methods for visual search and autonomous delivery
- (11) Publication number
- US11151632B2
- (21) Application number
- 16/158,940
- (22) Filing date
- 2018-10-12
- (30) Priority date
- 2017-07-28
- (43) Publication date
- 2021-10-19
- (45) Date of grant
- 2021-10-19
- (51) IPC
- A23L 2/52; A23L 5/00; A23L 7/109; A47J 37/06; A47J 47/00; B60H 1/00; B60P 1/36; B60P 3/00; B60P 3/025; B60R 19/18; B60R 19/48; B60R 21/34; B60R 21/36; B60R 25/25; B65G 67/24; G01C 21/20; G01C 21/34; G05D 1/00; G05D 1/02; G05D 1/12; G06F 16/955; G06F 3/0484; G06K 19/06; G06K 19/07; G06K 7/10; G06K 7/14; G06K 9/00; G06N 20/00; G06Q 10/06; G06Q 10/08; G06Q 20/00; G06Q 20/12; G06Q 20/18; G06Q 30/02; G06Q 30/06; G06Q 50/12; G07C 5/02; G07C 5/08; G07F 17/00; G07F 17/12; G08G 1/00; G08G 1/04; H04L 29/08; H04N 5/76; H04W 4/024; H04W 4/40; H05B 6/68
- (52) CPC
- G07C Time or attendance registers; registering or indicating the working of machines; generating random numbers; voting or lottery apparatus; arrangements, systems or apparatus for checking not provided for elsewhere: 9/00896, 2009/0092, 2209/63, 5/02, 5/08, 9/00563, 9/00571, 9/28
- A23L Foods, foodstuffs or non-alcoholic beverages, not otherwise provided for; preparation or treatment thereof: 2/52, 5/00, 7/109
- A23V Indexing scheme relating to foods, foodstuffs or non-alcoholic beverages and lactic or propionic acid bacteria used in foodstuffs or food preparation: 2002/00
- A47J Kitchen equipment; coffee mills; spice mills; apparatus for making beverages: 37/0658, 47/00
- B60H Arrangements of heating, cooling, ventilating or other air-treating devices specially adapted for passenger or goods spaces of vehicles: 1/00364, 1/00735
- B60P Vehicles adapted for load transportation or to transport, to carry, or to comprise special loads or objects: 1/36, 3/007, 3/0257
- B60R Vehicles, vehicle fittings, or vehicle parts, not otherwise provided for: 19/18, 19/483, 2021/346, 21/34, 21/36, 25/25, 25/252
- B60W Conjoint control of vehicle sub-units of different type or different function; control systems specially adapted for hybrid vehicles; road vehicle drive control systems for purposes not related to the control of a particular sub-unit: 2420/403, 2556/45, 30/08, 60/001, 60/0017, 60/00256, 60/0027, 60/00276
- B64U Unmanned aerial vehicles [uav]; equipment therefor: 2101/64, 2201/10
- B65G Transport or storage devices, e.g. conveyors for loading or tipping, shop conveyor systems or pneumatic tube conveyors: 2209/06, 67/24
- G01C Measuring distances, levels or bearings; surveying; navigation; gyroscopic instruments; photogrammetry or videogrammetry: 21/20, 21/343, 21/3438, 21/3453
- G05D Systems for controlling or regulating non-electric variables: 1/0027, 1/0033, 1/0038, 1/0061, 1/0088, 1/0094, 1/0212, 1/0214, 1/0223, 1/0231, 1/0276, 1/0291, 1/0295, 1/0297, 1/12, 1/2235, 1/2247, 1/6445, 1/651, 1/686, 2101/10, 2201/0207, 2201/0213
- G06F Electric digital data processing: 16/955, 3/017, 3/0484
- G06K Graphical data reading; presentation of data; record carriers; handling record carriers: 19/06028, 19/0723, 7/10297, 7/10722, 7/1413, 9/00201, 9/00791
- G06N Computing arrangements based on specific computational models: 20/00
- G06Q Information and communication technology [ICT] specially adapted for administrative, commercial, financial, managerial or supervisory purposes; systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not otherwise provided for: 10/00, 10/0631, 10/06315, 10/0635, 10/08, 10/083, 10/0832, 10/0833, 10/0834, 10/0835, 10/08355, 10/0837, 20/00, 20/127, 20/18, 30/0265, 30/0266, 30/0631, 30/0645, 50/12, 50/28, 50/30, 50/40
- G06V Image or video recognition or understanding: 20/56, 20/64
- G07F Coin-freed or like apparatus: 17/0057, 17/12
- G08G Traffic control systems: 1/04, 1/202, 1/22
- G09F Displaying; advertising; signs; labels or name-plates; seals: 21/048
- H04L Transmission of digital information, e.g. telegraphic communication: 67/12
- H04N Pictorial communication, e.g. television: 5/76
- H04W Wireless communication networks: 4/024, 4/40
- H05B Electric heating; electric light sources not otherwise provided for; circuit arrangements for electric light sources, in general: 6/688
- (73) Assignee
- Nuro Inc
- (72) Inventors
- David Ferguson; Jiajun Zhu; Nan RANSOHOFF
- (54) Title
- Systems and methods for visual search and autonomous delivery
- (57) Abstract
A delivery control system includes a communication device configured to receive an image of a product from a computing device of a customer, at least one processor, and a memory storing instructions which, when executed by the at least one processor, cause the delivery control system to perform an image recognition process on the image to identify at least one product based on the image, transmit the identified at least one product to the computing device via the communication device, receive from the computing device a product selected by the customer from among the identified at least one product, and instruct an robot vehicle to deliver the selected product to the customer.
- Full text
- View on Google Patents
Claims (20)
- A delivery control system comprising: a communication device configured to: obtain, from a plurality of merchants, a plurality of images of a plurality of products, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtain a first image of a first product from a computing device of a customer; and obtain an indication of one or more customer-preferred merchants from the computing device of the customer; at least one processor; and a memory storing instructions which, when executed by the at least one processor, cause the delivery control system to: train an image recognition process based on the plurality of images using machine learning; perform the image recognition process on the first image to identify at least one alternate product when the first product is not available based on at least one score of the at least one alternate product assigned by the image recognition process, wherein the image recognition process is constrained by the one or more customer-preferred merchants; provide an indication of the at least one alternate product to the computing device of the customer via the communication device; obtain, from the computing device of the customer, an indication of a product selected by the customer from among the at least one alternate product; and instruct a robot vehicle to deliver, to the customer, the product selected by the customer.
- The delivery control system according to claim 1, wherein the instructions, when executed by the at least one processor, further cause the delivery control system to: transmit an indication of the plurality of merchants to the computing device of the customer, wherein the communication device is configured to obtain the indication of the one or more customer-preferred merchants in response to transmitting the indication of the plurality of merchants to the computing device of the customer.
- The delivery control system according to claim 1, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.
- The delivery control system according to claim 3, wherein the instructions, when executed by the at least one processor, further cause the delivery control system to: transmit an indication of the at least one merchant to the computing device of the customer via the communication device.
- The delivery control system according to claim 4, wherein the communication device is further configured to: obtain, from the computing device of the customer, an indication of a merchant selected from the at least one merchant.
- The delivery control system of claim 1, wherein the plurality of images captures each of the plurality of products in a plurality of views.
- The delivery control system of claim 1, wherein the plurality of images captures each of the plurality of products in a plurality of surroundings.
- The delivery control system according to claim 1, wherein the instructions cause the delivery control system to: train the image recognition process based on the plurality of lightings using machine learning.
- A computing device for ordering a product, the computing device comprising: a communication device; a display screen; at least one processor; and a memory storing instructions which, when executed by the at least one processor, cause the computing device to: provide, via the communication device, a first image of a first product and an indication of one or more customer-preferred merchants to a delivery control system configured to obtain, from a plurality of merchants, a plurality of images of a plurality of products and train an image recognition process based on the plurality of images using machine learning, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtain an indication of at least one alternate product identified based on at least one score of the at least one alternate product assigned by the image recognition process when the first product is not available, wherein the image recognition process is constrained by the one or more customer-preferred merchants; display the at least one alternate product on the display screen; obtain a user selection from among the at least one alternate product, the user selection indicating a selected product; and provide, via the communication device, an order for the selected product to the delivery control system.
- The computing device according to claim 9, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.
- The computing device according to claim 10, wherein the instructions, when executed by the at least one processor, further cause the computing device to: obtain an indication of the plurality of merchants from the delivery control system.
- The computing device according to claim 11, wherein the communication device is further configured to: provide an indication of a merchant selected by a user from the plurality of merchants.
- The computing device according to claim 12, wherein the selected product is handled by the merchant selected by the user.
- The computing device according to claim 9, wherein the delivery control system is configured to train the image recognition process based on the plurality of lightings using machine learning.
- One or more non-transitory computer readable storage media including instructions which, when executed by a computer, cause the computer to perform a method comprising: obtaining, from a plurality of merchants, a plurality of images of a plurality of products, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtaining a first image of a first product from a computing device of a customer; obtaining an indication of one or more customer-preferred merchants from the computing device of the customer; training an image recognition process based on the plurality of images using machine learning; performing the image recognition process on the first image to identify at least one alternate product when the first product is not available based on at least one score of the at least one alternate product assigned by the image recognition process, wherein the image recognition process is constrained by the one or more customer-preferred merchants; providing an indication of the at least one alternate product to the computing device of the customer; obtaining, from the computing device of the customer, an indication of a product selected by the customer from among the at least one alternate product; and instructing a robot vehicle to deliver, to the customer, the product selected by the customer.
- The one or more non-transitory computer readable storage media of claim 15, wherein the method further comprises: transmitting an indication of the plurality of merchants to the computing device of the customer, wherein obtaining the indication of the one or more customer-preferred merchants includes obtaining the indication of the one or more customer-preferred merchants in response to transmitting the indication of the plurality of merchants to the computing device of the customer.
- The one or more non-transitory computer readable storage media of claim 15, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.
- The one or more non-transitory computer readable storage media of claim 17, wherein the method further comprises: transmitting an indication of the at least one merchant to the computing device of the customer.
- The one or more non-transitory computer readable storage media of claim 18, wherein the method further comprises: obtaining, from the computing device of the customer, an indication of a merchant selected from the at least one merchant.
- The one or more non-transitory computer readable storage media of claim 15, wherein the plurality of images includes images of each of the plurality of products in a plurality of views.
Description
The present application relates to systems and methods for autonomous delivery, and in particular, to systems and methods for autonomous delivery based on a visual search.
Businesses have been focusing on easing customer burden in placing orders. For example, customers have traditionally had to travel to brick and mortar stores to buy items. But now, consumers are able to go to online stores for shopping needs and thereby reduce, by clicking hyperlinks in websites, traveling time and effort in looking around a physical store to find items. Ease of ordering and paying for products lead more consumers to online shopping. Due to this shopping trend, most products are sold in online shopping websites.
Although online shopping has proliferated, customers may still experience difficulty in trying to locate and purchase a particular product they see outside of online shopping websites. For example, a customer may see an item of interest in a real world setting or in a web page that is not a shopping website, and to find the item, she would have to go through a series of steps to search for the product on the Internet, such as searching using brand names, product names, or other specifics about the product. Further, it may also be difficult for the consumer to locate a merchant which carries the product. Accordingly, there is continuing interest in reducing the burden of consumers in locating and purchase products.
This disclosure relates to improvement in online shopping with image recognition processing on images of products transmitted from customers' devices.
Citations (13)
- US20100082447A1
- US20100084426A1
- US20110246328A1
- US20130282844A1
- US20140081445A1
- US20140104379A1
- US20140337176A1
- US20150006005A1
- US20150242922A1
- US20180336605A1
- US20170174343A1
- US20170286892A1
- US20180197223A1
Record as JSON
{
"publication_number": "US11151632B2",
"country": "US",
"kind": "B2",
"title": "Systems and methods for visual search and autonomous delivery",
"abstract": "A delivery control system includes a communication device configured to receive an image of a product from a computing device of a customer, at least one processor, and a memory storing instructions which, when executed by the at least one processor, cause the delivery control system to perform an image recognition process on the image to identify at least one product based on the image, transmit the identified at least one product to the computing device via the communication device, receive from the computing device a product selected by the customer from among the identified at least one product, and instruct an robot vehicle to deliver the selected product to the customer.",
"claims": [
"1. A delivery control system comprising: a communication device configured to: obtain, from a plurality of merchants, a plurality of images of a plurality of products, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtain a first image of a first product from a computing device of a customer; and obtain an indication of one or more customer-preferred merchants from the computing device of the customer; at least one processor; and a memory storing instructions which, when executed by the at least one processor, cause the delivery control system to: train an image recognition process based on the plurality of images using machine learning; perform the image recognition process on the first image to identify at least one alternate product when the first product is not available based on at least one score of the at least one alternate product assigned by the image recognition process, wherein the image recognition process is constrained by the one or more customer-preferred merchants; provide an indication of the at least one alternate product to the computing device of the customer via the communication device; obtain, from the computing device of the customer, an indication of a product selected by the customer from among the at least one alternate product; and instruct a robot vehicle to deliver, to the customer, the product selected by the customer.",
"2. The delivery control system according to claim 1, wherein the instructions, when executed by the at least one processor, further cause the delivery control system to: transmit an indication of the plurality of merchants to the computing device of the customer, wherein the communication device is configured to obtain the indication of the one or more customer-preferred merchants in response to transmitting the indication of the plurality of merchants to the computing device of the customer.",
"3. The delivery control system according to claim 1, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.",
"4. The delivery control system according to claim 3, wherein the instructions, when executed by the at least one processor, further cause the delivery control system to: transmit an indication of the at least one merchant to the computing device of the customer via the communication device.",
"5. The delivery control system according to claim 4, wherein the communication device is further configured to: obtain, from the computing device of the customer, an indication of a merchant selected from the at least one merchant.",
"6. The delivery control system of claim 1, wherein the plurality of images captures each of the plurality of products in a plurality of views.",
"7. The delivery control system of claim 1, wherein the plurality of images captures each of the plurality of products in a plurality of surroundings.",
"8. The delivery control system according to claim 1, wherein the instructions cause the delivery control system to: train the image recognition process based on the plurality of lightings using machine learning.",
"9. A computing device for ordering a product, the computing device comprising: a communication device; a display screen; at least one processor; and a memory storing instructions which, when executed by the at least one processor, cause the computing device to: provide, via the communication device, a first image of a first product and an indication of one or more customer-preferred merchants to a delivery control system configured to obtain, from a plurality of merchants, a plurality of images of a plurality of products and train an image recognition process based on the plurality of images using machine learning, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtain an indication of at least one alternate product identified based on at least one score of the at least one alternate product assigned by the image recognition process when the first product is not available, wherein the image recognition process is constrained by the one or more customer-preferred merchants; display the at least one alternate product on the display screen; obtain a user selection from among the at least one alternate product, the user selection indicating a selected product; and provide, via the communication device, an order for the selected product to the delivery control system.",
"10. The computing device according to claim 9, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.",
"11. The computing device according to claim 10, wherein the instructions, when executed by the at least one processor, further cause the computing device to: obtain an indication of the plurality of merchants from the delivery control system.",
"12. The computing device according to claim 11, wherein the communication device is further configured to: provide an indication of a merchant selected by a user from the plurality of merchants.",
"13. The computing device according to claim 12, wherein the selected product is handled by the merchant selected by the user.",
"14. The computing device according to claim 9, wherein the delivery control system is configured to train the image recognition process based on the plurality of lightings using machine learning.",
"15. One or more non-transitory computer readable storage media including instructions which, when executed by a computer, cause the computer to perform a method comprising: obtaining, from a plurality of merchants, a plurality of images of a plurality of products, wherein the plurality of images captures each of the plurality of products in a plurality of lightings; obtaining a first image of a first product from a computing device of a customer; obtaining an indication of one or more customer-preferred merchants from the computing device of the customer; training an image recognition process based on the plurality of images using machine learning; performing the image recognition process on the first image to identify at least one alternate product when the first product is not available based on at least one score of the at least one alternate product assigned by the image recognition process, wherein the image recognition process is constrained by the one or more customer-preferred merchants; providing an indication of the at least one alternate product to the computing device of the customer; obtaining, from the computing device of the customer, an indication of a product selected by the customer from among the at least one alternate product; and instructing a robot vehicle to deliver, to the customer, the product selected by the customer.",
"16. The one or more non-transitory computer readable storage media of claim 15, wherein the method further comprises: transmitting an indication of the plurality of merchants to the computing device of the customer, wherein obtaining the indication of the one or more customer-preferred merchants includes obtaining the indication of the one or more customer-preferred merchants in response to transmitting the indication of the plurality of merchants to the computing device of the customer.",
"17. The one or more non-transitory computer readable storage media of claim 15, wherein the at least one alternate product is handled by at least one merchant of the one or more customer-preferred merchants.",
"18. The one or more non-transitory computer readable storage media of claim 17, wherein the method further comprises: transmitting an indication of the at least one merchant to the computing device of the customer.",
"19. The one or more non-transitory computer readable storage media of claim 18, wherein the method further comprises: obtaining, from the computing device of the customer, an indication of a merchant selected from the at least one merchant.",
"20. The one or more non-transitory computer readable storage media of claim 15, wherein the plurality of images includes images of each of the plurality of products in a plurality of views."
],
"description_excerpt": "The present application relates to systems and methods for autonomous delivery, and in particular, to systems and methods for autonomous delivery based on a visual search.\n\nBusinesses have been focusing on easing customer burden in placing orders. For example, customers have traditionally had to travel to brick and mortar stores to buy items. But now, consumers are able to go to online stores for shopping needs and thereby reduce, by clicking hyperlinks in websites, traveling time and effort in looking around a physical store to find items. Ease of ordering and paying for products lead more consumers to online shopping. Due to this shopping trend, most products are sold in online shopping websites.\n\nAlthough online shopping has proliferated, customers may still experience difficulty in trying to locate and purchase a particular product they see outside of online shopping websites. For example, a customer may see an item of interest in a real world setting or in a web page that is not a shopping website, and to find the item, she would have to go through a series of steps to search for the product on the Internet, such as searching using brand names, product names, or other specifics about the product. Further, it may also be difficult for the consumer to locate a merchant which carries the product. Accordingly, there is continuing interest in reducing the burden of consumers in locating and purchase products.\n\nThis disclosure relates to improvement in online shopping with image recognition processing on images of products transmitted from customers' devices.",
"cpc": [
"G07C 9/00896",
"A23L 2/52",
"A23L 5/00",
"A23L 7/109",
"A23V 2002/00",
"A47J 37/0658",
"A47J 47/00",
"B60H 1/00364",
"B60H 1/00735",
"B60P 1/36",
"B60P 3/007",
"B60P 3/0257",
"B60R 19/18",
"B60R 19/483",
"B60R 2021/346",
"B60R 21/34",
"B60R 21/36",
"B60R 25/25",
"B60R 25/252",
"B60W 2420/403",
"B60W 2556/45",
"B60W 30/08",
"B60W 60/001",
"B60W 60/0017",
"B60W 60/00256",
"B60W 60/0027",
"B60W 60/00276",
"B64U 2101/64",
"B64U 2201/10",
"B65G 2209/06",
"B65G 67/24",
"G01C 21/20",
"G01C 21/343",
"G01C 21/3438",
"G01C 21/3453",
"G05D 1/0027",
"G05D 1/0033",
"G05D 1/0038",
"G05D 1/0061",
"G05D 1/0088",
"G05D 1/0094",
"G05D 1/0212",
"G05D 1/0214",
"G05D 1/0223",
"G05D 1/0231",
"G05D 1/0276",
"G05D 1/0291",
"G05D 1/0295",
"G05D 1/0297",
"G05D 1/12",
"G05D 1/2235",
"G05D 1/2247",
"G05D 1/6445",
"G05D 1/651",
"G05D 1/686",
"G05D 2101/10",
"G05D 2201/0207",
"G05D 2201/0213",
"G06F 16/955",
"G06F 3/017",
"G06F 3/0484",
"G06K 19/06028",
"G06K 19/0723",
"G06K 7/10297",
"G06K 7/10722",
"G06K 7/1413",
"G06K 9/00201",
"G06K 9/00791",
"G06N 20/00",
"G06Q 10/00",
"G06Q 10/0631",
"G06Q 10/06315",
"G06Q 10/0635",
"G06Q 10/08",
"G06Q 10/083",
"G06Q 10/0832",
"G06Q 10/0833",
"G06Q 10/0834",
"G06Q 10/0835",
"G06Q 10/08355",
"G06Q 10/0837",
"G06Q 20/00",
"G06Q 20/127",
"G06Q 20/18",
"G06Q 30/0265",
"G06Q 30/0266",
"G06Q 30/0631",
"G06Q 30/0645",
"G06Q 50/12",
"G06Q 50/28",
"G06Q 50/30",
"G06Q 50/40",
"G06V 20/56",
"G06V 20/64",
"G07C 2009/0092",
"G07C 2209/63",
"G07C 5/02",
"G07C 5/08",
"G07C 9/00563",
"G07C 9/00571",
"G07C 9/28",
"G07F 17/0057",
"G07F 17/12",
"G08G 1/04",
"G08G 1/202",
"G08G 1/22",
"G09F 21/048",
"H04L 67/12",
"H04N 5/76",
"H04W 4/024",
"H04W 4/40",
"H05B 6/688"
],
"ipc": [
"A23L 2/52",
"A23L 5/00",
"A23L 7/109",
"A47J 37/06",
"A47J 47/00",
"B60H 1/00",
"B60P 1/36",
"B60P 3/00",
"B60P 3/025",
"B60R 19/18",
"B60R 19/48",
"B60R 21/34",
"B60R 21/36",
"B60R 25/25",
"B65G 67/24",
"G01C 21/20",
"G01C 21/34",
"G05D 1/00",
"G05D 1/02",
"G05D 1/12",
"G06F 16/955",
"G06F 3/0484",
"G06K 19/06",
"G06K 19/07",
"G06K 7/10",
"G06K 7/14",
"G06K 9/00",
"G06N 20/00",
"G06Q 10/06",
"G06Q 10/08",
"G06Q 20/00",
"G06Q 20/12",
"G06Q 20/18",
"G06Q 30/02",
"G06Q 30/06",
"G06Q 50/12",
"G07C 5/02",
"G07C 5/08",
"G07F 17/00",
"G07F 17/12",
"G08G 1/00",
"G08G 1/04",
"H04L 29/08",
"H04N 5/76",
"H04W 4/024",
"H04W 4/40",
"H05B 6/68"
],
"assignees": [
"Nuro Inc"
],
"inventors": [
"David Ferguson",
"Jiajun Zhu",
"Nan RANSOHOFF"
],
"filing_date": "2018-10-12",
"publication_date": "2021-10-19",
"grant_date": "2021-10-19",
"priority_date": "2017-07-28",
"application_number": "US-201816158940-A",
"family_id": "63207881",
"cited_by_count": 2,
"citations": [
"US20100082447A1",
"US20100084426A1",
"US20110246328A1",
"US20130282844A1",
"US20140081445A1",
"US20140104379A1",
"US20140337176A1",
"US20150006005A1",
"US20150242922A1",
"US20180336605A1",
"US20170174343A1",
"US20170286892A1",
"US20180197223A1"
]
}
Record 1,444 of 8,000 in Patents full text (MLC-0201). Request the full dataset.