Patent · US12096733B2 · B2 · US

Robotic fruit picking system

(11) Publication number

US12096733B2

(21) Application number

18/114,681

(22) Filing date

2023-02-27

(30) Priority date

2016-11-08

(43) Publication date

2024-09-24

(45) Date of grant

2024-09-24

(51) IPC

A01D 46/22; A01D 46/24; A01D 46/253; A01D 46/28; A01D 46/30; A01G 9/14; B25J 11/00; B25J 15/00; B25J 9/00; B25J 9/06; B25J 9/16; G05D 1/00; G06F 18/214; G06F 18/24; G06F 18/243; G06Q 30/0283; G06T 7/00; G06T 7/11; G06T 7/50; G06T 7/60; G06T 7/70; G06T 7/90; G06V 20/10; G06V 20/68

(52) CPC

A01G Horticulture; cultivation of vegetables, flowers, rice, fruit, vines, hops or seaweed; forestry; watering: 9/143
A01D Harvesting; mowing: 46/22, 46/243, 46/253, 46/28, 46/30
B25J Manipulators; chambers provided with manipulation devices: 11/00, 15/0019, 15/0033, 5/005, 9/0084, 9/06, 9/1679, 9/1697
G05B Control or regulating systems in general; functional elements of such systems; monitoring or testing arrangements for such systems or elements: 2219/45003
G05D Systems for controlling or regulating non-electric variables: 1/0094, 1/0219, 1/648
G06F Electric digital data processing: 18/2148, 18/24323, 18/24765
G06Q Information and communication technology [ICT] specially adapted for administrative, commercial, financial, managerial or supervisory purposes; systems or methods specially adapted for administrative, commercial, financial, managerial or supervisory purposes, not otherwise provided for: 30/0283
G06T Image data processing or generation, in general: 2207/10048, 2207/20081, 2207/20084, 2207/30128, 7/0004, 7/11, 7/50, 7/60, 7/70, 7/90
G06V Image or video recognition or understanding: 20/10, 20/68
Y02A Technologies for adaptation to climate change: 40/25

(73) Assignee

Dogtooth Technologies Ltd

(72) Inventors

Duncan Robertson; Matthew Cook; Edward Herbert; Frank TULLY

(54) Title

Robotic fruit picking system

(57) Abstract

A robotic fruit picking system includes an autonomous robot that includes a positioning subsystem that enables autonomous positioning of the robot using a computer vision guidance system. The robot also includes at least one picking arm and at least one picking head, or other type of end effector, mounted on each picking arm to either cut a stem or branch for a specific fruit or bunch of fruits or pluck that fruit or bunch. A computer vision subsystem analyses images of the fruit to be picked or stored and a control subsystem is programmed with or learns picking strategies using machine learning techniques. A quality control (QC) subsystem monitors the quality of fruit and grades that fruit according to size and/or quality. The robot has a storage subsystem for storing fruit in containers for storage or transportation, or in punnets for retail.

Full text

View on Google Patents

Claims (30)

A robotic fruit picking system in which the robot automatically navigates along rows of fruit-producing plants grown on table-top systems, and in which the fruit picking system includes a computer vision system that includes a camera and that is configured to detect one or more features of the table top systems in order to determine the position and/or orientation of the camera with respect to a coordinate system defined by the features of the table-top systems, such that the fruit picking system is configured to automatically determine its position and/or orientation with respect to the rows of fruit producing plants.
The fruit picking system of claim 1, in which the computer vision system uses images obtained by a forwards or backwards facing camera for determining the heading and lateral position of the robot with respect to one or more crop rows.
The fruit picking system of claim 1, in which the robot is configured to automatically navigate substantially in the middle of two crop rows, or at a fixed distance from a single crop row.
The fruit picking system of claim 1, in which the feature is or includes one or more legs or parts of one or more legs of the table-top systems.
The fruit picking system of claim 4, in which the one or more legs or parts of one or more legs of the table-top systems are approximately or substantially vertical, and/or evenly spaced, and/or arranged in a straight line.
The fruit picking system of claim 4, in which one or more legs or parts of one or more legs of the table-top systems are at a known orientation angle.
The fruit picking system of claim 1, in which the system measures the degree to which the robot is leaning over and compensates for the degree of lean by adapting models of the scene's geometry and camera viewpoints accordingly.
The fruit picking system of claim 1, in which the system includes an imaging or analysis chamber in which a fruit or bunch of fruits is positioned by a robot arm and is then imaged or analysed for grading or quality control purposes.
The fruit picking system of claim 1, in which the system includes a quality control (QC) subsystem configured to monitor the quality of fruit or bunch of fruits that has been picked or could be picked.
The fruit picking system of claim 9, in which the QC subsystem is responsible for grading picked fruit or bunch of fruits, determining its suitability for retail or other use, and discarding unusable fruit.
The fruit picking system of claim 9, in which the QC subsystem is configured to determine the fruit suitability for retail or other use and to discard unsuitable fruit.
The fruit picking system of claim 9, in which the QC determines the fruit's size and shape as a means of estimating the fruit's mass and thereby of ensuring that the require mass of fruit is placed in each punnet according to the requirements of the intended customer for average or minimum mass per punnet.
The fruit picking system of claim 9, in which the QC subsystem configured to classify a fruit or bunch of fruits that has been picked or could be picked, and in which the system allows a grower to adjust thresholds for classifying the quality of the fruit in which quality is a function of one or more properties of the fruit: size, ripeness color, hardness, symmetry, and stem length.
The fruit picking system of claim 9, in which the QC subsystem predicts the flavour or quality of a fruit or bunch of fruits and places the fruit or bunch of fruits in a specific storage container according to the flavour or quality prediction.
The fruit picking system of claim 1, in which the picked fruit or bunch of fruits is automatically allocated into specific punnets (or containers) based on size and quality measures of the picked fruit to minimize the statistical expectation of total cost according to a metric of maximizing the expected profitability for a grower from supplying a specific punnet to a customer in which the expected profitability is a function of one or more of the following: excess weight of fruits in the punnet compared to a target weight, number of fruits with a size that is outside of a desired size range, a measure of time it takes to place a fruit in the punnet and whether the punnet is underweight or not.
The fruit picking system of claim 1, in which a picked fruit or bunch of fruits is automatically allocated into specific punnets or containers based on size and quality measures of the picked fruit to minimize the statistical expectation of total cost according to a metric of maximizing the expected profitability for a grower.
The fruit picking system of claim 1, in which the system includes: (i) at least one picking arm; and (ii) at least one end effector, mounted on each picking arm configured either to cut a stem or branch for a specific fruit or bunch of fruits or pluck that fruit or bunch.
The fruit picking system of claim 1, in which picking head is configured to pull a target fruit or bunch of fruits away from a plant, to facilitate more reliable determination of the fruit's suitability for picking before the fruit is permanently severed from the plant.
The fruit picking system of claim 1, in which a control system software uses lighting conditions inferred or derived from the weather forecast as an input to the control subsystem or computer vision subsystem to control picking strategies or operations.
The fruit picking system of claim 1, in which the system is configured to estimate suitability for picking on the basis of a statistical probability that an attempt to pick a target fruit or bunch of fruits will be successful.
The fruit picking system of claim 20, in which a successful picking attempt means that the picked fruit is suitable for sale and is ripe and undamaged.
The fruit picking system of claim 20, in which a successful picking attempt means that no other part of the plant or growing infrastructure is damaged during picking.
The fruit picking system of claim 20, in which a successful picking attempt means that the picking arm does not undergo any collisions that could damage the crop or interfere with its continuing operation.
The fruit picking system of claim 1, in which the system includes a storage subsystem that is configured for receiving picked fruit and storing that fruit in containers for storage or transportation, or in punnets for retail.
The fruit picking system of claim 1, in which the system is configured to classify a fruit, and in which the system allows a grower to adjust thresholds for classifying the quality of the fruit in which quality is a function of one or more properties of the fruit: size, ripeness, color, hardness, symmetry, and stem length.
The fruit picking system of claim 1, in which system is configured to analyse fruit's size and shape used to estimate mass to work out punnet to place the fruit into.
The fruit picking system of claim 1, in which the system includes a communication network or central server.
The fruit picking system of claim 1, in which the system stores locations of all detected fruit, whether ripe or unripe, in computer memory in order to generate a yield map.
The fruit picking system of claim 1, in which the system stores a map coordinate system position of unripe fruits that have been detected but not picked in computer memory.
The fruit picking system of claim 28, in which the yield map takes into account the impact on time on the ripeness of previously unripe fruits.

Description

The field of the invention relates to systems and methods for robotic fruit picking.

A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

Horticultural producers depend critically on manual labour for harvesting their crops. Many types of fresh produce are harvested manually including berry fruits such as strawberries and raspberries, asparagus, table grapes and eating apples. Manual picking is currently necessary because the produce is prone to damage and requires delicate handling, or because the plant itself is valuable, producing fruit continuously over one or more growing seasons. Thus the efficient but destructive mechanical methods used to harvest crops such as wheat are not feasible.

Reliance on manual labour creates several problems for producers:

Current technologies for robotic soft fruit harvesting tend to rely on sophisticated hardware and naive robot control systems. In consequence, other soft fruit picking systems have not been commercially successful because they are expensive and require carefully controlled environments.

A small number of groups have developed robotic strawberry harvesting technology. However, the robots often come at a high cost and still need human operators to grade and post-process the fruit.

Citations (30)

US4519193A
US5544474A
US6145291A
JP2004180554A
US20050126144A1
US20060213167A1
US20090079839A1
US20100326363A1
US20090033457A1
US20100050585A1
US20110252760A1
US20110022231A1
US20110231017A1
US20110112730A1
US20120095651A1
US20140116469A1
US20140121836A1
US20140277694A1
CN103503639A
KR20150105661A
US20170273241A1
US9468152B1
US20170000027A1
US20180201157A1
US20170243466A1
US20170253259A1
US20170344017A1
US20170364524A1
US20200323140A1
US20180088586A1

Record as JSON

{
  "publication_number": "US12096733B2",
  "country": "US",
  "kind": "B2",
  "title": "Robotic fruit picking system",
  "abstract": "A robotic fruit picking system includes an autonomous robot that includes a positioning subsystem that enables autonomous positioning of the robot using a computer vision guidance system. The robot also includes at least one picking arm and at least one picking head, or other type of end effector, mounted on each picking arm to either cut a stem or branch for a specific fruit or bunch of fruits or pluck that fruit or bunch. A computer vision subsystem analyses images of the fruit to be picked or stored and a control subsystem is programmed with or learns picking strategies using machine learning techniques. A quality control (QC) subsystem monitors the quality of fruit and grades that fruit according to size and/or quality. The robot has a storage subsystem for storing fruit in containers for storage or transportation, or in punnets for retail.",
  "claims": [
    "1. A robotic fruit picking system in which the robot automatically navigates along rows of fruit-producing plants grown on table-top systems, and in which the fruit picking system includes a computer vision system that includes a camera and that is configured to detect one or more features of the table top systems in order to determine the position and/or orientation of the camera with respect to a coordinate system defined by the features of the table-top systems, such that the fruit picking system is configured to automatically determine its position and/or orientation with respect to the rows of fruit producing plants.",
    "2. The fruit picking system of claim 1, in which the computer vision system uses images obtained by a forwards or backwards facing camera for determining the heading and lateral position of the robot with respect to one or more crop rows.",
    "3. The fruit picking system of claim 1, in which the robot is configured to automatically navigate substantially in the middle of two crop rows, or at a fixed distance from a single crop row.",
    "4. The fruit picking system of claim 1, in which the feature is or includes one or more legs or parts of one or more legs of the table-top systems.",
    "5. The fruit picking system of claim 4, in which the one or more legs or parts of one or more legs of the table-top systems are approximately or substantially vertical, and/or evenly spaced, and/or arranged in a straight line.",
    "6. The fruit picking system of claim 4, in which one or more legs or parts of one or more legs of the table-top systems are at a known orientation angle.",
    "7. The fruit picking system of claim 1, in which the system measures the degree to which the robot is leaning over and compensates for the degree of lean by adapting models of the scene's geometry and camera viewpoints accordingly.",
    "8. The fruit picking system of claim 1, in which the system includes an imaging or analysis chamber in which a fruit or bunch of fruits is positioned by a robot arm and is then imaged or analysed for grading or quality control purposes.",
    "9. The fruit picking system of claim 1, in which the system includes a quality control (QC) subsystem configured to monitor the quality of fruit or bunch of fruits that has been picked or could be picked.",
    "10. The fruit picking system of claim 9, in which the QC subsystem is responsible for grading picked fruit or bunch of fruits, determining its suitability for retail or other use, and discarding unusable fruit.",
    "11. The fruit picking system of claim 9, in which the QC subsystem is configured to determine the fruit suitability for retail or other use and to discard unsuitable fruit.",
    "12. The fruit picking system of claim 9, in which the QC determines the fruit's size and shape as a means of estimating the fruit's mass and thereby of ensuring that the require mass of fruit is placed in each punnet according to the requirements of the intended customer for average or minimum mass per punnet.",
    "13. The fruit picking system of claim 9, in which the QC subsystem configured to classify a fruit or bunch of fruits that has been picked or could be picked, and in which the system allows a grower to adjust thresholds for classifying the quality of the fruit in which quality is a function of one or more properties of the fruit: size, ripeness color, hardness, symmetry, and stem length.",
    "14. The fruit picking system of claim 9, in which the QC subsystem predicts the flavour or quality of a fruit or bunch of fruits and places the fruit or bunch of fruits in a specific storage container according to the flavour or quality prediction.",
    "15. The fruit picking system of claim 1, in which the picked fruit or bunch of fruits is automatically allocated into specific punnets (or containers) based on size and quality measures of the picked fruit to minimize the statistical expectation of total cost according to a metric of maximizing the expected profitability for a grower from supplying a specific punnet to a customer in which the expected profitability is a function of one or more of the following: excess weight of fruits in the punnet compared to a target weight, number of fruits with a size that is outside of a desired size range, a measure of time it takes to place a fruit in the punnet and whether the punnet is underweight or not.",
    "16. The fruit picking system of claim 1, in which a picked fruit or bunch of fruits is automatically allocated into specific punnets or containers based on size and quality measures of the picked fruit to minimize the statistical expectation of total cost according to a metric of maximizing the expected profitability for a grower.",
    "17. The fruit picking system of claim 1, in which the system includes: (i) at least one picking arm; and (ii) at least one end effector, mounted on each picking arm configured either to cut a stem or branch for a specific fruit or bunch of fruits or pluck that fruit or bunch.",
    "18. The fruit picking system of claim 1, in which picking head is configured to pull a target fruit or bunch of fruits away from a plant, to facilitate more reliable determination of the fruit's suitability for picking before the fruit is permanently severed from the plant.",
    "19. The fruit picking system of claim 1, in which a control system software uses lighting conditions inferred or derived from the weather forecast as an input to the control subsystem or computer vision subsystem to control picking strategies or operations.",
    "20. The fruit picking system of claim 1, in which the system is configured to estimate suitability for picking on the basis of a statistical probability that an attempt to pick a target fruit or bunch of fruits will be successful.",
    "21. The fruit picking system of claim 20, in which a successful picking attempt means that the picked fruit is suitable for sale and is ripe and undamaged.",
    "22. The fruit picking system of claim 20, in which a successful picking attempt means that no other part of the plant or growing infrastructure is damaged during picking.",
    "23. The fruit picking system of claim 20, in which a successful picking attempt means that the picking arm does not undergo any collisions that could damage the crop or interfere with its continuing operation.",
    "24. The fruit picking system of claim 1, in which the system includes a storage subsystem that is configured for receiving picked fruit and storing that fruit in containers for storage or transportation, or in punnets for retail.",
    "25. The fruit picking system of claim 1, in which the system is configured to classify a fruit, and in which the system allows a grower to adjust thresholds for classifying the quality of the fruit in which quality is a function of one or more properties of the fruit: size, ripeness, color, hardness, symmetry, and stem length.",
    "26. The fruit picking system of claim 1, in which system is configured to analyse fruit's size and shape used to estimate mass to work out punnet to place the fruit into.",
    "27. The fruit picking system of claim 1, in which the system includes a communication network or central server.",
    "28. The fruit picking system of claim 1, in which the system stores locations of all detected fruit, whether ripe or unripe, in computer memory in order to generate a yield map.",
    "29. The fruit picking system of claim 1, in which the system stores a map coordinate system position of unripe fruits that have been detected but not picked in computer memory.",
    "30. The fruit picking system of claim 28, in which the yield map takes into account the impact on time on the ripeness of previously unripe fruits."
  ],
  "description_excerpt": "The field of the invention relates to systems and methods for robotic fruit picking.\n\nA portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.\n\nHorticultural producers depend critically on manual labour for harvesting their crops. Many types of fresh produce are harvested manually including berry fruits such as strawberries and raspberries, asparagus, table grapes and eating apples. Manual picking is currently necessary because the produce is prone to damage and requires delicate handling, or because the plant itself is valuable, producing fruit continuously over one or more growing seasons. Thus the efficient but destructive mechanical methods used to harvest crops such as wheat are not feasible.\n\nReliance on manual labour creates several problems for producers:\n\nCurrent technologies for robotic soft fruit harvesting tend to rely on sophisticated hardware and naive robot control systems. In consequence, other soft fruit picking systems have not been commercially successful because they are expensive and require carefully controlled environments.\n\nA small number of groups have developed robotic strawberry harvesting technology. However, the robots often come at a high cost and still need human operators to grade and post-process the fruit.",
  "cpc": [
    "A01G 9/143",
    "A01D 46/22",
    "A01D 46/243",
    "A01D 46/253",
    "A01D 46/28",
    "A01D 46/30",
    "B25J 11/00",
    "B25J 15/0019",
    "B25J 15/0033",
    "B25J 5/005",
    "B25J 9/0084",
    "B25J 9/06",
    "B25J 9/1679",
    "B25J 9/1697",
    "G05B 2219/45003",
    "G05D 1/0094",
    "G05D 1/0219",
    "G05D 1/648",
    "G06F 18/2148",
    "G06F 18/24323",
    "G06F 18/24765",
    "G06Q 30/0283",
    "G06T 2207/10048",
    "G06T 2207/20081",
    "G06T 2207/20084",
    "G06T 2207/30128",
    "G06T 7/0004",
    "G06T 7/11",
    "G06T 7/50",
    "G06T 7/60",
    "G06T 7/70",
    "G06T 7/90",
    "G06V 20/10",
    "G06V 20/68",
    "Y02A 40/25"
  ],
  "ipc": [
    "A01D 46/22",
    "A01D 46/24",
    "A01D 46/253",
    "A01D 46/28",
    "A01D 46/30",
    "A01G 9/14",
    "B25J 11/00",
    "B25J 15/00",
    "B25J 9/00",
    "B25J 9/06",
    "B25J 9/16",
    "G05D 1/00",
    "G06F 18/214",
    "G06F 18/24",
    "G06F 18/243",
    "G06Q 30/0283",
    "G06T 7/00",
    "G06T 7/11",
    "G06T 7/50",
    "G06T 7/60",
    "G06T 7/70",
    "G06T 7/90",
    "G06V 20/10",
    "G06V 20/68"
  ],
  "assignees": [
    "Dogtooth Technologies Ltd"
  ],
  "inventors": [
    "Duncan Robertson",
    "Matthew Cook",
    "Edward Herbert",
    "Frank TULLY"
  ],
  "filing_date": "2023-02-27",
  "publication_date": "2024-09-24",
  "grant_date": "2024-09-24",
  "priority_date": "2016-11-08",
  "application_number": "US-202318114681-A",
  "family_id": "60702825",
  "cited_by_count": 2,
  "citations": [
    "US4519193A",
    "US5544474A",
    "US6145291A",
    "JP2004180554A",
    "US20050126144A1",
    "US20060213167A1",
    "US20090079839A1",
    "US20100326363A1",
    "US20090033457A1",
    "US20100050585A1",
    "US20110252760A1",
    "US20110022231A1",
    "US20110231017A1",
    "US20110112730A1",
    "US20120095651A1",
    "US20140116469A1",
    "US20140121836A1",
    "US20140277694A1",
    "CN103503639A",
    "KR20150105661A",
    "US20170273241A1",
    "US9468152B1",
    "US20170000027A1",
    "US20180201157A1",
    "US20170243466A1",
    "US20170253259A1",
    "US20170344017A1",
    "US20170364524A1",
    "US20200323140A1",
    "US20180088586A1"
  ]
}

Record 368 of 8,000 in Patents full text (MLC-0201). Request the full dataset.