Neural network for bulk sorting

{
  "publication_number": "US12046010B2",
  "country": "US",
  "kind": "B2",
  "title": "Neural network for bulk sorting",
  "abstract": "A bulk sorting system for sorting objects in bulk is provided. The bulk sorting system includes: at least one radiation source arranged to radiate the objects, at least one optical sensor arranged to capture reflected radiation of the objects and acquire the reflected radiation as multi- or hyperspectral data; a processing circuit configured to analyze the reflected radiation of the objects by inputting the multi- or hyperspectral data into a convolutional neural network (CNN) with at least two convolutional layers in order to either detect and classify the objects in the multi- or hyperspectral data and/or semantically segment the multi- or hyperspectral data; and a mechanical sorter configured to sort the objects according to their classification and/or segmentation using the analysis of the processing circuit such that different overlapping and/or stacked objects are separated or treated as a single group of objects.",
  "claims": [
    "1. A bulk sorting system for sorting objects in bulk comprising: at least one radiation source arranged to radiate the objects, at least one optical sensor arranged to capture reflected radiation of the objects and acquire the reflected radiation as multi- or hyperspectral data, comprising a near-infrared (NIR) scanner arranged to scan the objects, wherein the multi- or hyperspectral data comprises the scan data; a processing circuit configured to analyze the reflected radiation of the objects by inputting the multi- or hyperspectral data into a convolutional neural network (CNN) with at least two convolutional layers in order to detect and classify the objects in the multi- or hyperspectral data, wherein the CNN is trained with sample images including images of overlapping objects with corresponding labels of the overlapping objects; and a mechanical sorter configured to sort the objects according to their classification using the analysis of the processing circuit such that different overlapping and/or stacked objects, as analyzed by the processing circuit, are treated as a single group of objects.",
    "2. The bulk sorting system according to claim 1, wherein the at least one optical sensor comprises an image sensor arranged to capture image data of the objects, wherein the multi- or hyperspectral data comprises the image data.",
    "3. The bulk sorting system according to claim 1, wherein the at least one optical sensor comprises a multi- or hyperspectral camera arranged to scan the objects, wherein the multi- or hyperspectral data comprises the scan.",
    "4. The bulk sorting system according to claim 1, wherein the at least one optical sensor comprises a laser triangulator arranged to measure 3D-data of the objects, wherein the multi- or hyperspectral data comprises the 3D-data, wherein the measurement of 3D-data may comprise laser height intensity scanning.",
    "5. The bulk sorting system according to claim 1, further comprising an electromagnetic detector arranged to measure electromagnetic properties of the objects, wherein the processing circuit is further configured to analyze the electromagnetic properties by inputting the measured electromagnetic properties into the CNN in order to detect and classify the objects in the multi- or hyperspectral data.",
    "6. The bulk sorting system according to claim 1, wherein the at least one optical sensor comprises a laser scanner (36) with a rotating polygon mirror arranged to measure laser scatter and/or anti-scatter properties of the objects, a pulsed LED emitter arranged to measure light anti-scatter properties of the objects and/or an X-ray camera arranged to measure X-ray transmission of the objects; wherein the multi- or hyperspectral data comprises the laser scatter and/or anti-scatter properties, the light anti-scatter properties and/or the X-ray transmission of the objects, respectively.",
    "7. The bulk sorting system according to claim 1, further comprising conveying the objects along detection range(s) of the at least one optical sensor to the mechanical sorter using a conveyor belt.",
    "8. The bulk sorting system according to claim 1, wherein the mechanical sorter is further configured to separate the objects into at least two streams and/or to eject unwanted objects from the bulk.",
    "9. The bulk sorting system according to claim 1, wherein for a group of objects comprising at least a first object type and a second object type, the mechanical sorter is configured to sort the group of objects as either the first object type or the second object type based on a preference in the bulk sorting system.",
    "10. The bulk sorting system according to claim 1, wherein the mechanical sorter is arranged to target the center of gravity or boundaries of the objects, as analyzed by the processing circuit.",
    "11. The bulk sorting system according to claim 1, further comprising post-processing the detected and classified multi- or hyperspectral data to configure the mechanical sorter before sorting occurs.",
    "12. The bulk sorting system according to claim 1, wherein the processing circuit is further configured to input at least a part of the multi- or hyperspectral data into a pattern recognition algorithm; and wherein the results of both the CNN and the pattern recognition algorithm are used by the mechanical sorter to sort the objects according to their detection and classification verified by the pattern recognition algorithm.",
    "13. A method for sorting objects in bulk comprising steps of: radiating the objects using at least one radiation source; capturing reflected radiation of the objects using at least one optical sensor, comprising capturing a scan using a multi- or hyperspectral scanner; acquiring the reflected radiation, comprising the scan, as multi- or hyperspectral data; analyzing the reflected radiation of the objects by inputting the multi- or hyperspectral data into a convolutional neural network (CNN) with at least two convolutional layers in order to detect and classify the objects in the multi- or hyperspectral data, wherein the CNN is trained with sample images including images of overlapping objects with corresponding labels of the overlapping objects; and sorting, by a mechanical sorter, the objects according to their classification using the results of the analysis step such that different overlapping and/or stacked objects, as analyzed by the processing circuit, are treated as a single group of objects.",
    "14. A method for sorting objects in bulk according to claim 13, wherein capturing reflected radiation of the objects using at least one optical sensor comprises capturing a scan using a NIR scanner."
  ],
  "description_excerpt": "The present invention relates to the field of sorting. More particularly, the present invention relates to bulk sorting aided by a convolutional neural network (CNN).\n\nSorting is a topical field of research with implications for e.g. recycling, mining or food processing. For a recycling implementation, sorting techniques are used to sort a mixture of garbage into the correct recycling bin. As technology evolves, this sorting may be done more accurately and faster than before.\n\nThere exists techniques such as those shown in US 2018/243800 AA, for using a machine learning system for sorting a stream of single objects. The machine learning system allows for an accurate identification of the objects being sorted. However, such techniques are slow as they may only process a single stream of objects at a time. Other examples of material characterization and segmentation techniques may be found in US 2019/130560 A1 and Matthieu Grard et al: “Object segmentation in depth maps with one user click and a synthetically trained fully convolutional network”, 2018.\n\nCurrent sorters separate individual particles. They require careful feed preparation so that individual particles may be detected and measured, and ejection is usually achieved by blasts of compressed air. Therefore, current sorters have very low capacity (up to 300 tonnes per hour for larger particles and much less for smaller particles), making them non-viable for higher tonnage pre-concentration or so-called bulk sorting. The sorting speed and throughput to be achieved in bulk sorting is directly related to the size of the objects to be sorted.",
  "cpc": [
    "G06V 10/143",
    "B07C 5/3422",
    "G06F 18/2414",
    "G06N 3/045",
    "G06N 3/08",
    "G06V 10/58",
    "G06V 10/764",
    "G06V 10/82",
    "G06V 20/64",
    "Y02W 90/00"
  ],
  "ipc": [
    "B07C 5/342",
    "G06V 10/143",
    "G06V 10/58",
    "G06V 10/764",
    "G06V 10/82",
    "G06V 20/64"
  ],
  "assignees": [
    "Tomra Sorting GmbH"
  ],
  "inventors": [
    "Dirk Balthasar",
    "Daniel Bender",
    "Frank Schmitt"
  ],
  "filing_date": "2020-11-04",
  "publication_date": "2024-07-23",
  "grant_date": "2024-07-23",
  "priority_date": "2019-11-04",
  "application_number": "US-202017773319-A",
  "family_id": "68426292",
  "cited_by_count": 3,
  "citations": [
    "US5242059A",
    "US5242059B1",
    "WO1996006689A2",
    "JP2002540397A",
    "US20060081510A1",
    "DE102010024784A1",
    "US20150160128A1",
    "WO2017140729A1",
    "CN106000904A",
    "US20180243800A1",
    "US20180100810A1",
    "CN206567239U",
    "US20190210067A1",
    "US20180322327A1",
    "US20180345324A1",
    "US20180365820A1",
    "US20190030571A1",
    "US20190130560A1",
    "CN107999405A",
    "US20190217342A1",
    "US20190299255A1"
  ]
}