Bounding box embedding for object identifying

{
  "publication_number": "US10922574B1",
  "country": "US",
  "kind": "B1",
  "title": "Bounding box embedding for object identifying",
  "abstract": "Techniques are disclosed for implementing a neural network that outputs embeddings. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn such embeddings. In some examples, the neural network may be trained to learn embeddings for instance segmentation of an object based on an embedding for a bounding box associated with the object being trained to match pixel embeddings for pixels associated with the object. The embeddings may be used for object identification, object matching, object classification, and/or object tracking in various examples.",
  "claims": [
    "1. A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: training a machine learned (ML) model based at least in part on a subset of examples, the training comprising: identifying a reference region associated with an object represented in an image; determining a training bounding box associated with the object; determining first embeddings for pixels of the reference region; associating a second embedding for the training bounding box with the first embeddings; and minimizing a loss function with respect to a difference between the first embeddings associated with the object and the second embedding associated with the training bounding box; and transmitting the ML model to an autonomous vehicle for detecting one or more additional objects in an environment.",
    "2. The system as claim 1 recites, the operations further comprising: determining a loss based at least in part on the difference, or a square of the difference, between the first embeddings and the second embedding.",
    "3. The system as claim 1 recites, the operations further comprising: determining, as an instance segmentation, pixels of the image having the first embeddings within a threshold distance of the second embedding.",
    "4. The system as claim 1 recites, wherein the reference region is comprised of pixels associated with the object.",
    "5. The system as claim 1 recites, the operations further comprising: averaging together the first embeddings for the pixels of the reference region to determine an average pixel embedding, wherein minimizing the loss function with respect to the difference between the first embeddings and the second embedding comprises minimizing the loss function with respect to a second difference between the average pixel embedding and the second embedding.",
    "6. The system as claim 5 recites, wherein the first embeddings and the second embedding comprise vectors.",
    "7. A computer-implemented method comprising: inputting an image comprising a representation of an object into a machine learned model trained to output: i) one or more pixel parameters associated with one or more pixels of the image; ii) a bounding box associated with the object; and iii) a box parameter associated with the bounding box; and determining, based at least in part on a difference between the one or more pixel parameters and the box parameter, an identifier for the object and object pixels associated with the object.",
    "8. The computer-implemented method of claim 7, wherein the one or more pixel parameters comprise one or more pixel embeddings, and wherein the box parameter comprises a box embedding, the method further comprising: determining a mask associated with the object based at least in part on the box embedding and the one or more pixel embeddings.",
    "9. The computer-implemented method of claim 7, wherein the machine learned model is further trained to: determine that the difference between the one or more pixel parameters and the box parameter satisfies a threshold difference; and determine, based at least in part on the determining that the difference satisfies the threshold difference, that the one or more pixels are associated with the bounding box.",
    "10. The computer-implemented method of claim 7, wherein the machine learned model is trained by: determining an average parameter of the one or more pixel parameters; and determining a loss function with respect to a second difference between the average parameter and the box parameter, wherein the one or more pixels comprise the object pixels associated with the object.",
    "11. The computer-implemented method of claim 7, wherein: the object is a first object and the one or more pixel parameters are one or more first pixel parameters, the machine learned model is further trained to output one or more second pixel parameters associated with one or more second pixels of a second representation of a second object in the image, and the computer-implemented method further comprises: determining, based at least in part on a second difference between the one or more first pixel parameters and the one or more second pixel parameters, that one or more first pixels are associated with the bounding box.",
    "12. The computer-implemented method of claim 11, wherein: the bounding box is associated with the first object, and the second object is at least partially within an area of the bounding box, the machine learned model is further trained to output a second bounding box associated with the second object and having second bounding box parameters, and the computer-implemented method further comprises: based at least in part on the difference between the one or more second pixel parameters and the second bounding box parameters, determining second object pixels associated with the second object and a second identifier associated with the second object.",
    "13. The computer-implemented method of claim 7, further comprising: capturing the image using an image sensor of an autonomous vehicle; and controlling the autonomous vehicle based at least in part on identifying the object in the image.",
    "14. The computer-implemented method of claim 7, wherein the box parameter is a vector.",
    "15. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: inputting an image comprising a representation of an object into a machine learned model trained to output: i) pixel parameters associated with pixels of the image; ii) a bounding box associated with the object; and iii) box parameters associated with the bounding box; and determining, based at least in part on a difference between the pixel parameters and the box parameters, a set of pixels associated with the object and an identifier associated with the object.",
    "16. The one or more non-transitory computer-readable media of claim 15, wherein: the pixel parameters comprise pixel embeddings, the box parameters comprise a box embedding, and the difference between the pixel embeddings and the box embedding further comprises a second difference between an average of the pixel embeddings and the box embedding.",
    "17. The one or more non-transitory computer-readable media of claim 16, further comprising: determining a mask associated with the object based at least in part on the box embedding and the pixel embeddings.",
    "18. The one or more non-transitory computer-readable media of claim 16, wherein the machine learned model is trained based at least in part on: determining an average pixel embedding of the pixel embeddings associated with the set of pixels associated with the object; and determining a loss function with respect to a second difference between the average pixel embedding and the box embedding.",
    "19. The one or more non-transitory computer-readable media of claim 15, wherein: the object is a first object, the pixel parameters are first pixel parameters associated with first pixels of the image, the machine learned model is further trained to output second pixel parameters associated with second pixels of a second representation of a second object in the image, and the operations further comprise: determining, based at least in part on a difference between the first pixel parameters and the second pixel parameters, that the first pixels are associated with the bounding box.",
    "20. The one or more non-transitory computer-readable media of claim 19, wherein: the bounding box is associated with the first object, and the second object is at least partially within an area of the bounding box, and the operations further comprise: determining a second bounding box associated with the second object; determining second box parameters associated with the second bounding box; and based at least in part on a second difference between the second pixel parameters and the second box parameters, determining a second set of pixels associated with the second object and a second identifier associated with the second object."
  ],
  "description_excerpt": "Methods, apparatuses, and systems utilize various types of data, such as image data or lidar data, just to name a few examples, to perform functions that allow robotic platforms to navigate through environments. For example, an autonomous vehicle may use image data and/or other sensor data for navigation to track objects and obstacles, such as other vehicles, buildings, pedestrians, etc. Equipping a computer with such functionality may include building software components that take an image, identify salient portions of the image, and represent the salient portions to the computer in a form that the computer can use to conduct further operations. One form of software that can be built to equip a computer with such functionality are machine-learned (ML) models.\n\nPrevious attempts to train ML models to identify salient portions of an image have resulted in flawed or inaccurate ML models. For example, some forms of ML model training result in a ML model that cannot properly distinguish between objects that are proximate to each other (e.g., a pedestrian that passes in front of another pedestrian in the camera's view), resulting in extraneous and/or inaccurate identifications of one or both objects.\n\nThe detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.",
  "cpc": [
    "G06T 7/12",
    "G06K 9/00791",
    "G06K 9/3233",
    "G06N 3/0464",
    "G06N 3/08",
    "G06N 3/088",
    "G06N 3/09",
    "G06T 2207/20084",
    "G06T 7/10",
    "G06V 10/25",
    "G06V 20/56"
  ],
  "ipc": [
    "G06N 3/08",
    "G06T 7/10",
    "G06V 10/25"
  ],
  "assignees": [
    "Zoox Inc"
  ],
  "inventors": [
    "Sarah Tariq"
  ],
  "filing_date": "2018-12-10",
  "publication_date": "2021-02-16",
  "grant_date": "2021-02-16",
  "priority_date": "2018-12-10",
  "application_number": "US-201816215554-A",
  "family_id": "74569875",
  "cited_by_count": 40,
  "citations": [
    "US20140376819A1",
    "US20180089505A1",
    "US20200118423A1",
    "US20190272645A1"
  ]
}