Systems and methods for parallel autonomy of a vehicle

{
  "publication_number": "US11531865B2",
  "country": "US",
  "kind": "B2",
  "title": "Systems and methods for parallel autonomy of a vehicle",
  "abstract": "Systems and methods for parallel autonomy of a vehicle are disclosed herein. One embodiment receives input data, the input data including at least one of sensor data and structured input data; encodes the input data into an intermediate embedding space using a first neural network; inputs the intermediate embedding space to a first behavior model and a second behavior model, the first behavior model producing a first behavior output, the second behavior model producing a second behavior output; combines the first behavior output and the second behavior output using an ideal-behavior model to produce an ideal behavior for the vehicle; and controls one or more aspects of operation of the vehicle based, at least in part, on the ideal behavior.",
  "claims": [
    "1. A system for parallel autonomy of a vehicle, the system comprising: one or more sensors; one or more processors; and a memory communicably coupled to the one or more processors and storing: a data encoding module including instructions that when executed by the one or more processors cause the one or more processors to: receive input data, the input data including at least one of sensor data from the one or more sensors and structured input data; and encode the input data into an intermediate embedding space using a first neural network, wherein the intermediate embedding space is an efficient, low-dimensionality representation of the input data; a first behavior module including instructions that when executed by the one or more processors cause the one or more processors to process the intermediate embedding space using a first behavior model to produce a first behavior output and a first risk estimate associated with the first behavior output; a second behavior module including instructions that when executed by the one or more processors cause the one or more processors to process the intermediate embedding space using a second behavior model to produce a second behavior output and a second risk estimate associated with the second behavior output; an ideal-behavior module including instructions that when executed by the one or more processors cause the one or more processors to combine the first behavior output and the second behavior output using an ideal-behavior model to produce an ideal behavior for the vehicle, wherein the ideal-behavior model considers the first risk estimate and the second risk estimate in producing the ideal behavior for the vehicle; and a control module including instructions that when executed by the one or more processors cause the one or more processors to control one or more aspects of operation of the vehicle based, at least in part, on the ideal behavior.",
    "2. The system of claim 1, wherein the one or more sensors include one or more of an image sensor, a radar sensor, a Light Detection and Ranging (LIDAR) sensor, and a Controller-Area-Network (CAN) sensor.",
    "3. The system of claim 1, wherein the structured input data includes at least one of (1) position data for at least one road agent, (2) lane geometries, and (3) image-segmentation data, and the data encoding module includes instructions to receive the structured input data from one or more of a perception pipeline of the vehicle and one or more sources external to the vehicle, the sources external to the vehicle including one or more of cloud servers, infrastructure systems, and other vehicles.",
    "4. The system of claim 1, wherein the first neural network is an encoder network of an autoencoder.",
    "5. The system of claim 4, further comprising a training module including instructions that when executed by the one or more processors cause the one or more processors to: input, during a training phase, the intermediate embedding space to a second neural network that is a decoder network of the autoencoder; and train the second neural network to predict the input data.",
    "6. The system of claim 1, further comprising a training module including instructions that when executed by the one or more processors cause the one or more processors to train, during a training phase, the ideal-behavior model to combine the first behavior output and the second behavior output in a manner that minimizes a predetermined loss function and wherein the ideal-behavior module includes instructions to update, during an operational phase, the ideal-behavior model in accordance with driver feedback.",
    "7. The system of claim 1, wherein: the first behavior model is an autonomous-behavior model for autonomous driving and the first behavior output is a planned behavior for the vehicle; and the second behavior model is a driver-behavior model and the second behavior output is a predicted behavior of the vehicle.",
    "8. The system of claim 1, wherein the first behavior model and the second behavior model are both autonomous-behavior models and the first behavior output and the second behavior output are different planned behaviors for the vehicle.",
    "9. The system of claim 1, wherein the ideal-behavior model includes at least one of a support vector machine, a regressive neural network, and a reinforcement-learning model.",
    "10. The system of claim 1, wherein the one or more aspects of operation of the vehicle include at least one of steering, acceleration, and braking.",
    "11. A non-transitory computer-readable medium for parallel autonomy of a vehicle and storing instructions that when executed by one or more processors cause the one or more processors to: receive input data, the input data including at least one of sensor data and structured input data; encode the input data into an intermediate embedding space using a first neural network, wherein the intermediate embedding space is an efficient, low-dimensionality representation of the input data; input the intermediate embedding space to a first behavior model and a second behavior model, the first behavior model producing a first behavior output and a first risk estimate associated with the first behavior output, the second behavior model producing a second behavior output and a second risk estimate associated with the second behavior output; combine, based at least in part on the first risk estimate and the second risk estimate, the first behavior output and the second behavior output using an ideal-behavior model to produce an ideal behavior for the vehicle; and control one or more aspects of operation of the vehicle based, at least in part, on the ideal behavior.",
    "12. The non-transitory computer-readable medium of claim 11, wherein the first neural network is an encoder network of an autoencoder.",
    "13. The non-transitory computer-readable medium of claim 12, further comprising instructions that when executed by the one or more processors cause the one or more processors to: input, during a training phase, the intermediate embedding space to a second neural network that is a decoder network of the autoencoder; and train the second neural network to predict the input data.",
    "14. A method of parallel autonomy of a vehicle, the method comprising: receiving input data, the input data including at least one of sensor data and structured input data; encoding the input data into an intermediate embedding space using a first neural network, wherein the intermediate embedding space is an efficient, low-dimensionality representation of the input data; inputting the intermediate embedding space to a first behavior model and a second behavior model, the first behavior model producing a first behavior output and a first risk estimate associated with the first behavior output, the second behavior model producing a second behavior output and a second risk estimate associated with the second behavior output; combining, based at least in part on the first risk estimate and the second risk estimate, the first behavior output and the second behavior output using an ideal-behavior model to produce an ideal behavior for the vehicle; and controlling one or more aspects of operation of the vehicle based, at least in part, on the ideal behavior.",
    "15. The method of claim 14, wherein the first neural network is an encoder network of an autoencoder.",
    "16. The method of claim 15, further comprising: inputting, during a training phase, the intermediate embedding space to a second neural network that is a decoder network of the autoencoder; and training the second neural network to predict the input data.",
    "17. The method of claim 14, wherein: the first behavior model is an autonomous-behavior model for autonomous driving and the first behavior output is a planned behavior for the vehicle; and the second behavior model is a driver-behavior model and the second behavior output is a predicted behavior of the vehicle.",
    "18. The method of claim 14, wherein the first behavior model and the second behavior model are both autonomous-behavior models and the first behavior output and the second behavior output are different planned behaviors for the vehicle.",
    "19. The method of claim 14, wherein the ideal-behavior model includes at least one of a support vector machine, a regressive neural network, and a reinforcement-learning model.",
    "20. The method of claim 14, further comprising: training, during a training phase, the ideal-behavior model to combine the first behavior output and the second behavior output in a manner that minimizes a predetermined loss function; and updating, during an operational phase, the ideal-behavior model in accordance with driver feedback."
  ],
  "description_excerpt": "The subject matter described herein generally relates to vehicles and, more particularly, to systems and methods for parallel autonomy of a vehicle.\n\nSome vehicles are designed to be driven manually but are equipped with an advanced driver-assistance system (ADAS) or other incident-avoidance system. Such a system can intervene by temporarily taking control of the steering system, the accelerator, and/or the braking system of the vehicle to prevent a potentially hazardous outcome (e.g., a collision). Unfortunately, the intervention can sometimes cause the driver to experience discomfort because the driver does not feel that he or she is in control of the vehicle.\n\nAn example of a system for parallel autonomy of a vehicle is presented herein. The system comprises one or more sensors, one or more processors, and a memory communicably coupled to the one or more processors. The memory stores a data encoding module including instructions that when executed by the one or more processors cause the one or more processors to receive input data, the input data including at least one of sensor data from the one or more sensors and structured input data. The data encoding module also includes instructions that when executed by the one or more processors cause the one or more processors to encode the input data into an intermediate embedding space using a first neural network. The memory also stores a first behavior module including instructions that when executed by the one or more processors cause the one or more processors to process the intermediate embedding space using a first behavior model to produce a first behavior output.",
  "cpc": [
    "G06N 3/045",
    "B60W 50/14",
    "B60W 60/0011",
    "B60W 60/0015",
    "B60W 60/0023",
    "B60W 60/0025",
    "G06N 20/10",
    "G06N 3/044",
    "G06N 3/0454",
    "G06N 3/0455",
    "G06N 3/08",
    "G06N 3/09",
    "G06N 3/092",
    "H04L 12/40",
    "H04L 12/40176",
    "H04L 2012/40215"
  ],
  "ipc": [
    "B60W 50/14",
    "B60W 60/00",
    "G06N 3/04",
    "G06N 3/08",
    "H04L 12/40"
  ],
  "assignees": [
    "Toyota Research Institute Inc"
  ],
  "inventors": [
    "Stephen G. McGill, JR.",
    "Guy Rosman",
    "Luke S. Fletcher",
    "John Joseph Leonard"
  ],
  "filing_date": "2020-02-28",
  "publication_date": "2022-12-20",
  "grant_date": "2022-12-20",
  "priority_date": "2020-02-28",
  "application_number": "US-202016805218-A",
  "family_id": "77463720",
  "cited_by_count": 2,
  "citations": [
    "US20180154899A1",
    "US20180260706A1",
    "US20180330178A1",
    "US20190009794A1",
    "US20210248514A1",
    "US20200023846A1",
    "US10981564B2",
    "US20200134442A1",
    "US20200151222A1",
    "US20200175691A1",
    "US20220126863A1",
    "US20200356835A1",
    "US20200379454A1",
    "US20210049452A1",
    "US20210053569A1",
    "WO2021089570A1",
    "US20210294323A1",
    "US20210342686A1",
    "US20210370962A1"
  ]
}