Patent · US10112606B2 · B2 · US

Scalable sensor fusion and autonomous x-by-wire control

(11) Publication number

US10112606B2

(21) Application number

15/004,718

(22) Filing date

2016-01-22

(30) Priority date

2016-01-22

(43) Publication date

2018-10-30

(45) Date of grant

2018-10-30

(51) IPC

B60T 7/22; B60W 30/00; B62D 15/02; G01S 17/87; G01S 19/13; G05D 1/00; H04L 29/08; B60T 7/18; B60T 8/171; B60T 8/88

(52) CPC

B60W Conjoint control of vehicle sub-units of different type or different function; control systems specially adapted for hybrid vehicles; road vehicle drive control systems for purposes not related to the control of a particular sub-unit: 30/00, 2420/403, 2420/408, 2420/42, 2420/52
B60T Vehicle brake control systems or parts thereof; brake control systems or parts thereof, in general; arrangement of braking elements on vehicles in general; portable devices for preventing unwanted movement of vehicles; vehicle modifications to facilitate cooling of brakes: 2270/402, 2270/82, 7/18, 7/22, 8/171, 8/885
B62D Motor vehicles; trailers: 15/025
G01S Radio direction-finding; radio navigation; determining distance or velocity by use of radio waves; locating or presence-detecting by use of the reflection or reradiation of radio waves; analogous arrangements using other waves: 17/87, 19/13
G05D Systems for controlling or regulating non-electric variables: 1/0088
H04L Transmission of digital information, e.g. telegraphic communication: 67/10, 67/12

(73) Assignee

International Business Machines Corp

(72) Inventors

Rakesh Jain; Gabor Madl; Ramani R. Routray; Yang Song

(54) Title

Scalable sensor fusion and autonomous x-by-wire control

(57) Abstract

An autonomous system for x-by-wire control includes processing nodes distributed and connected to one another. Sensors are connected to the processing nodes. Actuators are configured to directly control the autonomous system driven by and connected to the processing nodes for x-by-wire control. The processing nodes are configured to: partition and map processing tasks between the processing nodes, and merge and reduce results of individual processing tasks into an actionable table that specifies objectives for the autonomous system.

Full text

View on Google Patents

Claims (20)

An autonomous system comprising: a plurality of processing nodes distributed and coupled to one another; a plurality of sensors coupled to the processing nodes; and a plurality of actuators including at least one power control actuator, at least one brake actuator, and at least one steering actuator, the plurality of actuators configured to directly control the autonomous system driven by and coupled to the plurality of processing nodes for throttle-by-wire, brake-by-wire and steer-by-wire control; wherein each processing node of the plurality of processing nodes is configured to: partition and map processing tasks between the plurality of processing nodes; merge and reduce results of individual processing tasks into an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control the plurality of actuators for the autonomous system; and transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.
The autonomous system of claim 1, wherein each of the processing nodes comprise: at least one memory device configured to store sensor data received from one or more of the plurality of sensors; at least one processor configured to perform processing using the sensor data received from sensors, and generate the control values to directly drive any of the plurality of actuators; and a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.
The autonomous system of claim 2, wherein each processing node of the plurality of processing nodes is further configured to: read signals from a set of sensors of the plurality of sensors; interpret the signals as sets of sensor values over time; and perform the sensor fusion processing over the set of sensors based on the sets of sensor values, wherein each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.
The autonomous system of claim 3, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.
The autonomous system of claim 4, wherein each processing node of the plurality of processing nodes is configured to: determine control values based on a result of the sensor fusion processing; communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and provide processing to minimize the difference, wherein throttle-by-wire, brake-by-wire and steer-by-wire control is provided for a vehicle.
The autonomous system of claim 4, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).
The autonomous system of claim 1, wherein each processing node of the plurality of processing nodes is further configured to: determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting.
A method for sensor fusion and autonomous control comprising: receiving sensor signals from a plurality of sensors; interpreting sensor signals as sets of sensor values over time; performing distributed sensor fusion on the sets of sensor values by assigning a weight to each sensor of the plurality of sensors; partitioning and mapping processing tasks for the sets of sensor values to a set of distributed processing nodes; merging and reducing results of individual processing tasks performed by the distributed processing nodes into an actionable table, where such actionable table is used to specify objectives comprising control values resulting from sensor fusion processing, the objectives are used for an autonomous system; directly controlling a plurality of actuators using the actionable table for throttle-by-wire, brake-by-wire and steer-by-wire control, wherein the plurality of actuators includes at least one power control actuator, at least one brake actuator and at least one steering actuator; and transferring, by each of the distributed processing nodes, a workload from one or more processing nodes determined to have a fault to one or more active processing nodes wherein a reconfiguration of processing nodes is achieved within a bounded time.
The method of claim 8, further comprising: storing, by each of the distributed processing nodes, sensor data received from at least one sensor of the plurality of sensors; processing, by each of the distributed processing nodes, the sets of sensor values; generating, by each of the distributed processing nodes, the control values to directly drive any of the plurality of actuators; and communicating, by each of the distributed processing nodes, between the plurality of processing nodes, with the plurality of sensors, and with the plurality of actuators.
The method of claim 9, further comprising: reading, by each of the distributed processing nodes, signals from a set of sensors of the plurality of sensors; interpreting, by each of the distributed processing nodes, the signals as sets of sensor values over time; and performing, by each of the distributed processing nodes, sensor fusion processing over the set of sensors based on the sets of sensor values, wherein each processing node of the distributed processing nodes uses map-reduce operations for the distributed sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.
The method of claim 10, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and determining estimations based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.
The method of claim 11, further comprising: determining, by each of the distributed processing nodes, control values based on a result of the sensor fusion processing; communicating, by each of the distributed processing nodes, the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measuring, by each of the distributed processing nodes, a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; processing, by each of the distributed processing nodes, to minimize the difference; and providing throttle-by-wire, brake-by-wire and steer-by-wire control for a vehicle.
The method of claim 11, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).
The method of claim 8, further comprising: determining, by each of the distributed processing nodes, sensor failure; and compensating, by each of the distributed processing nodes, for the sensor failure by decreasing failed sensor importance weighting.
An autonomous computing environment for x-by-wire control of a vehicle comprising: an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control a plurality of actuators for the computing environment, the plurality of actuators including at least one power control actuator, at least one brake actuator and at least one steering actuator; and a plurality of processing nodes distributed and coupled to one another, each of the processing nodes are configured to receive sensor data from a plurality of sensors and to control the plurality of actuators to directly control vehicle devices for throttle-by-wire, brake-by-wire and steer-by-wire control; wherein each processing node of the plurality of processing nodes is configured to: partition and map processing tasks for distribution between the plurality of processing nodes; merge and reduce results of individual processing tasks into the actionable table; and transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.
The autonomous computing environment of claim 15, wherein each of the processing nodes comprise: at least one memory device configured to store sensor data received from one or more of the plurality of sensors; at least one processor configured to perform processing using the sensor data received from sensors, and generate control values to directly drive any of the plurality of actuators; and a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.
The autonomous computing environment system of claim 16, wherein each processing node of the plurality of processing nodes is further configured to: read signals from a set of sensors of the plurality of sensors; interpret the signals as sets of sensor values over time; and perform sensor fusion processing over the set of sensors based on the sets of sensor values, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the set of sensors, and each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.
The autonomous computing environment of claim 17, wherein each processing node of the plurality of processing nodes is configured to: determine control values based on a result of the sensor fusion processing; communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and provide processing to minimize the difference.
The autonomous computing environment of claim 17, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).
The autonomous computing environment of claim 15, wherein the plurality of processing nodes is further configured to: determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting.

Description

Safety-critical systems are increasingly connected, networked, and may rely on advanced computers for control. Fly-by-wire systems were developed within the aerospace industry for the purpose of decoupling control surfaces from direct pilot input, focusing on the intent of the pilot. Similarly, in the automotive industry, Anti-lock Brake Systems (ABS) have evolved into sophisticated Advanced Driving Assistance Systems (ADAS) that help drivers better deal with emergency situations, as well as routine tasks such as a “smart” cruise control.

Recently, there has been interest in autonomous vehicles. A notable example is the Self-driving Car, that is able to navigate through public streets through the use of an advanced Light Detection and Ranging (LIDAR) system coupled with other sensors. Several proposals have propped up suggesting deployment of autonomous drones for the purpose of delivering products to customers, and performing autonomous machine work for the hazard-prone mining environment.

Embodiments of the invention relate to scalable sensor fusion and autonomous x-by-wire control. In one embodiment, an autonomous system for x-by-wire control includes processing nodes distributed and connected to one another. Sensors are connected to the processing nodes. Actuators are configured to directly control the autonomous system driven by and connected to the processing nodes for x-by-wire control.

Citations (14)

US6151539A
GB2345153A
US6434698B1
US20030195938A1
US20030184158A1
US6747365B2
US8166096B1
US20040099468A1
US7272496B2
US8179787B2
US20140306833A1
US20160321848A1
US9720419B2
US20140306814A1

Record as JSON

{
  "publication_number": "US10112606B2",
  "country": "US",
  "kind": "B2",
  "title": "Scalable sensor fusion and autonomous x-by-wire control",
  "abstract": "An autonomous system for x-by-wire control includes processing nodes distributed and connected to one another. Sensors are connected to the processing nodes. Actuators are configured to directly control the autonomous system driven by and connected to the processing nodes for x-by-wire control. The processing nodes are configured to: partition and map processing tasks between the processing nodes, and merge and reduce results of individual processing tasks into an actionable table that specifies objectives for the autonomous system.",
  "claims": [
    "1. An autonomous system comprising: a plurality of processing nodes distributed and coupled to one another; a plurality of sensors coupled to the processing nodes; and a plurality of actuators including at least one power control actuator, at least one brake actuator, and at least one steering actuator, the plurality of actuators configured to directly control the autonomous system driven by and coupled to the plurality of processing nodes for throttle-by-wire, brake-by-wire and steer-by-wire control; wherein each processing node of the plurality of processing nodes is configured to: partition and map processing tasks between the plurality of processing nodes; merge and reduce results of individual processing tasks into an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control the plurality of actuators for the autonomous system; and transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.",
    "2. The autonomous system of claim 1, wherein each of the processing nodes comprise: at least one memory device configured to store sensor data received from one or more of the plurality of sensors; at least one processor configured to perform processing using the sensor data received from sensors, and generate the control values to directly drive any of the plurality of actuators; and a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.",
    "3. The autonomous system of claim 2, wherein each processing node of the plurality of processing nodes is further configured to: read signals from a set of sensors of the plurality of sensors; interpret the signals as sets of sensor values over time; and perform the sensor fusion processing over the set of sensors based on the sets of sensor values, wherein each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.",
    "4. The autonomous system of claim 3, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.",
    "5. The autonomous system of claim 4, wherein each processing node of the plurality of processing nodes is configured to: determine control values based on a result of the sensor fusion processing; communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and provide processing to minimize the difference, wherein throttle-by-wire, brake-by-wire and steer-by-wire control is provided for a vehicle.",
    "6. The autonomous system of claim 4, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).",
    "7. The autonomous system of claim 1, wherein each processing node of the plurality of processing nodes is further configured to: determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting.",
    "8. A method for sensor fusion and autonomous control comprising: receiving sensor signals from a plurality of sensors; interpreting sensor signals as sets of sensor values over time; performing distributed sensor fusion on the sets of sensor values by assigning a weight to each sensor of the plurality of sensors; partitioning and mapping processing tasks for the sets of sensor values to a set of distributed processing nodes; merging and reducing results of individual processing tasks performed by the distributed processing nodes into an actionable table, where such actionable table is used to specify objectives comprising control values resulting from sensor fusion processing, the objectives are used for an autonomous system; directly controlling a plurality of actuators using the actionable table for throttle-by-wire, brake-by-wire and steer-by-wire control, wherein the plurality of actuators includes at least one power control actuator, at least one brake actuator and at least one steering actuator; and transferring, by each of the distributed processing nodes, a workload from one or more processing nodes determined to have a fault to one or more active processing nodes wherein a reconfiguration of processing nodes is achieved within a bounded time.",
    "9. The method of claim 8, further comprising: storing, by each of the distributed processing nodes, sensor data received from at least one sensor of the plurality of sensors; processing, by each of the distributed processing nodes, the sets of sensor values; generating, by each of the distributed processing nodes, the control values to directly drive any of the plurality of actuators; and communicating, by each of the distributed processing nodes, between the plurality of processing nodes, with the plurality of sensors, and with the plurality of actuators.",
    "10. The method of claim 9, further comprising: reading, by each of the distributed processing nodes, signals from a set of sensors of the plurality of sensors; interpreting, by each of the distributed processing nodes, the signals as sets of sensor values over time; and performing, by each of the distributed processing nodes, sensor fusion processing over the set of sensors based on the sets of sensor values, wherein each processing node of the distributed processing nodes uses map-reduce operations for the distributed sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.",
    "11. The method of claim 10, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, and determining estimations based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the plurality of sensors.",
    "12. The method of claim 11, further comprising: determining, by each of the distributed processing nodes, control values based on a result of the sensor fusion processing; communicating, by each of the distributed processing nodes, the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measuring, by each of the distributed processing nodes, a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; processing, by each of the distributed processing nodes, to minimize the difference; and providing throttle-by-wire, brake-by-wire and steer-by-wire control for a vehicle.",
    "13. The method of claim 11, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).",
    "14. The method of claim 8, further comprising: determining, by each of the distributed processing nodes, sensor failure; and compensating, by each of the distributed processing nodes, for the sensor failure by decreasing failed sensor importance weighting.",
    "15. An autonomous computing environment for x-by-wire control of a vehicle comprising: an actionable table that specifies objectives comprising control values resulting from sensor fusion processing, the objectives are used to directly control a plurality of actuators for the computing environment, the plurality of actuators including at least one power control actuator, at least one brake actuator and at least one steering actuator; and a plurality of processing nodes distributed and coupled to one another, each of the processing nodes are configured to receive sensor data from a plurality of sensors and to control the plurality of actuators to directly control vehicle devices for throttle-by-wire, brake-by-wire and steer-by-wire control; wherein each processing node of the plurality of processing nodes is configured to: partition and map processing tasks for distribution between the plurality of processing nodes; merge and reduce results of individual processing tasks into the actionable table; and transfer a workload from one or more processing nodes determined to have a fault to one or more active processing nodes, wherein a reconfiguration of processing nodes is achieved within a bounded time.",
    "16. The autonomous computing environment of claim 15, wherein each of the processing nodes comprise: at least one memory device configured to store sensor data received from one or more of the plurality of sensors; at least one processor configured to perform processing using the sensor data received from sensors, and generate control values to directly drive any of the plurality of actuators; and a networking interface configured to provide communication with the plurality of processing nodes, the plurality of sensors, and the plurality of actuators.",
    "17. The autonomous computing environment system of claim 16, wherein each processing node of the plurality of processing nodes is further configured to: read signals from a set of sensors of the plurality of sensors; interpret the signals as sets of sensor values over time; and perform sensor fusion processing over the set of sensors based on the sets of sensor values, wherein sensor fusion processing comprises weighting importance of the sets of sensor values received from the set of sensors towards a pre-defined objective, estimation determination based on Kalman filtering with sensor value matrice sizing that is proportional to a total number of sensors in the set of sensors, and each processing node of the plurality of processing nodes uses map-reduce operations for the sensor fusion processing to process the sensor data to provide the control values from the actionable table to the plurality of actuators.",
    "18. The autonomous computing environment of claim 17, wherein each processing node of the plurality of processing nodes is configured to: determine control values based on a result of the sensor fusion processing; communicate the control values to the plurality of actuators within a bounded time period from a time when the signals are received from the set of sensors; measure a difference between an intended behavior for the plurality of actuators based on the control values communicated and an actual behavior output from the plurality of actuators based on the control values; and provide processing to minimize the difference.",
    "19. The autonomous computing environment of claim 17, wherein: the results of the individual processing tasks stored in the actionable table are determined in real-time within a bounded amount of time from a time when the sensor signals are read; the plurality of sensors comprises one or more: camera devices, light detection and ranging (LIDAR) sensors, and a global positioning sensor (GPS); and the plurality of actuators comprises one or more: an electronic power control, a steering control unit (SCU), and braking units (BUs).",
    "20. The autonomous computing environment of claim 15, wherein the plurality of processing nodes is further configured to: determine sensor failure and compensate for the sensor failure by decreasing failed sensor importance weighting."
  ],
  "description_excerpt": "Safety-critical systems are increasingly connected, networked, and may rely on advanced computers for control. Fly-by-wire systems were developed within the aerospace industry for the purpose of decoupling control surfaces from direct pilot input, focusing on the intent of the pilot. Similarly, in the automotive industry, Anti-lock Brake Systems (ABS) have evolved into sophisticated Advanced Driving Assistance Systems (ADAS) that help drivers better deal with emergency situations, as well as routine tasks such as a “smart” cruise control.\n\nRecently, there has been interest in autonomous vehicles. A notable example is the Self-driving Car, that is able to navigate through public streets through the use of an advanced Light Detection and Ranging (LIDAR) system coupled with other sensors. Several proposals have propped up suggesting deployment of autonomous drones for the purpose of delivering products to customers, and performing autonomous machine work for the hazard-prone mining environment.\n\nEmbodiments of the invention relate to scalable sensor fusion and autonomous x-by-wire control. In one embodiment, an autonomous system for x-by-wire control includes processing nodes distributed and connected to one another. Sensors are connected to the processing nodes. Actuators are configured to directly control the autonomous system driven by and connected to the processing nodes for x-by-wire control.",
  "cpc": [
    "B60W 30/00",
    "B60T 2270/402",
    "B60T 2270/82",
    "B60T 7/18",
    "B60T 7/22",
    "B60T 8/171",
    "B60T 8/885",
    "B60W 2420/403",
    "B60W 2420/408",
    "B60W 2420/42",
    "B60W 2420/52",
    "B62D 15/025",
    "G01S 17/87",
    "G01S 19/13",
    "G05D 1/0088",
    "H04L 67/10",
    "H04L 67/12"
  ],
  "ipc": [
    "B60T 7/22",
    "B60W 30/00",
    "B62D 15/02",
    "G01S 17/87",
    "G01S 19/13",
    "G05D 1/00",
    "H04L 29/08",
    "B60T 7/18",
    "B60T 8/171",
    "B60T 8/88"
  ],
  "assignees": [
    "International Business Machines Corp"
  ],
  "inventors": [
    "Rakesh Jain",
    "Gabor Madl",
    "Ramani R. Routray",
    "Yang Song"
  ],
  "filing_date": "2016-01-22",
  "publication_date": "2018-10-30",
  "grant_date": "2018-10-30",
  "priority_date": "2016-01-22",
  "application_number": "US-201615004718-A",
  "family_id": "59358818",
  "cited_by_count": 3,
  "citations": [
    "US6151539A",
    "GB2345153A",
    "US6434698B1",
    "US20030195938A1",
    "US20030184158A1",
    "US6747365B2",
    "US8166096B1",
    "US20040099468A1",
    "US7272496B2",
    "US8179787B2",
    "US20140306833A1",
    "US20160321848A1",
    "US9720419B2",
    "US20140306814A1"
  ]
}

Record 3,216 of 8,000 in Patents full text (MLC-0201). Request the full dataset.