Patent · US10974389B2 · B2 · US

Methods and apparatus for early sensory integration and robust acquisition of real world knowledge

(11) Publication number

US10974389B2

(21) Application number

16/376,109

(22) Filing date

2019-04-05

(30) Priority date

2013-05-22

(43) Publication date

2021-04-13

(45) Date of grant

2021-04-13

(51) IPC

B25J 9/16; G06F 16/22; G06F 16/29; G06K 9/46; G06N 3/00; G06N 3/02; G06N 5/02

(52) CPC

G06N Computing arrangements based on specific computational models: 5/02, 3/008, 3/02
B25J Manipulators; chambers provided with manipulation devices: 9/1664, 9/1694, 9/1697
G06F Electric digital data processing: 16/2228, 16/29
G06K Graphical data reading; presentation of data; record carriers; handling record carriers: 9/4619, 9/4671
G06V Image or video recognition or understanding: 10/449, 10/462
Y04S Systems integrating technologies related to power network operation, communication or information technologies for improving the electrical power generation, transmission, distribution, management or usage, i.e. smart grids: 10/50
Y10S Technical subjects covered by former uspc cross-reference art collections [xracs] and digests: 901/01, 901/09, 901/46, 901/47

(73) Assignee

NEURALA INC

(72) Inventors

GORSHECHNIKOV ANATOLY; VERSACE MASSIMILIANO

(54) Title

Methods and apparatus for early sensory integration and robust acquisition of real world knowledge

(57) Abstract

The systems and methods disclosed herein include a path integration system that calculates optic flow, infers angular velocity from the flow field, and incorporates this velocity estimate into heading calculations. The resulting system fuses heading estimates from accelerometers, 5 gyroscopes, engine torques, and optic flow to determine self-localization. The system also includes a motivational system that implements a reward drive, both positive and negative, into the system. In some implementations, the drives can include: a) a curiosity drive that encourages exploration of new areas, b) a resource drive that attracts the agent towards the recharging base when the battery is low, and c) a mineral reward drive that attracts the agent 10 towards previously explored scientific targets.

Full text

View on Google Patents

Claims (24)

A system for training a neural network in a virtual environment before deploying the neural network on a first platform, the system comprising: a virtual engine to simulate sensory information using a simulated platform; a proxy engine to: transmit the simulated sensory information to the neural network; extract neural information from the neural network based on the simulated sensory information; and transmit control information to the simulated platform based on the extracted neural information; and an Application Programming Interface, communicatively coupled to the virtual engine and the proxy engine, to enable communication between the neural network and the simulated platform, wherein the virtual engine and the proxy engine system are configured to test the simulated platform.
The system of claim 1, wherein the Application Programming Interface is further configured to deploy the neural network to the first platform.
The system of claim 2, wherein the first platform is a robotic platform.
The system of claim 1, wherein the simulated platform includes at least one animat controlled by the neural network model.
The system of claim 4, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.
The system of claim 1, further comprising: a graphical user interface configured to enable a user to the control the system.
The system of claim 6, wherein the graphical user interface is further configured to enable a user to create at least one animat controlled by the neural network model.
A system for training a neural network in a virtual environment before deploying the neural network on a first platform, the system comprising: a virtual engine to simulate sensory information using a simulated platform; a proxy engine to: transmit the simulated sensory information to the neural network; extract neural information from the neural network based on the simulated sensory information; and transmit control information to the simulated platform based on the extracted neural information; an Application Programming Interface, communicatively coupled to the virtual engine and the proxy engine, to enable communication between the neural network and the simulated platform; and a simulation engine to generate the simulated platform.
A method for training a neural network in a virtual environment before deploying the neural network on a first platform, the method comprising: simulating, via a virtual engine, sensory information using a simulated platform; transmitting, via a proxy engine, the simulated sensory information to the neural network; extracting, via the proxy engine, neural information from the neural network based on the simulated sensory information; transmitting, via the proxy engine, control information to the simulated platform based on the extracted neural information; and testing, via the virtual engine and the proxy engine, the simulated platform.
The method of claim 9, further comprising: deploying, via an Application Programming Interface, the neural network to the first platform.
The method of claim 10, wherein the first platform is a robotic platform.
The method of claim 9, further comprising: controlling, via the neural network, at least one animat included in the simulated platform.
The method of claim 12, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.
A method for training a neural network in a virtual environment before deploying the neural network on a first platform, the method comprising: generating, via a simulation engine, a simulated platform; simulating, via a virtual engine, sensory information using the simulated platform; transmitting, via a proxy engine, the simulated sensory information to the neural network; extracting, via the proxy engine, neural information from the neural network based on the simulated sensory information; and transmitting, via the proxy engine, control information to the simulated platform based on the extracted neural information.
The method of claim 14, further comprising: deploying, via an Application Programming Interface, the neural network to the first platform.
The method of claim 15, wherein the first platform is a robotic platform.
The method of claim 14, further comprising: controlling, via the neural network, at least one animat included in the simulated platform.
The method of claim 17, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.
The system of claim 8, wherein the Application Programming Interface is further configured to deploy the neural network to the first platform.
The system of claim 19, wherein the first platform is a robotic platform.
The system of claim 8, wherein the simulated platform includes at least one animat controlled by the neural network model.
The system of claim 21, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.
The system of claim 8, further comprising: a graphical user interface configured to enable a user to the control the system.
The system of claim 23, wherein the graphical user interface is further configured to enable a user to create at least one animat controlled by the neural network model.

Description

To date there has been little success in development of robotic intelligent behaviors without reliance on resource-intensive active sensors such as but not limited to laser range finders, sonars, etc. Furthermore, these sensors are often tailored and used exclusively for individual tasks. At the same time, robots do have other multi-purpose passive sensors, but the precision of knowledge based on these sensors leaves much to be desired. The lack of reliable cross-task sensory system is partially due to sensory noise and partially to an approach of independent, stove-piped processing of individual sensory streams.

The system and methods described herein is based on a combination of an arbitrary number of sensory inputs correcting each other by compensating weaknesses of one sensor with the strengths of other sensors in the process of early fusion, then processing these sensory inputs through a redundant and robust neural-like system. An exemplary system may be used for a variety of applications, including but not limited to spatial navigation, visual object segmentation and recognition, or robotic attention. In some implementations, the system may generate a noise-tolerant distributed data representation model via generating a set of data cells collectively representing a sensory input data point in the range of measurement for a particular type of data, and may use the data to define a spatial or temporal resolution of the set of data cells (e.g., a data representation model) from the data range of the incoming sensory information and the number of cells in the set.

Citations (73)

EP1224622B1
US10083523B2
US10300603B2
US2002046271A1
US2002050518A1
US2002064314A1
US2002168100A1
US2003026588A1
US2003078754A1
US2004015334A1
US2005166042A1
US2006184273A1
US2007052713A1
US2007198222A1
US2007279429A1
US2008033897A1
US2008066065A1
US2008117220A1
US2008117720A1
US2008258880A1
US2009080695A1
US2009089030A1
US2009116688A1
US2010048242A1
US2010138153A1
US2011004341A1
US2011173015A1
US2011279682A1
US2012072215A1
US2012089295A1
US2012089552A1
US2012197596A1
US2012316786A1
US2013080641A1
US2013126703A1
US2013131985A1
US2014019392A1
US2014032461A1
US2014052679A1
US2014089232A1
US2014192073A1
US2015127149A1
US2015134232A1
US2015224648A1
US2015269439A1
US2016075017A1
US2016096270A1
US2016198000A1
US2017024877A1
US2017076194A1
US2017193298A1
US5063603A
US5136687A
US5172253A
US5388206A
US6018696A
US6336051B1
US6647508B2
US7765029B2
US7861060B1
US7873650B1
US8392346B2
US8510244B2
US8583286B2
US8648828B2
US8648867B2
US9031692B2
US9177246B2
US9189828B2
US9626566B2
WO2014204615A2
WO2015143173A2
WO2016014137A2

Record as JSON

{
  "publication_number": "US10974389B2",
  "country": "US",
  "kind": "B2",
  "title": "Methods and apparatus for early sensory integration and robust acquisition of real world knowledge",
  "abstract": "The systems and methods disclosed herein include a path integration system that calculates optic flow, infers angular velocity from the flow field, and incorporates this velocity estimate into heading calculations. The resulting system fuses heading estimates from accelerometers, 5 gyroscopes, engine torques, and optic flow to determine self-localization. The system also includes a motivational system that implements a reward drive, both positive and negative, into the system. In some implementations, the drives can include: a) a curiosity drive that encourages exploration of new areas, b) a resource drive that attracts the agent towards the recharging base when the battery is low, and c) a mineral reward drive that attracts the agent 10 towards previously explored scientific targets.",
  "claims": [
    "1. A system for training a neural network in a virtual environment before deploying the neural network on a first platform, the system comprising: a virtual engine to simulate sensory information using a simulated platform; a proxy engine to: transmit the simulated sensory information to the neural network; extract neural information from the neural network based on the simulated sensory information; and transmit control information to the simulated platform based on the extracted neural information; and an Application Programming Interface, communicatively coupled to the virtual engine and the proxy engine, to enable communication between the neural network and the simulated platform, wherein the virtual engine and the proxy engine system are configured to test the simulated platform.",
    "2. The system of claim 1, wherein the Application Programming Interface is further configured to deploy the neural network to the first platform.",
    "3. The system of claim 2, wherein the first platform is a robotic platform.",
    "4. The system of claim 1, wherein the simulated platform includes at least one animat controlled by the neural network model.",
    "5. The system of claim 4, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.",
    "6. The system of claim 1, further comprising: a graphical user interface configured to enable a user to the control the system.",
    "7. The system of claim 6, wherein the graphical user interface is further configured to enable a user to create at least one animat controlled by the neural network model.",
    "8. A system for training a neural network in a virtual environment before deploying the neural network on a first platform, the system comprising: a virtual engine to simulate sensory information using a simulated platform; a proxy engine to: transmit the simulated sensory information to the neural network; extract neural information from the neural network based on the simulated sensory information; and transmit control information to the simulated platform based on the extracted neural information; an Application Programming Interface, communicatively coupled to the virtual engine and the proxy engine, to enable communication between the neural network and the simulated platform; and a simulation engine to generate the simulated platform.",
    "9. A method for training a neural network in a virtual environment before deploying the neural network on a first platform, the method comprising: simulating, via a virtual engine, sensory information using a simulated platform; transmitting, via a proxy engine, the simulated sensory information to the neural network; extracting, via the proxy engine, neural information from the neural network based on the simulated sensory information; transmitting, via the proxy engine, control information to the simulated platform based on the extracted neural information; and testing, via the virtual engine and the proxy engine, the simulated platform.",
    "10. The method of claim 9, further comprising: deploying, via an Application Programming Interface, the neural network to the first platform.",
    "11. The method of claim 10, wherein the first platform is a robotic platform.",
    "12. The method of claim 9, further comprising: controlling, via the neural network, at least one animat included in the simulated platform.",
    "13. The method of claim 12, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.",
    "14. A method for training a neural network in a virtual environment before deploying the neural network on a first platform, the method comprising: generating, via a simulation engine, a simulated platform; simulating, via a virtual engine, sensory information using the simulated platform; transmitting, via a proxy engine, the simulated sensory information to the neural network; extracting, via the proxy engine, neural information from the neural network based on the simulated sensory information; and transmitting, via the proxy engine, control information to the simulated platform based on the extracted neural information.",
    "15. The method of claim 14, further comprising: deploying, via an Application Programming Interface, the neural network to the first platform.",
    "16. The method of claim 15, wherein the first platform is a robotic platform.",
    "17. The method of claim 14, further comprising: controlling, via the neural network, at least one animat included in the simulated platform.",
    "18. The method of claim 17, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.",
    "19. The system of claim 8, wherein the Application Programming Interface is further configured to deploy the neural network to the first platform.",
    "20. The system of claim 19, wherein the first platform is a robotic platform.",
    "21. The system of claim 8, wherein the simulated platform includes at least one animat controlled by the neural network model.",
    "22. The system of claim 21, wherein the at least one animat includes a plurality of simulated sensory organs and a plurality of animat controls.",
    "23. The system of claim 8, further comprising: a graphical user interface configured to enable a user to the control the system.",
    "24. The system of claim 23, wherein the graphical user interface is further configured to enable a user to create at least one animat controlled by the neural network model."
  ],
  "description_excerpt": "To date there has been little success in development of robotic intelligent behaviors without reliance on resource-intensive active sensors such as but not limited to laser range finders, sonars, etc. Furthermore, these sensors are often tailored and used exclusively for individual tasks. At the same time, robots do have other multi-purpose passive sensors, but the precision of knowledge based on these sensors leaves much to be desired. The lack of reliable cross-task sensory system is partially due to sensory noise and partially to an approach of independent, stove-piped processing of individual sensory streams.\n\nThe system and methods described herein is based on a combination of an arbitrary number of sensory inputs correcting each other by compensating weaknesses of one sensor with the strengths of other sensors in the process of early fusion, then processing these sensory inputs through a redundant and robust neural-like system. An exemplary system may be used for a variety of applications, including but not limited to spatial navigation, visual object segmentation and recognition, or robotic attention. In some implementations, the system may generate a noise-tolerant distributed data representation model via generating a set of data cells collectively representing a sensory input data point in the range of measurement for a particular type of data, and may use the data to define a spatial or temporal resolution of the set of data cells (e.g., a data representation model) from the data range of the incoming sensory information and the number of cells in the set.",
  "cpc": [
    "G06N 5/02",
    "B25J 9/1664",
    "B25J 9/1694",
    "B25J 9/1697",
    "G06F 16/2228",
    "G06F 16/29",
    "G06K 9/4619",
    "G06K 9/4671",
    "G06N 3/008",
    "G06N 3/02",
    "G06V 10/449",
    "G06V 10/462",
    "Y04S 10/50",
    "Y10S 901/01",
    "Y10S 901/09",
    "Y10S 901/46",
    "Y10S 901/47"
  ],
  "ipc": [
    "B25J 9/16",
    "G06F 16/22",
    "G06F 16/29",
    "G06K 9/46",
    "G06N 3/00",
    "G06N 3/02",
    "G06N 5/02"
  ],
  "assignees": [
    "NEURALA INC"
  ],
  "inventors": [
    "GORSHECHNIKOV ANATOLY",
    "VERSACE MASSIMILIANO"
  ],
  "filing_date": "2019-04-05",
  "publication_date": "2021-04-13",
  "grant_date": "2021-04-13",
  "priority_date": "2013-05-22",
  "application_number": "US-201916376109-A",
  "family_id": "51934352",
  "citations": [
    "EP1224622B1",
    "US10083523B2",
    "US10300603B2",
    "US2002046271A1",
    "US2002050518A1",
    "US2002064314A1",
    "US2002168100A1",
    "US2003026588A1",
    "US2003078754A1",
    "US2004015334A1",
    "US2005166042A1",
    "US2006184273A1",
    "US2007052713A1",
    "US2007198222A1",
    "US2007279429A1",
    "US2008033897A1",
    "US2008066065A1",
    "US2008117220A1",
    "US2008117720A1",
    "US2008258880A1",
    "US2009080695A1",
    "US2009089030A1",
    "US2009116688A1",
    "US2010048242A1",
    "US2010138153A1",
    "US2011004341A1",
    "US2011173015A1",
    "US2011279682A1",
    "US2012072215A1",
    "US2012089295A1",
    "US2012089552A1",
    "US2012197596A1",
    "US2012316786A1",
    "US2013080641A1",
    "US2013126703A1",
    "US2013131985A1",
    "US2014019392A1",
    "US2014032461A1",
    "US2014052679A1",
    "US2014089232A1",
    "US2014192073A1",
    "US2015127149A1",
    "US2015134232A1",
    "US2015224648A1",
    "US2015269439A1",
    "US2016075017A1",
    "US2016096270A1",
    "US2016198000A1",
    "US2017024877A1",
    "US2017076194A1",
    "US2017193298A1",
    "US5063603A",
    "US5136687A",
    "US5172253A",
    "US5388206A",
    "US6018696A",
    "US6336051B1",
    "US6647508B2",
    "US7765029B2",
    "US7861060B1",
    "US7873650B1",
    "US8392346B2",
    "US8510244B2",
    "US8583286B2",
    "US8648828B2",
    "US8648867B2",
    "US9031692B2",
    "US9177246B2",
    "US9189828B2",
    "US9626566B2",
    "WO2014204615A2",
    "WO2015143173A2",
    "WO2016014137A2"
  ]
}

Record 682 of 5,000 in Patents full text (MLC-0201). Request the full dataset.