Systems, devices, articles, and methods for creating and using trained robots with augmented reality

{
  "publication_number": "US10671240B2",
  "country": "US",
  "kind": "B2",
  "title": "Systems, devices, articles, and methods for creating and using trained robots with augmented reality",
  "abstract": "Substantially as described and illustrated herein including devices, methods of operation for the systems or devices, articles of manufacture including stores processor-executable instructions, and a system including a robot. The system includes at least one processor. The system may further include a nontransitory processor-readable storage device communicatively coupled to at least one processor and which stores processor-executable instructions which, when executed by the at least one processor, cause the at least one processor to composite environment information that represents an environment and virtual item information that represents the virtual item to produce composited information, present to an agent the composited information, and receive action information that represents an action for the robot to perform via the output system.",
  "claims": [
    "1. A system comprising: a robot including an input subsystem and an output subsystem; at least one processor communicatively coupled to the input subsystem and the output subsystem; and at least one nontransitory processor-readable storage device communicatively coupled to the at least one processor and which stores processor-executable instructions which, when executed by the at least one processor, cause the at least one processor to: receive, at the input subsystem, at least one input signal from at least one sensor wherein the input signal includes environment information that represents an environment to the robot; create virtual item information that represents a virtual item; composite the environment information that represents an environment and the virtual item information that represents the virtual item to produce composited information; train a machine learning model with at least a portion of the composited information to generate a trained machine learning model; and create at least one output signal that includes information that represents an action for the robot to perform via the output subsystem based, at least in part, on the trained machine learning model.",
    "2. The system of claim 1 wherein the input subsystem is selected from the group consisting of a sensory subsystem and a network interface device.",
    "3. The system of claim 1 wherein the output subsystem is selected from the group consisting of a motion subsystem and a manipulation subsystem.",
    "4. The system of claim 1 further comprising an operator interface communicatively coupled to the at least one processor.",
    "5. The system of claim 4, wherein, when executed, the processor-executable instructions further cause the at least one processor to: send to the operator interface the composited information; and receive, at the robot from the operator interface, operator generated processor-executable robot control instructions which, when executed, cause the robot to perform the action.",
    "6. The system of claim 1, wherein, when executed, the processor-executable instructions further cause the at least one processor to: receive, at the robot, autonomous processor-executable robot control instructions which, when executed, cause the robot to perform the action.",
    "7. The system of claim 1 further comprising: a communication channel communicatively coupled to the at least one processor; and wherein, when executed, the processor-executable instructions further cause the at least one processor to: send the at least one output signal that includes information that represents the action for the robot to perform via the output subsystem.",
    "8. The system of claim 1, wherein, when executed, the processor-executable instructions further cause the at least one processor to: select the action for the robot to perform via the output subsystem based on the composited information.",
    "9. The system of claim 8, wherein, in order to select the action for the robot to perform via the output subsystem based on the composited information, when executed, the processor-executable instructions cause the at least one processor to further: select the action in view of the composited information wherein the composited information includes training data; select the action in view of the composited information wherein the composited information includes validation data; or select the action in view of the composited information wherein the composited information includes test data.",
    "10. The system of claim 1, wherein to composite the environment information that represents an environment to the robot and the virtual item information that represents the virtual item, when executed, the processor-executable instructions cause the at least one processor to further: receive pose information that represents a pose of the robot in the environment; and update the virtual item information that represents the virtual item based on the pose information that represents a pose of the robot in the environment.",
    "11. A method of operation for a system including at least one processor, at least one sensor in communication with the at least one processor, and a robot including an output subsystem in communication with the at least one processor, the method comprising: receiving, at the at least one processor, at least one input signal from the at least one sensor wherein the input signal includes environment information that represents an environment to the robot; creating, by the at least one processor, item information that represents a virtual item; compositing, by the at least one processor, the environment information that represents the environment to the robot and the item information that represents the virtual item to produce composited information that represents the environment to the robot and the item information that represents the virtual item; training, by the at least one processor, a machine learning model with at least a portion of the composited information to generate a trained machine learning model; and generating, by the at least one processor, at least one output signal that includes action information that represents an action for the robot to perform via the output subsystem based, at least in part, on the trained machine learning model.",
    "12. The method of claim 11 wherein: the robot is in communication with an operator interface; and the method further comprises: sending to the operator interface the composited information; and receiving, from the operator interface, operator generated processor-executable robot control instructions which, when executed, cause the robot to perform at least one of the action or another action.",
    "13. The method of claim 11, further comprising: receiving autonomous processor-executable robot control instructions which, when executed, cause the robot to perform the action via the output subsystem.",
    "14. The method of claim 11 further comprising: determining, by the at least one processor, the action information that represents an action for the robot to perform via the output subsystem from the composited information.",
    "15. The method of claim 14 wherein determining the action information that represents an action for the robot to perform via the output subsystem further comprises at least one of: selecting the action in view of the composited information, wherein the environment information that represents the environment to the robot is training data; selecting the action in view of the composited information, wherein the environment information that represents the environment to the robot is validation data; or selecting the action in view of the composited information, wherein the environment information that represents the environment to the robot is test data.",
    "16. The method of claim 11 wherein compositing the environment information that represents the environment to the robot and the item information that represents the virtual item further comprises: receiving, at the at least one processor, pose information that represents a pose of the robot in the environment; and updating, at the at least one processor, the item information based on the pose information.",
    "17. The method of claim 11 wherein compositing the environment information that represents the environment to the robot and the item information that represents the virtual item further comprises: rendering a representation of the virtual item disposed within the environment; and wherein, the representation is selected from the group consisting of: visual representation, haptic representation, and audio representation.",
    "18. The method of claim 11 wherein compositing the environment information that represents the environment to the robot and the item information that represents the virtual item further comprises at least one of: appending at least a part of the item information that represents the virtual item to the environment information that represents the environment to the robot; overlaying at least a part of the environment information that represents the environment to the robot with at least a part of the item information that represents the virtual item; or replacing at least a part of the environment information that represents the environment to the robot with at least a part of the item information that represents the virtual item.",
    "19. The method of claim 11 wherein the action information includes processor-executable instructions which, when executed, cause the robot to perform the action via the output subsystem.",
    "20. The method of claim 11 wherein the at least one processor is in communication with a nontransitory processor-readable storage device, the method further comprising: updating the nontransitory processor-readable storage device based on the at least one output signal that includes the action information that represents the action for the robot to perform via the output subsystem."
  ],
  "description_excerpt": "This disclosure generally relates to the field(s) of machine learning, data collection, augmented reality, and/or operation of robots.\n\nRobots\n\nRobots are systems, machines, or devices that are capable of carrying out one or more tasks. A robot is an electro-mechanical machine controlled by circuitry, for example a processor following processor-executable instructions; a human operator controllable electro-mechanical machine; a robotic subsystem of another machine including another robot; or the like. A robot has the ability to move in a physical space and to accomplish physical tasks. Robots may be operated by a human operator, such as, via remote control, or may operate autonomously without control of an operator. Hybrid robots exist in which some functions are autonomous while others are operator controlled or control switches between autonomous and operator controlled modes. As well, a robot includes computational resources to preform computational tasks. The computational tasks can be in aid of the physical tasks.\n\nMachine Learning\n\nA computer, which is a machine, can perform or succeed at one or more related tasks as defined by a measure. The computer learns if after exposure to information characterizing an event the computer improves under the measure at performing the one or more related tasks. Further, the computer learns without updates to any processor-executable instructions by imperative programming.",
  "cpc": [
    "G06F 3/04815",
    "B25J 13/089",
    "B25J 19/023",
    "B25J 5/007",
    "B25J 9/1697",
    "G06K 9/00671",
    "G06T 17/00",
    "G06T 19/006",
    "G06T 7/00",
    "G06V 20/20"
  ],
  "ipc": [
    "B25J 13/08",
    "B25J 19/02",
    "B25J 5/00",
    "B25J 9/16",
    "G06F 3/0481",
    "G06K 9/00",
    "G06T 17/00",
    "G06T 19/00",
    "G06T 7/00"
  ],
  "assignees": [
    "Kindred Systems Inc"
  ],
  "inventors": [
    "Suzanne Gildert",
    "Geordie S. Rose",
    "Dmytro Korenkevych",
    "Miles F. H. Steininger"
  ],
  "filing_date": "2018-06-05",
  "publication_date": "2020-06-02",
  "grant_date": "2020-06-02",
  "priority_date": "2017-06-05",
  "application_number": "US-201816000375-A",
  "family_id": "64459892",
  "cited_by_count": 4,
  "citations": [
    "US7298385B2",
    "US20160257000A1"
  ]
}