Interaction with physical objects as proxy objects representing virtual objects

{
  "publication_number": "US11069079B2",
  "country": "US",
  "kind": "B2",
  "title": "Interaction with physical objects as proxy objects representing virtual objects",
  "abstract": "Systems and techniques for enabling interaction with physical objects as proxy objects representing virtual objects are provided herein. Virtual reality application data associated with a virtual reality application executed on a virtual reality device, a first virtual reality object data associated with a first virtual reality object from the virtual reality application, and virtual reality event data associated with one or more events from the virtual reality application may be received. Robotic arms including a robotic hand may grasp a first physical object which corresponds to the first virtual reality object of the virtual reality application. Sensors may detect a user interaction with the first physical object. Force feedback instructions commanding the robotic arms to move while maintaining grasp of the first physical object may be generated and executed based on detecting the user interaction with the first physical object and based on the virtual reality event data.",
  "claims": [
    "1. A robotic system for enabling interaction with physical objects as proxy objects representing virtual objects, comprising: a robotic arm including a portion grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application; a sensor detecting a user interaction with the first physical object; and a force feedback module of a processor generating a force feedback instruction for the robotic arm, wherein the force feedback instruction commands the robotic arm and the portion to maintain grasp of the first physical object and move according to virtual reality event data, the processor executing the force feedback instruction based on detecting the user interaction with the first physical object.",
    "2. The robotic system of claim 1, comprising a motor control module of the processor generating a motor control release instruction for the robotic arm, wherein the motor control release instructions command the robotic arm to release grasping of the first physical object, wherein the processor executes the motor control release instruction based on the virtual reality event data.",
    "3. The robotic system of claim 2, wherein the motor control module of the processor generates a motor control grasp instruction for the robotic arm, wherein the motor control grasp instructions command the robotic arm to grasp a second physical object, wherein the processor executes the motor control grasp instruction based on the virtual reality event data.",
    "4. The robotic system of claim 2, comprising a movement portion, wherein the sensor detects a position of a second physical object relative to the robotic system; wherein the motor control module of the processor generates a motor control movement instruction for the movement portion; wherein the motor control movement instructions command the movement portion to move the robotic system within a predetermined distance of the second physical object; and wherein the processor executes the motor control movement instruction based on the virtual reality event data and a position of the user relative to the second physical object.",
    "5. The robotic system of claim 1, wherein the sensor detects a position of a user relative to the first physical object.",
    "6. The robotic system of claim 1, comprising a communication interface receiving a position of a user relative to the first physical object from a virtual reality device.",
    "7. The robotic system of claim 6, wherein the sensor measures an amount of force associated with the user interaction of the first physical object.",
    "8. The robotic system of claim 7, wherein the communication interface transmits data associated with the amount of force from the user interaction to the virtual reality device as virtual reality event data.",
    "9. The robotic system of claim 1, comprising: a virtual reality event module of the processor generating a second set of virtual reality event data based on the user interaction with the first physical object, wherein the first virtual reality object is a control device within the virtual reality application; and a communication interface transmitting the second set of virtual reality event data to a virtual reality device.",
    "10. The robotic system of claim 1, wherein the virtual reality event data is indicative of an event from the virtual reality application including a user approaching the first virtual reality object.",
    "11. A method for enabling interaction with physical objects as proxy objects representing virtual objects using a robotic system, comprising: grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application using a robotic arm including a portion; detecting a user interaction with the first physical object; generating a force feedback instruction for the robotic arm, wherein the force feedback instruction commands the robotic arm and the portion to maintain grasp of the first physical object and move according to virtual reality event data; and executing the force feedback instruction based on detecting the user interaction with the first physical object.",
    "12. The method of claim 11, comprising: generating a motor control release instruction for the robotic arm, wherein the motor control release instructions command the robotic arm to release grasping of the first physical object; and executing the motor control release instruction based on the virtual reality event data.",
    "13. The method of claim 12, comprising: generating a motor control grasp instruction for the robotic arm, wherein the motor control grasp instructions command the robotic arm to grasp a second physical object; and executing the motor control grasp instruction based on the virtual reality event data.",
    "14. The method of claim 12, comprising: detecting a position of a second physical object relative to the robotic system; generating a motor control movement instruction for a movement portion to command the movement portion to move the robotic system within a predetermined distance of the second physical object; and executing the motor control movement instruction based on the virtual reality event data and a position of the user relative to the second physical object.",
    "15. The method of claim 11, comprising detecting a position of a user relative to the first physical object.",
    "16. The method of claim 11, comprising receiving a position of a user relative to the first physical object from a virtual reality device.",
    "17. The method of claim 11, comprising measuring an amount of force associated with the user interaction of the first physical object.",
    "18. The method of claim 17, comprising transmitting data associated with the amount of force from the user interaction to the virtual reality device as virtual reality event data.",
    "19. The method of claim 11, wherein the virtual reality event data is indicative of an event from the virtual reality application including a user approaching the first virtual reality object.",
    "20. A robotic system for enabling interaction with physical objects as proxy objects representing virtual objects, comprising: a first robotic arm including a portion grasping a first physical object which corresponds to a first virtual reality object of a virtual reality application; a second robotic arm including a portion grasping the first physical object which corresponds to the first virtual reality object of the virtual reality application; a sensor detecting a user interaction with the first physical object; and a force feedback module of a processor generating force feedback instructions for each of the first robotic arm and the second robotic arm, wherein the force feedback instructions commands the first robotic arm and the second robotic arm to maintain grasp of the first physical object and move or rotate according to virtual reality event data, the processor executing the force feedback instruction based on detecting the user interaction with the first physical object."
  ],
  "description_excerpt": "Robots have been built for many purposes, and are often utilized in situations or scenarios as substitutes for humans. For example, robots may be used in environments such as bomb detection and de-activation, manufacturing processes, or other similar environments. Robots are often built to replicate human actions, such as walking, lifting, speech, cognition, etc. For example, a robot may recognize moving objects, postures, gestures, a surrounding environment, sounds or faces, or interact with humans, such as by facing a person when approached.\n\nAccording to one aspect, a robotic system for enabling interaction with physical objects as proxy objects representing virtual objects is provided. The robotic system may include a communication interface, a memory, one or more robotic arms, one or more sensors, and a processor. The communication interface may receive virtual reality application data associated with a virtual reality application executed on a virtual reality device, a first virtual reality object data associated with a first virtual reality object from the virtual reality application, and virtual reality event data associated with one or more events from the virtual reality application. The memory may store the virtual reality application data, the first virtual reality object data, and the virtual reality event data as one or more instructions. One or more of the robotic arms may include a robotic hand grasping a first physical object which corresponds to the first virtual reality object of the virtual reality application. One or more of the sensors may detect a user interaction with the first physical object.",
  "cpc": [
    "G06T 7/70",
    "B25J 9/1694",
    "G05B 2219/39014",
    "G06F 3/011",
    "G06F 3/016",
    "G06F 3/04815"
  ],
  "ipc": [
    "B25J 9/16",
    "G06F 3/01",
    "G06F 3/0481",
    "G06T 7/70"
  ],
  "assignees": [
    "HONDA MOTOR CO LTD"
  ],
  "inventors": [
    "MURRISH ROBERT WESLEY",
    "MOON JOHN P"
  ],
  "filing_date": "2019-05-10",
  "publication_date": "2021-07-20",
  "grant_date": "2021-07-20",
  "priority_date": "2017-03-31",
  "application_number": "US-201916408917-A",
  "family_id": "63670786",
  "citations": [
    "DE102020101675A1",
    "JP2020095703A",
    "US2002133264A1",
    "US2005148432A1",
    "US2007078564A1",
    "US2009234788A1",
    "US2015235423A1",
    "US2016349835A1",
    "US2017010607A1",
    "US2017010671A1",
    "US2017161555A1",
    "US2018150387A1",
    "US2018182094A1",
    "US2018284760A1",
    "US2018341386A1",
    "US2019000578A1",
    "US2019005838A1",
    "US2019005848A1",
    "US2019022864A1",
    "US2019361518A1",
    "US2020082628A1",
    "US4839838A",
    "US5577981A",
    "US5694013A",
    "US5858291A",
    "US6104379A",
    "US6126373A",
    "US7920124B2",
    "US8108190B2",
    "US8244402B2",
    "US8487873B2",
    "WO2017030840A1"
  ]
}