Voice controlled keyboard typing using computer vision

{
  "publication_number": "US10766143B1",
  "country": "US",
  "kind": "B1",
  "title": "Voice controlled keyboard typing using computer vision",
  "abstract": "Techniques for systems and methods that provide for utilizing a robotic system to type commands that correspond to voice commands using a keyboard of a device are described herein. In embodiments, an image of a device may be received from a camera. A keyboard region of the device may be determined based on a keyboard detection algorithm that uses the image. One or more characters in a portion of the image that corresponds to the keyboard region of the device may be detected based on a character detection algorithm. The one or more characters may be grouped into one or more groups based on the portion of the image. A character of a portion of character associated with a group may be edited based on an error detection algorithm.",
  "claims": [
    "1. A robot-assisted system for implementing speech-to-text commands on a device, the robot-assisted system comprising: a device mounting station configured to physically hold the device; a camera configured to captures images of the device while the device is mounted in the device mounting station; a robot arm having a tip configured to contact the device; a controller having a processor and memory containing executable instructions that, when executed by the processor, cause the controller to: receive an image of the device from the camera; detect a keyboard region of the device based at least in part on a keyboard detection algorithm that uses the image; detect one or more characters in the keyboard region based at least in part on a character detection algorithm that uses a portion of the image that corresponds to the keyboard region; generate one or more groups for the one or more characters based at least in part on the portion of the image that corresponds to the keyboard region and the one or more characters, a group of the one or more groups including characters of the one or more characters; update the characters in the one or more groups based at least in part on an error detection algorithm; and instruct the robot arm to perform a first plurality of movements that correspond to input provided by a speech-to-text device in communication with the robot-assisted system, the first plurality of movements including moving the robot arm to cause the tip to touch a plurality of specific locations on the device that correspond to the keyboard region.",
    "2. The robot-assisted system of claim 1, wherein the error detection algorithm includes identifying an occluded character in the one or more characters of the keyboard region based at least in part on the group, and wherein updating the characters in the one or more groups includes inserting a character that corresponds to the occluded character.",
    "3. The robot-assisted system of claim 1, wherein the error detection algorithm includes identifying a false positive character in the one or more characters of the keyboard region based at least in part on the group, and wherein updating the characters in the one or more groups includes removing a character that corresponds to the false positive character.",
    "4. The robot-assisted system of claim 1, wherein the error detection algorithm includes identifying a character in the one or more characters of the keyboard region based at least in part on the group, and wherein updating the characters in the one or more groups includes replacing the character.",
    "5. The robot-assisted system of claim 1, wherein the character detection algorithm includes associating each character of the one or more characters with a unique class, the unique class associated with a particular alpha-numeric character.",
    "6. The robot-assisted system of claim 1, wherein the executable instructions when executed by the processor further cause the controller to verify the input provided by the speech-to-text device based at least in part on comparing alpha-numeric characters of the input to the updated characters in the one or more groups.",
    "7. The robot-assisted system of claim 1, wherein the executable instructions when executed by the processor further cause the controller to instruct the robot arm to perform a second plurality of movements that initiate a state for the device that is associated with receiving the input.",
    "8. The robot-assisted system of claim 1, wherein the executable instructions when executed by the processor further cause the controller to generate a pixel-to-coordinate map of the device based at least in part on the portion of the image that corresponds to the keyboard region and the detected one or more characters.",
    "9. A computer-implemented method, comprising: receiving, by a computer system, an image of a device from a camera; detecting, by the computer system, a keyboard region of the device based at least in part on a keyboard detection algorithm that uses the image; detecting, by the computer system, one or more characters in a portion of the image that correspond to the keyboard region of the device based at least in part on a character detection algorithm; generating, by the computer system, one or more groups for the one or more characters based at least in part on the portion of the image that corresponds to the keyboard region and the one or more characters, a group of the one or more groups including a portion of characters of the one or more characters; and editing, by the computer system, a character of the portion of characters based at least in part on an error detection algorithm.",
    "10. The computer-implemented method of claim 9, wherein the keyboard detection algorithm includes identifying an orientation of the device.",
    "11. The computer-implemented method of claim 10, wherein generating the one or more groups for the one or more characters is further based at least in part on a horizontal orientation of the device.",
    "12. The computer-implemented method of claim 9, further comprising detecting the keyboard region of the device based at least in part on receiving a new image of the device.",
    "13. The computer-implemented method of claim 9, further comprising: identifying a first portion of upper case characters of the one or more characters and identifying a second portion of lower case characters of the one or more characters; and assigning an upper case or a lower case to the one or more characters based at least in part on a comparison of the first portion of the upper case characters and the second portion of the lower case characters.",
    "14. The computer-implemented method of claim 9, wherein the character detection algorithm utilizes a plurality of characters from a plurality of languages.",
    "15. A non-transitory computer readable medium storing specific computer-executable instructions that, when executed by a processor, cause a computer system to at least: detect a keyboard region of a device in an image of the device based at least in part on a keyboard detection algorithm that uses the image; detect one or more characters in a portion of the image that corresponds to the keyboard region of the device based at least in part on a character detection algorithm; and editing a character of the one or more characters in the portion of the image based at least in part on an error detection algorithm and in response to generating one or more groups for the one or more characters based at least in part on the portion of the image that corresponds to the keyboard region and the one or more characters, a group of the one or more groups including a portion of characters of the one or more characters.",
    "16. The non-transitory computer readable medium of claim 15, wherein the computer-executable instructions when executed by the processor further cause the computer system to instruct a robot arm with a tip to interact with the device, the interaction including a plurality of movements that correspond to input provided by a speech-to-text device in communication with the robot-arm and associated with the device.",
    "17. The non-transitory computer readable medium of claim 16, wherein the computer-executable instructions when executed by the processor further cause the computer system to detect the keyboard region in response to executing a movement of the plurality of movements.",
    "18. The non-transitory computer readable medium of claim 16, wherein the computer-executable instructions when executed by the processor further cause the computer system to verify the input based at least in part on a historical analysis of previous input provided via the speech-to-text device compared to the input.",
    "19. The non-transitory computer readable medium of claim 16, wherein the computer-executable instructions when executed by the processor further cause the computer system to verify the input based at least in part by generating and transmitting an audio representation of the input via an output component associated with the device.",
    "20. The non-transitory computer readable medium of claim 15, wherein the keyboard detection algorithm is configured to detect a plurality of keyboard regions of the device based at least in part on the image."
  ],
  "description_excerpt": "The use of robots to perform a variety of tasks has increased as advancements in technology are made. A robot can be pre-programmed to perform actions that are highly repetitive. However, a pre-programmed robot may fail to perform a desired action if the robot encounters new environments or circumstances. For example, a pre-programmed robot that encounters a situation for the first time may lack the intelligence to perform any function or perform an incorrect function given the circumstances. Conventional robots require human oversight and correction to aid in the decision-making ability of the robot.\n\nVarious embodiments in accordance with the present disclosure will be described with reference to the drawings, in which: FIG. 1 depicts an example robot assisted system for implementing a keyboard detection feature that includes a device mounting station, a camera, a robot arm, and a controller, in accordance with at least one embodiment; FIG. 2 depicts an example workflow for a keyboard detection feature, in accordance with at least one embodiment; FIG. 3 depicts an example workflow for a keyboard detection feature, in accordance with at least one embodiment; FIG. 4 depicts an example output from a conventional character recognition system using an image of a device; FIG. 5 depicts an example output from the keyboard detection feature using an image of a device, in accordance with at least one embodiment; FIG. 6 depicts a flowchart of a keyboard detection feature, in accordance with at least one embodiment; FIG. 7 depicts a flowchart of a keyboard detection feature, in accordance with at least one embodiment;",
  "cpc": [
    "G06F 3/0393",
    "B25J 9/163",
    "B25J 9/1697",
    "G05B 2219/35453",
    "G05B 2219/40041",
    "G05B 2219/40065",
    "G06F 3/023",
    "G06F 3/0233",
    "G06K 9/00671",
    "G06V 20/20",
    "G06V 30/10",
    "G06V 30/1456",
    "G10L 15/26"
  ],
  "ipc": [
    "B25J 9/16",
    "G06F 3/023",
    "G06V 30/10",
    "G10L 15/26"
  ],
  "assignees": [
    "AMAZON TECH INC"
  ],
  "inventors": [
    "LIU ZONGYI",
    "FERRY BRUCE",
    "CHEN KUN",
    "BECKER ANDREW"
  ],
  "filing_date": "2018-11-13",
  "publication_date": "2020-09-08",
  "grant_date": "2020-09-08",
  "priority_date": "2018-11-13",
  "application_number": "US-201816188738-A",
  "family_id": "72289984",
  "citations": [
    "US2008212877A1",
    "US7822990B2",
    "US9053350B1",
    "US9645655B2"
  ]
}