Systems and methods for presenting augmented reality in a display of a teleoperational system

{
  "publication_number": "US11351005B2",
  "country": "US",
  "kind": "B2",
  "title": "Systems and methods for presenting augmented reality in a display of a teleoperational system",
  "abstract": "A method comprises displaying a surgical environment image. The surgical environment image includes a virtual control element for controlling a component of a surgical system, and the virtual control element includes a real-time image of the component of the surgical system in the surgical environment image. The method further comprises displaying an image of a body part of a user. The body part is used to interact with the virtual control element. The method further comprises receiving a gesture of the body part of the user in a predetermined motion, via a gesture based input device registering movement of the body part of the user, while the body part interacts with the virtual control element. The method further comprises adjusting a setting of the component of the surgical system based on the received gesture.",
  "claims": [
    "1. A method comprising: displaying a surgical environment image, wherein the surgical environment image includes a virtual control element for controlling a component of a surgical system, and wherein the virtual control element includes a real-time image of the component of the surgical system in the surgical environment image; displaying an image of a body part of a user, the body part used to interact with the virtual control element; receiving a gesture of the body part of the user in a predetermined motion, via a gesture based input device registering movement of the body part of the user, while the body part interacts with the virtual control element; and adjusting a setting of the component of the surgical system based on the received gesture.",
    "2. The method of claim 1 wherein the virtual control element includes a graphical element superimposed on the surgical environment image.",
    "3. The method of claim 1 wherein the gesture based input device is configured to receive a three dimensional user input.",
    "4. The method of claim 1 wherein the gesture based input device includes at least one of a tablet device or a user wearable device.",
    "5. The method of claim 1 wherein the body part used to provide input to the gesture based input device is at least one of a user hand or a user foot.",
    "6. The method of claim 1 wherein the surgical environment image is displayed on a head-mounted display device.",
    "7. A method comprising: displaying a surgical environment image, wherein the surgical environment image includes a virtual marking element; displaying a body part of a user, the body part used to interact with the virtual marking element; receiving a user input, via a gesture based input device registering movement of the body part of the user, while the body part interacts with the virtual marking element; generating a patient anatomy mark on a patient anatomy based on the received user input, wherein the virtual marking element indicates a potential incision location on the patient anatomy; and evaluating a feasibility of the potential incision location.",
    "8. The method of claim 7 wherein the virtual marking element includes a plurality of virtual port markers for marking locations of a plurality of anatomic entry ports on the patient anatomy.",
    "9. The method of claim 7 wherein the surgical environment image is an endoscopic image inside the patient anatomy.",
    "10. The method of claim 7 wherein the surgical environment image is an external image of the patient anatomy.",
    "11. The method of claim 7 wherein the gesture based input device is configured to receive a three dimensional user input.",
    "12. The method of claim 7 wherein the gesture based input device includes at least one of a tablet device or a user wearable device.",
    "13. The method of claim 7 wherein the body part used to provide input to the gesture based input device is at least one of a user hand or a user foot.",
    "14. A method comprising: displaying a first image on a display while a patient is located in a surgical environment, wherein the first image includes: an image of an internal patient anatomy received from a first imaging device, a virtual control element for controlling a component of a surgical system in the surgical environment, and a real-time image of the component of the surgical system; displaying a second image on the display while the patient is located in the surgical environment, wherein the second image includes an image of the surgical environment external of the patient anatomy received from a second imaging device, wherein the displayed first image is at least partially surrounded by the displayed second image; receiving a gesture of a body part of a user in a predetermined motion, via a gesture based input device registering movement of the body part of the user, while the body part interacts with the virtual control element; and adjusting a setting of the component of the surgical system based on the received gesture.",
    "15. The method of claim 14 wherein the second imaging device is positioned on an instrument in the surgical environment.",
    "16. The method of claim 14 wherein the display is included in at least one of a head-mounted device or a patient-side device.",
    "17. The method of claim 14 wherein the first imaging device is an endoscopic device having a view axis and the second imaging device is oriented along the view axis.",
    "18. The method of claim 14 wherein the image of the internal patient anatomy is preoperatively obtained by the first imaging device.",
    "19. The method of claim 14 wherein the image of the internal patient anatomy is at least one of a CT image or an X-ray image.",
    "20. The method of claim 14, wherein receiving the gesture of the body part of the user includes receiving a gesture from at least one of a user hand or a user finger."
  ],
  "description_excerpt": "The present disclosure is directed to systems and methods for performing a teleoperational medical procedure and more particularly to systems and methods for presenting augmented reality in a display of a teleoperational system.\n\nMinimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments. Imaging instruments provide a user with a field of view within the patient anatomy. Some minimally invasive medical tools and imaging instruments may be teleoperated or otherwise computer-assisted. In existing teleoperational medical systems, the surgeon's view of his or her own extremities may be blocked by a display at a surgical control console, limiting the surgeon's awareness of body position relative to control input devices. Systems and methods are needed to augment the images on the display to create better body position awareness.\n\nThe embodiments of the invention are summarized by the claims that follow below.",
  "cpc": [
    "A61B 90/36",
    "A61B 1/00045",
    "A61B 17/02",
    "A61B 17/0218",
    "A61B 2017/00207",
    "A61B 2017/00973",
    "A61B 2034/105",
    "A61B 2034/107",
    "A61B 2034/2065",
    "A61B 2034/2068",
    "A61B 2034/2072",
    "A61B 2034/256",
    "A61B 2090/3614",
    "A61B 2090/365",
    "A61B 2090/371",
    "A61B 2090/3762",
    "A61B 2090/502",
    "A61B 34/20",
    "A61B 34/25",
    "A61B 34/35",
    "A61B 34/74",
    "A61B 90/361",
    "G06F 3/017",
    "G06T 19/006",
    "G06T 2210/41"
  ],
  "ipc": [
    "A61B 34/00",
    "A61B 34/35",
    "A61B 90/00",
    "G06F 3/01",
    "G06T 19/00"
  ],
  "assignees": [
    "Intuitive Surgical Operations Inc"
  ],
  "inventors": [
    "Brandon D. Itkowitz",
    "Simon P. DiMaio",
    "Paul W. Mohr",
    "Theodore W. Rogers"
  ],
  "filing_date": "2018-10-18",
  "publication_date": "2022-06-07",
  "grant_date": "2022-06-07",
  "priority_date": "2017-10-23",
  "application_number": "US-201816757746-A",
  "family_id": "66246673",
  "cited_by_count": 5,
  "citations": [
    "US6377011B1",
    "US20040111183A1",
    "US20130245375A1",
    "US8398541B2",
    "US9788909B2",
    "US20090177452A1",
    "US20110118753A1",
    "US20110218774A1",
    "US20120280988A1",
    "US20110264107A1",
    "US20110282141A1",
    "US8520027B2",
    "US9858475B2",
    "KR20120098342A",
    "US20150032414A1",
    "US20170348061A1",
    "US20170108930A1",
    "WO2014176403A1",
    "US20170273549A1",
    "US20170154158A1",
    "US9833254B1",
    "US10650594B2",
    "WO2016133644A1",
    "US20170203438A1",
    "US20160324580A1",
    "US20160331584A1",
    "US20160349845A1",
    "WO2017031132A1",
    "US20170367771A1",
    "US20170186157A1",
    "WO2017114834A1",
    "US20190206134A1",
    "US20180078034A1",
    "US20180092706A1",
    "US10499997B2",
    "US20200138518A1",
    "US20190355278A1"
  ]
}