Enhanced ophthalmic surgical experience using a virtual reality head-mounted display

{
  "publication_number": "US9844321B1",
  "country": "US",
  "kind": "B1",
  "title": "Enhanced ophthalmic surgical experience using a virtual reality head-mounted display",
  "abstract": "An ophthalmic surgical system comprises: a camera optically coupled to a surgical microscope; a virtual reality (VR) headset worn by a surgeon; and a VR data processing unit configured to communicate with the surgical microscope, the VR headset, and an ophthalmic surgical apparatus, wherein the VR data processing unit is configured to: project a real time video screen of video received from the camera into the VR headset; project a patient information screen into the VR headset to provide the patient information directly to the surgeon during ophthalmic surgery; project a surgical apparatus information screen into the VR headset; project a surgical apparatus input control screen into the VR headset to provide the surgeon with direct control over the surgical apparatus; and control which ones of the screens are visible in the VR headset based inputs indicating head movements of the surgeon as detected by the VR headset.",
  "claims": [
    "1. An ophthalmic surgical system, comprising: a 3D camera optically coupled to a surgical microscope; a virtual reality (VR) headset worn by a surgeon; and a VR data processing unit configured to communicate with the 3D camera, the VR headset, and an surgical apparatus for performing ophthalmic surgery on a patient, wherein the VR data processing unit is configured to: project a real time video screen of video received from the 3D camera into the VR headset; project a patient information screen into the VR headset to provide the patient information directly to the surgeon during ophthalmic surgery; project a surgical apparatus information screen into the VR headset showing a current context of the surgical apparatus; project a surgical apparatus input control screen into the VR headset that provides the surgeon with direct control over the surgical apparatus; and control which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset based inputs indicating head movements of the surgeon as detected by the VR headset.",
    "2. The system of claim 1, wherein the surgeon is provided with control over the surgical apparatus from the VR headset through at least one of: gesture interpretation; surgeon eye tracking; a footswitch; a trackball; and electroencephalographic pattern recognition.",
    "3. The system of claim 1, wherein the VR data processing unit is further configured to: receive feedback from accelerometers located in the VR headset indicating forward or backward movement of the surgeon's head; and adjust the zoom of the 3D camera based on the received feedback.",
    "4. The system of claim 1, wherein the VR data processing unit controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset; and determining, based on the received sensor outputs, surgeon side-to-side and up-and-down head movements.",
    "5. The system of claim 1, wherein: each of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen is assigned a corresponding range of rotation angles and/or inclination angles; and the VR data processing controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset indicating a measured rotation angle and/or inclination angle of the surgeon's head; and displaying each of the screens having a corresponding range of rotation angles and/or inclination angles covering the measured rotation angle and/or inclination angle of the surgeon's head.",
    "6. The system of claim 1, wherein the patient information displayed on the patient information screen comprises any combination of a patient's medical history, doctor's notes, and images.",
    "7. The system of claim 1, wherein the current context of the surgical apparatus includes device operating parameters.",
    "8. The system of claim 1, wherein the VR data processing unit is further configured to display the video received from the 3D camera on a monitor located in an operating room.",
    "9. A method for providing an enhanced ophthalmic surgical experience for a surgeon, the method comprising: responsive to a VR data processing unit receiving real time video from a 3D camera optically coupled to a surgical microscope, projecting, by the VR data processing unit, a real time video screen of the video received from the 3D camera into a VR headset; responsive to the VR data processing unit receiving patient data over a network, projecting, by the VR data processing unit, a patient information screen into the VR headset during ophthalmic surgery; responsive to the VR data processing unit receiving surgical apparatus current context information from an surgical apparatus for performing ophthalmic surgery on the patient, projecting, by the VR data processing unit, a surgical apparatus information screen into the VR headset showing the current context of the surgical apparatus; responsive to the VR data processing unit receiving surgical apparatus information, projecting, by the VR data processing unit, a surgical apparatus input control screen into the VR headset that provides the surgeon with direct control over the surgical apparatus; and controlling, by the VR data processing unit, which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset based inputs indicating head movements of the surgeon as detected by the VR headset.",
    "10. The method of claim 9, wherein the surgeon is provided with control over the surgical apparatus from the VR headset through at least one of: gesture interpretation; surgeon eye tracking; a footswitch; a trackball; and electroencephalographic pattern recognition.",
    "11. The method of claim 9, further comprising, responsive to the VR data processing unit receiving feedback from accelerometers located in the VR headset indicating forward or backward movement of the surgeon's head, adjusting the zoom of the 3D camera based on the received feedback.",
    "12. The method of claim 9, wherein the VR data processing unit controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset; and determining, based on the received sensor outputs, surgeon side-to-side and up-and-down head movements.",
    "13. The method of claim 9, wherein: each of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen is assigned a corresponding range of rotation angles and/or inclination angles; and the VR data processing controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset indicating a measured rotation angle and/or inclination angle of the surgeon's head; and displaying each of the screens having a corresponding range of rotation angles and/or inclination angles covering the measured rotation angle and/or inclination angle of the surgeon's head.",
    "14. The method of claim 9, wherein the patient information displayed on the patient information screen comprises any combination of a patient's medical history, doctor's notes, and images.",
    "15. The method of claim 9, wherein the current context of the surgical apparatus includes device operating parameters.",
    "16. The method of claim 9, further comprising displaying, by VR data processing unit, the video received from the 3D camera on a monitor located in an operating room.",
    "17. A non-transitory computer-readable medium containing program instructions for providing an enhanced ophthalmic surgical experience for a surgeon, the program instructions for: responsive to a VR data processing unit receiving real time video from a 3D camera optically coupled to a surgical microscope, projecting, by the VR data processing unit, a real time video screen of the video received from the 3D camera into a VR headset; responsive to the VR data processing unit receiving patient data over a network, projecting, by the VR data processing unit, a patient information screen into the VR headset during ophthalmic surgery; responsive to the VR data processing unit receiving surgical apparatus current context information from an surgical apparatus for performing ophthalmic surgery on the patient, projecting, by the VR data processing unit, a surgical apparatus information screen into the VR headset showing the current context of the surgical apparatus; responsive to the VR data processing unit receiving surgical apparatus information, projecting, by the VR data processing unit, a surgical apparatus input control screen into the VR headset that provides the surgeon with direct control over the surgical apparatus; and controlling, by the VR data processing unit, which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset based inputs indicating head movements of the surgeon as detected by the VR headset.",
    "18. The non-transitory computer-readable medium of claim 17, wherein the surgeon is provided with control over the surgical apparatus from the VR headset through at least one of: gesture interpretation; surgeon eye tracking; a footswitch; a trackball; and electroencephalographic pattern recognition.",
    "19. The non-transitory computer-readable medium of claim 17, the program instructions additionally for, responsive to the VR data processing unit receiving feedback from accelerometers located in the VR headset indicating forward or backward movement of the surgeon's head, adjusting the zoom of the 3D camera based on the received feedback.",
    "20. The non-transitory computer-readable medium of claim 17, wherein the VR data processing unit controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset; and determining, based on the received sensor outputs, surgeon side-to-side and up-and-down head movements.",
    "21. The non-transitory computer-readable medium of claim 17, wherein: each of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen is assigned a corresponding range of rotation angles and/or inclination angles; and the VR data processing controls which ones of the real time video screen, the patient information screen, the surgical apparatus information screen, and the surgical apparatus input control screen are visible in the VR headset by: receiving sensor outputs from the VR headset indicating a measured rotation angle and/or inclination angle of the surgeon's head; and displaying each of the screens having a corresponding range of rotation angles and/or inclination angles covering the measured rotation angle and/or inclination angle of the surgeon's head.",
    "22. The non-transitory computer-readable medium of claim 17, wherein the patient information displayed on the patient information screen comprises any combination of a patient's medical history, doctor's notes, and images.",
    "23. The non-transitory computer-readable medium of claim 17, wherein the current context of the surgical apparatus includes device operating parameters.",
    "24. The non-transitory computer-readable medium of claim 17, further comprising displaying, by VR data processing unit, the video received from the 3D camera on a monitor located in an operating room."
  ],
  "cpc": [
    "A61B 3/13",
    "A61B 2090/367",
    "A61B 2090/368",
    "A61B 2090/372",
    "A61B 3/0025",
    "A61B 3/0058",
    "A61B 3/145",
    "A61B 90/20",
    "A61B 90/37",
    "G06F 3/012",
    "G06T 19/006",
    "G16H 20/40",
    "G16H 30/40",
    "G16H 40/63"
  ],
  "assignees": [
    "NOVARTIS AG"
  ],
  "filing_date": "2016-08-04",
  "publication_date": "2017-12-19",
  "grant_date": "2017-12-19",
  "priority_date": "2016-08-04",
  "application_number": "US-201615228357-A",
  "family_id": "59762010"
}