Visual tracking and annotaton of clinically important anatomical landmarks for surgical interventions

{
  "publication_number": "US9814392B2",
  "country": "US",
  "kind": "B2",
  "title": "Visual tracking and annotaton of clinically important anatomical landmarks for surgical interventions",
  "abstract": "A visual tracking and annotation system for surgical intervention includes an image acquisition and display system arranged to obtain image streams of a surgical region of interest and of a surgical instrument proximate the surgical region of interest and to display acquired images to a user; a tracking system configured to track the surgical instrument relative to the surgical region of interest; a data storage system in communication with the image acquisition and display system and the tracking system; and a data processing system in communication with the data storage system, the image acquisition and display system and the tracking system. The data processing system is configured to annotate images displayed to the user in response to an input signal from the user.",
  "claims": [
    "1. A visual tracking and annotation system for surgical intervention in an eye, comprising: a surgical instrument comprising a surgical tool suitable for use in the eye and an optical sensor attached to the surgical tool, wherein the surgical tool has a reference portion and the optical sensor has an end fixed relative to said reference portion such that light reflected by said reference portion can be detected by the optical sensor along with light reflected by tissue that is proximate to or in contact with a distal end of said surgical tool while in use in order to provide a distance of the distal end of the surgical tool to the tissue; an image acquisition and display system arranged to obtain a plurality of image streams of a surgical region of interest and of said surgical instrument proximate to said surgical region of interest and to display said plurality of image streams to a user; a tracking system configured to track a position of said surgical instrument relative to said surgical region of interest based on processing said plurality of image streams; a data storage system in communication with said image acquisition and display system and said tracking system; and a data processing system in communication with said data storage system, said image acquisition and display system and said tracking system, wherein said data processing system is configured to generate an annotation from a first image of said plurality of image streams displayed to said user during said surgical intervention in response to an input signal from said user, the annotation being associated with the position of the surgical instrument and a location of the tissue relative to the surgical region of interest, wherein the data processing system is further configured to determine the location of the tissue based on the position of the surgical instrument and the distance of the distal end of the surgical tool to the tissue; wherein the tracking system is further configured to acquire a location of the annotation relative to at least one corresponding anatomical landmark location acquired from said plurality of image streams, wherein said data storage system is configured to store said annotation and said location of said annotation relative to the corresponding anatomical landmark location, wherein said data processing system is further configured to retrieve said annotation and said location of said annotation relative to the corresponding anatomical landmark location from said data storage system, and wherein said data processing system is further configured to overlay said annotation onto a second image of said plurality of image streams having at least one of a different direction and position from said first image based on said location of said annotation relative to said corresponding anatomical landmark and processing of said first image.",
    "2. A visual tracking and annotation system for surgical intervention according to claim 1, further comprising a user input device in communication with said data processing system to provide said input signal.",
    "3. A visual tracking and annotation system for surgical intervention according to claim 2, wherein said user input device comprises at least one of a pedal, a keypad, a microphone, an eye tracking system and a switch.",
    "4. A visual tracking and annotation system for surgical intervention according to claim 2, wherein said user input device comprises the surgical instrument.",
    "5. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said data processing system is further configured to annotate the first image displayed to said user to include at least one of a position or a track of said surgical instrument relative to said surgical region of interest.",
    "6. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said data processing system is further configured to retrieve information from said data storage system that corresponds to a position of said surgical instrument.",
    "7. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said surgical instrument comprises a sensor system comprising said optical sensor, said sensor system being constructed and arranged to collect data regarding localized portions of said surgical region of interest, and said surgical instrument being in communication with said data storage system such that data obtained by said surgical instrument can be saved for later retrieval.",
    "8. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said optical sensor comprises a visual imaging system.",
    "9. A visual tracking and annotation system for surgical intervention according to claim 8, wherein said visual imaging system comprises an optical fiber, said visual imaging system being configured to simultaneously image said reference portion of said surgical tool and said tissue proximate or in contact with said distal end of said surgical tool.",
    "10. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said optical sensor comprises an optical coherence tomography system.",
    "11. A visual tracking and annotation system for surgical intervention according to claim 10, wherein said optical coherence tomography system comprises an optical fiber that provides said fixed end of said optical sensor.",
    "12. A visual tracking and annotation system for surgical intervention according to claim 11, wherein said surgical tool comprises a pick.",
    "13. A visual tracking and annotation system for surgical intervention according to claim 12, wherein said distal end of said surgical tool is adapted to be held by a surgeon for performing manual surgery.",
    "14. A visual tracking and annotation system for surgical intervention according to claim 12, wherein said distal end of said surgical tool is adapted to be attached to a robotic system for at least one of robotic or robot-assisted surgery.",
    "15. A visual tracking and annotation system for surgical intervention according to claim 11, wherein said surgical tool is at least one of a pick, tweezers, a knife, a light delivery device, scissors, an injector, or a vitrectomy tool.",
    "16. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said surgical instrument comprises a sensor system comprising said optical sensor.",
    "17. A visual tracking and annotation system for surgical intervention according to claim 1, wherein said surgical instrument is a user input device that is in communication with said data processing system to provide said input signal.",
    "18. A visual tracking and annotation method for surgical intervention in an eye, comprising: acquiring a plurality of images of a surgical region of interest and of a surgical instrument proximate to said surgical region of interest, wherein the surgical instrument comprises a surgical tool suitable for use in an eye and an optical sensor attached to the surgical tool, and wherein the surgical tool has a reference portion and the optical sensor has an end fixed relative to said reference portion such that light reflected by said reference portion can be detected by the optical sensor along with light reflected by tissue that is proximate to or in contact with a distal end of said surgical tool while in use in order to provide a distance of the distal end of the surgical tool to the tissue; tracking a position of said surgical instrument relative to said surgical region of interest based on processing said plurality of said images; generating an annotation from a first image of said plurality of images during said surgical intervention in response to an input signal from a user, wherein the annotation is associated with the position of the surgical instrument and a location of the tissue relative to the surgical region of interest, the location of the tissue being based on the position of the surgical instrument and the distance of the distal end of the surgical tool to the tissue; acquiring and storing a location of said annotation relative to at least one corresponding anatomic landmark location from said plurality of images; retrieving said annotation and said location of said annotation relative to the corresponding anatomical landmark location; and overlaying said annotation onto a second image of said plurality of images having at least one of a different direction and position from said first image based on said location of said annotation relative to the corresponding anatomical landmark location and processing of said first image.",
    "19. A visual tracking and annotation method for surgical intervention according to claim 18, wherein said annotation includes at least one of a position or a track of said surgical instrument.",
    "20. A visual tracking and annotation method for surgical intervention according to claim 18, wherein said annotation includes information retrieved from a data storage system that corresponds to a position of said surgical instrument.",
    "21. A computer-readable medium comprising non-transient storage of software for visual tracking and annotation for surgical intervention, which when executed by a computer system, comprises: processing a plurality of images of a surgical region of interest and of a surgical instrument proximate to said surgical region of interest, wherein the surgical instrument comprises a surgical tool suitable for use in an eye and an optical sensor attached to the surgical tool, and wherein the surgical tool has a reference portion and the optical sensor has an end fixed relative to said reference portion such that light reflected by said reference portion can be detected by the optical sensor along with light reflected by tissue that is proximate to or in contact with a distal end of said surgical tool while in use in order to provide a distance of the distal end of the surgical tool to the tissue; processing tracking of a position of said surgical instrument relative to said surgical region of interest based on processing of said plurality of images; processing an input signal from a user to generate an annotation from a first image of said plurality of images during said surgical intervention, wherein the annotation is associated with the position of the surgical instrument a location of the tissue relative to the surgical region of interest, the location of the tissue being based on the position of the surgical instrument and the distance of the distal end of the surgical tool to the tissue; processing said annotation to store said annotation and a location of said annotation relative to at least one corresponding anatomical landmark location; processing said annotation to retrieve said annotation and said location of said annotation relative to the corresponding anatomical landmark location; and processing said annotation to overlay said annotation onto a second image of said plurality of images having at least one of a different direction and position from said first image based on said location of said annotation relative to said corresponding anatomical landmark location and said first image.",
    "22. A computer-readable medium according to claim 21, wherein said annotation includes at least one of a position or a track of said surgical instrument.",
    "23. A computer-readable medium according to claim 21, wherein said annotation includes information retrieved from a data storage system that corresponds to a position of said surgical instrument."
  ],
  "description_excerpt": "1. Field of Invention\n\nThe field of the currently claimed embodiments of this invention relates to systems and methods of acquiring and displaying information during surgical procedures, and more particularly to systems and methods of acquiring and displaying information that include visual tracking of surgical instruments and annotation of information displayed.\n\n2. Discussion of Related Art\n\nCurrently surgeons use a number of intraoperative diagnostic and surgical and/or treatment devices for operating on small tissue regions, e.g., laparoscopic ultrasound and/or RF ablation devices. The surgeon observes the device's position relative to the anatomy through a video feed. In the case of diagnostic instruments, the surgeon has to note this position and the imaging results and track this location with the anatomy over time. Over time, the surgeon may want to revisit particular anatomy for an intervention or inspection. If the surgeon does not recall or is not sure about the location or the content of a prior diagnostic image associated with a particular area, the diagnostic device might have to be reintroduced to the operating field, which is time consuming. In the case of surgical devices, the surgeon has to map the already treated area relative to some visible landmarks. This is difficult when the treatment is inside the target tissue, or does not alter the tissue's appearance. Missed, over treated or incorrect treatment locations are to be avoided. The surgeon may choose to inspect or treat a number of anatomical regions, which could be sparsely located, e.g. 10 landmarks.",
  "cpc": [
    "A61B 5/0066",
    "A61B 2034/2065",
    "A61B 2090/3614",
    "A61B 2090/373",
    "A61B 2090/3735",
    "A61B 2090/378",
    "A61B 2560/0295",
    "A61B 3/102",
    "A61B 3/1225",
    "A61B 3/13",
    "A61B 5/0084",
    "A61B 5/061",
    "A61B 5/1076",
    "A61B 5/743",
    "A61B 5/7475",
    "A61B 8/5261",
    "A61B 90/37",
    "A61F 9/007",
    "G02B 21/0012",
    "G06F 17/241",
    "G06F 40/169",
    "G06T 7/33",
    "G16H 20/40",
    "G16H 30/40"
  ],
  "ipc": [
    "A61B 5/103",
    "A61B 6/00",
    "A61B 6/02",
    "A61B 8/13",
    "A61B 34/20",
    "A61B 34/30",
    "A61B 90/00",
    "A61B 3/10",
    "A61B 3/12",
    "A61B 3/13",
    "A61B 5/00",
    "A61B 5/06",
    "A61B 5/107",
    "A61B 8/08",
    "G06F 17/24"
  ],
  "assignees": [
    "Johns Hopkins University"
  ],
  "inventors": [
    "Marcin A. Balicki",
    "Russell H. Taylor",
    "Gregory D. Hager",
    "Peter L. Gehlbach",
    "James T. Handa",
    "Rajesh Kumar"
  ],
  "filing_date": "2010-11-01",
  "publication_date": "2017-11-14",
  "grant_date": "2017-11-14",
  "priority_date": "2009-10-30",
  "application_number": "US-201013505058-A",
  "family_id": "43923049",
  "cited_by_count": 29,
  "citations": [
    "US5554155A",
    "US6167296A",
    "JP2002510230A",
    "US20110105898A1",
    "US20050200324A1",
    "US6428553B1",
    "US6975898B2",
    "US20050027199A1",
    "US20020171669A1",
    "US20040254454A1",
    "US20040097805A1",
    "US8401616B2",
    "US20050015005A1",
    "CN1802626A",
    "US20060174065A1",
    "US7126303B2",
    "US20050182295A1",
    "US8423120B2",
    "US7831294B2",
    "US20060100505A1",
    "CN101222882B",
    "US20060258938A1",
    "US20070021738A1",
    "US20080033240A1",
    "US20070115481A1",
    "US20070167702A1",
    "US20070238981A1",
    "US20070276226A1",
    "US20080039705A1",
    "US20070276195A1",
    "US8398541B2",
    "US20090036902A1",
    "US20080004603A1",
    "JP2008006169A",
    "US20080062429A1",
    "US20080119725A1",
    "US8528566B2",
    "US20080177256A1",
    "US20080287783A1",
    "US8073528B2",
    "US8108072B2",
    "US8147503B2",
    "EP2075763A1",
    "US20090220125A1",
    "US20110122365A1",
    "US20100168763A1",
    "US20100228123A1",
    "US20100249506A1",
    "US20120130258A1",
    "US20110106102A1",
    "US8554307B2"
  ]
}