Rendering tool information as graphic overlays on displayed images of tools

{
  "publication_number": "US10282881B2",
  "country": "US",
  "kind": "B2",
  "title": "Rendering tool information as graphic overlays on displayed images of tools",
  "abstract": "An operator telerobotically controls tools to perform a procedure on an object at a work site while viewing real-time images of the work site on a display. Tool information is provided in the operator's current gaze area on the display by rendering the tool information over the tool so as not to obscure objects being worked on at the time by the tool nor to require eyes of the user to refocus when looking at the tool information and the image of the tool on a stereo viewer.",
  "claims": [
    "1. A system comprising: a first input device; a first tool having a first working end, the first tool operatively coupled to the first input device; an image capture device adapted to generate a captured image from a perspective of an image reference frame, the captured image including an image of the first working end; a display; and a processor programmed to: cause the captured image to be displayed on the display; and cause information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by: determining a pose of the first working end in the image reference frame; registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.",
    "2. The system of claim 1, wherein the first input device is being operated by a user; and wherein the information of the first input device comprises information identifying the user.",
    "3. The system of claim 1, wherein the first input device is being operated by an appendage of a user; and wherein the information of the first input device comprises information identifying the appendage of the user.",
    "4. The system of claim 1, wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.",
    "5. The system of claim 1, further comprising: a second input device; a second tool having a second working end, the second tool operatively coupled to the second input device; wherein the captured image includes an image of the second working end; and wherein the processor is further programmed to: cause information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by: determining a pose of the second working end in the image reference frame; registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.",
    "6. The system of claim 5, wherein the first input device is being operated by a first user; wherein the second input device is being operated by a second user; wherein the information of the first input device comprises information identifying the first user; and wherein the information of the second input device comprises information identifying the second user.",
    "7. The system of claim 5, wherein the first input device is being operated by a first appendage of a user; wherein the second input device is being operated by a second appendage of the user; and wherein the information of the first input device comprises information identifying the first appendage; and wherein the information of the second input device comprises information identifying the second appendage.",
    "8. The system of claim 1, wherein the processor is further programmed to determine the pose of the first working end relative to the image reference frame by: determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.",
    "9. The system of claim 8, further comprising: a first manipulator adapted to manipulate the first tool; wherein the processor is further programmed to determine the pose of the first working end relative to the reference frame of the first tool by using kinematics information of the first manipulator.",
    "10. The system of claim 1, wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end; and wherein the display is a stereo viewer.",
    "11. A method comprising: a processor causing a captured image to be displayed on a display, the captured image generated from a perspective of an image reference frame by an image capture device, the captured image including an image of a first working end of a first tool, the first tool operatively coupled to a first input device; and the processor causing information of the first input device to be rendered on the image of the first working end in the captured image being displayed on the display by: determining a pose of the first working end in the image reference frame; registering the information of the first input device with a first designated place on the first working end in the image reference frame by using the pose of the first working end in the image reference frame; and rendering the information of the first input device over the first designated place on the first working end on the image of the first working end in the captured image being displayed on the display, after registering the information of the first input device with the first designated place on the first working end, so that the entire information of the first input device appears as a first decal that is rendered over the image of the first working end in the captured image being displayed on the display by bending the first decal to conform in shape and be applied to an outer surface of the first working end at the first designated place.",
    "12. The method of claim 11, wherein the first input device is being operated by a user; and wherein the information of the first input device comprises information identifying the user.",
    "13. The method of claim 11, wherein the first input device is being operated by an appendage of a user; and wherein the information of the first input device comprises information indicating the appendage of the user.",
    "14. The method of claim 11, wherein the first working end of the first tool includes an end effector, a wrist, and a distal end of a shaft, and the first decal is rendered over one of the end effector, the wrist, and the distal end of the shaft.",
    "15. The method of claim 11, wherein the captured image includes an image of a second working end of a second tool, the second tool operatively coupled to a second input device; and further comprising: the processor causing information of the second input device to be rendered on the image of the second working end in the captured image being displayed on the display by: determining a pose of the second working end in the image reference frame; registering the information of the second input device with a second designated place on the second working end in the image reference frame by using the pose of the second working end in the image reference frame; and rendering the information of the second input device over the second designated place on the second working end on the image of the second working end in the captured image being displayed on the display, after registering the information of the second input device with the second designated place on the second working end, so that the entire information of the second input device appears as a second decal that is rendered over the image of the second working end in the captured image being displayed on the display by bending the second decal to conform in shape and be applied to an outer surface of the second working end at the second designated place.",
    "16. The method of claim 15, wherein the first input device is being operated by a first user; wherein the second input device is being operated by a second user; wherein the information of the first input device comprises information identifying the first user; and wherein the information of the second input device comprises information identifying the second user.",
    "17. The method of claim 15, wherein the first input device is being operated by a first appendage of a user; wherein the second input device is being operated by a second appendage of the user; and wherein the information of the first input device comprises information identifying the first appendage; and wherein the information of the second input device comprises information identifying the second appendage.",
    "18. The method of claim 11, wherein the processor determines the pose of the first working end relative to the image reference frame by: determining the pose of the first working end in a reference frame of the first tool and translating the pose of the first working end from the reference frame of the first tool to the pose of the first working end in the image reference frame.",
    "19. The method of claim 18, wherein the processor determines the pose of the first working end relative to the reference frame of the first tool by using kinematics information of a first manipulator that is adapted to manipulate the first tool.",
    "20. The method of claim 11, wherein the image capture device comprises a stereoscopic camera, so that the captured image includes a stereo view of the first working end; and wherein the display is a stereo viewer."
  ],
  "description_excerpt": "The present invention generally relates to robotic systems and in particular, to a robotic system, and a method implemented therein, for rendering tool information as graphic overlays on displayed images of tools.\n\nIn a robotic system, one or more tools may be telerobotically controlled by an operator to perform a procedure on an object at a work site. A camera is provided at the work site to capture images of end effectors of the tools as they interact with the object to perform the procedure, so that the operator may view their movements on a display while telerobotically controlling the tools using associated input devices.\n\nDuring the procedure, it may be useful to provide the operator with tool information such as whether a tool is energized at the time or which of a plurality of tools is energized at the time or which robotic arm is operatively coupled to a specific tool at the time. The tool information may be provided as text or a graphic in an area on the display which is proximate to its corresponding tool so that the information may be readily associated with the tool. However, such positioning of the tool information may objectionably obstruct images of the tools and/or objects upon which the tools are performing a procedure at the time. Also, when the end effectors of two or more tools are in close proximity to one another, it may not be readily apparent which tool the tool information pertains to at the time. This may be especially problematic if the tool information is stationary and the end effectors are moving.",
  "cpc": [
    "G06T 11/60",
    "A61B 2034/2061",
    "A61B 2090/371",
    "A61B 34/20",
    "A61B 34/25",
    "A61B 34/30",
    "A61B 34/37",
    "A61B 90/36",
    "A61B 90/361",
    "B25J 9/1666",
    "B25J 9/1671",
    "B25J 9/1692",
    "G05B 19/4202",
    "G05B 2219/36432",
    "G05B 2219/39083",
    "G05B 2219/39096",
    "G05B 2219/39449",
    "G05B 2219/40607",
    "G05B 2219/45117",
    "G05B 2219/45123",
    "G06T 11/40",
    "Y02P 90/02",
    "Y02P 90/265"
  ],
  "ipc": [
    "A61B 34/00",
    "A61B 34/20",
    "A61B 34/30",
    "A61B 34/37",
    "A61B 90/00",
    "B25J 9/16",
    "G05B 19/42",
    "G06T 11/40",
    "G06T 11/60"
  ],
  "assignees": [
    "Intuitive Surgical Operations Inc"
  ],
  "inventors": [
    "Brandon D. Itkowitz",
    "Simon P. DiMaio",
    "Daniel J. Halabe",
    "Christopher J. Hasser",
    "Brian D. Hoffman",
    "David Q. Larkin",
    "Catherine J. Mohr",
    "Paul W. Mohr",
    "Tao Zhao",
    "Wenyi Zhao"
  ],
  "filing_date": "2018-04-05",
  "publication_date": "2019-05-07",
  "grant_date": "2019-05-07",
  "priority_date": "2009-03-31",
  "application_number": "US-201815946408-A",
  "family_id": "63037775",
  "cited_by_count": 111,
  "citations": [
    "US3628535A",
    "US3818284A",
    "US3923166A",
    "US3905215A",
    "US4150326A",
    "US4349837A",
    "US5493595A",
    "US4588348A",
    "US4577621A",
    "US4672963A",
    "US4644237A",
    "US4722056A",
    "US4837734A",
    "US4762456A",
    "US4837703A",
    "US4791934A",
    "US4858149A",
    "US4759074A",
    "US4853874A",
    "US4839838A",
    "US4860215A",
    "US4863133A",
    "US4762455A",
    "US4831549A",
    "US4833383A",
    "US5142930A",
    "US5230338A",
    "US5170347A",
    "US5079699A",
    "US5748767A",
    "US4815450A",
    "JPH01280449A",
    "US5251127A",
    "US5046022A",
    "US5187796A",
    "US4989253A",
    "US4979949A",
    "US4891767A",
    "US4984157A",
    "US5174276A",
    "US4942539A",
    "US5099846A",
    "US5098426A",
    "US5184009A",
    "US5053976A",
    "US5257203A",
    "US5337733A",
    "US5368428A",
    "US5086401A",
    "US5408409A",
    "US5299288A",
    "JPH04231034A",
    "US5235510A",
    "US5217003A",
    "US5368015A",
    "US5176702A",
    "US5251611A",
    "US5313306A",
    "EP0514584A2",
    "US5266875A",
    "US5695500A",
    "US6547782B1",
    "US6201984B1",
    "US5279309A",
    "US5950629A",
    "US5445166A",
    "US5402801A",
    "US5182641A",
    "US5261404A",
    "US5184601A",
    "US6876891B1",
    "US5230623A",
    "US5531742A",
    "US6574355B2",
    "US7107124B2",
    "US5808665A",
    "US5341950A",
    "US5737500A",
    "US5430643A",
    "US5321353A",
    "US5417210A",
    "US5749362A",
    "US5572999A",
    "US5482029A",
    "US5759153A",
    "US5239246A",
    "US5704897A",
    "US5657429A",
    "US5524180A",
    "US5815640A",
    "US5878193A",
    "US5515478A",
    "US5907664A",
    "US5754741A",
    "US5841950A",
    "US5397323A",
    "US5788688A",
    "US5831408A",
    "US5474571A",
    "US5877819A",
    "US5931832A",
    "US5791231A",
    "WO1995001757A1",
    "US5382885A",
    "US5343385A",
    "US5503320A",
    "WO1995007055A1",
    "US5755725A",
    "EP0646358A1",
    "JPH07265321A",
    "US6059718A",
    "US5876325A",
    "US5842473A",
    "US5964707A",
    "US5553198A",
    "US6241725B1",
    "JPH07184923A",
    "US5631973A",
    "US5859934A",
    "US5454827A",
    "US5835693A",
    "US6115053A",
    "JPH08107875A",
    "US5617858A",
    "US6120433A",
    "US5528955A",
    "JPH0889506A",
    "US5765561A",
    "JPH08132372A",
    "US5601549A",
    "US5715729A",
    "JPH08154321A",
    "US6036637A",
    "EP0732082B1",
    "JPH08215211A",
    "US5855553A",
    "EP1125557A2",
    "US6019724A",
    "US5836880A",
    "US5817022A",
    "US5797849A",
    "JPH08275958A",
    "US5887121A",
    "US6226566B1",
    "JPH08299363A",
    "US5814038A",
    "US5810880A",
    "US5759151A",
    "US5551432A",
    "US5820623A",
    "US6702736B2",
    "US7144367B2",
    "US6256529B1",
    "US5820545A",
    "US5638819A",
    "US5784542A",
    "US5911036A",
    "US5980460A",
    "JPH09141580A",
    "JP2000500679A",
    "EP0812662A1",
    "US5987591A",
    "US5624398A",
    "US20050228365A1",
    "US6699177B1",
    "US5762458A",
    "US5971976A",
    "US6063095A",
    "US6905460B2",
    "WO1997029690A1",
    "US5855583A",
    "US6436107B1",
    "US6083170A",
    "US5792135A",
    "US6371952B1",
    "US5807377A",
    "US6991627B2",
    "WO1997043942A1",
    "WO1997043943A1",
    "US5797900A",
    "US20030032878A1",
    "US6072466A",
    "US6642836B1",
    "JPH10146341A",
    "US7302288B1",
    "US5810008A",
    "US7963913B2",
    "US20060161138A1",
    "US5853367A",
    "US20040046916A1",
    "US5938678A",
    "JPH11309A",
    "US6827712B2",
    "US6330837B1",
    "US6307285B1",
    "US6786896B1",
    "US5993391A",
    "US6096025A",
    "US6129670A",
    "US6358749B1",
    "US5842993A",
    "US6292712B1",
    "US6196081B1",
    "US6371909B1",
    "US6402737B1",
    "US5980461A",
    "US20070265491A1",
    "US6425865B1",
    "US6246200B1",
    "JP2005303327A",
    "US6184868B1",
    "US5993390A",
    "US6434416B1",
    "US6665554B1",
    "US7806891B2",
    "US20020045888A1",
    "US20060178559A1",
    "US20030055410A1",
    "US6398726B1",
    "US6459926B1",
    "US6837883B2",
    "US6468265B1",
    "US6659939B2",
    "US6342889B1",
    "US6714839B2",
    "US6522906B1",
    "US6325808B1",
    "US6799065B1",
    "US20070038080A1",
    "US6620173B2",
    "US7574250B2",
    "US6331181B1",
    "US7107090B2",
    "US6270453B1",
    "US6224542B1",
    "US6394998B1",
    "US6602185B1",
    "US6084371A",
    "US6594522B1",
    "US20020044104A1",
    "US6243624B1",
    "US6569084B1",
    "US8944070B2",
    "US6493608B1",
    "US9101397B2",
    "US6594552B1",
    "US20150182287A1",
    "US7155315B2",
    "US20040039485A1",
    "US6424885B1",
    "US9232984B2",
    "US6671581B2",
    "JP2000300579A",
    "JP2001000448A",
    "US20040225183A1",
    "JP2001061850A",
    "US6522908B1",
    "JP2001104333A",
    "US6312435B1",
    "US6654031B1",
    "JP2001202531A",
    "US6442417B1",
    "US6204620B1",
    "US20020156345A1",
    "US6847922B1",
    "US6770081B1",
    "JP2001287183A",
    "US6648816B2",
    "US20060161045A1",
    "US6817973B2",
    "US20010035871A1",
    "US6984203B2",
    "US8221304B2",
    "US6926709B2",
    "US6645196B1",
    "US6599247B1",
    "US20020089544A1",
    "US20030023347A1",
    "JP2002103258A",
    "US7194118B1",
    "US20050251113A1",
    "US20040046711A1",
    "US20020120188A1",
    "US6676669B2",
    "US20070060879A1",
    "US6765569B2",
    "JP2002287613A",
    "US6456901B1",
    "US20060058988A1",
    "US20030109780A1",
    "US8170716B2",
    "US20020193800A1",
    "US7493153B2",
    "US20040254454A1",
    "US7996110B2",
    "US20040243147A1",
    "US6643563B2",
    "US6550757B2",
    "JP2003053684A",
    "US20030060927A1",
    "US20040238732A1",
    "EP1310844A1",
    "US20030114730A1",
    "US6852107B2",
    "US7413565B2",
    "WO2003061482A1",
    "US20030144649A1",
    "US20030167103A1",
    "US20040106916A1",
    "US20040024311A1",
    "US20060142657A1",
    "US20090012531A1",
    "US20040034283A1",
    "JP2003300444A",
    "JP2003339725A",
    "US20030225479A1",
    "US6960162B2",
    "WO2004014244A2",
    "US8005571B2",
    "US7155316B2",
    "US20080161830A1",
    "US20040176751A1",
    "US20040044295A1",
    "US7041053B2",
    "US20040049205A1",
    "JP2004105638A",
    "US20040077940A1",
    "EP1424173A2",
    "US20040189675A1",
    "US20050113640A1",
    "JP2004223128A",
    "US20050022158A1",
    "US20040254679A1",
    "US20040249508A1",
    "US7491198B2",
    "US20050166413A1",
    "US20050059960A1",
    "US20050054895A1",
    "WO2005037120A1",
    "JP2005110878A",
    "US7181315B2",
    "JP2007508913A",
    "JP2007531553A",
    "WO2005039391A2",
    "WO2005043319A2",
    "US20050096502A1",
    "US20050096892A1",
    "JP2005135278A",
    "US20050107680A1",
    "US20050203380A1",
    "JP2005334650A",
    "US20050267359A1",
    "US7979157B2",
    "US8256319B2",
    "US8062288B2",
    "US20080188986A1",
    "US20070177009A1",
    "US20060149129A1",
    "WO2006079108A1",
    "CN101160104A",
    "WO2006091494A1",
    "US20060258938A1",
    "US20100331855A1",
    "US8823308B2",
    "US8004229B2",
    "US20150032126A1",
    "US8541970B2",
    "US8624537B2",
    "US8749190B2",
    "US8749189B2",
    "US8786241B2",
    "US20150051733A1",
    "US8816628B2",
    "US20060261770A1",
    "US20070021738A1",
    "US20130245375A1",
    "WO2007030173A1",
    "WO2007005555A2",
    "US20070016174A1",
    "JP2007029232A",
    "US20070135803A1",
    "JP2007090481A",
    "US7998058B2",
    "US20070071310A1",
    "US20070106307A1",
    "US20070081714A1",
    "US20080033240A1",
    "WO2007047782A2",
    "US20070197896A1",
    "US20070142968A1",
    "US20070138992A1",
    "US7819859B2",
    "US20070142825A1",
    "US20070144298A1",
    "US20080287963A1",
    "US20070156285A1",
    "US20110290856A1",
    "WO2007088208A1",
    "WO2007088206A2",
    "JP2009525097A",
    "US20090024142A1",
    "US20100036198A1",
    "US20070229015A1",
    "US20070255454A1",
    "US20080140087A1",
    "US20070270650A1",
    "US20070270685A1",
    "WO2007136768A2",
    "US20070296366A1",
    "JP2009537229A",
    "US8398541B2",
    "US20080071291A1",
    "WO2007146987A2",
    "US20100198232A1",
    "US20080065105A1",
    "US20070287992A1",
    "US20070283970A1",
    "US20080065099A1",
    "US20070287884A1",
    "US7725214B2",
    "US9345387B2",
    "US20140055489A1",
    "US20140135792A1",
    "US20170209232A1",
    "US20090192524A1",
    "US20090192523A1",
    "US9718190B2",
    "US20170210012A1",
    "WO2008002830A2",
    "US20080081992A1",
    "US20080118115A1",
    "US20080247506A1",
    "KR20090111308A",
    "WO2008094766A2",
    "WO2008103383A1",
    "US20080243142A1",
    "US20100004505A1",
    "US8554368B2",
    "WO2009044287A2",
    "WO2009037576A2",
    "WO2009034477A2",
    "US20120132450A1",
    "US20160374767A1",
    "US9629520B2",
    "US20160242860A1",
    "US20160235486A1",
    "US9469034B2",
    "US8620473B2",
    "US9138129B2",
    "US9333042B2",
    "JP2009006410A",
    "US20090005640A1",
    "JP2009012106A",
    "JP2009039814A",
    "US8108072B2",
    "US20090088634A1",
    "US20090105750A1",
    "US20090228145A1",
    "US20090248036A1",
    "US8155479B2",
    "US20120154564A1",
    "US20090259105A1",
    "US8175861B2",
    "US20090326711A1",
    "US9089256B2",
    "US20090326552A1",
    "WO2009158164A1",
    "US9717563B2",
    "US9516996B2",
    "US20090326553A1",
    "US8864652B2",
    "US20090326322A1",
    "US8130907B2",
    "US8315720B2",
    "WO2010039394A1",
    "US20100106356A1",
    "US20100331856A1",
    "US20110313573A1",
    "US8335590B2",
    "US20100169815A1",
    "US8306656B1",
    "US8120301B2",
    "US20100228264A1",
    "US20100249657A1",
    "US20100317965A1",
    "US20150297300A1",
    "US8903546B2",
    "US9084623B2",
    "US9492927B2",
    "US20170035521A1",
    "US20110071675A1",
    "US20110196199A1",
    "US8918211B2",
    "US9622826B2",
    "US20170173788A1",
    "US20140052150A1",
    "US20130178868A1",
    "US20130289767A1",
    "US20140232824A1",
    "US9565990B2",
    "US9949798B2"
  ]
}