Line-based image registration and cross-image annotation devices, systems and methods

{
  "publication_number": "US10503868B2",
  "country": "US",
  "kind": "B2",
  "title": "Line-based image registration and cross-image annotation devices, systems and methods",
  "abstract": "The disclosure relates to devices, systems and methods for image registration and annotation. The devices include computer software products for aligning whole slide digital images on a common grid and transferring annotations from one aligned image to another aligned image on the basis of matching tissue structure. The systems include computer-implemented systems such as work stations and networked computers for accomplishing the tissue-structure based image registration and cross-image annotation. The methods include processes for aligning digital images corresponding to adjacent tissue sections on a common grid based on tissue structure, and transferring annotations from one of the adjacent tissue images to another of the adjacent tissue images. The basis for alignment may be a line-based registration process, wherein sets of lines are computed on the boundary regions computed for the two images, where the boundary regions are obtained using information from two domains - soft-weighted foreground images and gradient magnitude images. The binary mask image, based on whose boundary the line features are computed, may be generated by combining two binary masks - a first binary mask is obtained on thresholding a soft-weighted (continuous valued) foreground image, which is computed based on the stain content in an image, while a second binary mask is obtained after thresholding a gradient magnitude domain image, where the gradient is computed from the grayscale image obtained from the color image.",
  "claims": [
    "1. A computerized process, comprising: performing a line-based tissue matching image registration process to globally align a first image and a second image to a common grid, wherein the first image is a digital image of a tissue section and the second image is a digital image of an adjacent tissue section, and wherein the line-based tissue matching image registration process comprises a coarse registration mode comprising: generating a first foreground image having a boundary from the first digital image and generating a second foreground image having a boundary from the second digital image, computing a first set of line-based features from the boundary of the first foreground image and computing a second set of line-based features from the boundary of the second foreground image, computing global transformation parameters between the first set of line-based features and the second set of line-based features, and globally aligning the first digital image with the second digital image based on the global transformation parameters.",
    "2. A computerized process according to claim 1, wherein the first digital image is derived from an image obtained using a stain and an imaging mode, and the second digital image is derived from an image obtained using a different stain, a different imaging mode, or both as compared to the first digital image.",
    "3. A computerized image registration process according to claim 1, wherein the stain is chosen from a hematoxylin and eosin stain (‘HE’ stain) and an ImmunoHistoChemistry stain (‘IHC’ stain).",
    "4. A computerized image registration process according to claim 2, wherein the imaging mode is chosen from brightfield microscopy and fluorescent microscopy.",
    "5. A computerized image registration process according to claim 1, further comprising: a. selecting the first image of the tissue section from a set of digital images of adjacent tissue sections of a single patient; and, b. selecting the second digital image of the adjacent tissue section from the set.",
    "6. A computerized image registration process according to claim 1, wherein: a. generating a first foreground image comprises: generating a first soft-weighted foreground image from the first digital image, OTSU thresholding the first soft-weighted foreground image to generate a first soft-weighted binary mask, generating a first gradient magnitude image from the first digital image, OTSU thresholding the first gradient magnitude image to generate a first gradient binary mask, and combining the first soil-weighted binary mask with the first gradient binary mask using a binary OR operation to generate the first foreground image; and, b. generating a second foreground image comprises: generating a second soft-weighted foreground image from the second digital image, OTSU thresholding the second soft-weighted foreground image to generate a second soft-weighted binary mask, generating a second gradient magnitude image from the second digital image, OTSU thresholding the second gradient magnitude image to generate a second gradient binary mask, and combining the second soft-weighted binary mask with the second gradient binary image using a binary OR operation to generate the second foreground image.",
    "7. A computerized image registration process according to claim 1, further comprising a fine registration mode to refine alignment of the first digital image and the second digital image.",
    "8. A computerized image registration process according to claim 7, wherein the fine registration mode comprises: annotating the first digital image; mapping the annotation on the common big grid to a corresponding location in the second digital image; and, updating the location using correationbased distance matching in a gradient magnitude domain.",
    "9. A computerized image registration process according to claim 1, wherein computing global transformation parameters comprises: computing transformation parameters that minimize a distance between the first set of lines and the second set of lines for each of a first condition, second condition and third condition, wherein the first condition assumes rotation and translation without flipping, a second condition assumes rotation and translation with horizontal flipping, and the third condition assumes rotation and translation with vertical flipping; and, determining which of the first condition, second condition or third condition have a minimum cost.",
    "10. A method for snapping an annotation from a first digital image from a set of digital images of adjacent tissue sections to a second digital image of the set of digital images of adjacent tissue sections, comprising: a. selecting the first image from the set; b. selecting the second image from the set; c. optionally annotating the first image; d. instructing a computer processor to execute instructions resulting in aligning the first image with the second image on a common grid based using a tissue-matching registration process that is robust to mismatch between the first image and the second image, wherein aligning the first image and the second image comprises generating a first foreground image having a boundary from the first image and generating a second foreground image having a boundary from the second image, computing a first set of line-based features from the boundary of the first foreground image and computing a second set of line-based features from the boundary of the second foreground image, computing global transformation parameters between the first set of line-based features and the second set of line-based features, and globally aligning the first digital image with the second digital image based on the global transformation parameters; and e. instructing the computer processor to transfer an annotation from the first image to the second image.",
    "11. A method according to claim 10, wherein the instruction to transfer the annotation from the first image to the second image occurs automatically when the first image and the second image are aligned.",
    "12. A method according to claim 10, wherein transferring the annotation comprises: mapping an annotation on the first image to the common grid, and mapping the common grid annotation to a corresponding location on the second image.",
    "13. A method according to claim 12, wherein annotation further comprises adjusting the location of the annotation on the second image based on a fine registration process comprising defining a first window around the annotation in the first image, defining a second window in the second image, wherein the second window is larger than the first window but is substantially co-located with the first window on the common grid, and computing an optimized location for the first window in the second window using an iterative process based on distance transformation and cost function calculations.",
    "14. A method according to claim 10, wherein the mismatch can be due to rotation, translation, horizontal flip, and vertical flip.",
    "15. A method according to claim 10, wherein the mismatch can be due to Area of Interest mismatch, and aligning comprises computing a first subset in the first image, computing a second subset in the second image, and aligning the first subset in with the second subset in the second image.",
    "16. A method according to claim 10, wherein there are greater than 50% matching lines between the first set of line-based features and the second set of line-based features and the mismatch can be due to rotation, translation, horizontal flip, vertical flip, shift, wear-and-tear affects, Area of Interest mismatch, and combinations thereof.",
    "17. A computer-readable medium storing instructions which, when executed by one or more processors of a computing device, cause the computing device to perform a line-based tissue matching image registration process to globally align a first image and a second image to a common grid, wherein the first image is a digital image of a tissue section and the second image is a digital image of an adjacent tissue section, and wherein the line-based tissue matching image registration process comprises a coarse registration mode comprising: generating a first foreground image having a boundary from the first digital image and generating a second foreground image having a boundary from the second digital image, computing a first set of line-based features from the boundary of the first foreground image and computing a second set of line-based features from the boundary of the second foreground image, computing global transformation parameters between the first set of line-based features and the second set of line-based features, and globally aligning the first digital image with the second digital image based on the global transformation parameters.",
    "18. A computing system comprising one or more processors and coupled to one or more memory devices, the one or more memory devices storing instructions which, when executed by the one or more processors, cause the computing system to perform a line-based tissue matching image registration process to globally align a first image and a second image to a common grid, wherein the first image is a digital image of a tissue section and the second image is a digital image of an adjacent tissue section, and wherein the line-based tissue matching image registration process comprises a coarse registration mode comprising: generating a first foreground image having a boundary from the first digital image and generating a second foreground image having a boundary from the second digital image, computing a first set of line-based features from the boundary of the first foreground image and computing a second set of line-based features from the boundary of the second foreground image, computing global transformation parameters between the first set of line-based features and the second set of line-based features, and globally aligning the first digital image with the second digital image based on the global transformation parameters."
  ],
  "description_excerpt": "This specification relates to devices, systems, and methods for manipulation and/or analysis of digitized images of tissue samples. This specification also relates to devices, systems and methods for image registration of a set of digitized images of neighboring tissue section samples. This specification also relates to devices, systems and methods for transferring annotations from one image in the set of images of adjacent tissue section samples to other images in the set of images of adjacent tissue section samples.\n\nDigital Pathology refers to the management and interpretation of pathology information in a digital environment. Scanning devices are used to image slides of tissue sections, which may be stained, such that digital slides, e.g., whole slide images are generated. Digital Pathology software enables digital slides to be stored in a computer memory device, viewed on a computer monitor, and analyzed for pathology information.\n\nIt is expected that Digital Pathology may enable integration of various aspects of the pathology environment such as paper and electronic records, clinical background information, prior cases, images, and results, among other things. It is also expected that Digital Pathology may enable increased efficiencies such as increased workload capability, access to the right pathologist at the right time, rapid retrieval of cases and diagnoses, and improved workflow among other possible efficiencies. However, there are a number of impediments to the widespread adoption of Digital Pathology and the promise of its various benefits, such as imaging performance, scalability and management.",
  "cpc": [
    "G06T 3/14",
    "G06F 19/321",
    "G06K 9/0014",
    "G06T 2207/10024",
    "G06T 2207/10056",
    "G06T 2207/10064",
    "G06T 2207/20016",
    "G06T 2207/30024",
    "G06T 3/0068",
    "G06T 7/33",
    "G06V 20/695",
    "G16H 30/20",
    "G16H 30/40"
  ],
  "ipc": [
    "G06T 7/00",
    "G06K 9/00",
    "G06T 3/00",
    "G16H 30/40",
    "G06T 7/33"
  ],
  "assignees": [
    "Ventana Medical Systems Inc"
  ],
  "inventors": [
    "Srinivas Chukka",
    "Anindya Sarkar",
    "Quan Yuan"
  ],
  "filing_date": "2016-03-31",
  "publication_date": "2019-12-10",
  "grant_date": "2019-12-10",
  "priority_date": "2013-10-01",
  "application_number": "US-201615087753-A",
  "family_id": "51660471",
  "cited_by_count": 7,
  "citations": [
    "US6061476A",
    "US6251615B1",
    "JP2000342558A",
    "US20030118222A1",
    "US7899624B2",
    "US8131476B2",
    "US20100054560A1",
    "US20090028614A1",
    "US20100000383A1",
    "US20110040169A1",
    "US20130089249A1",
    "WO2011163624A1",
    "US20120082362A1",
    "US20120076390A1",
    "US20130156279A1",
    "EP2634749A2",
    "US20130230220A1",
    "US20140212037A1",
    "US20160019695A1",
    "US20170103521A1",
    "US20170309021A1"
  ]
}