HistoStitcher(©): an interactive program for accurate and rapid reconstruction of digitized whole histological sections from tissue fragments

Full resolution reconstruction of whole-mount sections from digitized individual tissue fragments

Whole-mount sectioning is a technique in histopathology where a full slice of tissue, such as a transversal cross-section of a prostate specimen, is prepared on a large microscope slide without further sectioning into smaller fragments. Although this technique can offer improved correlation with pre-operative imaging and is paramount for multimodal research, it is not commonly employed due to its technical difficulty, associated cost and cumbersome integration in (digital) pathology workflows. In this work, we present a computational tool named PythoStitcher which reconstructs artificial whole-mount sections from digitized tissue fragments, thereby bringing the benefits of whole-mount sections to pathology labs currently unable to employ this technique. Our proposed algorithm consists of a multi-step approach where it (i) automatically determines how fragments need to be reassembled, (ii) iteratively optimizes the stitch using a genetic algorithm and (iii) efficiently reconstructs the final artificial whole-mount section on full resolution (0.25 µm/pixel). PythoStitcher was validated on a total of 198 cases spanning five datasets with a varying number of tissue fragments originating from different organs from multiple centers. PythoStitcher successfully reconstructed the whole-mount section in 86-100% of cases for a given dataset with a residual registration mismatch of 0.65-2.76 mm on automatically selected landmarks. It is expected that our algorithm can aid pathology labs unable to employ whole-mount sectioning through faster clinical case evaluation and improved radiology-pathology correlation workflows.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Reconstructing virtual large slides can improve the accuracy and consistency of tumor bed evaluation for breast cancer after neoadjuvant therapy

BACKGROUND

To explore whether the "WSI Stitcher", a program we developed for reconstructing virtual large slide through whole slide imaging fragments stitching, can improve the efficiency and consistency of pathologists in evaluating the tumor bed after neoadjuvant treatment of breast cancer compared with the conventional methods through stack splicing of physical slides.

METHODS

This study analyzed the advantages of using software-assisted methods to evaluate the tumor bed after neoadjuvant treatment of breast cancer. This new method is to use "WSI Stitcher" to stitch all the WSI fragments together to reconstruct a virtual large slide and evaluate the tumor bed with the help of the built-in ruler and tumor proportion calculation functions.

RESULTS

Compared with the conventional method, the evaluation time of the software-assisted method was shortened by 35%(P < 0.001). In the process of tumor bed assessment after neoadjuvant treatment of breast cancer, the software-assisted method has higher intraclass correlation coefficient when measuring the length (0.994 versus 0.934), width (0.992 versus 0.927), percentage of residual tumor cells (0.947 versus 0.878), percentage of carcinoma in situ (0.983 versus 0.881) and RCB index(0.997 versus 0.772). The software-assisted method has higher kappa values when evaluating tumor staging(0.901 versus 0.687) and RCB grading (0.963 versus 0.857).

CONCLUSION

The "WSI Stitcher" is an effective tool to help pathologists with the assessment of breast cancer after neoadjuvant treatment.

Correlation of in-vivo imaging with histopathology: A review

Despite tremendous advancements in in vivo imaging modalities, there remains substantial uncertainty with respect to tumor delineation on in these images. Histopathology remains the gold standard for determining the extent of malignancy, with in vivo imaging to histopathologic correlation enabling spatial comparisons. In this review, the steps necessary for successful imaging to histopathologic correlation are described, including in vivo imaging, resection, fixation, specimen sectioning (sectioning technique, securing technique, orientation matching, slice matching), microtome sectioning and staining, correlation (including image registration) and performance evaluation. The techniques used for each of these steps are also discussed. Hundreds of publications from the past 20 years were surveyed, and 62 selected for detailed analysis. For these 62 publications, each stage of the correlative pathology process (and the sub-steps of specimen sectioning) are listed. A statistical analysis was conducted based on 19 studies that reported target registration error as their performance metric. While some methods promise greater accuracy, they may be expensive. Due to the complexity of the processes involved, correlative pathology studies generally include a small number of subjects, which hinders advanced developments in this field.

[New Trends of Development in Precision Pathological Diagnosis Promoted by Artificial Intelligence]

Precision pathological diagnosis plays a vital role in precision medicine. Both the limited resources available to pathologists and the incessant demands for further refinement and quantification of clinical diagnosis are posing new challenges for pathologists to meet the needs for precision pathological diagnosis. It is expected that artificial intelligence (AI) will be the powerful tool that will help find solutions to this problem from different angles. The author of this article elaborated on a number of ways in which AI can help promote precision pathological diagnosis, including AI-assisted precision extraction of tissue samples, AI-assisted precision histopathologic diagnosis, AI-assisted histological grading and quantitative scoring, AI-assisted precision assessment of tumor biomarkers, AI-assisted prediction of molecular features and precision interpretation of biological information based on hematoxylin-eosin (HE) stained images, the realization of in-depth precision diagnosis based on AI-assisted information integration, and AI-assisted accurate prediction of patient survival and prognosis based on HE-stained images. The paper presents to the readers the future of smart pathology that AI will help usher in.

3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

The use of MRI for prostate cancer diagnosis and treatment is increasing rapidly. However, identifying the presence and extent of cancer on MRI remains challenging, leading to high variability in detection even among expert radiologists. Improvement in cancer detection on MRI is essential to reducing this variability and maximizing the clinical utility of MRI. To date, such improvement has been limited by the lack of accurately labeled MRI datasets. Data from patients who underwent radical prostatectomy enables the spatial alignment of digitized histopathology images of the resected prostate with corresponding pre-surgical MRI. This alignment facilitates the delineation of detailed cancer labels on MRI via the projection of cancer from histopathology images onto MRI. We introduce a framework that performs 3D registration of whole-mount histopathology images to pre-surgical MRI in three steps. First, we developed a novel multi-image super-resolution generative adversarial network (miSRGAN), which learns information useful for 3D registration by producing a reconstructed 3D MRI. Second, we trained the network to learn information between histopathology slices to facilitate the application of 3D registration methods. Third, we registered the reconstructed 3D histopathology volumes to the reconstructed 3D MRI, mapping the extent of cancer from histopathology images onto MRI without the need for slice-to-slice correspondence. When compared to interpolation methods, our super-resolution reconstruction resulted in the highest PSNR relative to clinical 3D MRI (32.15 dB vs 30.16 dB for BSpline interpolation). Moreover, the registration of 3D volumes reconstructed via super-resolution for both MRI and histopathology images showed the best alignment of cancer regions when compared to (1) the state-of-the-art RAPSODI approach, (2) volumes that were not reconstructed, or (3) volumes that were reconstructed using nearest neighbor, linear, or BSpline interpolations. The improved 3D alignment of histopathology images and MRI facilitates the projection of accurate cancer labels on MRI, allowing for the development of improved MRI interpretation schemes and machine learning models to automatically detect cancer on MRI.

Nutil: A Pre- and Post-processing Toolbox for Histological Rodent Brain Section Images

With recent technological advances in microscopy and image acquisition of tissue sections, further developments of tools are required for viewing, transforming, and analyzing the ever-increasing amounts of high-resolution data produced. In the field of neuroscience, histological images of whole rodent brain sections are commonly used for investigating brain connections as well as cellular and molecular organization in the normal and diseased brain, but present a problem for the typical neuroscientist with no or limited programming experience in terms of the pre- and post-processing steps needed for analysis. To meet this need we have designed Nutil, an open access and stand-alone executable software that enables automated transformations, post-processing, and analyses of 2D section images using multi-core processing (OpenMP). The software is written in C++ for efficiency, and provides the user with a clean and easy graphical user interface for specifying the input and output parameters. Nutil currently contains four separate tools: (1) A transformation toolchain named “Transform” that allows for rotation, mirroring and scaling, resizing, and renaming of very large tiled tiff images. (2) “TiffCreator” enables the generation of tiled TIFF images from other image formats such as PNG and JPEG. (3) A “Resize” tool completes the preprocessing toolset and allows downscaling of PNG and JPEG images with output in PNG format. (4) The fourth tool is a post-processing method called “Quantifier” that enables the quantification of segmented objects in the context of regions defined by brain atlas maps generated with the QuickNII software based on a 3D reference atlas (mouse or rat). The output consists of a set of report files, point cloud coordinate files for visualization in reference atlas space, and reference atlas images superimposed with color-coded objects. The Nutil software is made available by the Human Brain Project (https://www.humanbrainproject.eu) at https://www.nitrc.org/projects/nutil/.

Cryogenic sequenced layering for the 3D reconstruction of biological objects

Three-dimensional (3D) visualization is applied throughout many specialities, prompting an important breakthrough in accessibility and modeling of data. Experimental rendering and computerized reconstruction of objects has influenced many scientific achievements, facilitating one of the greatest advancements in medical education since the first illustrated anatomy book changed specialist training forever. Modern medicine relies on detailed, high quality virtual models for educational, experimental and clinical purposes. Almost all current virtual visualization methods rely on object slicing producing serial sections, which can then be digitalized or analyzed manually. The tendency to computerize serial sections roots from convenience, accessibility, decent visualization quality and automation capabilities. Drawbacks of serial section imaging is tissue damage occurring within each consequent sectioning. To utilize the important aspects of real-life object reconstruction, and maintain integrity of biological structures, we suggest a novel method of low-temperature layering of objects for digitization and computerized virtual reconstruction. Here we show the process of consequent imaging of each novel layer of a biological object, which provides a computer with high quality data for virtual reconstruction and creation of a multidimensional real-life model. Our method prevents tissue deformation and biodegradation due to specific methods used in preparation of the biological object. The resulting images can be applied in surgical training, medical education and numerous scientific fields for realistic reconstruction of biological objects.

HISTOBREAST, a collection of brightfield microscopy images of Haematoxylin and Eosin stained breast tissue

Modern histopathology workflows rely on the digitization of histology slides. The quality of the resulting digital representations, in the form of histology slide image mosaics, depends on various specific acquisition conditions and on the image processing steps that underlie the generation of the final mosaic, e.g. registration and blending of the contained image tiles. We introduce HISTOBREAST, an extensive collection of brightfield microscopy images that we collected in a principled manner under different acquisition conditions on Haematoxylin - Eosin (H&E) stained breast tissue. HISTOBREAST is comprised of neighbour image tiles and ensemble of mosaics composed from different combinations of the available image tiles, exhibiting progressively degraded quality levels. HISTOBREAST can be used to benchmark image processing and computer vision techniques with respect to their robustness to image modifications specific to brightfield microscopy of H&E stained tissues. Furthermore, HISTOBREAST can serve in the development of new image processing methods, with the purpose of ensuring robustness to typical image artefacts that raise interpretation problems for expert histopathologists and affect the results of computerized image analysis.

Histo-genomics: digital pathology at the forefront of precision medicine

The toughest challenge OMICs face is that they provide extremely high molecular resolution but poor spatial information. Understanding the cellular/histological context of the overwhelming genetic data is critical for a full understanding of the clinical behavior of a malignant tumor. Digital pathology can add an extra layer of information to help visualize in a spatial and microenvironmental context the molecular information of cancer. Thus, histo-genomics provide a unique chance for data integration. In the era of a precision medicine, a four-dimensional (4D) (temporal/spatial) analysis of cancer aided by digital pathology can be a critical step to understand the evolution/progression of different cancers and consequently tailor individual treatment plans. For instance, the integration of molecular biomarkers expression into a three-dimensional (3D) image of a digitally scanned tumor can offer a better understanding of its subtype, behavior, host immune response and prognosis. Using advanced digital image analysis, a larger spectrum of parameters can be analyzed as potential predictors of clinical behavior. Correlation between morphological features and host immune response can be also performed with therapeutic implications. Radio-histomics, or the interface of radiological images and histology is another emerging exciting field which encompasses the integration of radiological imaging with digital pathological images, genomics, and clinical data to portray a more holistic approach to understating and treating disease. These advances in digital slide scanning are not without technical challenges, which will be addressed carefully in this review with quick peek at its future.

Identifying the morphologic basis for radiomic features in distinguishing different Gleason grades of prostate cancer on MRI: Preliminary findings

Translation of radiomics into the clinic may require a more comprehensive understanding of the underlying morphologic tissue characteristics they reflect. In the context of prostate cancer (PCa), some studies have correlated gross histological measurements of gland lumen, epithelium, and nuclei with disease appearance on MRI. Quantitative histomorphometry (QH), like radiomics for radiologic images, is the computer based extraction of features for describing tumor morphology on digitized tissue images. In this work, we attempt to establish the histomorphometric basis for radiomic features for prostate cancer by (1) identifying the radiomic features from T2w MRI most discriminating of low vs. intermediate/high Gleason score, (2) identifying QH features correlated with the most discriminating radiomic features previously identified, and (3) evaluating the discriminative ability of QH features found to be correlated with spatially co-localized radiomic features. On a cohort of 36 patients (23 for training, 13 for validation), Gabor texture features were identified as being most predictive of Gleason grade on MRI (AUC of 0.69) and gland lumen shape features were identified as the most predictive QH features (AUC = 0.75). Our results suggest that the PCa grade discriminability of Gabor features is a consequence of variations in gland shape and morphology at the tissue level.

High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: Application to invasive breast cancer detection

Precise detection of invasive cancer on whole-slide images (WSI) is a critical first step in digital pathology tasks of diagnosis and grading. Convolutional neural network (CNN) is the most popular representation learning method for computer vision tasks, which have been successfully applied in digital pathology, including tumor and mitosis detection. However, CNNs are typically only tenable with relatively small image sizes (200 × 200 pixels). Only recently, Fully convolutional networks (FCN) are able to deal with larger image sizes (500 × 500 pixels) for semantic segmentation. Hence, the direct application of CNNs to WSI is not computationally feasible because for a WSI, a CNN would require billions or trillions of parameters. To alleviate this issue, this paper presents a novel method, High-throughput Adaptive Sampling for whole-slide Histopathology Image analysis (HASHI), which involves: i) a new efficient adaptive sampling method based on probability gradient and quasi-Monte Carlo sampling, and, ii) a powerful representation learning classifier based on CNNs. We applied HASHI to automated detection of invasive breast cancer on WSI. HASHI was trained and validated using three different data cohorts involving near 500 cases and then independently tested on 195 studies from The Cancer Genome Atlas. The results show that (1) the adaptive sampling method is an effective strategy to deal with WSI without compromising prediction accuracy by obtaining comparative results of a dense sampling (∼6 million of samples in 24 hours) with far fewer samples (∼2,000 samples in 1 minute), and (2) on an independent test dataset, HASHI is effective and robust to data from multiple sites, scanners, and platforms, achieving an average Dice coefficient of 76%.

Co-Registration of ex vivo Surgical Histopathology and in vivo T2 weighted MRI of the Prostate via multi-scale spectral embedding representation

Multi-modal image co-registration via optimizing mutual information (MI) is based on the assumption that intensity distributions of multi-modal images follow a consistent relationship. However, images with a substantial difference in appearance violate this assumption, thus MI directly based on image intensity alone may be inadequate to drive similarity based co-registration. To address this issue, we introduce a novel approach for multi-modal co-registration called Multi-scale Spectral Embedding Registration (MSERg). MSERg involves the construction of multi-scale spectral embedding (SE) representations from multimodal images via texture feature extraction, scale selection, independent component analysis (ICA) and SE to create orthogonal representations that decrease the dissimilarity between the fixed and moving images to facilitate better co-registration. To validate the MSERg method, we aligned 45 pairs of in vivo prostate MRI and corresponding ex vivo histopathology images. The dataset was split into a learning set and a testing set. In the learning set, length scales of 5 × 5, 7 × 7 and 17 × 17 were selected. In the independent testing set, we compared MSERg with intensity-based registration, multi-attribute combined mutual information (MACMI) registration and scale-invariant feature transform (SIFT) flow registration. Our results suggest that multi-scale SE representations generated by MSERg are found to be more appropriate for radiology-pathology co-registration.

Deformable image registration between pathological images and MR image via an optical macro image

Computed tomography (CT) and magnetic resonance (MR) imaging have been widely used for visualizing the inside of the human body. However, in many cases, pathological diagnosis is conducted through a biopsy or resection of an organ to evaluate the condition of tissues as definitive diagnosis. To provide more advanced information onto CT or MR image, it is necessary to reveal the relationship between tissue information and image signals. We propose a registration scheme for a set of PT images of divided specimens and a 3D-MR image by reference to an optical macro image (OM image) captured by an optical camera. We conducted a fundamental study using a resected human brain after the death of a brain cancer patient. We constructed two kinds of registration processes using the OM image as the base for both registrations to make conversion parameters between the PT and MR images. The aligned PT images had shapes similar to the OM image. On the other hand, the extracted cross-sectional MR image was similar to the OM image. From these resultant conversion parameters, the corresponding region on the PT image could be searched and displayed when an arbitrary pixel on the MR image was selected. The relationship between the PT and MR images of the whole brain can be analyzed using the proposed method. We confirmed that same regions between the PT and MR images could be searched and displayed using resultant information obtained by the proposed method. In terms of the accuracy of proposed method, the TREs were 0.56±0.39mm and 0.87±0.42mm. We can analyze the relationship between tissue information and MR signals using the proposed method.

AutoStitcher: An Automated Program for Efficient and Robust Reconstruction of Digitized Whole Histological Sections from Tissue Fragments

In applications involving large tissue specimens that have been sectioned into smaller tissue fragments, manual reconstruction of a “pseudo whole-mount” histological section (PWMHS) can facilitate (a) pathological disease annotation, and (b) image registration and correlation with radiological images. We have previously presented a program called HistoStitcher, which allows for more efficient manual reconstruction than general purpose image editing tools (such as Photoshop). However HistoStitcher is still manual and hence can be laborious and subjective, especially when doing large cohort studies. In this work we present AutoStitcher, a novel automated algorithm for reconstructing PWMHSs from digitized tissue fragments. AutoStitcher reconstructs (“stitches”) a PWMHS from a set of 4 fragments by optimizing a novel cost function that is domain-inspired to ensure (i) alignment of similar tissue regions, and (ii) contiguity of the prostate boundary. The algorithm achieves computational efficiency by performing reconstruction in a multi-resolution hierarchy. Automated PWMHS reconstruction results (via AutoStitcher) were quantitatively and qualitatively compared to manual reconstructions obtained via HistoStitcher for 113 prostate pathology sections. Distances between corresponding fiducials placed on each of the automated and manual reconstruction results were between 2.7%–3.2%, reflecting their excellent visual similarity.

Development of a 2D Image Reconstruction and Viewing System for Histological Images from Multiple Tissue Blocks: Towards High-Resolution Whole-Organ 3D Histological Images

High-resolution 3D histology image reconstruction of the whole brain organ starts from reconstructing the high-resolution 2D histology images of a brain slice. In this paper, we introduced a method to automatically align the histology images of thin tissue sections cut from the multiple paraffin-embedded tissue blocks of a brain slice. For this method, we employed template matching and incorporated an optimization technique to further improve the accuracy of the 2D reconstructed image. In the template matching, we used the gross image of the brain slice as a reference to the reconstructed 2D histology image of the slice, while in the optimization procedure, we utilized the Jaccard index as the metric of the reconstruction accuracy. The results of our experiment on the initial 3 different whole-brain tissue slices showed that while the method works, it is also constrained by tissue deformations introduced during the tissue processing and slicing. The size of the reconstructed high-resolution 2D histology image of a brain slice is huge, and designing an image viewer that makes particularly efficient use of the computing power of a standard computer used in our laboratories is of interest. We also present the initial implementation of our 2D image viewer system in this paper.

Identifying in vivo DCE MRI markers associated with microvessel architecture and gleason grades of prostate cancer

BACKGROUND

To identify computer extracted in vivo dynamic contrast enhanced (DCE) MRI markers associated with quantitative histomorphometric (QH) characteristics of microvessels and Gleason scores (GS) in prostate cancer.

METHODS

This study considered retrospective data from 23 biopsy confirmed prostate cancer patients who underwent 3 Tesla multiparametric MRI before radical prostatectomy (RP). Representative slices from RP specimens were stained with vascular marker CD31. Tumor extent was mapped from RP sections onto DCE MRI using nonlinear registration methods. Seventy-seven microvessel QH features and 18 DCE MRI kinetic features were extracted and evaluated for their ability to distinguish low from intermediate and high GS. The effect of temporal sampling on kinetic features was assessed and correlations between those robust to temporal resolution and microvessel features discriminative of GS were examined.

RESULTS

A total of 12 microvessel architectural features were discriminative of low and intermediate/high grade tumors with area under the receiver operating characteristic curve (AUC) > 0.7. These features were most highly correlated with mean washout gradient (WG) (max rho = -0.62). Independent analysis revealed WG to be moderately robust to temporal resolution (intraclass correlation coefficient [ICC] = 0.63) and WG variance, which was poorly correlated with microvessel features, to be predictive of low grade tumors (AUC = 0.77). Enhancement ratio was the most robust (ICC = 0.96) and discriminative (AUC = 0.78) kinetic feature but was moderately correlated with microvessel features (max rho = -0.52).

CONCLUSION

Computer extracted features of prostate DCE MRI appear to be correlated with microvessel architecture and may be discriminative of low versus intermediate and high GS.

Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images

Technological advances in computing, imaging, and genomics have created new opportunities for exploring relationships between histology, molecular events, and clinical outcomes using quantitative methods. Slide scanning devices are now capable of rapidly producing massive digital image archives that capture histological details in high resolution. Commensurate advances in computing and image analysis algorithms enable mining of archives to extract descriptions of histology, ranging from basic human annotations to automatic and precisely quantitative morphometric characterization of hundreds of millions of cells. These imaging capabilities represent a new dimension in tissue-based studies, and when combined with genomic and clinical endpoints, can be used to explore biologic characteristics of the tumor microenvironment and to discover new morphologic biomarkers of genetic alterations and patient outcomes. In this paper, we review developments in quantitative imaging technology and illustrate how image features can be integrated with clinical and genomic data to investigate fundamental problems in cancer. Using motivating examples from the study of glioblastomas (GBMs), we demonstrate how public data from The Cancer Genome Atlas (TCGA) can serve as an open platform to conduct in silico tissue-based studies that integrate existing data resources. We show how these approaches can be used to explore the relation of the tumor microenvironment to genomic alterations and gene expression patterns and to define nuclear morphometric features that are predictive of genetic alterations and clinical outcomes. Challenges, limitations, and emerging opportunities in the area of quantitative imaging and integrative analyses are also discussed.

Histostitcher™: An informatics software platform for reconstructing whole-mount prostate histology using the extensible imaging platform framework

CONTEXT

Co-registration of ex-vivo histologic images with pre-operative imaging (e.g., magnetic resonance imaging [MRI]) can be used to align and map disease extent, and to identify quantitative imaging signatures. However, ex-vivo histology images are frequently sectioned into quarters prior to imaging.

AIMS

This work presents Histostitcher™, a software system designed to create a pseudo whole mount histology section (WMHS) from a stitching of four individual histology quadrant images.

MATERIALS AND METHODS

Histostitcher™ uses user-identified fiducials on the boundary of two quadrants to stitch such quadrants. An original prototype of Histostitcher™ was designed using the Matlab programming languages. However, clinical use was limited due to slow performance, computer memory constraints and an inefficient workflow. The latest version was created using the extensible imaging platform (XIP™) architecture in the C++ programming language. A fast, graphics processor unit renderer was designed to intelligently cache the visible parts of the histology quadrants and the workflow was significantly improved to allow modifying existing fiducials, fast transformations of the quadrants and saving/loading sessions.

RESULTS

The new stitching platform yielded significantly more efficient workflow and reconstruction than the previous prototype. It was tested on a traditional desktop computer, a Windows 8 Surface Pro table device and a 27 inch multi-touch display, with little performance difference between the different devices.

CONCLUSIONS

Histostitcher™ is a fast, efficient framework for reconstructing pseudo WMHS from individually imaged quadrants. The highly modular XIP™ framework was used to develop an intuitive interface and future work will entail mapping the disease extent from the pseudo WMHS onto pre-operative MRI.

Spatially weighted mutual information (SWMI) for registration of digitally reconstructed ex vivo whole mount histology and in vivo prostate MRI

In this work, we present a scheme for the registration of digitally reconstructed whole mount histology (WMH) to pre-operative in vivo multiprotocol prostate MR imagery (T2w and DCE) using spatially weighted mutual information (SWMI). Spatial alignment of ex vivo histological sections to pre-operative in vivo MRI for prostate cancer (CaP) patients undergoing radical prostatectomy is a necessary first step in the discovery of quantitative multiprotocol MRI signatures for CaP. This may be done by spatially mapping delineated extent of disease on ex vivo histopathology onto pre-operative in vivo MRI via image registration. Apart from the challenges in spatially registering multi-modal data (histology and MRI) on account of (a) modality specific differences, (b) deformation due to the endorectal coil and tissue loss on histology, another complication is that the ex vivo histological sections, in the lab, are usually obtained as quadrants. This means they need to be reconstituted as a pseudo-whole mount histologic section (WMHS) prior to registration with MRI. An additional challenge is that most registration techniques rely on availability of the pre-segmented prostate capsule on T2w MRI. The novel contribution of this paper is that it leverages a spatially weighted mutual information (SWMI) scheme to automatically register and map CaP extent from WMHS onto pre-operative, multiprotocol MRI. The SWMI scheme obviates the need for pre-segmentation of the prostate capsule on MRI. Additionally, we leverage a program developed by our group, Histostitcher©, for interactive stitching of individual histology quadrants to digitally reconstruct the pseudo WMHS. Our registration methodology comprises the following main steps, (1) affine registration of T2w and DCE MRI, (2) affine registration of stitched WMHS to multiprotocol T2w and DCE MRI, and (3) multimodal image registration of WMHS to multiprotocol T2w and DCE MRI using SWMI. We quantitatively and qualitatively evaluated all aspects of our methodology in the multimodal registration of a total of 7 corresponding histology and MRI sections from 2 different patients. For the 7 studies, we obtained an average Hausdorff distance of 1.85 mm, mean absolute distance of 0.99 mm, RMS of 1.65 mm, and DICE of 0.83, when comparing the capsular alignment on MRI to histology.

Global error minimization in image mosaicing using graph connectivity and its applications in microscopy

Several applications such as multiprojector displays and microscopy require the mosaicing of images (tiles) acquired by a camera as it traverses an unknown trajectory in 3D space. A homography relates the image coordinates of a point in each tile to those of a reference tile provided the 3D scene is planar. Our approach in such applications is to first perform pairwise alignment of the tiles that have imaged common regions in order to recover a homography relating the tile pair. We then find the global set of homographies relating each individual tile to a reference tile such that the homographies relating all tile pairs are kept as consistent as possible. Using these global homographies, one can generate a mosaic of the entire scene. We derive a general analytical solution for the global homographies by representing the pair-wise homographies on a connectivity graph. Our solution can accommodate imprecise prior information regarding the global homographies whenever such information is available. We also derive equations for the special case of translation estimation of an X-Y microscopy stage used in histology imaging and present examples of stitched microscopy slices of specimens obtained after radical prostatectomy or prostate biopsy. In addition, we demonstrate the superiority of our approach over tree-structured approaches for global error minimization.

Elastic registration of multimodal prostate MRI and histology via multiattribute combined mutual information

PURPOSE

By performing registration of preoperative multiprotocol in vivo magnetic resonance (MR) images of the prostate with corresponding whole-mount histology (WMH) sections from postoperative radical prostatectomy specimens, an accurate estimate of the spatial extent of prostate cancer (CaP) on in vivo MR imaging (MRI) can be retrospectively established. This could allow for definition of quantitative image-based disease signatures and lead to development of classifiers for disease detection on multiprotocol in vivo MRI. Automated registration of MR and WMH images of the prostate is complicated by dissimilar image intensities, acquisition artifacts, and nonlinear shape differences.

METHODS

The authors present a method for automated elastic registration of multiprotocol in vivo MRI and WMH sections of the prostate. The method, multiattribute combined mutual information (MACMI), leverages all available multiprotocol image data to drive image registration using a multivariate formulation of mutual information.

RESULTS

Elastic registration using the multivariate MI formulation is demonstrated for 150 corresponding sets of prostate images from 25 patient studies with T2-weighted and dynamic-contrast enhanced MRI and 85 image sets from 15 studies with an additional functional apparent diffusion coefficient MRI series. Qualitative results of MACMI evaluation via visual inspection suggest that an accurate delineation of CaP extent on MRI is obtained. Results of quantitative evaluation on 150 clinical and 20 synthetic image sets indicate improved registration accuracy using MACMI compared to conventional pairwise mutual information-based approaches.

CONCLUSIONS

The authors' approach to the registration of in vivo multiprotocol MRI and ex vivo WMH of the prostate using MACMI is unique, in that (1) information from all available image protocols is utilized to drive the registration with histology, (2) no additional, intermediate ex vivo radiology or gross histology images need be obtained in addition to the routinely acquired in vivo MRI series, and (3) no corresponding anatomical landmarks are required to be identified manually or automatically on the images.