Time-efficient sparse analysis of histopathological whole slide images

Multiplex Immunofluorescence and the Digital Image Analysis Workflow for Evaluation of the Tumor Immune Environment in Translational Research

A robust understanding of the tumor immune environment has important implications for cancer diagnosis, prognosis, research, and immunotherapy. Traditionally, immunohistochemistry (IHC) has been regarded as the standard method for detecting proteins in situ, but this technique allows for the evaluation of only one cell marker per tissue sample at a time. However, multiplexed imaging technologies enable the multiparametric analysis of a tissue section at the same time. Also, through the curation of specific antibody panels, these technologies enable researchers to study the cell subpopulations within a single immunological cell group. Thus, multiplexed imaging gives investigators the opportunity to better understand tumor cells, immune cells, and the interactions between them. In the multiplexed imaging technology workflow, once the protocol for a tumor immune micro environment study has been defined, histological slides are digitized to produce high-resolution images in which regions of interest are selected for the interrogation of simultaneously expressed immunomarkers (including those co-expressed by the same cell) by using an image analysis software and algorithm. Most currently available image analysis software packages use similar machine learning approaches in which tissue segmentation first defines the different components that make up the regions of interest and cell segmentation, then defines the different parameters, such as the nucleus and cytoplasm, that the software must utilize to segment single cells. Image analysis tools have driven dramatic evolution in the field of digital pathology over the past several decades and provided the data necessary for translational research and the discovery of new therapeutic targets. The next step in the growth of digital pathology is optimization and standardization of the different tasks in cancer research, including image analysis algorithm creation, to increase the amount of data generated and their accuracy in a short time as described herein. The aim of this review is to describe this process, including an image analysis algorithm creation for multiplex immunofluorescence analysis, as an essential part of the optimization and standardization of the different processes in cancer research, to increase the amount of data generated and their accuracy in a short time.

Computer-Aided Histopathological Image Analysis Techniques for Automated Nuclear Atypia Scoring of Breast Cancer: a Review

Breast cancer is the most common type of malignancy diagnosed in women. Through early detection and diagnosis, there is a great chance of recovery and thereby reduce the mortality rate. Many preliminary tests like non-invasive radiological diagnosis using ultrasound, mammography, and MRI are widely used for the diagnosis of breast cancer. However, histopathological analysis of breast biopsy specimen is inevitable and is considered to be the golden standard for the affirmation of cancer. With the advancements in the digital computing capabilities, memory capacity, and imaging modalities, the development of computer-aided powerful analytical techniques for histopathological data has increased dramatically. These automated techniques help to alleviate the laborious work of the pathologist and to improve the reproducibility and reliability of the interpretation. This paper reviews and summarizes digital image computational algorithms applied on histopathological breast cancer images for nuclear atypia scoring and explores the future possibilities. The algorithms for nuclear pleomorphism scoring of breast cancer can be widely grouped into two categories: handcrafted feature-based and learned feature-based. Handcrafted feature-based algorithms mainly include the computational steps like pre-processing the images, segmenting the nuclei, extracting unique features, feature selection, and machine learning-based classification. However, most of the recent algorithms are based on learned features, that extract high-level abstractions directly from the histopathological images utilizing deep learning techniques. In this paper, we discuss the various algorithms applied for the nuclear pleomorphism scoring of breast cancer, discourse the challenges to be dealt with, and outline the importance of benchmark datasets. A comparative analysis of some prominent works on breast cancer nuclear atypia scoring is done using a benchmark dataset which enables to quantitatively measure and compare the different features and algorithms used for breast cancer grading. Results show that improvements are still required, to have an automated cancer grading system suitable for clinical applications.

Machine Learning Methods for Histopathological Image Analysis: A Review

Histopathological images (HIs) are the gold standard for evaluating some types of tumors for cancer diagnosis. The analysis of such images is time and resource-consuming and very challenging even for experienced pathologists, resulting in inter-observer and intra-observer disagreements. One of the ways of accelerating such an analysis is to use computer-aided diagnosis (CAD) systems. This paper presents a review on machine learning methods for histopathological image analysis, including shallow and deep learning methods. We also cover the most common tasks in HI analysis, such as segmentation and feature extraction. Besides, we present a list of publicly available and private datasets that have been used in HI research.

Batch Mode Active Learning on the Riemannian Manifold for Automated Scoring of Nuclear Pleomorphism in Breast Cancer

Breast cancer is the most prevalent invasive type of cancer among women. The mortality rate of the disease can be reduced considerably through timely prognosis and felicitous treatment planning, by utilizing the computer aided detection and diagnosis techniques. With the advent of whole slide image (WSI) scanners for digitizing the histopathological tissue samples, there is a drastic increase in the availability of digital histopathological images. However, these samples are often unlabeled and hence they need labeling to be done through manual annotations by domain experts and experienced pathologists. But this annotation process required for acquiring high quality large labeled training set for nuclear atypia scoring is a tedious, expensive and time consuming job. Active learning techniques have achieved widespread acceptance in reducing this human effort in annotating the data samples. In this paper, we explore the possibilities of active learning on nuclear pleomorphism scoring over a non-Euclidean framework, the Riemannian manifold. Active learning technique adopted for the cancer grading is in the batch-mode framework, that adaptively identifies the apt batch size along with the batch of instances to be queried, following a submodular optimization framework. Samples for annotation are selected considering the diversity and redundancy between the pair of samples, based on the kernelized Riemannian distance measures such as log-Euclidean metrics and the two Bregman divergences - Stein and Jeffrey divergences. Results of the adaptive Batch Mode Active Learning on the Riemannian metric show a superior performance when compared with the state-of-the-art techniques for breast cancer nuclear pleomorphism scoring, as it makes use of the information from the unlabeled samples.

Sparse Representation Over Learned Dictionaries on the Riemannian Manifold for Automated Grading of Nuclear Pleomorphism in Breast Cancer

Breast cancer is found to be the most pervasive type of cancer among women. Computer aided detection and diagnosis of cancer at the initial stages can increase the chances of recovery and thus reduce the mortality rate through timely prognosis and adequate treatment planning. The nuclear atypia scoring or histopathological breast tumor grading remains to be a challenging problem due to the various artifacts and variabilities introduced during slide preparation and also because of the complexity in the structure of the underlying tissue patterns. Inspired by the success of symmetric positive definite (SPD) matrices in many of the challenging tasks in machine learning and computer vision, a sparse coding and dictionary learning on SPD matrices is proposed in this paper for the breast tumor grading. The proposed covariance-based SPD matrices form a Riemannian manifold and are represented as the sparse combination of Riemannian dictionary atoms. Non-linearity of the SPD manifold is tackled by embedding into the reproducing kernel Hilbert space using kernels derived from log-Euclidean metric, Jeffrey and Stein divergences and compared with the non-kernel-based affine invariant Riemannian metric. The novelty of the work lies in exploiting the kernel approach for the Hilbert space embedding of the Riemannian manifold, that can achieve a better discrimination of the breast cancer tissues, following a sparse representation over learned dictionaries and henceforth it outperforms many of the state-of-the-art algorithms in breast cancer grading in terms of quantitative and qualitative analysis.

Automated Computational Detection, Quantitation, and Mapping of Mitosis in Whole-Slide Images for Clinically Actionable Surgical Pathology Decision Support

Background

Determining mitotic index by counting mitotic figures (MFs) microscopically from tumor areas with most abundant MF (hotspots [HS]) produces a prognostically useful tumor grading biomarker. However, interobserver concordance identifying MF and HS can be poorly reproducible. Immunolabeling MF, coupled with computer-automated counting by image analysis, can improve reproducibility. A computational system for obtaining MF values across digitized whole-slide images (WSIs) was sought that would minimize impact of artifacts, generate values clinically relatable to counting ten high-power microscopic fields of view typical in conventional microscopy, and that would reproducibly map HS topography.

Materials and Methods

Relatively low-resolution WSI scans (0.50 μm/pixel) were imported in grid-tile format for feature-based MF segmentation, from naturally occurring canine melanomas providing a wide range of proliferative activity. MF feature extraction conformed to anti-phospho-histone H3-immunolabeled mitotic (M) phase cells. Computer vision image processing was established to subtract key artifacts, obtain MF counts, and employ rotationally invariant feature extraction to map MF topography.

Results

The automated topometric HS (TMHS) algorithm identified mitotic HS and mapped select tissue tiles with greatest MF counts back onto WSI thumbnail images to plot HS topographically. Influence of dye, pigment, and extraneous structure artifacts was minimized. TMHS diagnostic decision support included image overlay graphics of HS topography, as well as a spreadsheet and plot of tile-based MF count values. TMHS performance was validated examining both mitotic HS counting and mapping functions. Significantly correlated TMHS MF mapping and metrics were demonstrated using repeat analysis with WSI in different orientation (R ² = 0.9916) and by agreement with a pathologist (R ² = 0.8605) as well as through assessment of counting function using an independently tuned object counting algorithm (OCA) (R ² = 0.9482). Limits of agreement analysis support method interchangeability. MF counts obtained led to accurate patient survival prediction in all (n = 30) except one case. By contrast, more variable performance was documented when several pathologists examined similar cases using microscopy (pair-wise correlations, rho range = 0.7597-0.9286).

Conclusions

Automated TMHS MF segmentation and feature engineering performance were interchangeable with both observer and OCA in digital mode. Moreover, enhanced HS location accuracy and superior method reproducibility were achieved using the automated TMHS algorithm compared to the current practice employing clinical microscopy.

High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: Application to invasive breast cancer detection

Precise detection of invasive cancer on whole-slide images (WSI) is a critical first step in digital pathology tasks of diagnosis and grading. Convolutional neural network (CNN) is the most popular representation learning method for computer vision tasks, which have been successfully applied in digital pathology, including tumor and mitosis detection. However, CNNs are typically only tenable with relatively small image sizes (200 × 200 pixels). Only recently, Fully convolutional networks (FCN) are able to deal with larger image sizes (500 × 500 pixels) for semantic segmentation. Hence, the direct application of CNNs to WSI is not computationally feasible because for a WSI, a CNN would require billions or trillions of parameters. To alleviate this issue, this paper presents a novel method, High-throughput Adaptive Sampling for whole-slide Histopathology Image analysis (HASHI), which involves: i) a new efficient adaptive sampling method based on probability gradient and quasi-Monte Carlo sampling, and, ii) a powerful representation learning classifier based on CNNs. We applied HASHI to automated detection of invasive breast cancer on WSI. HASHI was trained and validated using three different data cohorts involving near 500 cases and then independently tested on 195 studies from The Cancer Genome Atlas. The results show that (1) the adaptive sampling method is an effective strategy to deal with WSI without compromising prediction accuracy by obtaining comparative results of a dense sampling (∼6 million of samples in 24 hours) with far fewer samples (∼2,000 samples in 1 minute), and (2) on an independent test dataset, HASHI is effective and robust to data from multiple sites, scanners, and platforms, achieving an average Dice coefficient of 76%.

Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: a Comparative Study

Whole slide digital imaging technology enables researchers to study pathologists' interpretive behavior as they view digital slides and gain new understanding of the diagnostic medical decision-making process. In this study, we propose a simple yet important analysis to extract diagnostically relevant regions of interest (ROIs) from tracking records using only pathologists' actions as they viewed biopsy specimens in the whole slide digital imaging format (zooming, panning, and fixating). We use these extracted regions in a visual bag-of-words model based on color and texture features to predict diagnostically relevant ROIs on whole slide images. Using a logistic regression classifier in a cross-validation setting on 240 digital breast biopsy slides and viewport tracking logs of three expert pathologists, we produce probability maps that show 74 % overlap with the actual regions at which pathologists looked. We compare different bag-of-words models by changing dictionary size, visual word definition (patches vs. superpixels), and training data (automatically extracted ROIs vs. manually marked ROIs). This study is a first step in understanding the scanning behaviors of pathologists and the underlying reasons for diagnostic errors.

Parallel multiple instance learning for extremely large histopathology image analysis

Background

Histopathology images are critical for medical diagnosis, e.g., cancer and its treatment. A standard histopathology slice can be easily scanned at a high resolution of, say, 200,000×200,000 pixels. These high resolution images can make most existing imaging processing tools infeasible or less effective when operated on a single machine with limited memory, disk space and computing power.

Results

In this paper, we propose an algorithm tackling this new emerging “big data” problem utilizing parallel computing on High-Performance-Computing (HPC) clusters. Experimental results on a large-scale data set (1318 images at a scale of 10 billion pixels each) demonstrate the efficiency and effectiveness of the proposed algorithm for low-latency real-time applications.

Conclusions

The framework proposed an effective and efficient system for extremely large histopathology image analysis. It is based on the multiple instance learning formulation for weakly-supervised learning for image classification, segmentation and clustering. When a max-margin concept is adopted for different clusters, we obtain further improvement in clustering performance.

Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: A review

With the advance of digital pathology, image analysis has begun to show its advantages in information analysis of hematoxylin and eosin histopathology images. Generally, histological features in hematoxylin and eosin images are measured to evaluate tumor grade and prognosis for breast cancer. This review summarized recent works in image analysis of hematoxylin and eosin histopathology images for breast cancer prognosis. First, prognostic factors for breast cancer based on hematoxylin and eosin histopathology images were summarized. Then, usual procedures of image analysis for breast cancer prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Finally, the prognostic value of image features and image feature–based prognostic models was evaluated. Moreover, we discussed the issues of current analysis, and some directions for future research.

Assisting the examination of large histopathological slides with adaptive forests

The examination of biopsy samples plays a central role in the diagnosis and staging of numerous diseases, including most cancer types. However, because of the large size of the acquired images, the localization and quantification of diseased portions of a tissue is usually time-consuming, as pathologists must scroll through the whole slide to look for objects of interest which are often only scarcely distributed. In this work, we introduce an approach to facilitate the visual inspection of large digital histopathological slides. Our method builds on a random forest classifier trained to segment the structures sought by the pathologist. However, moving beyond the pixelwise segmentation task, our main contribution is an interactive exploration framework including: (i) a region scoring function which is used to rank and sequentially display regions of interest to the user, and (ii) a relevance feedback capability which leverages human annotations collected on each suggested region. Thereby, an online domain adaptation of the learned pixelwise segmentation model is performed, so that the region scores adapt on-the-fly to possible discrepancies between the original training data and the slide at hand. Three real-time update strategies are compared, including a novel approach based on online gradient descent which supports faster user interaction than an accurate delineation of objects. Our method is evaluated on the task of extramedullary hematopoiesis quantification within mouse liver slides. We assess quantitatively the retrieval abilities of our approach and the benefit of the interactive adaptation scheme. Moreover, we demonstrate the possibility of extrapolating, after a partial exploration of the slide, the surface covered by hematopoietic cells within the whole tissue.

Integrative Analysis of Cellular Morphometric Context Reveals Clinically Relevant Signatures in Lower Grade Glioma

Integrative analysis based on quantitative representation of whole slide images (WSIs) in a large histology cohort may provide predictive models of clinical outcome. On one hand, the efficiency and effectiveness of such representation is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. On the other hand, perceptual interpretation/validation of important multivariate phenotypic signatures are often difficult due to the loss of visual information during feature transformation in hyperspace. To address these issues, we propose a novel approach for integrative analysis based on cellular morphometric context, which is a robust representation of WSI, with the emphasis on tumor architecture and tumor heterogeneity, built upon cellular level morphometric features within the spatial pyramid matching (SPM) framework. The proposed approach is applied to The Cancer Genome Atlas (TCGA) lower grade glioma (LGG) cohort, where experimental results (i) reveal several clinically relevant cellular morphometric types, which enables both perceptual interpretation/validation and further investigation through gene set enrichment analysis; and (ii) indicate the significantly increased survival rates in one of the cellular morphometric context subtypes derived from the cellular morphometric context.

When machine vision meets histology: A comparative evaluation of model architecture for classification of histology sections

Classification of histology sections in large cohorts, in terms of distinct regions of microanatomy (e.g., stromal) and histopathology (e.g., tumor, necrosis), enables the quantification of tumor composition, and the construction of predictive models of genomics and clinical outcome. To tackle the large technical variations and biological heterogeneities, which are intrinsic in large cohorts, emerging systems utilize either prior knowledge from pathologists or unsupervised feature learning for invariant representation of the underlying properties in the data. However, to a large degree, the architecture for tissue histology classification remains unexplored and requires urgent systematical investigation. This paper is the first attempt to provide insights into three fundamental questions in tissue histology classification: I. Is unsupervised feature learning preferable to human engineered features? II. Does cellular saliency help? III. Does the sparse feature encoder contribute to recognition? We show that (a) in I, both Cellular Morphometric Feature and features from unsupervised feature learning lead to superior performance when compared to SIFT and [Color, Texture]; (b) in II, cellular saliency incorporation impairs the performance for systems built upon pixel-/patch-level features; and (c) in III, the effect of the sparse feature encoder is correlated with the robustness of features, and the performance can be consistently improved by the multi-stage extension of systems built upon both Cellular Morphmetric Feature and features from unsupervised feature learning. These insights are validated with two cohorts of Glioblastoma Multiforme (GBM) and Kidney Clear Cell Carcinoma (KIRC).

PHENOTYPIC CHARACTERIZATION OF BREAST INVASIVE CARCINOMA VIA TRANSFERABLE TISSUE MORPHOMETRIC PATTERNS LEARNED FROM GLIOBLASTOMA MULTIFORME

Quantitative analysis of whole slide images (WSIs) in a large cohort may provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. Although unsupervised feature learning provides a promising way in learning pertinent features without human intervention, its capability can be greatly limited due to the lack of well-curated examples. In this paper, we explored the transferability of knowledge acquired from a well-curated Glioblastoma Multiforme (GBM) dataset through its application to the representation and characterization of tissue histology from the Cancer Genome Atlas (TCGA) Breast Invasive Carcinoma (BRCA) cohort. Our experimental results reveals two major phenotypic subtypes with statistically significantly different survival curves. Further differential expression analysis of these two subtypes indicates enrichment of genes regulated by NF-kB in response to TNF and genes up-regulated in response to IFNG.

Development of a 2D Image Reconstruction and Viewing System for Histological Images from Multiple Tissue Blocks: Towards High-Resolution Whole-Organ 3D Histological Images

High-resolution 3D histology image reconstruction of the whole brain organ starts from reconstructing the high-resolution 2D histology images of a brain slice. In this paper, we introduced a method to automatically align the histology images of thin tissue sections cut from the multiple paraffin-embedded tissue blocks of a brain slice. For this method, we employed template matching and incorporated an optimization technique to further improve the accuracy of the 2D reconstructed image. In the template matching, we used the gross image of the brain slice as a reference to the reconstructed 2D histology image of the slice, while in the optimization procedure, we utilized the Jaccard index as the metric of the reconstruction accuracy. The results of our experiment on the initial 3 different whole-brain tissue slices showed that while the method works, it is also constrained by tissue deformations introduced during the tissue processing and slicing. The size of the reconstructed high-resolution 2D histology image of a brain slice is huge, and designing an image viewer that makes particularly efficient use of the computing power of a standard computer used in our laboratories is of interest. We also present the initial implementation of our 2D image viewer system in this paper.

CD30 cell graphs of Hodgkin lymphoma are not scale-free--an image analysis approach

MOTIVATION

Hodgkin lymphoma (HL) is a type of B-cell lymphoma. To diagnose the subtypes, biopsies are taken and immunostained. The slides are scanned to produce high-resolution digital whole slide images (WSI). Pathologists manually inspect the spatial distribution of cells, but little is known on the statistical properties of cell distributions in WSIs. Such properties would give valuable information for the construction of theoretical models that describe the invasion of malignant cells in the lymph node and the intercellular interactions.

RESULTS

In this work, we define and discuss HL cell graphs. We identify CD30(+) cells in HL WSIs, bringing together the fields of digital imaging and network analysis. We define special graphs based on the positions of the immunostained cells. We present an automatic analysis of complete WSIs to determine significant morphological and immunohistochemical features of HL cells and their spatial distribution in the lymph node tissue under three different medical conditions: lymphadenitis (LA) and two types of HL. We analyze the vertex degree distributions of CD30 cell graphs and compare them to a null model. CD30 cell graphs show higher vertex degrees than expected by a random unit disk graph, suggesting clustering of the cells. We found that a gamma distribution is suitable to model the vertex degree distributions of CD30 cell graphs, meaning that they are not scale-free. Moreover, we compare the graphs for LA and two subtypes of HL. LA and classical HL showed different vertex degree distributions. The vertex degree distributions of the two HL subtypes NScHL and mixed cellularity HL (MXcHL) were similar.

AVAILABILITY AND IMPLEMENTATION

The CellProfiler pipeline used for cell detection is available at https://sourceforge.net/projects/cellgraphs/.

CONTACT

ina.koch@bioinformatik.uni-frankfurt.de

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Automated image based prominent nucleoli detection

Introduction:

Nucleolar changes in cancer cells are one of the cytologic features important to the tumor pathologist in cancer assessments of tissue biopsies. However, inter-observer variability and the manual approach to this work hamper the accuracy of the assessment by pathologists. In this paper, we propose a computational method for prominent nucleoli pattern detection.

Materials and Methods:

Thirty-five hematoxylin and eosin stained images were acquired from prostate cancer, breast cancer, renal clear cell cancer and renal papillary cell cancer tissues. Prostate cancer images were used for the development of a computer-based automated prominent nucleoli pattern detector built on a cascade farm. An ensemble of approximately 1000 cascades was constructed by permuting different combinations of classifiers such as support vector machines, eXclusive component analysis, boosting, and logistic regression. The output of cascades was then combined using the RankBoost algorithm. The output of our prominent nucleoli pattern detector is a ranked set of detected image patches of patterns of prominent nucleoli.

Results:

The mean number of detected prominent nucleoli patterns in the top 100 ranked detected objects was 58 in the prostate cancer dataset, 68 in the breast cancer dataset, 86 in the renal clear cell cancer dataset, and 76 in the renal papillary cell cancer dataset. The proposed cascade farm performs twice as good as the use of a single cascade proposed in the seminal paper by Viola and Jones. For comparison, a naive algorithm that randomly chooses a pixel as a nucleoli pattern would detect five correct patterns in the first 100 ranked objects.

Conclusions:

Detection of sparse nucleoli patterns in a large background of highly variable tissue patterns is a difficult challenge our method has overcome. This study developed an accurate prominent nucleoli pattern detector with the potential to be used in the clinical settings.

Breast cancer histopathology image analysis: a review

This paper presents an overview of methods that have been proposed for the analysis of breast cancer histopathology images. This research area has become particularly relevant with the advent of whole slide imaging (WSI) scanners, which can perform cost-effective and high-throughput histopathology slide digitization, and which aim at replacing the optical microscope as the primary tool used by pathologist. Breast cancer is the most prevalent form of cancers among women, and image analysis methods that target this disease have a huge potential to reduce the workload in a typical pathology lab and to improve the quality of the interpretation. This paper is meant as an introduction for nonexperts. It starts with an overview of the tissue preparation, staining and slide digitization processes followed by a discussion of the different image processing techniques and applications, ranging from analysis of tissue staining to computer-aided diagnosis, and prognosis of breast cancer patients.

Stacked Predictive Sparse Decomposition for Classification of Histology Sections

Image-based classification of histology sections, in terms of distinct components (e.g., tumor, stroma, normal), provides a series of indices for histology composition (e.g., the percentage of each distinct components in histology sections), and enables the study of nuclear properties within each component. Furthermore, the study of these indices, constructed from each whole slide image in a large cohort, has the potential to provide predictive models of clinical outcome. For example, correlations can be established between the constructed indices and the patients' survival information at cohort level, which is a fundamental step towards personalized medicine. However, performance of the existing techniques is hindered as a result of large technical variations (e.g., variations of color/textures in tissue images due to non-standard experimental protocols) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. We propose a system that automatically learns a series of dictionary elements for representing the underlying spatial distribution using stacked predictive sparse decomposition. The learned representation is then fed into the spatial pyramid matching framework with a linear support vector machine classifier. The system has been evaluated for classification of distinct histological components for two cohorts of tumor types. Throughput has been increased by using of graphical processing unit (GPU), and evaluation indicates a superior performance results, compared with previous research.

Methods for nuclei detection, segmentation, and classification in digital histopathology: a review-current status and future potential

Digital pathology represents one of the major evolutions in modern medicine. Pathological examinations constitute the gold standard in many medical protocols, and also play a critical and legal role in the diagnosis process. In the conventional cancer diagnosis, pathologists analyze biopsies to make diagnostic and prognostic assessments, mainly based on the cell morphology and architecture distribution. Recently, computerized methods have been rapidly evolving in the area of digital pathology, with growing applications related to nuclei detection, segmentation, and classification. In cancer research, these approaches have played, and will continue to play a key (often bottleneck) role in minimizing human intervention, consolidating pertinent second opinions, and providing traceable clinical information. Pathological studies have been conducted for numerous cancer detection and grading applications, including brain, breast, cervix, lung, and prostate cancer grading. Our study presents, discusses, and extracts the major trends from an exhaustive overview of various nuclei detection, segmentation, feature computation, and classification techniques used in histopathology imagery, specifically in hematoxylin-eosin and immunohistochemical staining protocols. This study also enables us to measure the challenges that remain, in order to reach robust analysis of whole slide images, essential high content imaging with diagnostic biomarkers and prognosis support in digital pathology.

Stacked Predictive Sparse Coding for Classification of Distinct Regions of Tumor Histopathology

Image-based classification of tissue histology, in terms of distinct histopathology (e.g., tumor or necrosis regions), provides a series of indices for tumor composition. Furthermore, aggregation of these indices from each whole slide image (WSI) in a large cohort can provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. We suggest that, compared with human engineered features widely adopted in existing systems, unsupervised feature learning is more tolerant to batch effect (e.g., technical variations associated with sample preparation) and pertinent features can be learned without user intervention. This leads to a novel approach for classification of tissue histology based on unsupervised feature learning and spatial pyramid matching (SPM), which utilize sparse tissue morphometric signatures at various locations and scales. This approach has been evaluated on two distinct datasets consisting of different tumor types collected from The Cancer Genome Atlas (TCGA), and the experimental results indicate that the proposed approach is (i) extensible to different tumor types; (ii) robust in the presence of wide technical variations and biological heterogeneities; and (iii) scalable with varying training sample sizes.

Pathology imaging informatics for quantitative analysis of whole-slide images

Objectives

With the objective of bringing clinical decision support systems to reality, this article reviews histopathological whole-slide imaging informatics methods, associated challenges, and future research opportunities.

Target audience

This review targets pathologists and informaticians who have a limited understanding of the key aspects of whole-slide image (WSI) analysis and/or a limited knowledge of state-of-the-art technologies and analysis methods.

Scope

First, we discuss the importance of imaging informatics in pathology and highlight the challenges posed by histopathological WSI. Next, we provide a thorough review of current methods for: quality control of histopathological images; feature extraction that captures image properties at the pixel, object, and semantic levels; predictive modeling that utilizes image features for diagnostic or prognostic applications; and data and information visualization that explores WSI for de novo discovery. In addition, we highlight future research directions and discuss the impact of large public repositories of histopathological data, such as the Cancer Genome Atlas, on the field of pathology informatics. Following the review, we present a case study to illustrate a clinical decision support system that begins with quality control and ends with predictive modeling for several cancer endpoints. Currently, state-of-the-art software tools only provide limited image processing capabilities instead of complete data analysis for clinical decision-making. We aim to inspire researchers to conduct more research in pathology imaging informatics so that clinical decision support can become a reality.

Classification of Tumor Histology via Morphometric Context

Image-based classification of tissue histology, in terms of different components (e.g., normal signature, categories of aberrant signatures), provides a series of indices for tumor composition. Subsequently, aggregation of these indices in each whole slide image (WSI) from a large cohort can provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical and biological variations that are always present in a large cohort. In this paper, we propose two algorithms for classification of tissue histology based on robust representations of morphometric context, which are built upon nuclear level morphometric features at various locations and scales within the spatial pyramid matching (SPM) framework. These methods have been evaluated on two distinct datasets of different tumor types collected from The Cancer Genome Atlas (TCGA), and the experimental results indicate that our methods are (i) extensible to different tumor types; (ii) robust in the presence of wide technical and biological variations; (iii) invariant to different nuclear segmentation strategies; and (iv) scalable with varying training sample size. In addition, our experiments suggest that enforcing sparsity, during the construction of morphometric context, further improves the performance of the system.

Classification of Tumor Histology via Morphometric Context

Image-based classification of tissue histology, in terms of different components (e.g., normal signature, categories of aberrant signatures), provides a series of indices for tumor composition. Subsequently, aggregation of these indices in each whole slide image (WSI) from a large cohort can provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical and biological variations that are always present in a large cohort. In this paper, we propose two algorithms for classification of tissue histology based on robust representations of morphometric context, which are built upon nuclear level morphometric features at various locations and scales within the spatial pyramid matching (SPM) framework. These methods have been evaluated on two distinct datasets of different tumor types collected from The Cancer Genome Atlas (TCGA), and the experimental results indicate that our methods are (i) extensible to different tumor types; (ii) robust in the presence of wide technical and biological variations; (iii) invariant to different nuclear segmentation strategies; and (iv) scalable with varying training sample size. In addition, our experiments suggest that enforcing sparsity, during the construction of morphometric context, further improves the performance of the system.

CLASSIFICATION OF TUMOR HISTOPATHOLOGY VIA SPARSE FEATURE LEARNING

Our goal is to decompose whole slide images (WSI) of histology sections into distinct patches (e.g., viable tumor, necrosis) so that statistics of distinct histopathology can be linked with the outcome. Such an analysis requires a large cohort of histology sections that may originate from different laboratories, which may not use the same protocol in sample preparation. We have evaluated a method based on a variation of the restricted Boltzmann machine (RBM) that learns intrinsic features of the image signature in an unsupervised fashion. Computed code, from the learned representation, is then utilized to classify patches from a curated library of images. The system has been evaluated against a dataset of small image blocks of 1k-by-1k that have been extracted from glioblastoma multiforme (GBM) and clear cell kidney carcinoma (KIRC) from the cancer genome atlas (TCGA) archive. The learned model is then projected on each whole slide image (e.g., of size 20k-by-20k pixels or larger) for characterizing and visualizing tumor architecture. In the case of GBM, each WSI is decomposed into necrotic, transition into necrosis, and viable. In the case of the KIRC, each WSI is decomposed into tumor types, stroma, normal, and others. Evaluation of 1400 and 2500 samples of GBM and KIRC indicates a performance of 84% and 81%, respectively.

Multi-field-of-view framework for distinguishing tumor grade in ER+ breast cancer from entire histopathology slides

Modified Bloom-Richardson (mBR) grading is known to have prognostic value in breast cancer (BCa), yet its use in clinical practice has been limited by intra- and interobserver variability. The development of a computerized system to distinguish mBR grade from entire estrogen receptor-positive (ER+) BCa histopathology slides will help clinicians identify grading discrepancies and improve overall confidence in the diagnostic result. In this paper, we isolate salient image features characterizing tumor morphology and texture to differentiate entire hematoxylin and eosin (H and E) stained histopathology slides based on mBR grade. The features are used in conjunction with a novel multi-field-of-view (multi-FOV) classifier--a whole-slide classifier that extracts features from a multitude of FOVs of varying sizes--to identify important image features at different FOV sizes. Image features utilized include those related to the spatial arrangement of cancer nuclei (i.e., nuclear architecture) and the textural patterns within nuclei (i.e., nuclear texture). Using slides from 126 ER+ patients (46 low, 60 intermediate, and 20 high mBR grade), our grading system was able to distinguish low versus high, low versus intermediate, and intermediate versus high grade patients with area under curve values of 0.93, 0.72, and 0.74, respectively. Our results suggest that the multi-FOV classifier is able to 1) successfully discriminate low, medium, and high mBR grade and 2) identify specific image features at different FOV sizes that are important for distinguishing mBR grade in H and E stained ER+ BCa histology slides.

How bioinformatics influences health informatics: usage of biomolecular sequences, expression profiles and automated microscopic image analyses for clinical needs and public health

ABSTRACT

The currently hyped expectation of personalized medicine is often associated with just achieving the information technology led integration of biomolecular sequencing, expression and histopathological bioimaging data with clinical records at the individual patients' level as if the significant biomedical conclusions would be its more or less mandatory result. It remains a sad fact that many, if not most biomolecular mechanisms that translate the human genomic information into phenotypes are not known and, thus, most of the molecular and cellular data cannot be interpreted in terms of biomedically relevant conclusions. Whereas the historical trend will certainly be into the general direction of personalized diagnostics and cures, the temperate view suggests that biomedical applications that rely either on the comparison of biomolecular sequences and/or on the already known biomolecular mechanisms have much greater chances to enter clinical practice soon. In addition to considering the general trends, we exemplarily review advances in the area of cancer biomarker discovery, in the clinically relevant characterization of patient-specific viral and bacterial pathogens (with emphasis on drug selection for influenza and enterohemorrhagic E. coli) as well as progress in the automated assessment of histopathological images. As molecular and cellular data analysis will become instrumental for achieving desirable clinical outcomes, the role of bioinformatics and computational biology approaches will dramatically grow.

AUTHOR SUMMARY

With DNA sequencing and computers becoming increasingly cheap and accessible to the layman, the idea of integrating biomolecular and clinical patient data seems to become a realistic, short-term option that will lead to patient-specific diagnostics and treatment design for many diseases such as cancer, metabolic disorders, inherited conditions, etc. These hyped expectations will fail since many, if not most biomolecular mechanisms that translate the human genomic information into phenotypes are not known yet and, thus, most of the molecular and cellular data collected will not lead to biomedically relevant conclusions. At the same time, less spectacular biomedical applications based on biomolecular sequence comparison and/or known biomolecular mechanisms have the potential to unfold enormous potential for healthcare and public health. Since the analysis of heterogeneous biomolecular data in context with clinical data will be increasingly critical, the role of bioinformatics and computational biology will grow correspondingly in this process.

[Whole slide imaging technology: from digitization to online applications]

As e-health becomes essential to modern care, whole slide images (virtual slides) are now an important clinical, teaching and research tool in pathology. Virtual microscopy consists of digitizing a glass slide by acquiring hundreds of tiles of regions of interest at different zoom levels and assembling them into a structured file. This gigapixel image can then be remotely viewed over a terminal, exactly the way pathologists use a microscope. In this article, we will first describe the key elements of this technology, from the acquisition, using a scanner or a motorized microscope, to the broadcasting of virtual slides through a local or distant viewer over an intranet or Internet connection. As virtual slides are now commonly used in virtual classrooms, clinical data and research databases, we will highlight the main issues regarding its uses in modern pathology. Emphasis will be made on quality assurance policies, standardization and scaling.

A parallel solution for high resolution histological image analysis

This paper describes a general methodology for developing parallel image processing algorithms based on message passing for high resolution images (on the order of several Gigabytes). These algorithms have been applied to histological images and must be executed on massively parallel processing architectures. Advances in new technologies for complete slide digitalization in pathology have been combined with developments in biomedical informatics. However, the efficient use of these digital slide systems is still a challenge. The image processing that these slides are subject to is still limited both in terms of data processed and processing methods. The work presented here focuses on the need to design and develop parallel image processing tools capable of obtaining and analyzing the entire gamut of information included in digital slides. Tools have been developed to assist pathologists in image analysis and diagnosis, and they cover low and high-level image processing methods applied to histological images. Code portability, reusability and scalability have been tested by using the following parallel computing architectures: distributed memory with massive parallel processors and two networks, INFINIBAND and Myrinet, composed of 17 and 1024 nodes respectively. The parallel framework proposed is flexible, high performance solution and it shows that the efficient processing of digital microscopic images is possible and may offer important benefits to pathology laboratories.

Clustering methods applied in the detection of Ki67 hot-spots in whole tumor slide images: an efficient way to characterize heterogeneous tissue-based biomarkers

Whole-slide scanners allow the digitization of an entire histological slide at very high resolution. This new acquisition technique opens a wide range of possibilities for addressing challenging image analysis problems, including the identification of tissue-based biomarkers. In this study, we use whole-slide scanner technology for imaging the proliferating activity patterns in tumor slides based on Ki67 immunohistochemistry. Faced with large images, pathologists require tools that can help them identify tumor regions that exhibit high proliferating activity, called "hot-spots" (HSs). Pathologists need tools that can quantitatively characterize these HS patterns. To respond to this clinical need, the present study investigates various clustering methods with the aim of identifying Ki67 HSs in whole tumor slide images. This task requires a method capable of identifying an unknown number of clusters, which may be highly variable in terms of shape, size, and density. We developed a hybrid clustering method, referred to as Seedlink. Compared to manual HS selections by three pathologists, we show that Seedlink provides an efficient way of detecting Ki67 HSs and improves the agreement among pathologists when identifying HSs.