Invariant delineation of nuclear architecture in glioblastoma multiforme for clinical and molecular association

BoNuS: Boundary Mining for Nuclei Segmentation With Partial Point Labels

Nuclei segmentation is a fundamental prerequisite in the digital pathology workflow. The development of automated methods for nuclei segmentation enables quantitative analysis of the wide existence and large variances in nuclei morphometry in histopathology images. However, manual annotation of tens of thousands of nuclei is tedious and time-consuming, which requires significant amount of human effort and domain-specific expertise. To alleviate this problem, in this paper, we propose a weakly-supervised nuclei segmentation method that only requires partial point labels of nuclei. Specifically, we propose a novel boundary mining framework for nuclei segmentation, named BoNuS, which simultaneously learns nuclei interior and boundary information from the point labels. To achieve this goal, we propose a novel boundary mining loss, which guides the model to learn the boundary information by exploring the pairwise pixel affinity in a multiple-instance learning manner. Then, we consider a more challenging problem, i.e., partial point label, where we propose a nuclei detection module with curriculum learning to detect the missing nuclei with prior morphological knowledge. The proposed method is validated on three public datasets, MoNuSeg, CPM, and CoNIC datasets. Experimental results demonstrate the superior performance of our method to the state-of-the-art weakly-supervised nuclei segmentation methods. Code: https://github.com/hust-linyi/bonus.

A Review of Nuclei Detection and Segmentation on Microscopy Images Using Deep Learning With Applications to Unbiased Stereology Counting

The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.

Toward Source-Free Cross Tissues Histopathological Cell Segmentation via Target-Specific Finetuning

Recognition and quantitative analytics of histopathological cells are the golden standard for diagnosing multiple cancers. Despite recent advances in deep learning techniques that have been widely investigated for the automated segmentation of various types of histopathological cells, the heavy dependency on specific histopathological image types with sufficient supervised annotations, as well as the limited access to clinical data in hospitals, still pose significant challenges in the application of computer-aided diagnosis in pathology. In this paper, we focus on the model generalization of cell segmentation towards cross-tissue histopathological images. Remarkably, a novel target-specific finetuning-based self-supervised domain adaptation framework is proposed to transfer the cell segmentation model to unlabeled target datasets, without access to source datasets and annotations. When performed on the target unlabeled histopathological image set, the proposed method only needs to tune very few parameters of the pre-trained model in a self-supervised manner. Considering the morphological properties of pathological cells, we introduce two constraint terms at both local and global levels into this framework to access more reliable predictions. The proposed cross-domain framework is validated on three different types of histopathological tissues, showing promising performance in self-supervised cell segmentation. Additionally, the whole framework can be further applied to clinical tools in pathology without accessing the original training image data. The code and dataset are released at: https://github.com/NeuronXJTU/SFDA-CellSeg.

Biomarkers of Tumor Heterogeneity in Glioblastoma Multiforme Cohort of TCGA

Tumor Whole Slide Images (WSI) are often heterogeneous, which hinders the discovery of biomarkers in the presence of confounding clinical factors. In this study, we present a pipeline for identifying biomarkers from the Glioblastoma Multiforme (GBM) cohort of WSIs from TCGA archive. The GBM cohort endures many technical artifacts while the discovery of GBM biomarkers is challenged because "age" is the single most confounding factor for predicting outcomes. The proposed approach relies on interpretable features (e.g., nuclear morphometric indices), effective similarity metrics for heterogeneity analysis, and robust statistics for identifying biomarkers. The pipeline first removes artifacts (e.g., pen marks) and partitions each WSI into patches for nuclear segmentation via an extended U-Net for subsequent quantitative representation. Given the variations in fixation and staining that can artificially modulate hematoxylin optical density (HOD), we extended Navab's Lab method to normalize images and reduce the impact of batch effects. The heterogeneity of each WSI is then represented either as probability density functions (PDF) per patient or as the composition of a dictionary predicted from the entire cohort of WSIs. For PDF- or dictionary-based methods, morphometric subtypes are constructed based on distances computed from optimal transport and linkage analysis or consensus clustering with Euclidean distances, respectively. For each inferred subtype, Kaplan-Meier and/or the Cox regression model are used to regress the survival time. Since age is the single most important confounder for predicting survival in GBM and there is an observed violation of the proportionality assumption in the Cox model, we use both age and age-squared coupled with the Likelihood ratio test and forest plots for evaluating competing statistics. Next, the PDF- and dictionary-based methods are combined to identify biomarkers that are predictive of survival. The combined model has the advantage of integrating global (e.g., cohort scale) and local (e.g., patient scale) attributes of morphometric heterogeneity, coupled with robust statistics, to reveal stable biomarkers. The results indicate that, after normalization of the GBM cohort, mean HOD, eccentricity, and cellularity are predictive of survival. Finally, we also stratified the GBM cohort as a function of EGFR expression and published genomic subtypes to reveal genomic-dependent morphometric biomarkers.

Clinical significance and molecular annotation of cellular morphometric subtypes in lower-grade gliomas discovered by machine learning

BACKGROUND

Lower-grade gliomas (LGG) are heterogeneous diseases by clinical, histological, and molecular criteria. We aimed to personalize the diagnosis and therapy of LGG patients by developing and validating robust cellular morphometric subtypes (CMS) and to uncover the molecular signatures underlying these subtypes.

METHODS

Cellular morphometric biomarkers (CMBs) were identified with artificial intelligence technique from TCGA-LGG cohort. Consensus clustering was used to define CMS. Survival analysis was performed to assess the clinical impact of CMBs and CMS. A nomogram was constructed to predict 3- and 5-year overall survival (OS) of LGG patients. Tumor mutational burden (TMB) and immune cell infiltration between subtypes were analyzed using the Mann-Whitney U test. The double-blinded validation for important immunotherapy-related biomarkers was executed using immunohistochemistry (IHC).

RESULTS

We developed a machine learning (ML) pipeline to extract CMBs from whole-slide images of tissue histology; identifying and externally validating robust CMS of LGGs in multicenter cohorts. The subtypes had independent predicted OS across all three independent cohorts. In the TCGA-LGG cohort, patients within the poor-prognosis subtype responded poorly to primary and follow-up therapies. LGGs within the poor-prognosis subtype were characterized by high mutational burden, high frequencies of copy number alterations, and high levels of tumor-infiltrating lymphocytes and immune checkpoint genes. Higher levels of PD-1/PD-L1/CTLA-4 were confirmed by IHC staining. In addition, the subtypes learned from LGG demonstrate translational impact on glioblastoma (GBM).

CONCLUSIONS

We developed and validated a framework (CMS-ML) for CMS discovery in LGG associated with specific molecular alterations, immune microenvironment, prognosis, and treatment response.

An interpretable semi-supervised framework for patch-based classification of breast cancer

Developing effective invasive Ductal Carcinoma (IDC) detection methods remains a challenging problem for breast cancer diagnosis. Recently, there has been notable success in utilizing deep neural networks in various application domains; however, it is well-known that deep neural networks require a large amount of labelled training data to achieve high accuracy. Such amounts of manually labelled data are time-consuming and expensive, especially when domain expertise is required. To this end, we present a novel semi-supervised learning framework for IDC detection using small amounts of labelled training examples to take advantage of cheap available unlabeled data. To gain trust in the prediction of the framework, we explain the prediction globally. Our proposed framework consists of five main stages: data augmentation, feature selection, dividing co-training data labelling, deep neural network modelling, and the interpretability of neural network prediction. The data cohort used in this study contains digitized BCa histopathology slides from 162 women with IDC at the Hospital of the University of Pennsylvania and the Cancer Institute of New Jersey. To evaluate the effectiveness of the deep neural network model used by the proposed approach, we compare it to different state-of-the-art network architectures; AlexNet and a shallow VGG network trained only on the labelled data. The results show that the deep neural network used in our proposed approach outperforms the state-of-the-art techniques achieving balanced accuracy of 0.73 and F-measure of 0.843. In addition, we compare the performance of the proposed semi-supervised approach to state-of-the-art semi-supervised DCGAN technique and self-learning technique. The experimental evaluation shows that our framework outperforms both semi-supervised techniques and detects IDC with an accuracy of 85.75%, a balanced accuracy of 0.865, and an F-measure of 0.773 using only 10% labelled instances from the training dataset while the rest of the training dataset is treated as unlabeled.

Mathematical approach for segmenting chromosome clusters in metaspread images

Karyotyping is an examination that helps in detecting chromosomal abnormalities. Chromosome analysis is a very challenging task which requires various steps to obtain a karyotype. The challenges associated with chromosome analysis are overlapping and touching of chromosomes. The input considered for chromosome analysis is the metaspread G band chromosomes. The proposed work mainly focus on separation the overlapped and touching chromosomes which is considered to be the major challenge in karyotype. There are various research contribution in chromosome analysis in progress which includes both low (Machine Learning) and high level (Deep Learning) methods. This paper proposes a mathematical based approaches which is very effective in segmentation of clustered chromosomes. The accuracy of segmentation is robust compared to high level approaches.

Quantitative Nuclear Histomorphometry Predicts Molecular Subtype and Clinical Outcome in Medulloblastomas: Preliminary Findings

Molecular subtypes of medulloblastoma [Sonic Hedgehog (SHH), Wingless/INT (WNT), Group 3, and Group 4] are defined by common patterns of gene expression. These differential gene expression patterns appear to result in different histomorphology and prognosis. Quantitative histomorphometry is a well-known method of computer-aided pathology image analysis. The hypotheses we sought to examine in this preliminary proof of concept study were whether computer extracted nuclear morphological features of medulloblastomas from digitized tissue slide images could independently: (1) distinguish between molecularly determined subgroups and (2) identify patterns within these subgroups that correspond with clinical outcome. Our dataset was composed of 46 medulloblastoma patients: 16 SHH (5 dead, 11 survived), 3 WNT (0 dead, 3 survived), 12 Group 3 (4 dead, 8 survived), and 15 were Group 4 (5 dead, 10 survived). A watershed-based thresholding scheme was used to automatically identify individual nuclei within digitized whole slide hematoxylin and eosin tissue images. Quantitative histomorphometric features corresponding to the texture (variation in pixel intensity), shape (variations in size, roundness), and architectural rearrangement (distances between, and number of connected neighbors) of nuclei were subsequently extracted. These features were ranked using feature selection schemes and these top-ranked features were then used to train machine-learning classifiers via threefold cross-validation to separate patients based on: (1) molecular subtype and (2) disease-specific outcomes within the individual molecular subtype groups. SHH and WNT tumors were separated from Groups 3 and 4 tumors with a maximum area under the receiver operating characteristic curve (AUC) of 0.7, survival within Group 3 tumors was predicted with an AUC of 0.92, and Group 3 and 4 patients were separated into high- and low-risk groups with p = 0.002. Model prediction was quantitatively compared with age, stage, and histological subtype using univariate and multivariate Cox hazard ratio models. Age was the most statistically significant variable for predicting survival in Group 3 and 4 tumors, but model predictions had the highest hazard ratio value. Quantitative nuclear histomorphometry can be used to study medulloblastoma genetic expression phenotypes as it may distinguish meaningful features of disease pathology.

MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge

Detecting various types of cells in and around the tumor matrix holds a special significance in characterizing the tumor micro-environment for cancer prognostication and research. Automating the tasks of detecting, segmenting, and classifying nuclei can free up the pathologists' time for higher value tasks and reduce errors due to fatigue and subjectivity. To encourage the computer vision research community to develop and test algorithms for these tasks, we prepared a large and diverse dataset of nucleus boundary annotations and class labels. The dataset has over 46,000 nuclei from 37 hospitals, 71 patients, four organs, and four nucleus types. We also organized a challenge around this dataset as a satellite event at the International Symposium on Biomedical Imaging (ISBI) in April 2020. The challenge saw a wide participation from across the world, and the top methods were able to match inter-human concordance for the challenge metric. In this paper, we summarize the dataset and the key findings of the challenge, including the commonalities and differences between the methods developed by various participants. We have released the MoNuSAC2020 dataset to the public.

Accurate Segmentation of Nuclear Regions with Multi-Organ Histopathology Images Using Artificial Intelligence for Cancer Diagnosis in Personalized Medicine

Accurate nuclear segmentation in histopathology images plays a key role in digital pathology. It is considered a prerequisite for the determination of cell phenotype, nuclear morphometrics, cell classification, and the grading and prognosis of cancer. However, it is a very challenging task because of the different types of nuclei, large intraclass variations, and diverse cell morphologies. Consequently, the manual inspection of such images under high-resolution microscopes is tedious and time-consuming. Alternatively, artificial intelligence (AI)-based automated techniques, which are fast and robust, and require less human effort, can be used. Recently, several AI-based nuclear segmentation techniques have been proposed. They have shown a significant performance improvement for this task, but there is room for further improvement. Thus, we propose an AI-based nuclear segmentation technique in which we adopt a new nuclear segmentation network empowered by residual skip connections to address this issue. Experiments were performed on two publicly available datasets: (1) The Cancer Genome Atlas (TCGA), and (2) Triple-Negative Breast Cancer (TNBC). The results show that our proposed technique achieves an aggregated Jaccard index (AJI) of 0.6794, Dice coefficient of 0.8084, and F1-measure of 0.8547 on TCGA dataset, and an AJI of 0.7332, Dice coefficient of 0.8441, precision of 0.8352, recall of 0.8306, and F1-measure of 0.8329 on the TNBC dataset. These values are higher than those of the state-of-the-art methods.

DeepDistance: A multi-task deep regression model for cell detection in inverted microscopy images

This paper presents a new deep regression model, which we call DeepDistance, for cell detection in images acquired with inverted microscopy. This model considers cell detection as a task of finding most probable locations that suggest cell centers in an image. It represents this main task with a regression task of learning an inner distance metric. However, different than the previously reported regression based methods, the DeepDistance model proposes to approach its learning as a multi-task regression problem where multiple tasks are learned by using shared feature representations. To this end, it defines a secondary metric, normalized outer distance, to represent a different aspect of the problem and proposes to define its learning as complementary to the main cell detection task. In order to learn these two complementary tasks more effectively, the DeepDistance model designs a fully convolutional network (FCN) with a shared encoder path and end-to-end trains this FCN to concurrently learn the tasks in parallel. For further performance improvement on the main task, this paper also presents an extended version of the DeepDistance model that includes an auxiliary classification task and learns it in parallel to the two regression tasks by also sharing feature representations with them. DeepDistance uses the inner distances estimated by these FCNs in a detection algorithm to locate individual cells in a given image. In addition to this detection algorithm, this paper also suggests a cell segmentation algorithm that employs the estimated maps to find cell boundaries. Our experiments on three different human cell lines reveal that the proposed multi-task learning models, the DeepDistance model and its extended version, successfully identify the locations of cell as well as delineate their boundaries, even for the cell line that was not used in training, and improve the results of its counterparts.

A Multi-Organ Nucleus Segmentation Challenge

Generalized nucleus segmentation techniques can contribute greatly to reducing the time to develop and validate visual biomarkers for new digital pathology datasets. We summarize the results of MoNuSeg 2018 Challenge whose objective was to develop generalizable nuclei segmentation techniques in digital pathology. The challenge was an official satellite event of the MICCAI 2018 conference in which 32 teams with more than 80 participants from geographically diverse institutes participated. Contestants were given a training set with 30 images from seven organs with annotations of 21,623 individual nuclei. A test dataset with 14 images taken from seven organs, including two organs that did not appear in the training set was released without annotations. Entries were evaluated based on average aggregated Jaccard index (AJI) on the test set to prioritize accurate instance segmentation as opposed to mere semantic segmentation. More than half the teams that completed the challenge outperformed a previous baseline. Among the trends observed that contributed to increased accuracy were the use of color normalization as well as heavy data augmentation. Additionally, fully convolutional networks inspired by variants of U-Net, FCN, and Mask-RCNN were popularly used, typically based on ResNet or VGG base architectures. Watershed segmentation on predicted semantic segmentation maps was a popular post-processing strategy. Several of the top techniques compared favorably to an individual human annotator and can be used with confidence for nuclear morphometrics.

A survey and evaluation of Web-based tools/databases for variant analysis of TCGA data

The Cancer Genome Atlas (TCGA) is a publicly funded project that aims to catalog and discover major cancer-causing genomic alterations with the goal of creating a comprehensive ‘atlas’ of cancer genomic profiles. The availability of this genome-wide information provides an unprecedented opportunity to expand our knowledge of tumourigenesis. Computational analytics and mining are frequently used as effective tools for exploring this byzantine series of biological and biomedical data. However, some of the more advanced computational tools are often difficult to understand or use, thereby limiting their application by scientists who do not have a strong computational background. Hence, it is of great importance to build user-friendly interfaces that allow both computational scientists and life scientists without a computational background to gain greater biological and medical insights. To that end, this survey was designed to systematically present available Web-based tools and facilitate the use TCGA data for cancer research.

Histo-genomics: digital pathology at the forefront of precision medicine

The toughest challenge OMICs face is that they provide extremely high molecular resolution but poor spatial information. Understanding the cellular/histological context of the overwhelming genetic data is critical for a full understanding of the clinical behavior of a malignant tumor. Digital pathology can add an extra layer of information to help visualize in a spatial and microenvironmental context the molecular information of cancer. Thus, histo-genomics provide a unique chance for data integration. In the era of a precision medicine, a four-dimensional (4D) (temporal/spatial) analysis of cancer aided by digital pathology can be a critical step to understand the evolution/progression of different cancers and consequently tailor individual treatment plans. For instance, the integration of molecular biomarkers expression into a three-dimensional (3D) image of a digitally scanned tumor can offer a better understanding of its subtype, behavior, host immune response and prognosis. Using advanced digital image analysis, a larger spectrum of parameters can be analyzed as potential predictors of clinical behavior. Correlation between morphological features and host immune response can be also performed with therapeutic implications. Radio-histomics, or the interface of radiological images and histology is another emerging exciting field which encompasses the integration of radiological imaging with digital pathological images, genomics, and clinical data to portray a more holistic approach to understating and treating disease. These advances in digital slide scanning are not without technical challenges, which will be addressed carefully in this review with quick peek at its future.

An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images

BACKGROUND

Since nuclei segmentation in histopathology images can provide key information for identifying the presence or stage of a disease, the images need to be assessed carefully. However, color variation in histopathology images, and various structures of nuclei are two major obstacles in accurately segmenting and analyzing histopathology images. Several machine learning methods heavily rely on hand-crafted features which have limitations due to manual thresholding.

RESULTS

To obtain robust results, deep learning based methods have been proposed. Deep convolutional neural networks (DCNN) used for automatically extracting features from raw image data have been proven to achieve great performance. Inspired by such achievements, we propose a nuclei segmentation method based on DCNNs. To normalize the color of histopathology images, we use a deep convolutional Gaussian mixture color normalization model which is able to cluster pixels while considering the structures of nuclei. To segment nuclei, we use Mask R-CNN which achieves state-of-the-art object segmentation performance in the field of computer vision. In addition, we perform multiple inference as a post-processing step to boost segmentation performance. We evaluate our segmentation method on two different datasets. The first dataset consists of histopathology images of various organ while the other consists histopathology images of the same organ. Performance of our segmentation method is measured in various experimental setups at the object-level and the pixel-level. In addition, we compare the performance of our method with that of existing state-of-the-art methods. The experimental results show that our nuclei segmentation method outperforms the existing methods.

CONCLUSIONS

We propose a nuclei segmentation method based on DCNNs for histopathology images. The proposed method which uses Mask R-CNN with color normalization and multiple inference post-processing provides robust nuclei segmentation results. Our method also can facilitate downstream nuclei morphological analyses as it provides high-quality features extracted from histopathology images.

Tumor Malignancy Detection Using Histopathology Imaging

Image segmentation and classification in the biomedical imaging field has high worth in cancer diagnosis and grading. The proposed method classifies the images based on a combination of handcrafted features and shape features using bag of visual words (BoW). The multistage segmentation technique to localize the nuclei in histopathology images includes the stain decomposition and histogram equalization to highlight the nucleus region, followed by the nuclei key point extraction using fast radial symmetry transform, normalized graph cut based on the nuclei region estimation, and nuclei boundary estimation using modified gradient. Subsequently, features from localized regions termed as handcrafted features and the shape features using BoW are extracted for classification. The experiments are performed using both the handcrafted features and BoW to take the advantages of both local nuclei features and globally spatial features. The simulation is performed on the Bisque and BreakHis data sets (with corresponding average accuracies of 93.87% and 96.96%, respectively) and confirms better diagnosis performance using the proposed method.

Automated and Reproducible Detection of Vascular Endothelial Growth Factor (VEGF) in Renal Tissue Sections

Background

Manual analysis of tissue sections, such as for pathological diagnosis, requires an analyst with substantial knowledge and experience. Reproducible image analysis of biological samples is steadily gaining scientific importance. The aim of the present study was to employ image analysis followed by machine learning to identify vascular endothelial growth factor (VEGF) in kidney tissue that had been subjected to hypoxia.

Methods

Light microscopy images of renal tissue sections stained for VEGF were analyzed. Subsequently, machine learning classified the cells as VEGF⁺ and VEGF⁻ cells.

Results

VEGF was detected and cells were counted with high sensitivity and specificity.

Conclusion

With great clinical, diagnostic, and research potential, automatic image analysis offers a new quantitative capability, thereby adding numerical information to a mostly qualitative diagnostic approach.

NHL Pathological Image Classification Based on Hierarchical Local Information and GoogLeNet-Based Representations

Background

Accurate classification for different non-Hodgkin lymphomas (NHL) is one of the main challenges in clinical pathological diagnosis due to its intrinsic complexity. Therefore, this paper proposes an effective classification model for three types of NHL pathological images, including mantle cell lymphoma (MCL), follicular lymphoma (FL), and chronic lymphocytic leukemia (CLL).

Methods

There are three main parts with respect to our model. First, NHL pathological images stained by hematoxylin and eosin (H&E) are transferred into blue ratio (BR) and Lab spaces, respectively. Then specific patch-level textural and statistical features are extracted from BR images and color features are obtained from Lab images both using a hierarchical way, yielding a set of hand-crafted representations corresponding to different image spaces. A random forest classifier is subsequently trained for patch-level classification. Second, H&E images are cropped and fed into a pretrained google inception net (GoogLeNet) for learning high-level representations and a softmax classifier is used for patch-level classification. Finally, three image-level classification strategies based on patch-level results are discussed including a novel method for calculating the weighted sum of patch results. Different classification results are fused at both feature 1 and image levels to obtain a more satisfactory result.

Results

The proposed model is evaluated on a public IICBU Malignant Lymphoma Dataset and achieves an improved overall accuracy of 0.991 and area under the receiver operating characteristic curve of 0.998.

Conclusion

The experimentations demonstrate the significantly increased classification performance of the proposed model, indicating that it is a suitable classification approach for NHL pathological images.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network

Detection of nuclei is an important step in phenotypic profiling of 1) histology sections imaged in bright field; and 2) colony formation of the 3-D cell culture models that are imaged using confocal microscopy. It is shown that feature-based representation of the original image improves color decomposition (CD) and subsequent nuclear detection using convolutional neural networks independent of the imaging modality. The feature-based representation utilizes the Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Moreover, in the case of samples imaged in bright field, the LoG response also provides the necessary initial statistics for CD using nonnegative matrix factorization. Several permutations of input data representations and network architectures are evaluated to show that by coupling improved CD and the LoG response of this representation, detection of nuclei is advanced. In particular, the frequencies of detection of nuclei with the vesicular or necrotic phenotypes, or poor staining, are improved. The overall system has been evaluated against manually annotated images, and the F-scores for alternative representations and architectures are reported.

Object-Oriented Segmentation of Cell Nuclei in Fluorescence Microscopy Images

Cell nucleus segmentation remains an open and challenging problem especially to segment nuclei in cell clumps. Splitting a cell clump would be straightforward if the gradients of boundary pixels in-between the nuclei were always higher than the others. However, imperfections may exist: inhomogeneities of pixel intensities in a nucleus may cause to define spurious boundaries whereas insufficient pixel intensity differences at the border of overlapping nuclei may cause to miss some true boundary pixels. In contrast, these imperfections are typically observed at the pixel-level, causing local changes in pixel values without changing the semantics on a large scale. In response to these issues, this article introduces a new nucleus segmentation method that relies on using gradient information not at the pixel level but at the object level. To this end, it proposes to decompose an image into smaller homogeneous subregions, define edge-objects at four different orientations to encode the gradient information at the object level, and devise a merging algorithm, in which the edge-objects vote for subregion pairs along their orientations and the pairs are iteratively merged if they get sufficient votes from multiple orientations. Our experiments on fluorescence microscopy images reveal that this high-level representation and the design of a merging algorithm using edge-objects (gradients at the object level) improve the segmentation results.

Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes

BACKGROUND

Nuclear segmentation is an important step for profiling aberrant regions of histology sections. If nuclear segmentation can be resolved, then new biomarkers of nuclear phenotypes and their organization can be predicted for the application of precision medicine. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), nuclear phenotypes (e.g., vesicular, aneuploidy), and overlapping nuclei. The problem is further complicated as a result of variations in sample preparation (e.g., fixation, staining). Our hypothesis is that (i) deep learning techniques can learn complex phenotypic signatures that rise in tumor sections, and (ii) fusion of different representations (e.g., regions, boundaries) contributes to improved nuclear segmentation.

RESULTS

We have demonstrated that training of deep encoder-decoder convolutional networks overcomes complexities associated with multiple nuclear phenotypes, where we evaluate alternative architecture of deep learning for an improved performance against the simplicity of the design. In addition, improved nuclear segmentation is achieved by color decomposition and combining region- and boundary-based features through a fusion network. The trained models have been evaluated against approximately 19,000 manually annotated nuclei, and object-level Precision, Recall, F1-score and Standard Error are reported with the best F1-score being 0.91. Raw training images, annotated images, processed images, and source codes are released as a part of the Additional file 1.

CONCLUSIONS

There are two intrinsic barriers in nuclear segmentation to H&E stained images, which correspond to the diversity of nuclear phenotypes and perceptual boundaries between adjacent cells. We demonstrate that (i) the encoder-decoder architecture can learn complex phenotypes that include the vesicular type; (ii) delineation of overlapping nuclei is enhanced by fusion of region- and edge-based networks; (iii) fusion of ENets produces an improved result over the fusion of UNets; and (iv) fusion of networks is better than multitask learning. We suggest that our protocol enables processing a large cohort of whole slide images for applications in precision medicine.

Unsupervised Learning for Cell-Level Visual Representation in Histopathology Images With Generative Adversarial Networks

The visual attributes of cells, such as the nuclear morphology and chromatin openness, are critical for histopathology image analysis. By learning cell-level visual representation, we can obtain a rich mix of features that are highly reusable for various tasks, such as cell-level classification, nuclei segmentation, and cell counting. In this paper, we propose a unified generative adversarial networks architecture with a new formulation of loss to perform robust cell-level visual representation learning in an unsupervised setting. Our model is not only label-free and easily trained but also capable of cell-level unsupervised classification with interpretable visualization, which achieves promising results in the unsupervised classification of bone marrow cellular components. Based on the proposed cell-level visual representation learning, we further develop a pipeline that exploits the varieties of cellular elements to perform histopathology image classification, the advantages of which are demonstrated on bone marrow datasets.

Informatics Approaches to Address New Challenges in the Classification of Lymphoid Malignancies

Purpose

Lymphoid malignancies are remarkably heterogeneous, with variations in outcomes and clinical, biologic, and histologic presentation complicating classification according to the World Health Organization guidelines. Incorrect classification of lymphoid neoplasms can result in suboptimal therapeutic strategies for individual patients and confound the interpretation of clinical trials involving personalized, class-based treatments. This review discusses the potential role of pathology informatics in improving the classification accuracy and objectivity for lymphoid malignancies.

Design

We identified peer-reviewed publications examining pathology informatics approaches for the classification of lymphoid malignancies, reviewed developments in the lymphoma classification systems, and summarized computational methods for pathologic assessment that can impact practice.

Results

Computer-assisted pathology image analysis algorithms in lymphoma most commonly have been applied to follicular lymphoma to address biologic heterogeneity and subjectivity in the process of classification.

Conclusion

Objective methods are available to assist pathologists in lymphoma classification and grading, and have been demonstrated to provide measurable benefits in specific contexts. Future validation and extension of these approaches will require datasets that link high resolution pathology images available for image analysis algorithms with clinical variables and follow up outcomes.

The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research

Tissue-based cancer studies can generate large amounts of histology data in the form of glass slides. These slides contain important diagnostic, prognostic, and biological information and can be digitized into expansive and high-resolution whole-slide images using slide-scanning devices. Effectively utilizing digital pathology data in cancer research requires the ability to manage, visualize, share, and perform quantitative analysis on these large amounts of image data, tasks that are often complex and difficult for investigators with the current state of commercial digital pathology software. In this article, we describe the Digital Slide Archive (DSA), an open-source web-based platform for digital pathology. DSA allows investigators to manage large collections of histologic images and integrate them with clinical and genomic metadata. The open-source model enables DSA to be extended to provide additional capabilities. Cancer Res; 77(21); e75-78. ©2017 AACR.

Interactive phenotyping of large-scale histology imaging data with HistomicsML

Whole-slide imaging of histologic sections captures tissue microenvironments and cytologic details in expansive high-resolution images. These images can be mined to extract quantitative features that describe tissues, yielding measurements for hundreds of millions of histologic objects. A central challenge in utilizing this data is enabling investigators to train and evaluate classification rules for identifying objects related to processes like angiogenesis or immune response. In this paper we describe HistomicsML, an interactive machine-learning system for digital pathology imaging datasets. This framework uses active learning to direct user feedback, making classifier training efficient and scalable in datasets containing 10⁸+ histologic objects. We demonstrate how this system can be used to phenotype microvascular structures in gliomas to predict survival, and to explore the molecular pathways associated with these phenotypes. Our approach enables researchers to unlock phenotypic information from digital pathology datasets to investigate prognostic image biomarkers and genotype-phenotype associations.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.

A Dataset and a Technique for Generalized Nuclear Segmentation for Computational Pathology

Nuclear segmentation in digital microscopic tissue images can enable extraction of high-quality features for nuclear morphometrics and other analysis in computational pathology. Conventional image processing techniques, such as Otsu thresholding and watershed segmentation, do not work effectively on challenging cases, such as chromatin-sparse and crowded nuclei. In contrast, machine learning-based segmentation can generalize across various nuclear appearances. However, training machine learning algorithms requires data sets of images, in which a vast number of nuclei have been annotated. Publicly accessible and annotated data sets, along with widely agreed upon metrics to compare techniques, have catalyzed tremendous innovation and progress on other image classification problems, particularly in object recognition. Inspired by their success, we introduce a large publicly accessible data set of hematoxylin and eosin (H&E)-stained tissue images with more than 21000 painstakingly annotated nuclear boundaries, whose quality was validated by a medical doctor. Because our data set is taken from multiple hospitals and includes a diversity of nuclear appearances from several patients, disease states, and organs, techniques trained on it are likely to generalize well and work right out-of-the-box on other H&E-stained images. We also propose a new metric to evaluate nuclear segmentation results that penalizes object- and pixel-level errors in a unified manner, unlike previous metrics that penalize only one type of error. We also propose a segmentation technique based on deep learning that lays a special emphasis on identifying the nuclear boundaries, including those between the touching or overlapping nuclei, and works well on a diverse set of test images.

Detection of Nuclei in H&E Stained Sections Using Convolutional Neural Networks

Detection of nuclei is an important step in phenotypic profiling of histology sections that are usually imaged in bright field. However, nuclei can have multiple phenotypes, which are difficult to model. It is shown that convolutional neural networks (CNN)s can learn different phenotypic signatures for nuclear detection, and that the performance is improved with the feature-based representation of the original image. The feature-based representation utilizes Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Several combinations of input data representations are evaluated to show that by LoG representation, detection of nuclei is advanced. In addition, the efficacy of CNN for vesicular and hyperchromatic nuclei is evaluated. In particular, the frequency of detection of nuclei with the vesicular and apoptotic phenotypes is increased. The overall system has been evaluated against manually annotated nuclei and the F-Scores for alternative representations have been reported.

Graph-based segmentation of abnormal nuclei in cervical cytology

A general method is reported for improving the segmentation of abnormal cell nuclei in cervical cytology images. In automation-assisted reading of cervical cytology, one of the essential steps is the segmentation of nuclei. Despite some progress, there is a need to improve the sensitivity, particularly the segmentation of abnormal nuclei. Our method starts with pre-segmenting the nucleus to define the coarse center and size of nucleus, which is used to construct a graph by image unfolding that maps ellipse-like border in the Cartesian coordinate system to lines in the polar coordinate system. The cost function jointly reflects properties of nucleus border and nucleus region. The prior constraints regarding the context of nucleus-cytoplasm position are utilized to modify the local cost functions. The globally optimal path in the constructed graph is then identified by dynamic programming with an iterative approach ensuring an optimal closed contour. Validation of our method was performed on abnormal nuclei from two cervical cell image datasets, Herlev and H&E stained manual liquid-based cytology (HEMLBC). Compared with five state-of-the-art approaches, our graph-search based method shows superior performance.

Multi-Pass Adaptive Voting for Nuclei Detection in Histopathological Images

Nuclei detection is often a critical initial step in the development of computer aided diagnosis and prognosis schemes in the context of digital pathology images. While over the last few years, a number of nuclei detection methods have been proposed, most of these approaches make idealistic assumptions about the staining quality of the tissue. In this paper, we present a new Multi-Pass Adaptive Voting (MPAV) for nuclei detection which is specifically geared towards images with poor quality staining and noise on account of tissue preparation artifacts. The MPAV utilizes the symmetric property of nuclear boundary and adaptively selects gradient from edge fragments to perform voting for a potential nucleus location. The MPAV was evaluated in three cohorts with different staining methods: Hematoxylin &Eosin, CD31 &Hematoxylin, and Ki-67 and where most of the nuclei were unevenly and imprecisely stained. Across a total of 47 images and nearly 17,700 manually labeled nuclei serving as the ground truth, MPAV was able to achieve a superior performance, with an area under the precision-recall curve (AUC) of 0.73. Additionally, MPAV also outperformed three state-of-the-art nuclei detection methods, a single pass voting method, a multi-pass voting method, and a deep learning based method.

Transfer Shape Modeling Towards High-throughput Microscopy Image Segmentation

In order to deal with ambiguous image appearances in cell segmentation, high-level shape modeling has been introduced to delineate cell boundaries. However, shape modeling usually requires sufficient annotated training shapes, which are often labor intensive or unavailable. Meanwhile, when applying the model to different datasets, it is necessary to repeat the tedious annotation process to generate enough training data, and this will significantly limit the applicability of the model. In this paper, we propose to transfer shape modeling learned from an existing but different dataset (e.g. lung cancer) to assist cell segmentation in a new target dataset (e.g. skeletal muscle) without expensive manual annotations. Considering the intrinsic geometry structure of cell shapes, we incorporate the shape transfer model into a sparse representation framework with a manifold embedding constraint, and provide an efficient algorithm to solve the optimization problem. The proposed algorithm is tested on multiple microscopy image datasets with different tissue and staining preparations, and the experiments demonstrate its effectiveness.

Integrative Analysis of Cellular Morphometric Context Reveals Clinically Relevant Signatures in Lower Grade Glioma

Integrative analysis based on quantitative representation of whole slide images (WSIs) in a large histology cohort may provide predictive models of clinical outcome. On one hand, the efficiency and effectiveness of such representation is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. On the other hand, perceptual interpretation/validation of important multivariate phenotypic signatures are often difficult due to the loss of visual information during feature transformation in hyperspace. To address these issues, we propose a novel approach for integrative analysis based on cellular morphometric context, which is a robust representation of WSI, with the emphasis on tumor architecture and tumor heterogeneity, built upon cellular level morphometric features within the spatial pyramid matching (SPM) framework. The proposed approach is applied to The Cancer Genome Atlas (TCGA) lower grade glioma (LGG) cohort, where experimental results (i) reveal several clinically relevant cellular morphometric types, which enables both perceptual interpretation/validation and further investigation through gene set enrichment analysis; and (ii) indicate the significantly increased survival rates in one of the cellular morphometric context subtypes derived from the cellular morphometric context.

When machine vision meets histology: A comparative evaluation of model architecture for classification of histology sections

Classification of histology sections in large cohorts, in terms of distinct regions of microanatomy (e.g., stromal) and histopathology (e.g., tumor, necrosis), enables the quantification of tumor composition, and the construction of predictive models of genomics and clinical outcome. To tackle the large technical variations and biological heterogeneities, which are intrinsic in large cohorts, emerging systems utilize either prior knowledge from pathologists or unsupervised feature learning for invariant representation of the underlying properties in the data. However, to a large degree, the architecture for tissue histology classification remains unexplored and requires urgent systematical investigation. This paper is the first attempt to provide insights into three fundamental questions in tissue histology classification: I. Is unsupervised feature learning preferable to human engineered features? II. Does cellular saliency help? III. Does the sparse feature encoder contribute to recognition? We show that (a) in I, both Cellular Morphometric Feature and features from unsupervised feature learning lead to superior performance when compared to SIFT and [Color, Texture]; (b) in II, cellular saliency incorporation impairs the performance for systems built upon pixel-/patch-level features; and (c) in III, the effect of the sparse feature encoder is correlated with the robustness of features, and the performance can be consistently improved by the multi-stage extension of systems built upon both Cellular Morphmetric Feature and features from unsupervised feature learning. These insights are validated with two cohorts of Glioblastoma Multiforme (GBM) and Kidney Clear Cell Carcinoma (KIRC).

Integrative analysis of diffusion-weighted MRI and genomic data to inform treatment of glioblastoma

Gene expression profiling from glioblastoma (GBM) patients enables characterization of cancer into subtypes that can be predictive of response to therapy. An integrative analysis of imaging and gene expression data can potentially be used to obtain novel biomarkers that are closely associated with the genetic subtype and gene signatures and thus provide a noninvasive approach to stratify GBM patients. In this retrospective study, we analyzed the expression of 12,042 genes for 558 patients from The Cancer Genome Atlas (TCGA). Among these patients, 50 patients had magnetic resonance imaging (MRI) studies including diffusion weighted (DW) MRI in The Cancer Imaging Archive (TCIA). We identified the contrast enhancing region of the tumors using the pre- and post-contrast T1-weighted MRI images and computed the apparent diffusion coefficient (ADC) histograms from the DW-MRI images. Using the gene expression data, we classified patients into four molecular subtypes, determined the number and composition of genes modules using the gap statistic, and computed gene signature scores. We used logistic regression to find significant predictors of GBM subtypes. We compared the predictors for different subtypes using Mann-Whitney U tests. We assessed detection power using area under the receiver operating characteristic (ROC) analysis. We computed Spearman correlations to determine the associations between ADC and each of the gene signatures. We performed gene enrichment analysis using Ingenuity Pathway Analysis (IPA). We adjusted all p values using the Benjamini and Hochberg method. The mean ADC was a significant predictor for the neural subtype. Neural tumors had a significantly lower mean ADC compared to non-neural tumors ([Formula: see text]), with mean ADC of [Formula: see text] and [Formula: see text] for neural and non-neural tumors, respectively. Mean ADC showed an area under the ROC of 0.75 for detecting neural tumors. We found eight gene modules in the GBM cohort. The mean ADC was significantly correlated with the gene signature related with dendritic cell maturation ([Formula: see text], [Formula: see text]). Mean ADC could be used as a biomarker of a gene signature associated with dendritic cell maturation and to assist in identifying patients with neural GBMs, known to be resistant to aggressive standard of care.

Robust Cell Detection of Histopathological Brain Tumor Images Using Sparse Reconstruction and Adaptive Dictionary Selection

Successful diagnostic and prognostic stratification, treatment outcome prediction, and therapy planning depend on reproducible and accurate pathology analysis. Computer aided diagnosis (CAD) is a useful tool to help doctors make better decisions in cancer diagnosis and treatment. Accurate cell detection is often an essential prerequisite for subsequent cellular analysis. The major challenge of robust brain tumor nuclei/cell detection is to handle significant variations in cell appearance and to split touching cells. In this paper, we present an automatic cell detection framework using sparse reconstruction and adaptive dictionary learning. The main contributions of our method are: 1) A sparse reconstruction based approach to split touching cells; 2) An adaptive dictionary learning method used to handle cell appearance variations. The proposed method has been extensively tested on a data set with more than 2000 cells extracted from 32 whole slide scanned images. The automatic cell detection results are compared with the manually annotated ground truth and other state-of-the-art cell detection algorithms. The proposed method achieves the best cell detection accuracy with a F1 score = 0.96.

An Automatic Learning-Based Framework for Robust Nucleus Segmentation

Computer-aided image analysis of histopathology specimens could potentially provide support for early detection and improved characterization of diseases such as brain tumor, pancreatic neuroendocrine tumor (NET), and breast cancer. Automated nucleus segmentation is a prerequisite for various quantitative analyses including automatic morphological feature computation. However, it remains to be a challenging problem due to the complex nature of histopathology images. In this paper, we propose a learning-based framework for robust and automatic nucleus segmentation with shape preservation. Given a nucleus image, it begins with a deep convolutional neural network (CNN) model to generate a probability map, on which an iterative region merging approach is performed for shape initializations. Next, a novel segmentation algorithm is exploited to separate individual nuclei combining a robust selection-based sparse shape model and a local repulsive deformable model. One of the significant benefits of the proposed framework is that it is applicable to different staining histopathology images. Due to the feature learning characteristic of the deep CNN and the high level shape prior modeling, the proposed method is general enough to perform well across multiple scenarios. We have tested the proposed algorithm on three large-scale pathology image datasets using a range of different tissue and stain preparations, and the comparative experiments with recent state of the arts demonstrate the superior performance of the proposed approach.

Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Digital pathology and microscopy image analysis is widely used for comprehensive studies of cell morphology or tissue structure. Manual assessment is labor intensive and prone to interobserver variations. Computer-aided methods, which can significantly improve the objectivity and reproducibility, have attracted a great deal of interest in recent literature. Among the pipeline of building a computer-aided diagnosis system, nucleus or cell detection and segmentation play a very important role to describe the molecular morphological information. In the past few decades, many efforts have been devoted to automated nucleus/cell detection and segmentation. In this review, we provide a comprehensive summary of the recent state-of-the-art nucleus/cell segmentation approaches on different types of microscopy images including bright-field, phase-contrast, differential interference contrast, fluorescence, and electron microscopies. In addition, we discuss the challenges for the current methods and the potential future work of nucleus/cell detection and segmentation.

Stacked Sparse Autoencoder (SSAE) for Nuclei Detection on Breast Cancer Histopathology Images

Automated nuclear detection is a critical step for a number of computer assisted pathology related image analysis algorithms such as for automated grading of breast cancer tissue specimens. The Nottingham Histologic Score system is highly correlated with the shape and appearance of breast cancer nuclei in histopathological images. However, automated nucleus detection is complicated by 1) the large number of nuclei and the size of high resolution digitized pathology images, and 2) the variability in size, shape, appearance, and texture of the individual nuclei. Recently there has been interest in the application of "Deep Learning" strategies for classification and analysis of big image data. Histopathology, given its size and complexity, represents an excellent use case for application of deep learning strategies. In this paper, a Stacked Sparse Autoencoder (SSAE), an instance of a deep learning strategy, is presented for efficient nuclei detection on high-resolution histopathological images of breast cancer. The SSAE learns high-level features from just pixel intensities alone in order to identify distinguishing features of nuclei. A sliding window operation is applied to each image in order to represent image patches via high-level features obtained via the auto-encoder, which are then subsequently fed to a classifier which categorizes each image patch as nuclear or non-nuclear. Across a cohort of 500 histopathological images (2200 × 2200) and approximately 3500 manually segmented individual nuclei serving as the groundtruth, SSAE was shown to have an improved F-measure 84.49% and an average area under Precision-Recall curve (AveP) 78.83%. The SSAE approach also out-performed nine other state of the art nuclear detection strategies.

Sparse Non-negative Matrix Factorization (SNMF) based color unmixing for breast histopathological image analysis

Color deconvolution has emerged as a popular method for color unmixing as a pre-processing step for image analysis of digital pathology images. One deficiency of this approach is that the stain matrix is pre-defined which requires specific knowledge of the data. This paper presents an unsupervised Sparse Non-negative Matrix Factorization (SNMF) based approach for color unmixing. We evaluate this approach for color unmixing of breast pathology images. Compared to Non-negative Matrix Factorization (NMF), the sparseness constraint imposed on coefficient matrix aims to use more meaningful representation of color components for separating stained colors. In this work SNMF is leveraged for decomposing pure stained color in both Immunohistochemistry (IHC) and Hematoxylin and Eosin (H&E) images. SNMF is compared with Principle Component Analysis (PCA), Independent Component Analysis (ICA), Color Deconvolution (CD), and Non-negative Matrix Factorization (NMF) based approaches. SNMF demonstrated improved performance in decomposing brown diaminobenzidine (DAB) component from 36 IHC images as well as accurately segmenting about 1400 nuclei and 500 lymphocytes from H & E images.

Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images

Technological advances in computing, imaging, and genomics have created new opportunities for exploring relationships between histology, molecular events, and clinical outcomes using quantitative methods. Slide scanning devices are now capable of rapidly producing massive digital image archives that capture histological details in high resolution. Commensurate advances in computing and image analysis algorithms enable mining of archives to extract descriptions of histology, ranging from basic human annotations to automatic and precisely quantitative morphometric characterization of hundreds of millions of cells. These imaging capabilities represent a new dimension in tissue-based studies, and when combined with genomic and clinical endpoints, can be used to explore biologic characteristics of the tumor microenvironment and to discover new morphologic biomarkers of genetic alterations and patient outcomes. In this paper, we review developments in quantitative imaging technology and illustrate how image features can be integrated with clinical and genomic data to investigate fundamental problems in cancer. Using motivating examples from the study of glioblastomas (GBMs), we demonstrate how public data from The Cancer Genome Atlas (TCGA) can serve as an open platform to conduct in silico tissue-based studies that integrate existing data resources. We show how these approaches can be used to explore the relation of the tumor microenvironment to genomic alterations and gene expression patterns and to define nuclear morphometric features that are predictive of genetic alterations and clinical outcomes. Challenges, limitations, and emerging opportunities in the area of quantitative imaging and integrative analyses are also discussed.

NUCLEI SEGMENTATION VIA SPARSITY CONSTRAINED CONVOLUTIONAL REGRESSION

Automated profiling of nuclear architecture, in histology sections, can potentially help predict the clinical outcomes. However, the task is challenging as a result of nuclear pleomorphism and cellular states (e.g., cell fate, cell cycle), which are compounded by the batch effect (e.g., variations in fixation and staining). Present methods, for nuclear segmentation, are based on human-designed features that may not effectively capture intrinsic nuclear architecture. In this paper, we propose a novel approach, called sparsity constrained convolutional regression (SCCR), for nuclei segmentation. Specifically, given raw image patches and the corresponding annotated binary masks, our algorithm jointly learns a bank of convolutional filters and a sparse linear regressor, where the former is used for feature extraction, and the latter aims to produce a likelihood for each pixel being nuclear region or background. During classification, the pixel label is simply determined by a thresholding operation applied on the likelihood map. The method has been evaluated using the benchmark dataset collected from The Cancer Genome Atlas (TCGA). Experimental results demonstrate that our method outperforms traditional nuclei segmentation algorithms and is able to achieve competitive performance compared to the state-of-the-art algorithm built upon human-designed features with biological prior knowledge.

Coupled Segmentation of Nuclear and Membrane-bound Macromolecules through Voting and Multiphase Level Set

Membrane-bound macromolecules play an important role in tissue architecture and cell-cell communication, and is regulated by almost one-third of the genome. At the optical scale, one group of membrane proteins expresses themselves as linear structures along the cell surface boundaries, while others are sequestered; and this paper targets the former group. Segmentation of these membrane proteins on a cell-by-cell basis enables the quantitative assessment of localization for comparative analysis. However, such membrane proteins typically lack continuity, and their intensity distributions are often very heterogeneous; moreover, nuclei can form large clump, which further impedes the quantification of membrane signals on a cell-by-cell basis. To tackle these problems, we introduce a three-step process to (i) regularize the membrane signal through iterative tangential voting, (ii) constrain the location of surface proteins by nuclear features, where clumps of nuclei are segmented through a delaunay triangulation approach, and (iii) assign membrane-bound macromolecules to individual cells through an application of multi-phase geodesic level-set. We have validated our method using both synthetic data and a dataset of 200 images, and are able to demonstrate the efficacy of our approach with superior performance.

The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge

The Cancer Genome Atlas (TCGA) is a public funded project that aims to catalogue and discover major cancer-causing genomic alterations to create a comprehensive "atlas" of cancer genomic profiles. So far, TCGA researchers have analysed large cohorts of over 30 human tumours through large-scale genome sequencing and integrated multi-dimensional analyses. Studies of individual cancer types, as well as comprehensive pan-cancer analyses have extended current knowledge of tumorigenesis. A major goal of the project was to provide publicly available datasets to help improve diagnostic methods, treatment standards, and finally to prevent cancer. This review discusses the current status of TCGA Research Network structure, purpose, and achievements.