Iterative voting for inference of structural saliency and characterization of subcellular events

Evaluating Cancer-Related Biomarkers Based on Pathological Images: A Systematic Review

Many diseases are accompanied by changes in certain biochemical indicators called biomarkers in cells or tissues. A variety of biomarkers, including proteins, nucleic acids, antibodies, and peptides, have been identified. Tumor biomarkers have been widely used in cancer risk assessment, early screening, diagnosis, prognosis, treatment, and progression monitoring. For example, the number of circulating tumor cell (CTC) is a prognostic indicator of breast cancer overall survival, and tumor mutation burden (TMB) can be used to predict the efficacy of immune checkpoint inhibitors. Currently, clinical methods such as polymerase chain reaction (PCR) and next generation sequencing (NGS) are mainly adopted to evaluate these biomarkers, which are time-consuming and expansive. Pathological image analysis is an essential tool in medical research, disease diagnosis and treatment, functioning by extracting important physiological and pathological information or knowledge from medical images. Recently, deep learning-based analysis on pathological images and morphology to predict tumor biomarkers has attracted great attention from both medical image and machine learning communities, as this combination not only reduces the burden on pathologists but also saves high costs and time. Therefore, it is necessary to summarize the current process of processing pathological images and key steps and methods used in each process, including: (1) pre-processing of pathological images, (2) image segmentation, (3) feature extraction, and (4) feature model construction. This will help people choose better and more appropriate medical image processing methods when predicting tumor biomarkers.

Spatially Constrained Context-Aware Hierarchical Deep Correlation Filters for Nucleus Detection in Histology Images

Nucleus detection in histology images is a fundamental step for cellular-level analysis in computational pathology. In clinical practice, quantitative nuclear morphology can be used for diagnostic decision making, prognostic stratification, and treatment outcome prediction. Nucleus detection is a challenging task because of large variations in the shape of different types of nucleus such as nuclear clutter, heterogeneous chromatin distribution, and irregular and fuzzy boundaries. To address these challenges, we aim to accurately detect nuclei using spatially constrained context-aware correlation filters using hierarchical deep features extracted from multiple layers of a pre-trained network. During training, we extract contextual patches around each nucleus which are used as negative examples while the actual nucleus patch is used as a positive example. In order to spatially constrain the correlation filters, we propose to construct a spatial structural graph across different nucleus components encoding pairwise similarities. The correlation filters are constrained to act as eigenvectors of the Laplacian of the spatial graphs enforcing these to capture the nucleus structure. A novel objective function is proposed by embedding graph-based structural information as well as the contextual information within the discriminative correlation filter framework. The learned filters are constrained to be orthogonal to both the contextual patches and the spatial graph-Laplacian basis to improve the localization and discriminative performance. The proposed objective function trains a hierarchy of correlation filters on different deep feature layers to capture the heterogeneity in nuclear shape and texture. The proposed algorithm is evaluated on three publicly available datasets and compared with 15 current state-of-the-art methods demonstrating competitive performance in terms of accuracy, speed, and generalization.

A deep learning-based framework for lung cancer survival analysis with biomarker interpretation

BACKGROUND

Lung cancer is the leading cause of cancer-related deaths in both men and women in the United States, and it has a much lower five-year survival rate than many other cancers. Accurate survival analysis is urgently needed for better disease diagnosis and treatment management.

RESULTS

In this work, we propose a survival analysis system that takes advantage of recently emerging deep learning techniques. The proposed system consists of three major components. 1) The first component is an end-to-end cellular feature learning module using a deep neural network with global average pooling. The learned cellular representations encode high-level biologically relevant information without requiring individual cell segmentation, which is aggregated into patient-level feature vectors by using a locality-constrained linear coding (LLC)-based bag of words (BoW) encoding algorithm. 2) The second component is a Cox proportional hazards model with an elastic net penalty for robust feature selection and survival analysis. 3) The third commponent is a biomarker interpretation module that can help localize the image regions that contribute to the survival model's decision. Extensive experiments show that the proposed survival model has excellent predictive power for a public (i.e., The Cancer Genome Atlas) lung cancer dataset in terms of two commonly used metrics: log-rank test (p-value) of the Kaplan-Meier estimate and concordance index (c-index).

CONCLUSIONS

In this work, we have proposed a segmentation-free survival analysis system that takes advantage of the recently emerging deep learning framework and well-studied survival analysis methods such as the Cox proportional hazards model. In addition, we provide an approach to visualize the discovered biomarkers, which can serve as concrete evidence supporting the survival model's decision.

Assessing the Impact of Color Normalization in Convolutional Neural Network-Based Nuclei Segmentation Frameworks

Image analysis tools for cancer, such as automatic nuclei segmentation, are impacted by the inherent variation contained in pathology image data. Convolutional neural networks (CNN), demonstrate success in generalizing to variable data, illustrating great potential as a solution to the problem of data variability. In some CNN-based segmentation works for digital pathology, authors apply color normalization (CN) to reduce color variability of data as a preprocessing step prior to prediction, while others do not. Both approaches achieve reasonable performance and yet, the reasoning for utilizing this step has not been justified. It is therefore important to evaluate the necessity and impact of CN for deep learning frameworks, and its effect on downstream processes. In this paper, we evaluate the effect of popular CN methods on CNN-based nuclei segmentation frameworks.

Cell segmentation methods for label-free contrast microscopy: review and comprehensive comparison

BACKGROUND

Because of its non-destructive nature, label-free imaging is an important strategy for studying biological processes. However, routine microscopic techniques like phase contrast or DIC suffer from shadow-cast artifacts making automatic segmentation challenging. The aim of this study was to compare the segmentation efficacy of published steps of segmentation work-flow (image reconstruction, foreground segmentation, cell detection (seed-point extraction) and cell (instance) segmentation) on a dataset of the same cells from multiple contrast microscopic modalities.

RESULTS

We built a collection of routines aimed at image segmentation of viable adherent cells grown on the culture dish acquired by phase contrast, differential interference contrast, Hoffman modulation contrast and quantitative phase imaging, and we performed a comprehensive comparison of available segmentation methods applicable for label-free data. We demonstrated that it is crucial to perform the image reconstruction step, enabling the use of segmentation methods originally not applicable on label-free images. Further we compared foreground segmentation methods (thresholding, feature-extraction, level-set, graph-cut, learning-based), seed-point extraction methods (Laplacian of Gaussians, radial symmetry and distance transform, iterative radial voting, maximally stable extremal region and learning-based) and single cell segmentation methods. We validated suitable set of methods for each microscopy modality and published them online.

CONCLUSIONS

We demonstrate that image reconstruction step allows the use of segmentation methods not originally intended for label-free imaging. In addition to the comprehensive comparison of methods, raw and reconstructed annotated data and Matlab codes are provided.

Effective nuclei segmentation with sparse shape prior and dynamic occlusion constraint for glioblastoma pathology images

We propose a segmentation method for nuclei in glioblastoma histopathologic images based on a sparse shape prior guided variational level set framework. By spectral clustering and sparse coding, a set of shape priors is exploited to accommodate complicated shape variations. We automate the object contour initialization by a seed detection algorithm and deform contours by minimizing an energy functional that incorporates a shape term in a sparse shape prior representation, an adaptive contour occlusion penalty term, and a boundary term encouraging contours to converge to strong edges. As a result, our approach is able to deal with mutual occlusions and detect contours of multiple intersected nuclei simultaneously. Our method is applied to several whole-slide histopathologic image datasets for nuclei segmentation. The proposed method is compared with other state-of-the-art methods and demonstrates good accuracy for nuclei detection and segmentation, suggesting its promise to support biomedical image-based investigations.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network

Detection of nuclei is an important step in phenotypic profiling of 1) histology sections imaged in bright field; and 2) colony formation of the 3-D cell culture models that are imaged using confocal microscopy. It is shown that feature-based representation of the original image improves color decomposition (CD) and subsequent nuclear detection using convolutional neural networks independent of the imaging modality. The feature-based representation utilizes the Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Moreover, in the case of samples imaged in bright field, the LoG response also provides the necessary initial statistics for CD using nonnegative matrix factorization. Several permutations of input data representations and network architectures are evaluated to show that by coupling improved CD and the LoG response of this representation, detection of nuclei is advanced. In particular, the frequencies of detection of nuclei with the vesicular or necrotic phenotypes, or poor staining, are improved. The overall system has been evaluated against manually annotated images, and the F-scores for alternative representations and architectures are reported.

Contour-Seed Pairs Learning-Based Framework for Simultaneously Detecting and Segmenting Various Overlapping Cells/Nuclei in Microscopy Images

In this paper, we propose a novel contour-seed pairs learning-based framework for robust and automated cell/nucleus segmentation. Automated granular object segmentation in microscopy images has significant clinical importance for pathology grading of the cell carcinoma and gene expression. The focus of the past literature is dominated by either segmenting a certain type of cells/nuclei or simply splitting the clustered objects without contours inference of them. Our method addresses these issues by formulating the detection and segmentation tasks in terms of a unified regression problem, where a cascade sparse regression chain model is trained and then applied to return object locations and entire boundaries of clustered objects. In particular, we first learn a set of online convolutional features in each layer. Then, in the proposed cascade sparse regression chain, with the input from the learned features, we iteratively update the locations and clustered object boundaries until convergence. In this way, the boundary evidences of each individual object can be easily delineated and be further fed to a complete contour inference procedure optimized by the minimum description length principle. For any probe image, our method enables to analyze free-lying and overlapping cells with complex shapes. Experimental results show that the proposed method is very generic and performs well on contour inferences of various cell/nucleus types. Compared with the current segmentation techniques, our approach achieves state-of-the-art performances on four challenging datasets, i.e., the kidney renal cell carcinoma histopathology dataset, Drosophila Kc167 cellular dataset, differential interference contrast red blood cell dataset, and cervical cytology dataset.

Object-Oriented Segmentation of Cell Nuclei in Fluorescence Microscopy Images

Cell nucleus segmentation remains an open and challenging problem especially to segment nuclei in cell clumps. Splitting a cell clump would be straightforward if the gradients of boundary pixels in-between the nuclei were always higher than the others. However, imperfections may exist: inhomogeneities of pixel intensities in a nucleus may cause to define spurious boundaries whereas insufficient pixel intensity differences at the border of overlapping nuclei may cause to miss some true boundary pixels. In contrast, these imperfections are typically observed at the pixel-level, causing local changes in pixel values without changing the semantics on a large scale. In response to these issues, this article introduces a new nucleus segmentation method that relies on using gradient information not at the pixel level but at the object level. To this end, it proposes to decompose an image into smaller homogeneous subregions, define edge-objects at four different orientations to encode the gradient information at the object level, and devise a merging algorithm, in which the edge-objects vote for subregion pairs along their orientations and the pairs are iteratively merged if they get sufficient votes from multiple orientations. Our experiments on fluorescence microscopy images reveal that this high-level representation and the design of a merging algorithm using edge-objects (gradients at the object level) improve the segmentation results.

VOTING-BASED SEGMENTATION OF OVERLAPPING NUCLEI IN CLARITY IMAGES

New tissue-clearing techniques and improvements in optical microscopy have rapidly advanced capabilities to acquire volumetric imagery of neural tissue at resolutions of one micron or better. As sizes for data collections increase, accurate automatic segmentation of cell nuclei becomes increasingly important for quantitative analysis of imaged tissue. We present a cell nucleus segmentation method that is formulated as a parameter estimation problem with the goal of determining the count, shapes, and locations of nuclei that most accurately describe an image. We applied our new voting-based approach to fluorescence confocal microscopy images of neural tissue stained with DAPI, which highlights nuclei. Compared to manual counting of cells in three DAPI images, our method outperformed three existing approaches. On a manually labeled high-resolution DAPI image, our method also outperformed those methods and achieved a cell count accuracy of 98.99% and mean Dice coefficient of 0.6498.

Robust Cell Detection for Large-Scale 3D Microscopy Using GPU-Accelerated Iterative Voting

High-throughput imaging techniques, such as Knife-Edge Scanning Microscopy (KESM),are capable of acquiring three-dimensional whole-organ images at sub-micrometer resolution. These images are challenging to segment since they can exceed several terabytes (TB) in size, requiring extremely fast and fully automated algorithms. Staining techniques are limited to contrast agents that can be applied to large samples and imaged in a single pass. This requires maximizing the number of structures labeled in a single channel, resulting in images that are densely packed with spatial features. In this paper, we propose a three-dimensional approach for locating cells based on iterative voting. Due to the computational complexity of this algorithm, a highly efficient GPU implementation is required to make it practical on large data sets. The proposed algorithm has a limited number of input parameters and is highly parallel.

Measuring multiple parameters of CD8+ tumor-infiltrating lymphocytes in human cancers by image analysis

BACKGROUND

Immuno-oncology and cancer immunotherapies are areas of intense research. The numbers and locations of CD8+ tumor-infiltrating lymphocytes (TILs) are important measures of the immune response to cancer with prognostic, pharmacodynamic, and predictive potential. We describe the development, validation, and application of advanced image analysis methods to characterize multiple immunohistochemistry-derived CD8 parameters in clinical and nonclinical tumor tissues.

METHODS

Commercial resection tumors from nine cancer types, and paired screening/on-drug biopsies of non-small-cell lung carcinoma (NSCLC) patients enrolled in a phase 1/2 clinical trial investigating the PD-L1 antibody therapy durvalumab (NCT01693562), were immunostained for CD8. Additional NCT01693562 samples were immunostained with a CD8/PD-L1 dual immunohistochemistry assay. Whole-slide scanning was performed, tumor regions were annotated by a pathologist, and images were analyzed with customized algorithms using Definiens Developer XD software. Validation of image analysis data used cell-by-cell comparison to pathologist scoring across a range of CD8+ TIL densities of all nine cancers, relying primarily on 95% confidence in having at least moderate agreement regarding Lin concordance correlation coefficient (CCC = 0.88-0.99, CCC_lower = 0.65-0.96).

RESULTS

We found substantial variability in CD8+ TILs between individual patients and across the nine types of human cancer. Diffuse large B-cell lymphoma had several-fold more CD8+ TILs than some other cancers. TIL densities were significantly higher in the invasive margin versus tumor center for carcinomas of head and neck, kidney and pancreas, and NSCLC; the reverse was true only for prostate cancer. In paired patient biopsies, there were significantly increased CD8+ TILs 6 weeks after onset of durvalumab therapy (mean of 365 cells/mm2 over baseline; P = 0.009), consistent with immune activation. Image analysis accurately enumerated CD8+ TILs in PD-L1+ regions of lung tumors using the dual assay and also measured elongate CD8+ lymphocytes which constituted a fraction of overall TILs.

CONCLUSIONS

Validated image analysis accurately enumerates CD8+ TILs, permitting comparisons of CD8 parameters among tumor regions, individual patients, and cancer types. It also enables the more complex digital solutions needed to better understand cancer immunity, like analysis of multiplex immunohistochemistry and spatial evaluation of the various components comprising the tumor microenvironment.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT01693562 . Study code: CD-ON-MEDI4736-1108. Interventional study (ongoing but not currently recruiting). Actual study start date: August 29, 2012. Primary completion date: June 23, 2017 (final data collection date for primary outcome measure).

Efficient and robust cell detection: A structured regression approach

Efficient and robust cell detection serves as a critical prerequisite for many subsequent biomedical image analysis methods and computer-aided diagnosis (CAD). It remains a challenging task due to touching cells, inhomogeneous background noise, and large variations in cell sizes and shapes. In addition, the ever-increasing amount of available datasets and the high resolution of whole-slice scanned images pose a further demand for efficient processing algorithms. In this paper, we present a novel structured regression model based on a proposed fully residual convolutional neural network for efficient cell detection. For each testing image, our model learns to produce a dense proximity map that exhibits higher responses at locations near cell centers. Our method only requires a few training images with weak annotations (just one dot indicating the cell centroids). We have extensively evaluated our method using four different datasets, covering different microscopy staining methods (e.g., H & E or Ki-67 staining) or image acquisition techniques (e.g., bright-filed image or phase contrast). Experimental results demonstrate the superiority of our method over existing state of the art methods in terms of both detection accuracy and running time.

Automated Photoreceptor Cell Identification on Nonconfocal Adaptive Optics Images Using Multiscale Circular Voting

Purpose

Adaptive optics scanning light ophthalmoscopy (AOSLO) has enabled quantification of the photoreceptor mosaic in the living human eye using metrics such as cell density and average spacing. These rely on the identification of individual cells. Here, we demonstrate a novel approach for computer-aided identification of cone photoreceptors on nonconfocal split detection AOSLO images.

Methods

Algorithms for identification of cone photoreceptors were developed, based on multiscale circular voting (MSCV) in combination with a priori knowledge that split detection images resemble Nomarski differential interference contrast images, in which dark and bright regions are present on the two sides of each cell. The proposed algorithm locates dark and bright region pairs, iteratively refining the identification across multiple scales. Identification accuracy was assessed in data from 10 subjects by comparing automated identifications with manual labeling, followed by computation of density and spacing metrics for comparison to histology and published data.

Results

There was good agreement between manual and automated cone identifications with overall recall, precision, and F1 score of 92.9%, 90.8%, and 91.8%, respectively. On average, computed density and spacing values using automated identification were within 10.7% and 11.2% of the expected histology values across eccentricities ranging from 0.5 to 6.2 mm. There was no statistically significant difference between MSCV-based and histology-based density measurements (P = 0.96, Kolmogorov-Smirnov 2-sample test).

Conclusions

MSCV can accurately detect cone photoreceptors on split detection images across a range of eccentricities, enabling quick, objective estimation of photoreceptor mosaic metrics, which will be important for future clinical trials utilizing adaptive optics.

Computer-assisted quantification of CD3+ T cells in follicular lymphoma

The advance of high resolution digital scans of pathology slides allowed development of computer based image analysis algorithms that may help pathologists in IHC stains quantification. While very promising, these methods require further refinement before they are implemented in routine clinical setting. Particularly critical is to evaluate algorithm performance in a setting similar to current clinical practice. In this article, we present a pilot study that evaluates the use of a computerized cell quantification method in the clinical estimation of CD3 positive (CD3+) T cells in follicular lymphoma (FL). Our goal is to demonstrate the degree to which computerized quantification is comparable to the practice of estimation by a panel of expert pathologists. The computerized quantification method uses entropy based histogram thresholding to separate brown (CD3+) and blue (CD3-) regions after a color space transformation. A panel of four board-certified hematopathologists evaluated a database of 20 FL images using two different reading methods: visual estimation and manual marking of each CD3+ cell in the images. These image data and the readings provided a reference standard and the range of variability among readers. Sensitivity and specificity measures of the computer's segmentation of CD3+ and CD- T cell are recorded. For all four pathologists, mean sensitivity and specificity measures are 90.97 and 88.38%, respectively. The computerized quantification method agrees more with the manual cell marking as compared to the visual estimations. Statistical comparison between the computerized quantification method and the pathologist readings demonstrated good agreement with correlation coefficient values of 0.81 and 0.96 in terms of Lin's concordance correlation and Spearman's correlation coefficient, respectively. These values are higher than most of those calculated among the pathologists. In the future, the computerized quantification method may be used to investigate the relationship between the overall architectural pattern (i.e., interfollicular vs. follicular) and outcome measures (e.g., overall survival, and time to treatment). © 2017 International Society for Advancement of Cytometry.

Detection of Nuclei in H&E Stained Sections Using Convolutional Neural Networks

Detection of nuclei is an important step in phenotypic profiling of histology sections that are usually imaged in bright field. However, nuclei can have multiple phenotypes, which are difficult to model. It is shown that convolutional neural networks (CNN)s can learn different phenotypic signatures for nuclear detection, and that the performance is improved with the feature-based representation of the original image. The feature-based representation utilizes Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Several combinations of input data representations are evaluated to show that by LoG representation, detection of nuclei is advanced. In addition, the efficacy of CNN for vesicular and hyperchromatic nuclei is evaluated. In particular, the frequency of detection of nuclei with the vesicular and apoptotic phenotypes is increased. The overall system has been evaluated against manually annotated nuclei and the F-Scores for alternative representations have been reported.

Quantitative 3D analysis of complex single border cell behaviors in coordinated collective cell migration

Understanding the mechanisms of collective cell migration is crucial for cancer metastasis, wound healing and many developmental processes. Imaging a migrating cluster in vivo is feasible, but the quantification of individual cell behaviours remains challenging. We have developed an image analysis toolkit, CCMToolKit, to quantify the Drosophila border cell system. In addition to chaotic motion, previous studies reported that the migrating cells are able to migrate in a highly coordinated pattern. We quantify the rotating and running migration modes in 3D while also observing a range of intermediate behaviours. Running mode is driven by cluster external protrusions. Rotating mode is associated with cluster internal cell extensions that could not be easily characterized. Although the cluster moves slower while rotating, individual cells retain their mobility and are in fact slightly more active than in running mode. We also show that individual cells may exchange positions during migration.

Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: A review

With the advance of digital pathology, image analysis has begun to show its advantages in information analysis of hematoxylin and eosin histopathology images. Generally, histological features in hematoxylin and eosin images are measured to evaluate tumor grade and prognosis for breast cancer. This review summarized recent works in image analysis of hematoxylin and eosin histopathology images for breast cancer prognosis. First, prognostic factors for breast cancer based on hematoxylin and eosin histopathology images were summarized. Then, usual procedures of image analysis for breast cancer prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Finally, the prognostic value of image features and image feature–based prognostic models was evaluated. Moreover, we discussed the issues of current analysis, and some directions for future research.

Robust Nanoparticles Detection From Noisy Background by Fusing Complementary Image Information

This paper studies the problem of detecting the presence of nanoparticles in noisy transmission electron microscopic (TEM) images and then fitting each nanoparticle with an elliptic shape model. In order to achieve robustness while handling low contrast and high noise in the TEM images, we propose an approach to fuse two kinds of complementary image information, namely, the pixel intensity and the gradient (the first derivative in intensity). Our approach entails two main steps: 1) the first step is to, after necessary pre-processing, employ both intensity-based information and gradient-based information to process the same TEM image and produce two independent sets of results and 2) the subsequent step is to formulate a binary integer programming (BIP) problem for conflict resolution among the two sets of results. Solving the BIP problem determines the final nanoparticle identification. We apply our method to a set of TEM images taken under different microscopic resolutions and noise levels. The empirical results show the merit of the proposed method. It can process a TEM image of 1024×1024 pixels in a few minutes, and the processed outcomes appear rather robust.

Multi-Pass Adaptive Voting for Nuclei Detection in Histopathological Images

Nuclei detection is often a critical initial step in the development of computer aided diagnosis and prognosis schemes in the context of digital pathology images. While over the last few years, a number of nuclei detection methods have been proposed, most of these approaches make idealistic assumptions about the staining quality of the tissue. In this paper, we present a new Multi-Pass Adaptive Voting (MPAV) for nuclei detection which is specifically geared towards images with poor quality staining and noise on account of tissue preparation artifacts. The MPAV utilizes the symmetric property of nuclear boundary and adaptively selects gradient from edge fragments to perform voting for a potential nucleus location. The MPAV was evaluated in three cohorts with different staining methods: Hematoxylin &Eosin, CD31 &Hematoxylin, and Ki-67 and where most of the nuclei were unevenly and imprecisely stained. Across a total of 47 images and nearly 17,700 manually labeled nuclei serving as the ground truth, MPAV was able to achieve a superior performance, with an area under the precision-recall curve (AUC) of 0.73. Additionally, MPAV also outperformed three state-of-the-art nuclei detection methods, a single pass voting method, a multi-pass voting method, and a deep learning based method.

Size-Invariant Detection of Cell Nuclei in Microscopy Images

Accurate detection of individual cell nuclei in microscopy images is an essential and fundamental task for many biological studies. In particular, multivariate fluorescence microscopy is used to observe different aspects of cells in cultures. Manual detection of individual cell nuclei by visual inspection is time consuming, and prone to induce subjective bias. This makes automatic detection of cell nuclei essential for large-scale, objective studies of cell cultures. Blur, clutter, bleed-through, imaging noise and touching and partially overlapping nuclei with varying sizes and shapes make automated detection of individual cell nuclei a challenging task using image analysis. In this paper we propose a new automated method for fast and robust detection of individual cell nuclei based on their radial symmetric nature in fluorescence in-situ hybridization (FISH) images obtained via confocal microscopy. The main contributions are two-fold. 1) This work presents a more accurate cell nucleus detection system using the fast radial symmetry transform (FRST). 2) The proposed cell nucleus detection system is robust against most occlusions and variations in size and moderate shape deformations. We evaluate the performance of the proposed algorithm using precision/recall rates, Fβ-score and root-mean-squared distance (RMSD) and show that our algorithm provides improved detection accuracy compared to existing algorithms.

Robust Cell Segmentation for Histological Images of Glioblastoma

Glioblastoma (GBM) is a malignant brain tumor with uniformly dismal prognosis. Quantitative analysis of GBM cells is an important avenue to extract latent histologic disease signatures to correlate with molecular underpinnings and clinical outcomes. As a prerequisite, a robust and accurate cell segmentation is required. In this paper, we present an automated cell segmentation method that can satisfactorily address segmentation of overlapped cells commonly seen in GBM histology specimens. This method first detects cells with seed connectivity, distance constraints, image edge map, and a shape-based voting image. Initialized by identified seeds, cell boundaries are deformed with an improved variational level set method that can handle clumped cells. We test our method on 40 histological images of GBM with human annotations. The validation results suggest that our cell segmentation method is promising and represents an advance in quantitative cancer research.

Robust Cell Detection of Histopathological Brain Tumor Images Using Sparse Reconstruction and Adaptive Dictionary Selection

Successful diagnostic and prognostic stratification, treatment outcome prediction, and therapy planning depend on reproducible and accurate pathology analysis. Computer aided diagnosis (CAD) is a useful tool to help doctors make better decisions in cancer diagnosis and treatment. Accurate cell detection is often an essential prerequisite for subsequent cellular analysis. The major challenge of robust brain tumor nuclei/cell detection is to handle significant variations in cell appearance and to split touching cells. In this paper, we present an automatic cell detection framework using sparse reconstruction and adaptive dictionary learning. The main contributions of our method are: 1) A sparse reconstruction based approach to split touching cells; 2) An adaptive dictionary learning method used to handle cell appearance variations. The proposed method has been extensively tested on a data set with more than 2000 cells extracted from 32 whole slide scanned images. The automatic cell detection results are compared with the manually annotated ground truth and other state-of-the-art cell detection algorithms. The proposed method achieves the best cell detection accuracy with a F1 score = 0.96.

Segmentation of Pollen Tube Growth Videos Using Dynamic Bi-Modal Fusion and Seam Carving

The growth of pollen tubes is of significant interest in plant cell biology, as it provides an understanding of internal cell dynamics that affect observable structural characteristics such as cell diameter, length, and growth rate. However, these parameters can only be measured in experimental videos if the complete shape of the cell is known. The challenge is to accurately obtain the cell boundary in noisy video images. Usually, these measurements are performed by a scientist who manually draws regions-of-interest on the images displayed on a computer screen. In this paper, a new automated technique is presented for boundary detection by fusing fluorescence and brightfield images, and a new efficient method of obtaining the final cell boundary through the process of Seam Carving is proposed. This approach takes advantage of the nature of the fusion process and also the shape of the pollen tube to efficiently search for the optimal cell boundary. In video segmentation, the first two frames are used to initialize the segmentation process by creating a search space based on a parametric model of the cell shape. Updates to the search space are performed based on the location of past segmentations and a prediction of the next segmentation.Experimental results show comparable accuracy to a previous method, but significant decrease in processing time. This has the potential for real time applications in pollen tube microscopy.

Automatic Nuclei Detection Based on Generalized Laplacian of Gaussian Filters

Efficient and accurate detection of cell nuclei is an important step toward automatic analysis in histopathology. In this work, we present an automatic technique based on generalized Laplacian of Gaussian (gLoG) filter for nuclei detection in digitized histological images. The proposed technique first generates a bank of gLoG kernels with different scales and orientations and then performs convolution between directional gLoG kernels and the candidate image to obtain a set of response maps. The local maxima of response maps are detected and clustered into different groups by mean-shift algorithm based on their geometrical closeness. The point which has the maximum response in each group is finally selected as the nucleus seed. Experimental results on two datasets show that the proposed technique provides a superior performance in nuclei detection compared to existing techniques.

Iterative h-minima-based marker-controlled watershed for cell nucleus segmentation

Automated microscopy imaging systems facilitate high-throughput screening in molecular cellular biology research. The first step of these systems is cell nucleus segmentation, which has a great impact on the success of the overall system. The marker-controlled watershed is a technique commonly used by the previous studies for nucleus segmentation. These studies define their markers finding regional minima on the intensity/gradient and/or distance transform maps. They typically use the h-minima transform beforehand to suppress noise on these maps. The selection of the h value is critical; unnecessarily small values do not sufficiently suppress the noise, resulting in false and oversegmented markers, and unnecessarily large ones suppress too many pixels, causing missing and undersegmented markers. Because cell nuclei show different characteristics within an image, the same h value may not work to define correct markers for all the nuclei. To address this issue, in this work, we propose a new watershed algorithm that iteratively identifies its markers, considering a set of different h values. In each iteration, the proposed algorithm defines a set of candidates using a particular h value and selects the markers from those candidates provided that they fulfill the size requirement. Working with widefield fluorescence microscopy images, our experiments reveal that the use of multiple h values in our iterative algorithm leads to better segmentation results, compared to its counterparts. © 2016 International Society for Advancement of Cytometry.

Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Digital pathology and microscopy image analysis is widely used for comprehensive studies of cell morphology or tissue structure. Manual assessment is labor intensive and prone to interobserver variations. Computer-aided methods, which can significantly improve the objectivity and reproducibility, have attracted a great deal of interest in recent literature. Among the pipeline of building a computer-aided diagnosis system, nucleus or cell detection and segmentation play a very important role to describe the molecular morphological information. In the past few decades, many efforts have been devoted to automated nucleus/cell detection and segmentation. In this review, we provide a comprehensive summary of the recent state-of-the-art nucleus/cell segmentation approaches on different types of microscopy images including bright-field, phase-contrast, differential interference contrast, fluorescence, and electron microscopies. In addition, we discuss the challenges for the current methods and the potential future work of nucleus/cell detection and segmentation.

Stacked Sparse Autoencoder (SSAE) for Nuclei Detection on Breast Cancer Histopathology Images

Automated nuclear detection is a critical step for a number of computer assisted pathology related image analysis algorithms such as for automated grading of breast cancer tissue specimens. The Nottingham Histologic Score system is highly correlated with the shape and appearance of breast cancer nuclei in histopathological images. However, automated nucleus detection is complicated by 1) the large number of nuclei and the size of high resolution digitized pathology images, and 2) the variability in size, shape, appearance, and texture of the individual nuclei. Recently there has been interest in the application of "Deep Learning" strategies for classification and analysis of big image data. Histopathology, given its size and complexity, represents an excellent use case for application of deep learning strategies. In this paper, a Stacked Sparse Autoencoder (SSAE), an instance of a deep learning strategy, is presented for efficient nuclei detection on high-resolution histopathological images of breast cancer. The SSAE learns high-level features from just pixel intensities alone in order to identify distinguishing features of nuclei. A sliding window operation is applied to each image in order to represent image patches via high-level features obtained via the auto-encoder, which are then subsequently fed to a classifier which categorizes each image patch as nuclear or non-nuclear. Across a cohort of 500 histopathological images (2200 × 2200) and approximately 3500 manually segmented individual nuclei serving as the groundtruth, SSAE was shown to have an improved F-measure 84.49% and an average area under Precision-Recall curve (AveP) 78.83%. The SSAE approach also out-performed nine other state of the art nuclear detection strategies.

Segmentation of Overlapping Elliptical Objects in Silhouette Images

Segmentation of partially overlapping objects with a known shape is needed in an increasing amount of various machine vision applications. This paper presents a method for segmentation of clustered partially overlapping objects with a shape that can be approximated using an ellipse. The method utilizes silhouette images, which means that it requires only that the foreground (objects) and background can be distinguished from each other. The method starts with seedpoint extraction using bounded erosion and fast radial symmetry transform. Extracted seedpoints are then utilized to associate edge points to objects in order to create contour evidence. Finally, contours of the objects are estimated by fitting ellipses to the contour evidence. The experiments on one synthetic and two different real data sets showed that the proposed method outperforms two current state-of-art approaches in overlapping objects segmentation.

NUCKS1 is a novel RAD51AP1 paralog important for homologous recombination and genome stability

NUCKS1 (nuclear casein kinase and cyclin-dependent kinase substrate 1) is a 27 kD chromosomal, vertebrate-specific protein, for which limited functional data exist. Here, we demonstrate that NUCKS1 shares extensive sequence homology with RAD51AP1 (RAD51 associated protein 1), suggesting that these two proteins are paralogs. Similar to the phenotypic effects of RAD51AP1 knockdown, we find that depletion of NUCKS1 in human cells impairs DNA repair by homologous recombination (HR) and chromosome stability. Depletion of NUCKS1 also results in greatly increased cellular sensitivity to mitomycin C (MMC), and in increased levels of spontaneous and MMC-induced chromatid breaks. NUCKS1 is critical to maintaining wild type HR capacity, and, as observed for a number of proteins involved in the HR pathway, functional loss of NUCKS1 leads to a slow down in DNA replication fork progression with a concomitant increase in the utilization of new replication origins. Interestingly, recombinant NUCKS1 shares the same DNA binding preference as RAD51AP1, but binds to DNA with reduced affinity when compared to RAD51AP1. Our results show that NUCKS1 is a chromatin-associated protein with a role in the DNA damage response and in HR, a DNA repair pathway critical for tumor suppression.

An automated cell viability quantification method for low-resolution confocal images of closely packed cells based on a modified gradient flow tracking algorithm

Fluorescent-based live/dead labelling combined with fluorescent microscopy is one of the widely used and reliable methods for assessment of cell viability. This method is, however, not quantitative. Many image-processing methods have been proposed for cell quantification in an image. Among all these methods, several of them are capable of quantifying the number of cells in high-resolution images with closely packed cells. However, no method has addressed the quantification of the number of cells in low-resolution images containing closely packed cells with variable sizes. This paper presents a novel method for automatic quantification of live/dead cells in 2D fluorescent low-resolution images containing closely packed cells with variable sizes using a mean shift-based gradient flow tracking. Accuracy and performance of the method was tested on growth plate confocal images. Experimental results show that our algorithm has a better performance in comparison to other methods used in similar detection conditions.

Light sheet fluorescence microscopy (LSFM): past, present and future

Light sheet fluorescence microscopy (LSFM) has emerged as an important imaging modality to follow biology in live 3D samples over time with reduced phototoxicity and photobleaching. In particular, LSFM has been instrumental in revealing the detail of early embryonic development of Zebrafish, Drosophila, and C. elegans. Open access projects, DIY-SPIM, OpenSPIM, and OpenSPIN, now allow LSFM to be set-up easily and at low cost. The aim of this paper is to facilitate the set-up and use of LSFM by reviewing and comparing open access projects, image processing tools and future challenges.

Segmenting overlapping cell nuclei in digital histopathology images

Automatic quantification of cell nuclei in immunostained images is highly desired by pathologists in diagnosis. In this paper, we present a new approach for the segmentation of severely clustered overlapping nuclei. The proposed approach first involves applying a combined global and local threshold method to extract foreground regions. In order to segment clustered overlapping nuclei in the foreground regions, seed markers are obtained by utilizing morphological filtering and intensity based region growing. Seeded watershed is then applied and clustered nuclei are separated. As pixels corresponding to stained cellular cytoplasm can be falsely identified as belonging to nuclei, a post processing step identifying positive nuclei pixels is added to eliminate these false pixels. This new approach has been tested on a set of manually labeled Tissue Microarray (TMA) and Whole Slide Images (WSI) colorectal cancers stained for the biomarker P53. Experimental results show that it outperformed currently available state of the art methods in nuclei segmentation.

iCut: an Integrative Cut Algorithm Enables Accurate Segmentation of Touching Cells

Individual cells play essential roles in the biological processes of the brain. The number of neurons changes during both normal development and disease progression. High-resolution imaging has made it possible to directly count cells. However, the automatic and precise segmentation of touching cells continues to be a major challenge for massive and highly complex datasets. Thus, an integrative cut (iCut) algorithm, which combines information regarding spatial location and intervening and concave contours with the established normalized cut, has been developed. iCut involves two key steps: (1) a weighting matrix is first constructed with the abovementioned information regarding the touching cells and (2) a normalized cut algorithm that uses the weighting matrix is implemented to separate the touching cells into isolated cells. This novel algorithm was evaluated using two types of data: the open SIMCEP benchmark dataset and our micro-optical imaging dataset from a Nissl-stained mouse brain. It has achieved a promising recall/precision of 91.2 ± 2.1%/94.1 ± 1.8% and 86.8 ± 4.1%/87.5 ± 5.7%, respectively, for the two datasets. As quantified using the harmonic mean of recall and precision, the accuracy of iCut is higher than that of some state-of-the-art algorithms. The better performance of this fully automated algorithm can benefit studies of brain cytoarchitecture.

Segmentation of the Clustered Cells with Optimized Boundary Detection in Negative Phase Contrast Images

Cell image segmentation plays a central role in numerous biology studies and clinical applications. As a result, the development of cell image segmentation algorithms with high robustness and accuracy is attracting more and more attention. In this study, an automated cell image segmentation algorithm is developed to get improved cell image segmentation with respect to cell boundary detection and segmentation of the clustered cells for all cells in the field of view in negative phase contrast images. A new method which combines the thresholding method and edge based active contour method was proposed to optimize cell boundary detection. In order to segment clustered cells, the geographic peaks of cell light intensity were utilized to detect numbers and locations of the clustered cells. In this paper, the working principles of the algorithms are described. The influence of parameters in cell boundary detection and the selection of the threshold value on the final segmentation results are investigated. At last, the proposed algorithm is applied to the negative phase contrast images from different experiments. The performance of the proposed method is evaluated. Results show that the proposed method can achieve optimized cell boundary detection and highly accurate segmentation for clustered cells.

Automated profiling of individual cell-cell interactions from high-throughput time-lapse imaging microscopy in nanowell grids (TIMING)

MOTIVATION

There is a need for effective automated methods for profiling dynamic cell-cell interactions with single-cell resolution from high-throughput time-lapse imaging data, especially, the interactions between immune effector cells and tumor cells in adoptive immunotherapy.

RESULTS

Fluorescently labeled human T cells, natural killer cells (NK), and various target cells (NALM6, K562, EL4) were co-incubated on polydimethylsiloxane arrays of sub-nanoliter wells (nanowells), and imaged using multi-channel time-lapse microscopy. The proposed cell segmentation and tracking algorithms account for cell variability and exploit the nanowell confinement property to increase the yield of correctly analyzed nanowells from 45% (existing algorithms) to 98% for wells containing one effector and a single target, enabling automated quantification of cell locations, morphologies, movements, interactions, and deaths without the need for manual proofreading. Automated analysis of recordings from 12 different experiments demonstrated automated nanowell delineation accuracy >99%, automated cell segmentation accuracy >95%, and automated cell tracking accuracy of 90%, with default parameters, despite variations in illumination, staining, imaging noise, cell morphology, and cell clustering. An example analysis revealed that NK cells efficiently discriminate between live and dead targets by altering the duration of conjugation. The data also demonstrated that cytotoxic cells display higher motility than non-killers, both before and during contact.

CONTACT

broysam@central.uh.edu or nvaradar@central.uh.edu

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

An efficient technique for nuclei segmentation based on ellipse descriptor analysis and improved seed detection algorithm

In this paper, we propose an efficient method for segmenting cell nuclei in the skin histopathological images. The proposed technique consists of four modules. First, it separates the nuclei regions from the background with an adaptive threshold technique. Next, an elliptical descriptor is used to detect the isolated nuclei with elliptical shapes. This descriptor classifies the nuclei regions based on two ellipticity parameters. Nuclei clumps and nuclei with irregular shapes are then localized by an improved seed detection technique based on voting in the eroded nuclei regions. Finally, undivided nuclei regions are segmented by a marked watershed algorithm. Experimental results on 114 different image patches indicate that the proposed technique provides a superior performance in nuclei detection and segmentation.

Coupled Segmentation of Nuclear and Membrane-bound Macromolecules through Voting and Multiphase Level Set

Membrane-bound macromolecules play an important role in tissue architecture and cell-cell communication, and is regulated by almost one-third of the genome. At the optical scale, one group of membrane proteins expresses themselves as linear structures along the cell surface boundaries, while others are sequestered; and this paper targets the former group. Segmentation of these membrane proteins on a cell-by-cell basis enables the quantitative assessment of localization for comparative analysis. However, such membrane proteins typically lack continuity, and their intensity distributions are often very heterogeneous; moreover, nuclei can form large clump, which further impedes the quantification of membrane signals on a cell-by-cell basis. To tackle these problems, we introduce a three-step process to (i) regularize the membrane signal through iterative tangential voting, (ii) constrain the location of surface proteins by nuclear features, where clumps of nuclei are segmented through a delaunay triangulation approach, and (iii) assign membrane-bound macromolecules to individual cells through an application of multi-phase geodesic level-set. We have validated our method using both synthetic data and a dataset of 200 images, and are able to demonstrate the efficacy of our approach with superior performance.

Methods for nuclei detection, segmentation, and classification in digital histopathology: a review-current status and future potential

Digital pathology represents one of the major evolutions in modern medicine. Pathological examinations constitute the gold standard in many medical protocols, and also play a critical and legal role in the diagnosis process. In the conventional cancer diagnosis, pathologists analyze biopsies to make diagnostic and prognostic assessments, mainly based on the cell morphology and architecture distribution. Recently, computerized methods have been rapidly evolving in the area of digital pathology, with growing applications related to nuclei detection, segmentation, and classification. In cancer research, these approaches have played, and will continue to play a key (often bottleneck) role in minimizing human intervention, consolidating pertinent second opinions, and providing traceable clinical information. Pathological studies have been conducted for numerous cancer detection and grading applications, including brain, breast, cervix, lung, and prostate cancer grading. Our study presents, discusses, and extracts the major trends from an exhaustive overview of various nuclei detection, segmentation, feature computation, and classification techniques used in histopathology imagery, specifically in hematoxylin-eosin and immunohistochemical staining protocols. This study also enables us to measure the challenges that remain, in order to reach robust analysis of whole slide images, essential high content imaging with diagnostic biomarkers and prognosis support in digital pathology.

Prostate cancer grading: use of graph cut and spatial arrangement of nuclei

Tissue image grading is one of the most important steps in prostate cancer diagnosis, where the pathologist relies on the gland structure to assign a Gleason grade to the tissue image. In this grading scheme, the discrimination between grade 3 and grade 4 is the most difficult, and receives the most attention from researchers. In this study, we propose a novel method (called nuclei-based method) that 1) utilizes graph theory techniques to segment glands and 2) computes a gland-score (based on the spatial arrangement of nuclei) to estimate how similar a segmented region is to a gland. Next, we create a fusion method by combining this nuclei-based method with the lumen-based method presented in our previous work to improve the performance of grade 3 versus grade 4 classification problem (the accuracy is now improved to 87.3% compared to 81.1% of the lumen-based method alone). To segment glands, we build a graph of nuclei and lumina in the image, and use the normalized cut method to partition the graph into different components, each corresponding to a gland. Unlike most state-of-the-art lumen-based gland segmentation method, the nuclei-based method is able to segment glands without lumen or glands with multiple lumina. Moreover, another important contribution in this research is the development of a set of measures to exploit the difference in nuclei spatial arrangement between grade 3 images (where nuclei form closed chain structure on the gland boundary) and grade 4 image (where nuclei distribute more randomly in the gland). These measures are combined to generate a single gland-score value, which estimates how similar a segmented region (which is a set of nuclei and lumina) is to a gland.

An automated three-dimensional detection and segmentation method for touching cells by integrating concave points clustering and random walker algorithm

Characterizing cytoarchitecture is crucial for understanding brain functions and neural diseases. In neuroanatomy, it is an important task to accurately extract cell populations' centroids and contours. Recent advances have permitted imaging at single cell resolution for an entire mouse brain using the Nissl staining method. However, it is difficult to precisely segment numerous cells, especially those cells touching each other. As presented herein, we have developed an automated three-dimensional detection and segmentation method applied to the Nissl staining data, with the following two key steps: 1) concave points clustering to determine the seed points of touching cells; and 2) random walker segmentation to obtain cell contours. Also, we have evaluated the performance of our proposed method with several mouse brain datasets, which were captured with the micro-optical sectioning tomography imaging system, and the datasets include closely touching cells. Comparing with traditional detection and segmentation methods, our approach shows promising detection accuracy and high robustness.

Automatic Myonuclear Detection in Isolated Single Muscle Fibers Using Robust Ellipse Fitting and Sparse Representation

Accurate and robust detection of myonuclei in isolated single muscle fibers is required to calculate myonuclear domain size. However, this task is challenging because: 1) shape and size variations of the nuclei, 2) overlapping nuclear clumps, and 3) multiple z-stack images with out-of-focus regions. In this paper, we have proposed a novel automatic detection algorithm to robustly quantify myonuclei in isolated single skeletal muscle fibers. The original z-stack images are first converted into one all-in-focus image using multi-focus image fusion. A sufficient number of ellipse fitting hypotheses are then generated from the myonuclei contour segments using heteroscedastic errors-in-variables (HEIV) regression. A set of representative training samples and a set of discriminative features are selected by a two-stage sparse model. The selected samples with representative features are utilized to train a classifier to select the best candidates. A modified inner geodesic distance based mean-shift clustering algorithm is used to produce the final nuclei detection results. The proposed method was extensively tested using 42 sets of z-stack images containing over 1,500 myonuclei. The method demonstrates excellent results that are better than current state-of-the-art approaches.