Expectation-maximization-driven geodesic active contour with overlap resolution (EMaGACOR): application to lymphocyte segmentation on breast cancer histopathology

BreasTDLUSeg: A coarse-to-fine framework for segmentation of breast terminal duct lobular units on histopathological whole-slide images

Automatic segmentation of breast terminal duct lobular units (TDLUs) on histopathological whole-slide images (WSIs) is crucial for the quantitative evaluation of TDLUs in the diagnostic and prognostic analysis of breast cancer. However, TDLU segmentation remains a great challenge due to its highly heterogeneous sizes, structures, and morphologies as well as the small areas on WSIs. In this study, we propose BreasTDLUSeg, an efficient coarse-to-fine two-stage framework based on multi-scale attention to achieve localization and precise segmentation of TDLUs on hematoxylin and eosin (H&E)-stained WSIs. BreasTDLUSeg consists of two networks: a superpatch-based patch-level classification network (SPPC-Net) and a patch-based pixel-level segmentation network (PPS-Net). SPPC-Net takes a superpatch as input and adopts a sub-region classification head to classify each patch within the superpatch as TDLU positive or negative. PPS-Net takes the TDLU positive patches derived from SPPC-Net as input. PPS-Net deploys a multi-scale CNN-Transformer as an encoder to learn enhanced multi-scale morphological representations and an upsampler to generate pixel-wise segmentation masks for the TDLU positive patches. We also constructed two breast cancer TDLU datasets containing a total of 530 superpatch images with patch-level annotations and 2322 patch images with pixel-level annotations to enable the development of TDLU segmentation methods. Experiments on the two datasets demonstrate that BreasTDLUSeg outperforms other state-of-the-art methods with the highest Dice similarity coefficients of 79.97% and 92.93%, respectively. The proposed method shows great potential to assist pathologists in the pathological analysis of breast cancer. An open-source implementation of our approach can be found at https://github.com/Dian-kai/BreasTDLUSeg.

iIMPACT: integrating image and molecular profiles for spatial transcriptomics analysis

Current clustering analysis of spatial transcriptomics data primarily relies on molecular information and fails to fully exploit the morphological features present in histology images, leading to compromised accuracy and interpretability. To overcome these limitations, we have developed a multi-stage statistical method called iIMPACT. It identifies and defines histology-based spatial domains based on AI-reconstructed histology images and spatial context of gene expression measurements, and detects domain-specific differentially expressed genes. Through multiple case studies, we demonstrate iIMPACT outperforms existing methods in accuracy and interpretability and provides insights into the cellular spatial organization and landscape of functional genes within spatial transcriptomics data.

A Review of Nuclei Detection and Segmentation on Microscopy Images Using Deep Learning With Applications to Unbiased Stereology Counting

The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

DDTNet: A dense dual-task network for tumor-infiltrating lymphocyte detection and segmentation in histopathological images of breast cancer

The morphological evaluation of tumor-infiltrating lymphocytes (TILs) in hematoxylin and eosin (H& E)-stained histopathological images is the key to breast cancer (BCa) diagnosis, prognosis, and therapeutic response prediction. For now, the qualitative assessment of TILs is carried out by pathologists, and computer-aided automatic lymphocyte measurement is still a great challenge because of the small size and complex distribution of lymphocytes. In this paper, we propose a novel dense dual-task network (DDTNet) to simultaneously achieve automatic TIL detection and segmentation in histopathological images. DDTNet consists of a backbone network (i.e., feature pyramid network) for extracting multi-scale morphological characteristics of TILs, a detection module for the localization of TIL centers, and a segmentation module for the delineation of TIL boundaries, where a boundary-aware branch is further used to provide a shape prior to segmentation. An effective feature fusion strategy is utilized to introduce multi-scale features with lymphocyte location information from highly correlated branches for precise segmentation. Experiments on three independent lymphocyte datasets of BCa demonstrate that DDTNet outperforms other advanced methods in detection and segmentation metrics. As part of this work, we also propose a semi-automatic method (TILAnno) to generate high-quality boundary annotations for TILs in H& E-stained histopathological images. TILAnno is used to produce a new lymphocyte dataset that contains 5029 annotated lymphocyte boundaries, which have been released to facilitate computational histopathology in the future.

Whale Optimizer-Based Clustering for Breast Histopathology Image Segmentation

Breast histopathology image segmentation is a complex task due to indiscernibly correlated and noisy regions of interest. Breast histopathological images are composed of different types of cells. Some of these cells can be harmful for humans due to the presence of cancer. Under such circumstances, many segmentation techniques for automatic detection of cancer cells have been proposed considering clustering schemes. However, such clustering methodologies are sensitive to initial cluster centers, which promote false-positive solutions. This paper presents the use of the Whale Optimization Algorithm (WOA) for proper clustering segmentation of breast histopathological images to overcome clustering issues. Also, a rigorous comparative study is conducted among the proposed approach and several state-of-art Nature-Inspired Optimization Algorithms (NIOAs) and traditional clustering techniques. The numerical results indicate that the proposed approach outperforms the other utilized clustering methods in terms of precision, robustness, and quality of the segmented outputs.

Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review

Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.

Dynamic Learning Rate in Deep CNN Model for Metastasis Detection and Classification of Histopathology Images

Diagnosis of different breast cancer stages using histopathology whole slide images (WSI) is the gold standard in determining the grade of tissue metastasis. Computer-aided diagnosis (CAD) assists medical experts as a second opinion tool in early detection to prevent further proliferation. The field of pathology has advanced so rapidly that it is possible to obtain high-quality images from glass slides. Patches from the region of interest in histopathology images are extracted and trained using artificial neural network models. The trained model primarily analyzes and predicts the histology images for the benign or malignant class to which it belongs. Classification of medical images focuses on the training of models with layers of abstraction to distinguish between these two classes with less false-positive rates. The learning rate is the crucial hyperparameter used during the training of deep convolutional neural networks (DCNN) to improve model accuracy. This work emphasizes the relevance of the dynamic learning rate than the fixed learning rate during the training of networks. The dynamic learning rate varies with preset conditions between the lower and upper boundaries and repeats at different iterations. The performance of the model thus improves and attains comparatively high accuracy with fewer iterations.

System for quantitative evaluation of DAB&H-stained breast cancer biopsy digital images (CHISEL)

This study presents CHISEL (Computer-assisted Histopathological Image Segmentation and EvaLuation), an end-to-end system capable of quantitative evaluation of benign and malignant (breast cancer) digitized tissue samples with immunohistochemical nuclear staining of various intensity and diverse compactness. It stands out with the proposed seamless segmentation based on regions of interest cropping as well as the explicit step of nuclei cluster splitting followed by a boundary refinement. The system utilizes machine learning and recursive local processing to eliminate distorted (inaccurate) outlines. The method was validated using two labeled datasets which proved the relevance of the achieved results. The evaluation was based on the IISPV dataset of tissue from biopsy of breast cancer patients, with markers of T cells, along with Warwick Beta Cell Dataset of DAB&H-stained tissue from postmortem diabetes patients. Based on the comparison of the ground truth with the results of the detected and classified objects, we conclude that the proposed method can achieve better or similar results as the state-of-the-art methods. This system deals with the complex problem of nuclei quantification in digitalized images of immunohistochemically stained tissue sections, achieving best results for DAB&H-stained breast cancer tissue samples. Our method has been prepared with user-friendly graphical interface and was optimized to fully utilize the available computing power, while being accessible to users with fewer resources than needed by deep learning techniques.

Machine Learning Methods for Histopathological Image Analysis: A Review

Histopathological images (HIs) are the gold standard for evaluating some types of tumors for cancer diagnosis. The analysis of such images is time and resource-consuming and very challenging even for experienced pathologists, resulting in inter-observer and intra-observer disagreements. One of the ways of accelerating such an analysis is to use computer-aided diagnosis (CAD) systems. This paper presents a review on machine learning methods for histopathological image analysis, including shallow and deep learning methods. We also cover the most common tasks in HI analysis, such as segmentation and feature extraction. Besides, we present a list of publicly available and private datasets that have been used in HI research.

Vacuole Segmentation and Quantification in Liver Images of Wistar Rat

Accurate detection of macro and microvesicles in rat models of fatty liver disease is crucial in evaluating the progression of liver disease and identifying potential hepatotoxic findings during drug development. In this paper, we present a deep-learning-based framework for the segmentation of vacuoles in liver images of Wistar rat and study the correlation of automated quantification with expert pathologist's manual evaluation. To address the issue of misclassification of lumina (vascular and bile duct) as large vacuoles, we propose a selective tiling technique to generate tiles that include complete lumina and large vacuoles. A binary encoder-decoder convolution neural network is trained to detect individual vacuoles. We report a sensitivity of 85% and specificity of 98%. Furthermore, the diameter and roundness of the segmented vacuoles are estimated with an error of less than 8%, which supports the high potential of our method in drug development process.

Morphological Estimation of Cellularity on Neo-Adjuvant Treated Breast Cancer Histological Images

This paper describes a methodology that extracts key morphological features from histological breast cancer images in order to automatically assess Tumour Cellularity (TC) in Neo-Adjuvant treatment (NAT) patients. The response to NAT gives information on therapy efficacy and it is measured by the residual cancer burden index, which is composed of two metrics: TC and the assessment of lymph nodes. The data consist of whole slide images (WSIs) of breast tissue stained with Hematoxylin and Eosin (H&E) released in the 2019 SPIE Breast Challenge. The methodology proposed is based on traditional computer vision methods (K-means, watershed segmentation, Otsu’s binarisation, and morphological operations), implementing colour separation, segmentation, and feature extraction. Correlation between morphological features and the residual TC after a NAT treatment was examined. Linear regression and statistical methods were used and twenty-two key morphological parameters from the nuclei, epithelial region, and the full image were extracted. Subsequently, an automated TC assessment that was based on Machine Learning (ML) algorithms was implemented and trained with only selected key parameters. The methodology was validated with the score assigned by two pathologists through the intra-class correlation coefficient (ICC). The selection of key morphological parameters improved the results reported over other ML methodologies and it was very close to deep learning methodologies. These results are encouraging, as a traditionally-trained ML algorithm can be useful when limited training data are available preventing the use of deep learning approaches.

Nonlinear Image Registration and Pixel Classification Pipeline for the Study of Tumor Heterogeneity Maps

We present a novel method to assess the variations in protein expression and spatial heterogeneity of tumor biopsies with application in computational pathology. This was done using different antigen stains for each tissue section and proceeding with a complex image registration followed by a final step of color segmentation to detect the exact location of the proteins of interest. For proper assessment, the registration needs to be highly accurate for the careful study of the antigen patterns. However, accurate registration of histopathological images comes with three main problems: the high amount of artifacts due to the complex biopsy preparation, the size of the images, and the complexity of the local morphology. Our method manages to achieve an accurate registration of the tissue cuts and segmentation of the positive antigen areas.

A segmentation method combining probability map and boundary based on multiple fully convolutional networks and repetitive training

Cell nuclei image segmentation technology can help researchers observe each cell's stress response to drug treatment. However, it is still a challenge to accurately segment the adherent cell nuclei. At present, image segmentation based on a fully convolutional network (FCN) is attracting researchers' attention. We propose a multiple FCN architecture and repetitive training (M-FCN-RT) method to learn features of cell nucleus images. In M-FCN-RT, the multiple FCN (M-FCN) architecture is composed of several single FCNs (S-FCNs) with the same structure, and each FCN is used to learn the specific features of image datasets. In this paper, the M-FCN contains three FCNs; FCN_1-2, FCN₃ and FCN_B. FCN_1-2 and FCN₃ are respectively used to learn the spatial features of cell nuclei for generating probability maps to indicate nucleus regions of an image; FCN_B (boundary FCN) is used to learn the edge features of cell nuclei for generating the nucleus boundary. For the training of each FCN, we propose a repetitive training (RT) method to improve the classification accuracy of the model. To segment cell nuclei, we finally propose an algorithm combining the probability map and boundary (PMB) to segment the adherent nuclei. This paper uses a public opening nucleus image dataset to train, verify and evaluate the proposed M-FCN-RT and PMB methods. Our M-FCN-RT method achieves a high Dice similarity coefficient (DSC) of 92.11%, 95.64% and 87.99% on the three types of sub-datasets respectively for probability maps. In addition, segmentation experimental results show the PMB method is more effective and efficient compared with other methods.

Fast ScanNet: Fast and Dense Analysis of Multi-Gigapixel Whole-Slide Images for Cancer Metastasis Detection

Lymph node metastasis is one of the most important indicators in breast cancer diagnosis, that is traditionally observed under the microscope by pathologists. In recent years, with the dramatic advance of high-throughput scanning and deep learning technology, automatic analysis of histology from whole-slide images has received a wealth of interest in the field of medical image computing, which aims to alleviate pathologists' workload and simultaneously reduce misdiagnosis rate. However, the automatic detection of lymph node metastases from whole-slide images remains a key challenge because such images are typically very large, where they can often be multiple gigabytes in size. Also, the presence of hard mimics may result in a large number of false positives. In this paper, we propose a novel method with anchor layers for model conversion, which not only leverages the efficiency of fully convolutional architectures to meet the speed requirement in clinical practice but also densely scans the whole-slide image to achieve accurate predictions on both micro- and macro-metastases. Incorporating the strategies of asynchronous sample prefetching and hard negative mining, the network can be effectively trained. The efficacy of our method is corroborated on the benchmark dataset of 2016 Camelyon Grand Challenge. Our method achieved significant improvements in comparison with the state-of-the-art methods on tumor localization accuracy with a much faster speed and even surpassed human performance on both challenge tasks.

Multi-Path Dilated Residual Network for Nuclei Segmentation and Detection

As a typical biomedical detection task, nuclei detection has been widely used in human health management, disease diagnosis and other fields. However, the task of cell detection in microscopic images is still challenging because the nuclei are commonly small and dense with many overlapping nuclei in the images. In order to detect nuclei, the most important key step is to segment the cell targets accurately. Based on Mask RCNN model, we designed a multi-path dilated residual network, and realized a network structure to segment and detect dense small objects, and effectively solved the problem of information loss of small objects in deep neural network. The experimental results on two typical nuclear segmentation data sets show that our model has better recognition and segmentation capability for dense small targets.

Qualitative and Quantitative Assessment of Technical Skills in Percutaneous Coronary Intervention: In Vivo Porcine Studies

OBJECTIVE

Technical skill assessment plays an important role in the professional development of an interventionalist in percutaneous coronary intervention (PCI). However, most of the traditional assessment methods are time consuming and subjective. This paper aims to develop objective assessment techniques.

METHODS

In this study, a natural-behavior-based assessment framework is proposed to qualitatively and quantitatively assess technical skills in PCI. In vivo porcine studies were conducted to deliver a medical guidewire to two target coronaries of left circumflex arteries by six novice and four expert interventionalists. Simultaneously, four types of natural behaviors (i.e., hand motion, proximal force, muscle activity, and finger motion) were acquired from the subjects' dominant hand and arm. The features extracted from the behaviors of different skill-level groups were compared using the Mann-Whitney U-test for effective behavior selection. The effective ones were further applied in the Gaussian-mixture-model-based qualitative assessment and Mahalanobis-distance-based quantitative assessment.

RESULTS

The qualitative assessment achieves an accuracy of 92% to distinguish the novice and expert attempts, which is significantly higher than that of using single guidewire motions. Furthermore, the quantitative assessment can assign objective and effective scores for all attempts, indicating high correlation ( R = 0.9225) to those obtained by traditional methods.

CONCLUSION

The objective, effective, and comprehensive assessment of technical skills can be provided by qualitatively and quantitatively analyzing interventionalists' natural behaviors in PCI.

SIGNIFICANCE

This paper suggests a novel approach for the technical skill assessment and the promising results demonstrate the great importance and effectiveness of the proposed method for promoting the development of objective assessment techniques.

Introduction to Digital Image Analysis in Whole-slide Imaging: A White Paper from the Digital Pathology Association

The advent of whole-slide imaging in digital pathology has brought about the advancement of computer-aided examination of tissue via digital image analysis. Digitized slides can now be easily annotated and analyzed via a variety of algorithms. This study reviews the fundamentals of tissue image analysis and aims to provide pathologists with basic information regarding the features, applications, and general workflow of these new tools. The review gives an overview of the basic categories of software solutions available, potential analysis strategies, technical considerations, and general algorithm readouts. Advantages and limitations of tissue image analysis are discussed, and emerging concepts, such as artificial intelligence and machine learning, are introduced. Finally, examples of how digital image analysis tools are currently being used in diagnostic laboratories, translational research, and drug development are discussed.

Spatial Architecture and Arrangement of Tumor-Infiltrating Lymphocytes for Predicting Likelihood of Recurrence in Early-Stage Non-Small Cell Lung Cancer

PURPOSE

The presence of a high degree of tumor-infiltrating lymphocytes (TIL) has been proven to be associated with outcome in patients with non-small cell lung cancer (NSCLC). However, recent evidence indicates that tissue architecture is also prognostic of disease-specific survival and recurrence. We show a set of descriptors (spatial TIL, SpaTIL) that capture density, and spatial colocalization of TILs and tumor cells across digital images that can predict likelihood of recurrence in early-stage NSCLC.

EXPERIMENTAL DESIGN

The association between recurrence in early-stage NSCLC and SpaTIL features was explored on 301 patients across four different cohorts. Cohort D1 (n = 70) was used to identify the most prognostic SpaTIL features and to train a classifier to predict the likelihood of recurrence. The classifier performance was evaluated in cohorts D2 (n = 119), D3 (n = 112), and D4 (n = 112). Two pathologists graded each sample of D1 and D2; intraobserver agreement and association between manual grading and likelihood of recurrence were analyzed.

RESULTS

SpaTIL was associated with likelihood of recurrence in all test sets (log-rank P < 0.02). A multivariate Cox proportional hazards analysis revealed an HR of 3.08 (95% confidence interval, 2.1-4.5, P = 7.3 × 10-5). In contrast, agreement among expert pathologists using tumor grade was moderate (Kappa = 0.5), and the manual TIL grading was only prognostic for one reader in D2 (P = 8.0 × 10-3).

CONCLUSIONS

A set of features related to density and spatial architecture of TILs was found to be associated with a likelihood of recurrence of early-stage NSCLC. This information could potentially be used for helping in treatment planning and management of early-stage NSCLC.See related commentary by Peled et al., p. 1449.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Computer-Aided Laser Dissection: A Microdissection Workflow Leveraging Image Analysis Tools

Introduction

The development and application of new molecular diagnostic assays based on next-generation sequencing and proteomics require improved methodologies for procurement of target cells from histological sections. Laser microdissection can successfully isolate distinct cells from tissue specimens based on visual selection for many research and clinical applications. However, this can be a daunting task when a large number of cells are required for molecular analysis or when a sizeable number of specimens need to be evaluated.

Materials and Methods

To improve the efficiency of the cellular identification process, we describe a microdissection workflow that leverages recently developed and open source image analysis algorithms referred to as computer-aided laser dissection (CALD). CALD permits a computer algorithm to identify the cells of interest and drive the dissection process.

Results

We describe several "use cases" that demonstrate the integration of image analytic tools probabilistic pairwise Markov model, ImageJ, spatially invariant vector quantization (SIVQ), and eSeg onto the ThermoFisher Scientific ArcturusXT and Leica LMD7000 microdissection platforms.

Conclusions

The CALD methodology demonstrates the integration of image analysis tools with the microdissection workflow and shows the potential impact to clinical and life science applications.

Contour-Seed Pairs Learning-Based Framework for Simultaneously Detecting and Segmenting Various Overlapping Cells/Nuclei in Microscopy Images

In this paper, we propose a novel contour-seed pairs learning-based framework for robust and automated cell/nucleus segmentation. Automated granular object segmentation in microscopy images has significant clinical importance for pathology grading of the cell carcinoma and gene expression. The focus of the past literature is dominated by either segmenting a certain type of cells/nuclei or simply splitting the clustered objects without contours inference of them. Our method addresses these issues by formulating the detection and segmentation tasks in terms of a unified regression problem, where a cascade sparse regression chain model is trained and then applied to return object locations and entire boundaries of clustered objects. In particular, we first learn a set of online convolutional features in each layer. Then, in the proposed cascade sparse regression chain, with the input from the learned features, we iteratively update the locations and clustered object boundaries until convergence. In this way, the boundary evidences of each individual object can be easily delineated and be further fed to a complete contour inference procedure optimized by the minimum description length principle. For any probe image, our method enables to analyze free-lying and overlapping cells with complex shapes. Experimental results show that the proposed method is very generic and performs well on contour inferences of various cell/nucleus types. Compared with the current segmentation techniques, our approach achieves state-of-the-art performances on four challenging datasets, i.e., the kidney renal cell carcinoma histopathology dataset, Drosophila Kc167 cellular dataset, differential interference contrast red blood cell dataset, and cervical cytology dataset.

Predicting Invasive Disease-Free Survival for Early-stage Breast Cancer Patients Using Follow-up Clinical Data

OBJECTIVE

Chinese women are seriously threatened by breast cancer with high morbidity and mortality. The lack of robust prognosis models results in difficulty for doctors to prepare an appropriate treatment plan that may prolong patient survival time. An alternative prognosis model framework to predict Invasive Disease-Free Survival (iDFS) for early-stage breast cancer patients, called MP4Ei, is proposed. MP4Ei framework gives an excellent performance to predict the relapse or metastasis breast cancer of Chinese patients in 5 years.

METHODS

MP4Ei is built based on statistical theory and gradient boosting decision tree framework. 5246 patients, derived from the Clinical Research Center for Breast (CRCB) in West China Hospital of Sichuan University, with early-stage (stage I-III) breast cancer are eligible for inclusion. Stratified feature selection, including statistical and ensemble methods, is adopted to select 23 out of the 89 patient features about the patient' demographics, diagnosis, pathology and therapy. Then 23 selected features as the input variables are imported into the XGBoost algorithm, with Bayesian parameter tuning and cross validation, to find out the optimum simplified model for 5-year iDFS prediction.

RESULTS

For eligible data, with 4196 patients (80%) for training, and with 1050 patients (20%) for testing, MP4Ei achieves comparable accuracy with AUC 0.8451, which has a significant advantage (p < 0.05).

CONCLUSION

This work demonstrates the complete iDFS prognosis model with very competitive performance.

SIGNIFICANCE

The proposed method in this paper could be used in clinical practice to predict patients' prognosis and future surviving state, which may help doctors make treatment plan.

Advances in the computational and molecular understanding of the prostate cancer cell nucleus

Nuclear alterations are a hallmark of many types of cancers, including prostate cancer (PCa). Recent evidence shows that subvisual changes, ones that may not be visually perceptible to a pathologist, to the nucleus and its ultrastructural components can precede visual histopathological recognition of cancer. Alterations to nuclear features, such as nuclear size and shape, texture, and spatial architecture, reflect the complex molecular-level changes that occur during oncogenesis. Quantitative nuclear morphometry, a field that uses computational approaches to identify and quantify malignancy-induced nuclear changes, can enable a detailed and objective analysis of the PCa cell nucleus. Recent advances in machine learning-based approaches can now automatically mine data related to these changes to aid in the diagnosis, decision making, and prediction of PCa prognoses. In this review, we use PCa as a case study to connect the molecular-level mechanisms that underlie these nuclear changes to the machine learning computational approaches, bridging the gap between the clinical and computational understanding of PCa. First, we will discuss recent developments to our understanding of the molecular events that drive nuclear alterations in the context of PCa: the role of the nuclear matrix and lamina in size and shape changes, the role of 3-dimensional chromatin organization and epigenetic modifications in textural changes, and the role of the tumor microenvironment in altering nuclear spatial topology. We will then discuss the advances in the applications of machine learning algorithms to automatically segment nuclei in prostate histopathological images, extract nuclear features to aid in diagnostic decision making, and predict potential outcomes, such as biochemical recurrence and survival. Finally, we will discuss the challenges and opportunities associated with translation of the quantitative nuclear morphometry methodology into the clinical space. Ultimately, accurate identification and quantification of nuclear alterations can contribute to the field of nucleomics and has applications for computationally driven precision oncologic patient care.

Tumor Cell Load and Heterogeneity Estimation From Diffusion-Weighted MRI Calibrated With Histological Data: an Example From Lung Cancer

Diffusion-weighted magnetic resonance imaging (DWI) is a key non-invasive imaging technique for cancer diagnosis and tumor treatment assessment, reflecting Brownian movement of water molecules in tissues. Since densely packed cells restrict molecule mobility, tumor tissues produce usually higher signal (a.k.a. less attenuated signal) on isotropic maps compared with normal tissues. However, no general quantitative relation between DWI data and the cell density has been established. In order to link low-resolution clinical cross-sectional data with high-resolution histological information, we developed an image processing and analysis chain, which was used to study the correlation between the diffusion coefficient (D value) estimated from DWI and tumor cellularity from serial histological slides of a resected non-small cell lung cancer tumor. Color deconvolution followed by cell nuclei segmentation was performed on digitized histological images to determine local and cell-type specific 2d (two-dimensional) densities. From these, the 3d cell density was inferred by a model-based sampling technique, which is necessary for the calculation of local and global 3d tumor cell count. Next, DWI sequence information was overlaid with high-resolution CT data and the resected histology using prominent anatomical hallmarks for co-registration of histology tissue blocks and non-invasive imaging modalities' data. The integration of cell numbers information and DWI data derived from different tumor areas revealed a clear negative correlation between cell density and D value. Importantly, spatial tumor cell density can be calculated based on DWI data. In summary, our results demonstrate that tumor cell count and heterogeneity can be predicted from DWI data, which may open new opportunities for personalized diagnosis and therapy optimization.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.

Training a cell-level classifier for detecting basal-cell carcinoma by combining human visual attention maps with low-level handcrafted features

Computational histomorphometric approaches typically use low-level image features for building machine learning classifiers. However, these approaches usually ignore high-level expert knowledge. A computational model (M_im) combines low-, mid-, and high-level image information to predict the likelihood of cancer in whole slide images. Handcrafted low- and mid-level features are computed from area, color, and spatial nuclei distributions. High-level information is implicitly captured from the recorded navigations of pathologists while exploring whole slide images during diagnostic tasks. This model was validated by predicting the presence of cancer in a set of unseen fields of view. The available database was composed of 24 cases of basal-cell carcinoma, from which 17 served to estimate the model parameters and the remaining 7 comprised the evaluation set. A total of 274 fields of view of size [Formula: see text] were extracted from the evaluation set. Then 176 patches from this set were used to train a support vector machine classifier to predict the presence of cancer on a patch-by-patch basis while the remaining 98 image patches were used for independent testing, ensuring that the training and test sets do not comprise patches from the same patient. A baseline model (M_ex) estimated the cancer likelihood for each of the image patches. M_ex uses the same visual features as M_im, but its weights are estimated from nuclei manually labeled as cancerous or noncancerous by a pathologist. M_im achieved an accuracy of 74.49% and an [Formula: see text]-measure of 80.31%, while M_ex yielded corresponding accuracy and F-measures of 73.47% and 77.97%, respectively.

Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: A review

With the advance of digital pathology, image analysis has begun to show its advantages in information analysis of hematoxylin and eosin histopathology images. Generally, histological features in hematoxylin and eosin images are measured to evaluate tumor grade and prognosis for breast cancer. This review summarized recent works in image analysis of hematoxylin and eosin histopathology images for breast cancer prognosis. First, prognostic factors for breast cancer based on hematoxylin and eosin histopathology images were summarized. Then, usual procedures of image analysis for breast cancer prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Finally, the prognostic value of image features and image feature–based prognostic models was evaluated. Moreover, we discussed the issues of current analysis, and some directions for future research.

Connecting Markov random fields and active contour models: application to gland segmentation and classification

We introduce a Markov random field (MRF)-driven region-based active contour model (MaRACel) for histological image segmentation. This Bayesian segmentation method combines a region-based active contour (RAC) with an MRF. State-of-the-art RAC models assume that every spatial location in the image is statistically independent, thereby ignoring valuable contextual information among spatial locations. To address this shortcoming, we incorporate an MRF prior into energy term of the RAC. This requires a formulation of the Markov prior consistent with the continuous variational framework characteristic of active contours; consequently, we introduce a continuous analog to the discrete Potts model. Based on the automated segmentation boundary of glands by MaRACel model, explicit shape descriptors are then employed to distinguish prostate glands belonging to Gleason patterns 3 (G3) and 4 (G4). To demonstrate the effectiveness of MaRACel, we compare its performance to the popular models proposed by Chan and Vese (CV) and Rousson and Deriche (RD) with respect to the following tasks: (1) the segmentation of prostatic acini (glands) and (2) the differentiation of G3 and G4 glands. On almost 600 prostate biopsy needle images, MaRACel was shown to have higher average dice coefficients, overlap ratios, sensitivities, specificities, and positive predictive values both in terms of segmentation accuracy and ability to discriminate between G3 and G4 glands compared to the CV and RD models.

Texture Detection for Letter Carving Segmentation of Ancient Copper Inscriptions

As relics of history, ancient copper inscriptions are found in many countries. Information in the image or letter forms contained on copper ancient inscription has a very high value. The age and environmental factors caused damage to the surface of the inscription and also reduced the appearances of the image and letter. In this paper, we describe a novel segmentation methodology based on multi-texture features for ancient copper inscriptions which were severely damaged. The segmentation results of letters on ancient copper inscriptions by using the proposed method have an average accuracy of 90%. Based on these results, the proposed method is suitable for letter segmentation of the ancient copper inscriptions.

Computer-based image analysis in breast pathology

Whole slide imaging (WSI) has the potential to be utilized in telepathology, teleconsultation, quality assurance, clinical education, and digital image analysis to aid pathologists. In this paper, the potential added benefits of computer-assisted image analysis in breast pathology are reviewed and discussed. One of the major advantages of WSI systems is the possibility of doing computer-based image analysis on the digital slides. The purpose of computer-assisted analysis of breast virtual slides can be (i) segmentation of desired regions or objects such as diagnostically relevant areas, epithelial nuclei, lymphocyte cells, tubules, and mitotic figures, (ii) classification of breast slides based on breast cancer (BCa) grades, the invasive potential of tumors, or cancer subtypes, (iii) prognosis of BCa, or (iv) immunohistochemical quantification. While encouraging results have been achieved in this area, further progress is still required to make computer-based image analysis of breast virtual slides acceptable for clinical practice.

Multi-Pass Adaptive Voting for Nuclei Detection in Histopathological Images

Nuclei detection is often a critical initial step in the development of computer aided diagnosis and prognosis schemes in the context of digital pathology images. While over the last few years, a number of nuclei detection methods have been proposed, most of these approaches make idealistic assumptions about the staining quality of the tissue. In this paper, we present a new Multi-Pass Adaptive Voting (MPAV) for nuclei detection which is specifically geared towards images with poor quality staining and noise on account of tissue preparation artifacts. The MPAV utilizes the symmetric property of nuclear boundary and adaptively selects gradient from edge fragments to perform voting for a potential nucleus location. The MPAV was evaluated in three cohorts with different staining methods: Hematoxylin &Eosin, CD31 &Hematoxylin, and Ki-67 and where most of the nuclei were unevenly and imprecisely stained. Across a total of 47 images and nearly 17,700 manually labeled nuclei serving as the ground truth, MPAV was able to achieve a superior performance, with an area under the precision-recall curve (AUC) of 0.73. Additionally, MPAV also outperformed three state-of-the-art nuclei detection methods, a single pass voting method, a multi-pass voting method, and a deep learning based method.

Automated Segmentation of Nuclei in Breast Cancer Histopathology Images

The process of Nuclei detection in high-grade breast cancer images is quite challenging in the case of image processing techniques due to certain heterogeneous characteristics of cancer nuclei such as enlarged and irregularly shaped nuclei, highly coarse chromatin marginalized to the nuclei periphery and visible nucleoli. Recent reviews state that existing techniques show appreciable segmentation accuracy on breast histopathology images whose nuclei are dispersed and regular in texture and shape; however, typical cancer nuclei are often clustered and have irregular texture and shape properties. This paper proposes a novel segmentation algorithm for detecting individual nuclei from Hematoxylin and Eosin (H&E) stained breast histopathology images. This detection framework estimates a nuclei saliency map using tensor voting followed by boundary extraction of the nuclei on the saliency map using a Loopy Back Propagation (LBP) algorithm on a Markov Random Field (MRF). The method was tested on both whole-slide images and frames of breast cancer histopathology images. Experimental results demonstrate high segmentation performance with efficient precision, recall and dice-coefficient rates, upon testing high-grade breast cancer images containing several thousand nuclei. In addition to the optimal performance on the highly complex images presented in this paper, this method also gave appreciable results in comparison with two recently published methods-Wienert et al. (2012) and Veta et al. (2013), which were tested using their own datasets.

When machine vision meets histology: A comparative evaluation of model architecture for classification of histology sections

Classification of histology sections in large cohorts, in terms of distinct regions of microanatomy (e.g., stromal) and histopathology (e.g., tumor, necrosis), enables the quantification of tumor composition, and the construction of predictive models of genomics and clinical outcome. To tackle the large technical variations and biological heterogeneities, which are intrinsic in large cohorts, emerging systems utilize either prior knowledge from pathologists or unsupervised feature learning for invariant representation of the underlying properties in the data. However, to a large degree, the architecture for tissue histology classification remains unexplored and requires urgent systematical investigation. This paper is the first attempt to provide insights into three fundamental questions in tissue histology classification: I. Is unsupervised feature learning preferable to human engineered features? II. Does cellular saliency help? III. Does the sparse feature encoder contribute to recognition? We show that (a) in I, both Cellular Morphometric Feature and features from unsupervised feature learning lead to superior performance when compared to SIFT and [Color, Texture]; (b) in II, cellular saliency incorporation impairs the performance for systems built upon pixel-/patch-level features; and (c) in III, the effect of the sparse feature encoder is correlated with the robustness of features, and the performance can be consistently improved by the multi-stage extension of systems built upon both Cellular Morphmetric Feature and features from unsupervised feature learning. These insights are validated with two cohorts of Glioblastoma Multiforme (GBM) and Kidney Clear Cell Carcinoma (KIRC).

Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples

BACKGROUND

Immune cell infiltration in tumor is an emerging prognostic biomarker in breast cancer. The gold standard for quantification of immune cells in tissue sections is visual assessment through a microscope, which is subjective and semi-quantitative. In this study, we propose and evaluate an approach based on antibody-guided annotation and deep learning to quantify immune cell-rich areas in hematoxylin and eosin (H&E) stained samples.

METHODS

Consecutive sections of formalin-fixed parafin-embedded samples obtained from the primary tumor of twenty breast cancer patients were cut and stained with H&E and the pan-leukocyte CD45 antibody. The stained slides were digitally scanned, and a training set of immune cell-rich and cell-poor tissue regions was annotated in H&E whole-slide images using the CD45-expression as a guide. In analysis, the images were divided into small homogenous regions, superpixels, from which features were extracted using a pretrained convolutional neural network (CNN) and classified with a support of vector machine. The CNN approach was compared to texture-based classification and to visual assessments performed by two pathologists.

RESULTS

In a set of 123,442 labeled superpixels, the CNN approach achieved an F-score of 0.94 (range: 0.92-0.94) in discrimination of immune cell-rich and cell-poor regions, as compared to an F-score of 0.88 (range: 0.87-0.89) obtained with the texture-based classification. When compared to visual assessment of 200 images, an agreement of 90% (κ = 0.79) to quantify immune infiltration with the CNN approach was achieved while the inter-observer agreement between pathologists was 90% (κ = 0.78).

CONCLUSIONS

Our findings indicate that deep learning can be applied to quantify immune cell infiltration in breast cancer samples using a basic morphology staining only. A good discrimination of immune cell-rich areas was achieved, well in concordance with both leukocyte antigen expression and pathologists' visual assessment.

Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases

BACKGROUND

Deep learning (DL) is a representation learning approach ideally suited for image analysis challenges in digital pathology (DP). The variety of image analysis tasks in the context of DP includes detection and counting (e.g., mitotic events), segmentation (e.g., nuclei), and tissue classification (e.g., cancerous vs. non-cancerous). Unfortunately, issues with slide preparation, variations in staining and scanning across sites, and vendor platforms, as well as biological variance, such as the presentation of different grades of disease, make these image analysis tasks particularly challenging. Traditional approaches, wherein domain-specific cues are manually identified and developed into task-specific "handcrafted" features, can require extensive tuning to accommodate these variances. However, DL takes a more domain agnostic approach combining both feature discovery and implementation to maximally discriminate between the classes of interest. While DL approaches have performed well in a few DP related image analysis tasks, such as detection and tissue classification, the currently available open source tools and tutorials do not provide guidance on challenges such as (a) selecting appropriate magnification, (b) managing errors in annotations in the training (or learning) dataset, and (c) identifying a suitable training set containing information rich exemplars. These foundational concepts, which are needed to successfully translate the DL paradigm to DP tasks, are non-trivial for (i) DL experts with minimal digital histology experience, and (ii) DP and image processing experts with minimal DL experience, to derive on their own, thus meriting a dedicated tutorial.

AIMS

This paper investigates these concepts through seven unique DP tasks as use cases to elucidate techniques needed to produce comparable, and in many cases, superior to results from the state-of-the-art hand-crafted feature-based classification approaches.

RESULTS

Specifically, in this tutorial on DL for DP image analysis, we show how an open source framework (Caffe), with a singular network architecture, can be used to address: (a) nuclei segmentation (F-score of 0.83 across 12,000 nuclei), (b) epithelium segmentation (F-score of 0.84 across 1735 regions), (c) tubule segmentation (F-score of 0.83 from 795 tubules), (d) lymphocyte detection (F-score of 0.90 across 3064 lymphocytes), (e) mitosis detection (F-score of 0.53 across 550 mitotic events), (f) invasive ductal carcinoma detection (F-score of 0.7648 on 50 k testing patches), and (g) lymphoma classification (classification accuracy of 0.97 across 374 images).

CONCLUSION

This paper represents the largest comprehensive study of DL approaches in DP to date, with over 1200 DP images used during evaluation. The supplemental online material that accompanies this paper consists of step-by-step instructions for the usage of the supplied source code, trained models, and input data.

PHENOTYPIC CHARACTERIZATION OF BREAST INVASIVE CARCINOMA VIA TRANSFERABLE TISSUE MORPHOMETRIC PATTERNS LEARNED FROM GLIOBLASTOMA MULTIFORME

Quantitative analysis of whole slide images (WSIs) in a large cohort may provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. Although unsupervised feature learning provides a promising way in learning pertinent features without human intervention, its capability can be greatly limited due to the lack of well-curated examples. In this paper, we explored the transferability of knowledge acquired from a well-curated Glioblastoma Multiforme (GBM) dataset through its application to the representation and characterization of tissue histology from the Cancer Genome Atlas (TCGA) Breast Invasive Carcinoma (BRCA) cohort. Our experimental results reveals two major phenotypic subtypes with statistically significantly different survival curves. Further differential expression analysis of these two subtypes indicates enrichment of genes regulated by NF-kB in response to TNF and genes up-regulated in response to IFNG.

Image analysis and machine learning in digital pathology: Challenges and opportunities

With the rise in whole slide scanner technology, large numbers of tissue slides are being scanned and represented and archived digitally. While digital pathology has substantial implications for telepathology, second opinions, and education there are also huge research opportunities in image computing with this new source of "big data". It is well known that there is fundamental prognostic data embedded in pathology images. The ability to mine "sub-visual" image features from digital pathology slide images, features that may not be visually discernible by a pathologist, offers the opportunity for better quantitative modeling of disease appearance and hence possibly improved prediction of disease aggressiveness and patient outcome. However the compelling opportunities in precision medicine offered by big digital pathology data come with their own set of computational challenges. Image analysis and computer assisted detection and diagnosis tools previously developed in the context of radiographic images are woefully inadequate to deal with the data density in high resolution digitized whole slide images. Additionally there has been recent substantial interest in combining and fusing radiologic imaging and proteomics and genomics based measurements with features extracted from digital pathology images for better prognostic prediction of disease aggressiveness and patient outcome. Again there is a paucity of powerful tools for combining disease specific features that manifest across multiple different length scales. The purpose of this review is to discuss developments in computational image analysis tools for predictive modeling of digital pathology images from a detection, segmentation, feature extraction, and tissue classification perspective. We discuss the emergence of new handcrafted feature approaches for improved predictive modeling of tissue appearance and also review the emergence of deep learning schemes for both object detection and tissue classification. We also briefly review some of the state of the art in fusion of radiology and pathology images and also combining digital pathology derived image measurements with molecular "omics" features for better predictive modeling. The review ends with a brief discussion of some of the technical and computational challenges to be overcome and reflects on future opportunities for the quantitation of histopathology.

Stain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology

Digital histopathology slides have many sources of variance, and while pathologists typically do not struggle with them, computer aided diagnostic algorithms can perform erratically. This manuscript presents Stain Normalization using Sparse AutoEncoders (StaNoSA) for use in standardizing the color distributions of a test image to that of a single template image. We show how sparse autoencoders can be leveraged to partition images into tissue sub-types, so that color standardization for each can be performed independently. StaNoSA was validated on three experiments and compared against five other color standardization approaches and shown to have either comparable or superior results.

A resolution adaptive deep hierarchical (RADHicaL) learning scheme applied to nuclear segmentation of digital pathology images

Deep learning (DL) has recently been successfully applied to a number of image analysis problems. However, DL approaches tend to be inefficient for segmentation on large image data, such as high-resolution digital pathology slide images. For example, typical breast biopsy images scanned at 40× magnification contain billions of pixels, of which usually only a small percentage belong to the class of interest. For a typical naïve deep learning scheme, parsing through and interrogating all the image pixels would represent hundreds if not thousands of hours of compute time using high performance computing environments. In this paper, we present a resolution adaptive deep hierarchical (RADHicaL) learning scheme wherein DL networks at lower resolutions are leveraged to determine if higher levels of magnification, and thus computation, are necessary to provide precise results. We evaluate our approach on a nuclear segmentation task with a cohort of 141 ER+ breast cancer images and show we can reduce computation time on average by about 85%. Expert annotations of 12,000 nuclei across these 141 images were employed for quantitative evaluation of RADHicaL. A head-to-head comparison with a naïve DL approach, operating solely at the highest magnification, yielded the following performance metrics: .9407 vs .9854 Detection Rate, .8218 vs .8489 F-score, .8061 vs .8364 true positive rate and .8822 vs 0.8932 positive predictive value. Our performance indices compare favourably with state of the art nuclear segmentation approaches for digital pathology images.