A Complete Color Normalization Approach to Histopathology Images Using Color Cues Computed From Saturation-Weighted Statistics

A High-Resolution Digital Pathological Image Staining Style Transfer Model Based on Gradient Guidance

Digital pathology images have long been regarded as the gold standard for cancer diagnosis in clinical medicine. A highly generalized digital pathological image diagnosis system can provide strong support for cancer diagnosis, help to improve the diagnostic efficiency and accuracy of doctors, and has important research value. The whole slide image of different centers can lead to very large staining differences due to different scanners and dyes, which pose a challenge to the generalization performance of the model application in multi-center data testing. In order to achieve the normalization of multi-center data, this paper proposes a style transfer algorithm based on an adversarial generative network for high-resolution images. The gradient-guided dye migration model proposed in this paper introduces a gradient-enhanced regularized term in the loss function design of the algorithm. A style transfer algorithm was applied to the source data, and the diagnostic performance of the multi-example learning model based on the domain data was significantly improved by validation in the pathological image datasets of two centers. The proposed method improved the AUC of the best classification model from 0.8856 to 0.9243, and another set of experiments improved the AUC from 0.8012 to 0.8313.

Bridging Histopathology and Radiomics Toward Prognosis of Metastasis in Early Breast Cancer

Tumor histomorphology is crucial for the prognostication of breast cancer outcomes because it contains histological, cellular, and molecular tumor heterogeneity related to metastatic potential. To enhance breast cancer prognosis, we aimed to apply radiomics analysis-traditionally used in 3D scans-to 2D histopathology slides. This study tested radiomics analysis in a cohort of 92 breast tumor specimens for outcome prognosis, addressing -omics dimensionality by comparing models with moderate and high feature counts, using least absolute shrinkage and selection operator for feature selection and machine learning for prognostic modeling. In the test folds, models with radiomics features [area under the curves (AUCs) range 0.799-0.823] significantly outperformed the benchmark model, which only included clinicopathological (CP) parameters (AUC = 0.584). The moderate-dimensionality model with 11 CP + 93 radiomics features matched the performance of the highly dimensional models with 1,208 radiomics or 11 CP + 1,208 radiomics features, showing average AUCs of 0.823, 0.799, and 0.807 and accuracies of 79.8, 79.3, and 76.6%, respectively. In conclusion, our application of deep texture radiomics analysis to 2D histopathology showed strong prognostic performance with a moderate-dimensionality model, surpassing a benchmark based on standard CP parameters, indicating that this deep texture histomics approach could potentially become a valuable prognostic tool.

Review of computer-assisted diagnosis model to classify follicular lymphoma histology

The field of image processing is experiencing significant advancements to support professionals in analyzing histological images obtained from biopsies. The primary objective is to enhance the process of diagnosis and prognostic evaluations. Various forms of cancer can be diagnosed by employing different segmentation techniques followed by postprocessing approaches that can identify distinct neoplastic areas. Using computer approaches facilitates a more objective and efficient study of experts. The progressive advancement of histological image analysis holds significant importance in modern medicine. This paper provides an overview of the current advances in segmentation and classification approaches for images of follicular lymphoma. This research analyzes the primary image processing techniques utilized in the various stages of preprocessing, segmentation of the region of interest, classification, and postprocessing as described in the existing literature. The study also examines the strengths and weaknesses associated with these approaches. Additionally, this study encompasses an examination of validation procedures and an exploration of prospective future research roads in the segmentation of neoplasias.

RandStainNA++: Enhance Random Stain Augmentation and Normalization Through Foreground and Background Differentiation

The wide prevalence of staining variations in digital pathology presents a significant obstacle, often undermining the effectiveness of diagnosis and analysis. The current strategies to counteract this issue primarily revolve around Stain Normalization (SN) and Stain Augmentation (SA). Nonetheless, these methodologies come with inherent limitations. They struggle to adapt to the vast array of staining styles, tend to presuppose linear associations between color spaces, and often lead to unrealistic color transformations. In response to these challenges, we introduce RandStainNA++, a novel method seamlessly integrating SN and SA. This method exploits the versatility of random SN and SA within randomly selected color spaces, effectively managing variations for the foreground and background independently. By refining the transformations of staining styles for the foreground and background within a realistic scope, this strategy promotes the generation of more practical staining transformations during the training phase. Further enhancing our approach, we propose a unique self-distillation method. This technique incorporates prior knowledge of stain variation, substantially augmenting the generalization capability of the network. The striking results yield that, compared to conventional classification models, our method boosts performance by a significant margin of 16-25%. Furthermore, when juxtaposed with baseline segmentation models, the Dice score registers an increase of 0.06.

A Robust Method for the Unsupervised Scoring of Immunohistochemical Staining

Immunohistochemistry is a powerful technique that is widely used in biomedical research and clinics; it allows one to determine the expression levels of some proteins of interest in tissue samples using color intensity due to the expression of biomarkers with specific antibodies. As such, immunohistochemical images are complex and their features are difficult to quantify. Recently, we proposed a novel method, including a first separation stage based on non-negative matrix factorization (NMF), that achieved good results. However, this method was highly dependent on the parameters that control sparseness and non-negativity, as well as on algorithm initialization. Furthermore, the previously proposed method required a reference image as a starting point for the NMF algorithm. In the present work, we propose a new, simpler and more robust method for the automated, unsupervised scoring of immunohistochemical images based on bright field. Our work is focused on images from tumor tissues marked with blue (nuclei) and brown (protein of interest) stains. The new proposed method represents a simpler approach that, on the one hand, avoids the use of NMF in the separation stage and, on the other hand, circumvents the need for a control image. This new approach determines the subspace spanned by the two colors of interest using principal component analysis (PCA) with dimension reduction. This subspace is a two-dimensional space, allowing for color vector determination by considering the point density peaks. A new scoring stage is also developed in our method that, again, avoids reference images, making the procedure more robust and less dependent on parameters. Semi-quantitative image scoring experiments using five categories exhibit promising and consistent results when compared to manual scoring carried out by experts.

Fractal-Based Analysis of Histological Features of Brain Tumors

The structural complexity of brain tumor tissue represents a major challenge for effective histopathological diagnosis. Tumor vasculature is known to be heterogeneous, and mixtures of patterns are usually present. Therefore, extracting key descriptive features for accurate quantification is not a straightforward task. Several steps are involved in the texture analysis process where tissue heterogeneity contributes to the variability of the results. One of the interesting aspects of the brain lies in its fractal nature. Many regions within the brain tissue yield similar statistical properties at different scales of magnification. Fractal-based analysis of the histological features of brain tumors can reveal the underlying complexity of tissue structure and angiostructure, also providing an indication of tissue abnormality development. It can further be used to quantify the chaotic signature of disease to distinguish between different temporal tumor stages and histopathological grades.Brain meningioma subtype classifications' improvement from histopathological images is the main focus of this chapter. Meningioma tissue texture exhibits a wide range of histological patterns whereby a single slide may show a combination of multiple patterns. Distinctive fractal patterns quantified in a multiresolution manner would be for better spatial relationship representation. Fractal features extracted from textural tissue patterns can be useful in characterizing meningioma tumors in terms of subtype classification, a challenging problem compared to histological grading, and furthermore can provide an objective measure for quantifying subtle features within subtypes that are hard to discriminate.

Deep learning-enabled realistic virtual histology with ultraviolet photoacoustic remote sensing microscopy

The goal of oncologic surgeries is complete tumor resection, yet positive margins are frequently found postoperatively using gold standard H&E-stained histology methods. Frozen section analysis is sometimes performed for rapid intraoperative margin evaluation, albeit with known inaccuracies. Here, we introduce a label-free histological imaging method based on an ultraviolet photoacoustic remote sensing and scattering microscope, combined with unsupervised deep learning using a cycle-consistent generative adversarial network for realistic virtual staining. Unstained tissues are scanned at rates of up to 7 mins/cm2, at resolution equivalent to 400x digital histopathology. Quantitative validation suggests strong concordance with conventional histology in benign and malignant prostate and breast tissues. In diagnostic utility studies we demonstrate a mean sensitivity and specificity of 0.96 and 0.91 in breast specimens, and respectively 0.87 and 0.94 in prostate specimens. We also find virtual stain quality is preferred (P = 0.03) compared to frozen section analysis in a blinded survey of pathologists.

Preparing pathological data to develop an artificial intelligence model in the nonclinical study

Artificial intelligence (AI)-based analysis has recently been adopted in the examination of histological slides via the digitization of glass slides using a digital scanner. In this study, we examined the effect of varying the staining color tone and magnification level of a dataset on the result of AI model prediction in hematoxylin and eosin stained whole slide images (WSIs). The WSIs of liver tissues with fibrosis were used as an example, and three different datasets (N20, B20, and B10) were prepared with different color tones and magnifications. Using these datasets, we built five models trained Mask R-CNN algorithm by a single or mixed dataset of N20, B20, and B10. We evaluated their model performance using the test dataset of three datasets. It was found that the models that were trained with mixed datasets (models B20/N20 and B10/B20), which consist of different color tones or magnifications, performed better than the single dataset trained models. Consequently, superior performance of the mixed models was obtained from the actual prediction results of the test images. We suggest that training the algorithm with various staining color tones and multi-scaled image datasets would be more optimized for consistent remarkable performance in predicting pathological lesions of interest.

Stain color translation of multi-domain OSCC histopathology images using attention gated cGAN

Oral Squamous Cell Carcinoma (OSCC) is the most prevalent type of oral cancer across the globe. Histopathology examination is the gold standard for OSCC examination, where stained histopathology slides help in studying and analyzing the cell structures under a microscope to determine the stages and grading of OSCC. One of the staining methods popularly known as H&E staining is used to produce differential coloration, highlight key tissue features, and improve contrast, which makes cell analysis easier. However, the stained H&E histopathology images exhibit inter and intra-variation due to staining techniques, incubation times, and staining reagents. These variations negatively impact computer-aided diagnosis (CAD) and Machine learning algorithm's accuracy and development. A pre-processing procedure called stain normalization must be employed to reduce stain variance's negative impacts. Numerous state-of-the-art stain normalization methods are introduced. However, a robust multi-domain stain normalization approach is still required because, in a real-world situation, the OSCC histopathology images will include more than two color variations involving several domains. In this paper, a multi-domain stain translation method is proposed. The proposed method is an attention gated generator based on a Conditional Generative Adversarial Network (cGAN) with a novel objective function to enforce color distribution and the perpetual resemblance between the source and target domains. Instead of using WSI scanner images like previous techniques, the proposed method is experimented on OSCC histopathology images obtained by several conventional microscopes coupled with cameras. The proposed method receives the L* channel from the L*a*b* color space in inference mode and generates the G(a*b*) channel, which are color-adapted. The proposed technique uses mappings learned during training phases to translate the source domain to the target domain; mapping are learned using the whole color distribution of the target domain instead of one reference image. The suggested technique outperforms the four state-of-the-art methods in multi-domain OSCC histopathological translation, the claim is supported by results obtained after assessment in both quantitative and qualitative ways.

Artificial intelligence techniques for neuropathological diagnostics and research

Artificial intelligence (AI) research began in theoretical neurophysiology, and the resulting classical paper on the McCulloch-Pitts mathematical neuron was written in a psychiatry department almost 80 years ago. However, the application of AI in digital neuropathology is still in its infancy. Rapid progress is now being made, which prompted this article. Human brain diseases represent distinct system states that fall outside the normal spectrum. Many differ not only in functional but also in structural terms, and the morphology of abnormal nervous tissue forms the traditional basis of neuropathological disease classifications. However, only a few countries have the medical specialty of neuropathology, and, given the sheer number of newly developed histological tools that can be applied to the study of brain diseases, a tremendous shortage of qualified hands and eyes at the microscope is obvious. Similarly, in neuroanatomy, human observers no longer have the capacity to process the vast amounts of connectomics data. Therefore, it is reasonable to assume that advances in AI technology and, especially, whole-slide image (WSI) analysis will greatly aid neuropathological practice. In this paper, we discuss machine learning (ML) techniques that are important for understanding WSI analysis, such as traditional ML and deep learning, introduce a recently developed neuropathological AI termed PathoFusion, and present thoughts on some of the challenges that must be overcome before the full potential of AI in digital neuropathology can be realized.

Vision-Based Performance Analysis of an Active Microfluidic Droplet Generation System Using Droplet Images

This paper discusses an active droplet generation system, and the presented droplet generator successfully performs droplet generation using two fluid phases: continuous phase fluid and dispersed phase fluid. The performance of an active droplet generation system is analysed based on the droplet morphology using vision sensing and digital image processing. The proposed system in the study includes a droplet generator, camera module with image pre-processing and identification algorithm, and controller and control algorithm with a workstation computer. The overall system is able to control, sense, and analyse the generation of droplets. The main controller consists of a microcontroller, motor controller, voltage regulator, and power supply. Among the morphological features of droplets, the diameter is extracted from the images to observe the system performance. The MATLAB-based image processing algorithm consists of image acquisition, image enhancement, droplet identification, feature extraction, and analysis. RGB band filtering, thresholding, and opening are used in image pre-processing. After the image enhancement, droplet identification is performed by tracing the boundary of the droplets. The average droplet diameter varied from ~3.05 mm to ~4.04 mm in the experiments, and the average droplet diameter decrement presented a relationship of a second-order polynomial with the droplet generation time.

Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions

Removing the bias and variance of multicentre data has always been a challenge in large scale digital healthcare studies, which requires the ability to integrate clinical features extracted from data acquired by different scanners and protocols to improve stability and robustness. Previous studies have described various computational approaches to fuse single modality multicentre datasets. However, these surveys rarely focused on evaluation metrics and lacked a checklist for computational data harmonisation studies. In this systematic review, we summarise the computational data harmonisation approaches for multi-modality data in the digital healthcare field, including harmonisation strategies and evaluation metrics based on different theories. In addition, a comprehensive checklist that summarises common practices for data harmonisation studies is proposed to guide researchers to report their research findings more effectively. Last but not least, flowcharts presenting possible ways for methodology and metric selection are proposed and the limitations of different methods have been surveyed for future research.

A Method for Unsupervised Semi-Quantification of Inmunohistochemical Staining with Beta Divergences

In many research laboratories, it is essential to determine the relative expression levels of some proteins of interest in tissue samples. The semi-quantitative scoring of a set of images consists of establishing a scale of scores ranging from zero or one to a maximum number set by the researcher and assigning a score to each image that should represent some predefined characteristic of the IHC staining, such as its intensity. However, manual scoring depends on the judgment of an observer and therefore exposes the assessment to a certain level of bias. In this work, we present a fully automatic and unsupervised method for comparative biomarker quantification in histopathological brightfield images. The method relies on a color separation method that discriminates between two chromogens expressed as brown and blue colors robustly, independent of color variation or biomarker expression level. For this purpose, we have adopted a two-stage stain separation approach in the optical density space. First, a preliminary separation is performed using a deconvolution method in which the color vectors of the stains are determined after an eigendecomposition of the data. Then, we adjust the separation using the non-negative matrix factorization method with beta divergences, initializing the algorithm with the matrices resulting from the previous step. After that, a feature vector of each image based on the intensity of the two chromogens is determined. Finally, the images are annotated using a systematically initialized k-means clustering algorithm with beta divergences. The method clearly defines the initial boundaries of the categories, although some flexibility is added. Experiments for the semi-quantitative scoring of images in five categories have been carried out by comparing the results with the scores of four expert researchers yielding accuracies that range between 76.60% and 94.58%. These results show that the proposed automatic scoring system, which is definable and reproducible, produces consistent results.

Histological image color normalization using a skewed normal distribution mixed model

Color variation between histological images may influence the performance of computer-aided histological image analysis. Therefore, among the most essential and challenging tasks in histological image analysis are the reduction of the color variation between images and the preservation of the histological information contained in the images. In recent years, many methods have been introduced with respect to the color normalization of histological images. In this study, we introduce a new clustering method referred to as the skewed normal distribution mixed model clustering algorithm. Realizing that the color distribution of hue values approximates the combination of several skewed normal distributions, we propose to use the skewed normal distribution mixture model to analyze the hue distribution. The proposed skewed normal distribution mixture model clustering algorithm includes saturation-weighted hue histograms because it takes into account the saturation and hue information of a particular histogram image, which can diminish the influence of achromatic pixels. Finally, we conducted extensive experiments based on three data sets and compared them with commonly used color normalization methods. The experiments show that the proposed algorithm has better performance in stain separation and color normalization compared to other methods.

Automated knowledge-assisted mitosis cells detection framework in breast histopathology images

Based on the Nottingham Histopathology Grading (NHG) system, mitosis cells detection is one of the important criteria to determine the grade of breast carcinoma. Mitosis cells detection is a challenging task due to the heterogeneous microenvironment of breast histopathology images. Recognition of complex and inconsistent objects in the medical images could be achieved by incorporating domain knowledge in the field of interest. In this study, the strategies of the histopathologist and domain knowledge approach were used to guide the development of the image processing framework for automated mitosis cells detection in breast histopathology images. The detection framework starts with color normalization and hyperchromatic nucleus segmentation. Then, a knowledge-assisted false positive reduction method is proposed to eliminate the false positive (i.e., non-mitosis cells). This stage aims to minimize the percentage of false positive and thus increase the F1-score. Next, features extraction was performed. The mitosis candidates were classified using a Support Vector Machine (SVM) classifier. For evaluation purposes, the knowledge-assisted detection framework was tested using two datasets: a custom dataset and a publicly available dataset (i.e., MITOS dataset). The proposed knowledge-assisted false positive reduction method was found promising by eliminating at least 87.1% of false positive in both the dataset producing promising results in the F1-score. Experimental results demonstrate that the knowledge-assisted detection framework can achieve promising results in F1-score (custom dataset: 89.1%; MITOS dataset: 88.9%) and outperforms the recent works.

Fast hybrid optomechanical scanning photoacoustic remote sensing microscopy for virtual histology

A rapid scanning microscopy method for hematoxylin and eosin (H&E) like images is sought after for interoperative diagnosis of solid tumor margins. The rapid observation and diagnosis of histological samples can greatly lower surgical risk and improve patient outcomes from solid tumor resection surgeries. Photoacoustic remote sensing (PARS) has recently been demonstrated to provide images of virtual H&E stains with excellent concordance with true H&E staining of formalin-fixed, paraffin embedded tissues. By using PARS with constant velocity and 1D galvanometer mirror scanning we acquire large virtual H&E images (10mm x 5mm) of prostate tissue in less than 3.5 minutes without staining, and over two orders of magnitude faster data acquisition than the current PARS imaging speed.

Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review

Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.

Reduced and stable feature sets selection with random forest for neurons segmentation in histological images of macaque brain

In preclinical research, histology images are produced using powerful optical microscopes to digitize entire sections at cell scale. Quantification of stained tissue relies on machine learning driven segmentation. However, such methods require multiple additional information, or features, which are increasing the quantity of data to process. As a result, the quantity of features to deal with represents a drawback to process large series or massive histological images rapidly in a robust manner. Existing feature selection methods can reduce the amount of required information but the selected subsets lack reproducibility. We propose a novel methodology operating on high performance computing (HPC) infrastructures and aiming at finding small and stable sets of features for fast and robust segmentation of high-resolution histological images. This selection has two steps: (1) selection at features families scale (an intermediate pool of features, between spaces and individual features) and (2) feature selection performed on pre-selected features families. We show that the selected sets of features are stables for two different neuron staining. In order to test different configurations, one of these dataset is a mono-subject dataset and the other is a multi-subjects dataset to test different configurations. Furthermore, the feature selection results in a significant reduction of computation time and memory cost. This methodology will allow exhaustive histological studies at a high-resolution scale on HPC infrastructures for both preclinical and clinical research.

Virtual histopathology with ultraviolet scattering and photoacoustic remote sensing microscopy

Realistic label-free virtual histopathology has been a long sought-after goal not yet achieved with current methods. Here, we introduce high-resolution hematoxylin and eosin (H&E)-like virtual histology of unstained human breast lumpectomy specimen sections using ultraviolet scattering-augmented photoacoustic remote sensing microscopy. Together with a colormap-matching algorithm based on blind stain separation from a reference true H&E image, we are able to produce virtual H&E images of unstained tissues with close concordance to true H&E-stained sections, with promising diagnostic utility.

The utility of color normalization for AI-based diagnosis of hematoxylin and eosin-stained pathology images

The color variation of hematoxylin and eosin (H&E)-stained tissues has presented a challenge for applications of artificial intelligence (AI) in digital pathology. Many color normalization algorithms have been developed in recent years in order to reduce the color variation between H&E images. However, previous efforts in benchmarking these algorithms have produced conflicting results and none have sufficiently assessed the efficacy of the various color normalization methods for improving diagnostic performance of AI systems. In this study, we systematically investigated eight color normalization algorithms for AI-based classification of H&E-stained histopathology slides, in the context of using images both from one center and from multiple centers. Our results show that color normalization does not consistently improve classification performance when both training and testing data are from a single center. However, using four multi-center datasets of two cancer types (ovarian and pleural) and objective functions, we show that color normalization can significantly improve the classification accuracy of images from external datasets (ovarian cancer: 0.25 AUC increase, p = 1.6 e-05; pleural cancer: 0.21 AUC increase, p = 1.4 e-10). Furthermore, we introduce a novel augmentation strategy by mixing color-normalized images using three easily accessible algorithms that consistently improves the diagnosis of test images from external centers, even when the individual normalization methods had varied results. We anticipate our study to be a starting point for reliable use of color normalization to improve AI-based, digital pathology-empowered diagnosis of cancers sourced from multiple centers. © 2021 The Pathological Society of Great Britain and Ireland. Published by John Wiley & Sons, Ltd.

Recent technical advances in whole slide imaging instrumentation

Microscopic observation of biological specimen smears is the mainstay of diagnostic pathology, as defined by the Digital Pathology Association. Though automated systems for this are commercially available, their bulky size and high cost renders them unusable for remote areas. The research community is investing much effort towards building equivalent but portable, low-cost systems. An overview of such research is presented here, including a comparative analysis of recent reports. This paper also reviews recently reported systems for automated staining and smear formation, including microfluidic devices; and optical and computational automated microscopy systems including smartphone-based devices. Image pre-processing and analysis methods for automated diagnosis are also briefly discussed. It concludes with a set of foreseeable research directions that could lead to affordable, integrated and accurate whole slide imaging systems.

Role of AI and Histopathological Images in Detecting Prostate Cancer: A Survey

Prostate cancer is one of the most identified cancers and second most prevalent among cancer-related deaths of men worldwide. Early diagnosis and treatment are substantial to stop or handle the increase and spread of cancer cells in the body. Histopathological image diagnosis is a gold standard for detecting prostate cancer as it has different visual characteristics but interpreting those type of images needs a high level of expertise and takes too much time. One of the ways to accelerate such an analysis is by employing artificial intelligence (AI) through the use of computer-aided diagnosis (CAD) systems. The recent developments in artificial intelligence along with its sub-fields of conventional machine learning and deep learning provide new insights to clinicians and researchers, and an abundance of research is presented specifically for histopathology images tailored for prostate cancer. However, there is a lack of comprehensive surveys that focus on prostate cancer using histopathology images. In this paper, we provide a very comprehensive review of most, if not all, studies that handled the prostate cancer diagnosis using histopathological images. The survey begins with an overview of histopathological image preparation and its challenges. We also briefly review the computing techniques that are commonly applied in image processing, segmentation, feature selection, and classification that can help in detecting prostate malignancies in histopathological images.

The Effect of Quality Control on Accuracy of Digital Pathology Image Analysis

Digital slide images produced from routine diagnostic histopathological preparations suffer from variation arising at every step of the processing pipeline. Typically, pathologists compensate for such variation using expert knowledge and experience, which is difficult to replicate in automated solutions. The extent to which inconsistencies affect image analysis is explored in this work, examining in detail, the results from a previously published algorithm automating the generation of tumor:stroma ratio (TSR) in colorectal clinical trial datasets. One dataset consisting of 2,211 cases and 106,268 expert-labelled images is used to identify quality issues, by visually inspecting cases where algorithm-pathologist agreement is lowest. Twelve categories are identified and used to analyze pathologist-algorithm agreement in relation to these categories. Of the 2,211 cases, 701 were found to be free from any image quality issues. Algorithm performance was then assessed, comparing pathologist agreement with image quality classification. It was found that agreement was lowest on poorly differentiated tissue, with a mean TSR difference of 0.25 (sd = 0.24). Removing images that contained quality issues increased accuracy from 80% to 83%, at the expense of reducing the dataset to 33,736 images (32%). Training the algorithm on the optimized dataset, prior to testing on all images saw a decrease in accuracy of 4%, indicating that the optimized dataset did not contain enough variation to generate a fully representative model. The results provide an in-depth perspective on image quality, highlighting the importance of the effects on downstream image analysis.

Stain Standardization Capsule for Application-Driven Histopathological Image Normalization

Color consistency is crucial to developing robust deep learning methods for histopathological image analysis. With the increasing application of digital histopathological slides, the deep learning methods are probably developed based on the data from multiple medical centers. This requirement makes it a challenging task to normalize the color variance of histopathological images from different medical centers. In this paper, we propose a novel color standardization module named stain standardization capsule based on the capsule network and the corresponding dynamic routing algorithm. The proposed module can learn and generate uniform stain separation outputs for histopathological images in various color appearance without the reference to manually selected template images. The proposed module is light and can be jointly trained with the application-driven CNN model. The proposed method was validated on three histopathology datasets and a cytology dataset, and was compared with state-of-the-art methods. The experimental results have demonstrated that the SSC module is effective in improving the performance of histopathological image analysis and has achieved the best performance in the compared methods.

Style transfer strategy for developing a generalizable deep learning application in digital pathology

BACKGROUND AND OBJECTIVES

Despite recent advances in artificial intelligence for medical images, the development of a robust deep learning model for identifying malignancy on pathology slides has been limited by problems related to substantial inter- and intra-institutional heterogeneity attributable to tissue preparation. The paucity of available data aggravates this limitation for relatively rare cancers. Here, using ovarian cancer pathology images, we explored the effect of image-to-image style transfer approaches on diagnostic performance.

METHODS

We leveraged a relatively large public image set for 142 patients with ovarian cancer from The Cancer Image Archive (TCIA) to fine-tune the renowned deep learning model Inception V3 for identifying malignancy on tissue slides. As an external validation, the performance of the developed classifier was tested using a relatively small institutional pathology image set for 32 patients. To reduce deterioration of the performance associated with the inter-institutional heterogeneity of pathology slides, we translated the style of the small image set of the local institution into the large image set style of the TCIA using cycle-consistent generative adversarial networks.

RESULTS

Without style transfer, the performance of the classifier was as follows: area under the receiver operating characteristic curve (AUROC) = 0.737 and area under the precision recall curve (AUPRC) = 0.710. After style transfer, AUROC and AUPRC improved to 0.916 and 0.898, respectively.

CONCLUSIONS

This study provides a case of the successful application of style transfer technology to generalize a deep learning model into small image sets in the field of digital pathology. Researchers at local institutions can select this collaborative system to make their small image sets acceptable to the deep learning model.

Single image super-resolution for whole slide image using convolutional neural networks and self-supervised color normalization

High-quality whole slide scanners used for animal and human pathology scanning are expensive and can produce massive datasets, which limits the access to and adoption of this technique. As a potential solution to these challenges, we present a deep learning-based approach making use of single image super-resolution (SISR) to reconstruct high-resolution histology images from low-resolution inputs. Such low-resolution images can easily be shared, require less storage, and can be acquired quickly using widely available low-cost slide scanners. The network consists of multi-scale fully convolutional networks capable of capturing hierarchical features. Conditional generative adversarial loss is incorporated to penalize blurriness in the output images. The network is trained using a progressive strategy where the scaling factor is sampled from a normal distribution with an increasing mean. The results are evaluated with quantitative metrics and are used in a clinical histopathology diagnosis procedure which shows that the SISR framework can be used to reconstruct high-resolution images with clinical level quality. We further propose a self-supervised color normalization method that can remove staining variation artifacts. Quantitative evaluations show that the SISR framework can generalize well on unseen data collected from other patient tissue cohorts by incorporating the color normalization method.

Class-Agnostic Weighted Normalization of Staining in Histopathology Images Using a Spatially Constrained Mixture Model

The colorless biopsied tissue samples are usually stained in order to visualize different microscopic structures for diagnostic purposes. But color variations associated with the process of sample preparation, usage of raw materials, diverse staining protocols, and using different slide scanners may adversely influence both visual inspection and computer-aided image analysis. As a result, many methods are proposed for histopathology image stain normalization in recent years. In this study, we introduce a novel approach for stain normalization based on learning a mixture of multivariate skew-normal distributions for stain clustering and parameter estimation alongside a stain transformation technique. The proposed method, labeled "Class-Agnostic Weighted Normalization" (short CLAW normalization), has the ability to normalize a source image by learning the color distribution of both source and target images within an expectation-maximization framework. The novelty of this approach is its flexibility to quantify the underlying both symmetric and nonsymmetric distributions of the different stain components while it is considering the spatial information. The performance of this new stain normalization scheme is tested on several publicly available digital pathology datasets to compare it against state-of-the-art normalization algorithms in terms of ability to preserve the image structure and information. All in all, our proposed method performed superior more consistently in comparison with existing methods in terms of information preservation, visual quality enhancement, and boosting computer-aided diagnosis algorithm performance.

Automatic Quantification of Cardiomyocyte Dimensions and Connexin 43 Lateralization in Fluorescence Images

Cardiomyocytes' geometry and connexin 43 (CX43) amount and distribution are structural features that play a pivotal role in electrical conduction. Their quantitative assessment is of high interest in the study of arrhythmias, but it is usually hampered by the lack of automatic tools. In this work, we propose a software algorithm (Myocyte Automatic Retrieval and Tissue Analyzer, MARTA) to automatically detect myocytes from fluorescent microscopy images of cardiac tissue, measure their morphological features and evaluate the expression of CX43 and its degree of lateralization. The proposed software is based on the generation of cell masks, contouring of individual cells, enclosing of cells in minimum area rectangles and splitting of these rectangles into end-to-end and middle compartments to estimate CX43 lateral-to-total ratio. Application to human ventricular tissue images shows that mean differences between automatic and manual methods in terms of cardiomyocyte length and width are below 4 μm. The percentage of lateral CX43 also agrees between automatic and manual evaluation, with the interquartile range approximately covering from 3% to 30% in both cases. MARTA is not limited by fiber orientation and has an optimized speed by using contour filtering, which makes it run hundreds of times faster than a trained expert. Developed for CX43 studies in the left ventricle, MARTA is a flexible tool applicable to morphometric and lateralization studies of other markers in any heart chamber or even skeletal muscle. This open-access software is available online.

Analysis of Spatial Distribution and Prognostic Value of Different Pan Cytokeratin Immunostaining Intensities in Breast Tumor Tissue Sections

Cancer risk prognosis could improve patient survival through early personalized treatment decisions. This is the first systematic analysis of the spatial and prognostic distribution of different pan cytokeratin immunostaining intensities in breast tumors. The prognostic model included 102 breast carcinoma patients, with distant metastasis occurrence as the endpoint. We segmented the full intensity range (0–255) of pan cytokeratin digitized immunostaining into seven discrete narrow grey level ranges: 0–130, 130–160, 160–180, 180–200, 200–220, 220–240, and 240–255. These images were subsequently examined by 33 major (GLCM), fractal and first-order statistics computational analysis features. Interestingly, while moderate intensities were strongly associated with metastasis outcome, high intensities of pan cytokeratin immunostaining provided no prognostic value even after an exhaustive computational analysis. The intense pan cytokeratin immunostaining was also relatively rare, suggesting the low differentiation state of epithelial cells. The observed variability in immunostaining intensities highlighted the intratumoral heterogeneity of the malignant cells and its association with a poor disease outcome. The prognostic importance of the moderate intensity range established by complex computational morphology analyses was supported by simple measurements of its immunostaining area which was associated with favorable disease outcome. This study reveals intratumoral heterogeneity of the pan cytokeratin immunostaining together with the prognostic evaluation and spatial distribution of its discrete intensities.

Towards Population-Based Histologic Stain Normalization of Glioblastoma

Glioblastoma ( 'GBM' ) is the most aggressive type of primary malignant adult brain tumor, with very heterogeneous radio-graphic, histologic, and molecular profiles. A growing body of advanced computational analyses are conducted towards further understanding the biology and variation in glioblastoma. To address the intrinsic heterogeneity among different computational studies, reference standards have been established to facilitate both radiographic and molecular analyses, e.g., anatomical atlas for image registration and housekeeping genes, respectively. However, there is an apparent lack of reference standards in the domain of digital pathology, where each independent study uses an arbitrarily chosen slide from their evaluation dataset for normalization purposes. In this study, we introduce a novel stain normalization approach based on a composite reference slide comprised of information from a large population of anatomically annotated hematoxylin and eosin ( 'H&E' ) whole-slide images from the Ivy Glioblastoma Atlas Project ( 'IvyGAP' ). Two board-certified neuropathologists manually reviewed and selected annotations in 509 slides, according to the World Health Organization definitions. We computed summary statistics from each of these approved annotations and weighted them based on their percent contribution to overall slide ( 'PCOS' ), to form a global histogram and stain vectors. Quantitative evaluation of pre- and post-normalization stain density statistics for each annotated region with PCOS > 0.05% yielded a significant (largest p = 0.001, two-sided Wilcoxon rank sum test) reduction of its intensity variation for both 'H' & 'E' . Subject to further large-scale evaluation, our findings support the proposed approach as a potentially robust population-based reference for stain normalization.

Circular Clustering in Fuzzy Approximation Spaces for Color Normalization of Histological Images

One of the foremost and challenging tasks in hematoxylin and eosin stained histological image analysis is to reduce color variation present among images, which may significantly affect the performance of computer-aided histological image analysis. In this regard, the paper introduces a new rough-fuzzy circular clustering algorithm for stain color normalization. It judiciously integrates the merits of both fuzzy and rough sets. While the theory of rough sets deals with uncertainty, vagueness, and incompleteness in stain class definition, fuzzy set handles the overlapping nature of histochemical stains. The proposed circular clustering algorithm works on a weighted hue histogram, which considers both saturation and local neighborhood information of the given image. A new dissimilarity measure is introduced to deal with the circular nature of hue values. Some new quantitative measures are also proposed to evaluate the color constancy after normalization. The performance of the proposed method, along with a comparison with other state-of-the-art methods, is demonstrated on several publicly available standard data sets consisting of hematoxylin and eosin stained histological images.

SDCT-AuxNetθ: DCT augmented stain deconvolutional CNN with auxiliary classifier for cancer diagnosis

Acute lymphoblastic leukemia (ALL) is a pervasive pediatric white blood cell cancer across the globe. With the popularity of convolutional neural networks (CNNs), computer-aided diagnosis of cancer has attracted considerable attention. Such tools are easily deployable and are cost-effective. Hence, these can enable extensive coverage of cancer diagnostic facilities. However, the development of such a tool for ALL cancer was challenging so far due to the non-availability of a large training dataset. The visual similarity between the malignant and normal cells adds to the complexity of the problem. This paper discusses the recent release of a large dataset and presents a novel deep learning architecture for the classification of cell images of ALL cancer. The proposed architecture, namely, SDCT-AuxNet^θ is a 2-module framework that utilizes a compact CNN as the main classifier in one module and a Kernel SVM as the auxiliary classifier in the other one. While CNN classifier uses features through bilinear-pooling, spectral-averaged features are used by the auxiliary classifier. Further, this CNN is trained on the stain deconvolved quantity images in the optical density domain instead of the conventional RGB images. A novel test strategy is proposed that exploits both the classifiers for decision making using the confidence scores of their predicted class labels. Elaborate experiments have been carried out on our recently released public dataset of 15114 images of ALL cancer and healthy cells to establish the validity of the proposed methodology that is also robust to subject-level variability. A weighted F1 score of 94.8% is obtained that is best so far on this challenging dataset.

A computer-aided diagnosis system for differentiation and delineation of malignant regions on whole-slide prostate histopathology image using spatial statistics and multidimensional DenseNet

PURPOSE

Prostate cancer (PCa) is a major health concern in aging males, and proper management of the disease depends on accurately interpreting pathology specimens. However, reading prostatectomy histopathology slides, which is basically for staging, is usually time consuming and differs from reading small biopsy specimens, which is mainly used for diagnosis. Generally, each prostatectomy specimen generates tens of large tissue sections and for each section, the malignant region needs to be delineated to assess the amount of tumor and its burden. With the aim of reducing the workload of pathologists, in this study, we focus on developing a computer-aided diagnosis (CAD) system based on a densely connected convolutional neural network (DenseNet) for whole-slide histopathology images to outline the malignant regions.

METHODS

We use an efficient color normalization process based on ranklet transformation to automatically correct the intensity of the images. Additionally, we use spatial probability to segment the tissue structure regions for different tissue recognition patterns. Based on the segmentation, we incorporate a multidimensional structure into DenseNet to determine if a particular prostatic region is benign or malignant.

RESULTS

As demonstrated by the experimental results with a test set of 2,663 images from 32 whole-slide prostate histopathology images, our proposed system achieved 0.726, 0.6306, and 0.5209 in the average of the Dice coefficient, Jaccard similarity coefficient, and Boundary F1 score measures, respectively. Then, the accuracy, sensitivity, specificity, and the area under the ROC curve (AUC) of the proposed classification method were observed to be 95.0% (2544/2663), 96.7% (1210/1251), 93.9% (1334/1412), and 0.9831, respectively.

DISCUSSIONS

We provide a detailed discussion on how our proposed system demonstrates considerable improvement compared with similar methods considered in previous researches as well as how it can be used for delineating malignant regions.

Assessing the Impact of Color Normalization in Convolutional Neural Network-Based Nuclei Segmentation Frameworks

Image analysis tools for cancer, such as automatic nuclei segmentation, are impacted by the inherent variation contained in pathology image data. Convolutional neural networks (CNN), demonstrate success in generalizing to variable data, illustrating great potential as a solution to the problem of data variability. In some CNN-based segmentation works for digital pathology, authors apply color normalization (CN) to reduce color variability of data as a preprocessing step prior to prediction, while others do not. Both approaches achieve reasonable performance and yet, the reasoning for utilizing this step has not been justified. It is therefore important to evaluate the necessity and impact of CN for deep learning frameworks, and its effect on downstream processes. In this paper, we evaluate the effect of popular CN methods on CNN-based nuclei segmentation frameworks.

Size and Shape Filtering of Malignant Cell Clusters within Breast Tumors Identifies Scattered Individual Epithelial Cells as the Most Valuable Histomorphological Clue in the Prognosis of Distant Metastasis Risk

Survival and life quality of breast cancer patients could be improved by more aggressive chemotherapy for those at high metastasis risk and less intense treatments for low-risk patients. Such personalized treatment cannot be currently achieved due to the insufficient reliability of metastasis risk prognosis. The purpose of this study was therefore, to identify novel histopathological prognostic markers of metastasis risk through exhaustive computational image analysis of 80 size and shape subsets of epithelial clusters in breast tumors. The group of 102 patients had a follow-up median of 12.3 years, without lymph node spread and systemic treatments. Epithelial cells were stained by the AE1/AE3 pan-cytokeratin antibody cocktail. The size and shape subsets of the stained epithelial cell clusters were defined in each image by use of the circularity and size filters and analyzed for prognostic performance. Epithelial areas with the optimal prognostic performance were uniformly small and round and could be recognized as individual epithelial cells scattered in tumor stroma. Their count achieved an area under the receiver operating characteristic curve (AUC) of 0.82, total area (AUC = 0.77), average size (AUC = 0.63), and circularity (AUC = 0.62). In conclusion, by use of computational image analysis as a hypothesis-free discovery tool, this study reveals the histomorphological marker with a high prognostic value that is simple and therefore easy to quantify by visual microscopy.

A High-Performance System for Robust Stain Normalization of Whole-Slide Images in Histopathology

Stain normalization is an important processing task for computer-aided diagnosis (CAD) systems in modern digital pathology. This task reduces the color and intensity variations present in stained images from different laboratories. Consequently, stain normalization typically increases the prediction accuracy of CAD systems. However, there are computational challenges that this normalization step must overcome, especially for real-time applications: the memory and run-time bottlenecks associated with the processing of images in high resolution, e.g., 40X. Moreover, stain normalization can be sensitive to the quality of the input images, e.g., when they contain stain spots or dirt. In this case, the algorithm may fail to accurately estimate the stain vectors. We present a high-performance system for stain normalization using a state-of-the-art unsupervised method based on stain-vector estimation. Using a highly-optimized normalization engine, our architecture enables high-speed and large-scale processing of high-resolution whole-slide images. This optimized engine integrates an automated thresholding technique to determine the useful pixels and uses a novel pixel-sampling method that significantly reduces the processing time of the normalization algorithm. We demonstrate the performance of our architecture using measurements from images of different sizes and scanner formats that belong to four different datasets. The results show that our optimizations achieve up to 58x speedup compared to a baseline implementation. We also prove the scalability of our system by showing that the processing time scales almost linearly with the amount of tissue pixels present in the image. Furthermore, we show that the output of the normalization algorithm can be adversely affected when the input images include artifacts. To address this issue, we enhance the stain normalization pipeline by introducing a parameter cross-checking technique that automatically detects the distortion of the algorithm's critical parameters. To assess the robustness of the proposed method we employ a machine learning (ML) pipeline that classifies images for detection of prostate cancer. The results show that the enhanced normalization algorithm increases the classification accuracy of the ML pipeline in the presence of poor-quality input images. For an exemplary ML pipeline, our new method increases the accuracy on an unseen dataset from 0.79 to 0.87.

Staining Invariant Features for Improving Generalization of Deep Convolutional Neural Networks in Computational Pathology

One of the main obstacles for the implementation of deep convolutional neural networks (DCNNs) in the clinical pathology workflow is their low capability to overcome variability in slide preparation and scanner configuration, that leads to changes in tissue appearance. Some of these variations may not be not included in the training data, which means that the models have a risk to not generalize well. Addressing such variations and evaluating them in reproducible scenarios allows understanding of when the models generalize better, which is crucial for performance improvements and better DCNN models. Staining normalization techniques (often based on color deconvolution and deep learning) and color augmentation approaches have shown improvements in the generalization of the classification tasks for several tissue types. Domain-invariant training of DCNN's is also a promising technique to address the problem of training a single model for different domains, since it includes the source domain information to guide the training toward domain-invariant features, achieving state-of-the-art results in classification tasks. In this article, deep domain adaptation in convolutional networks (DANN) is applied to computational pathology and compared with widely used staining normalization and color augmentation methods in two challenging classification tasks. The classification tasks rely on two openly accessible datasets, targeting Gleason grading in prostate cancer, and mitosis classification in breast tissue. The benchmark of the different techniques and their combination in two DCNN architectures allows us to assess the generalization abilities and advantages of each method in the considered classification tasks. The code for reproducing our experiments and preprocessing the data is publicly available¹. Quantitative and qualitative results show that the use of DANN helps model generalization to external datasets. The combination of several techniques to manage color heterogeneity suggests that several methods together, such as color augmentation methods with DANN training, can generalize even further. The results do not show a single best technique among the considered methods, even when combining them. However, color augmentation and DANN training obtain most often the best results (alone or combined with color normalization and color augmentation). The statistical significance of the results and the embeddings visualizations provide useful insights to design DCNN that generalizes to unseen staining appearances. Furthermore, in this work, we release for the first time code for DANN evaluation in open access datasets for computational pathology. This work opens the possibility for further research on using DANN models together with techniques that can overcome the tissue preparation differences across datasets to tackle limited generalization.

Machine learning approaches for pathologic diagnosis

Machine learning techniques, especially deep learning techniques such as convolutional neural networks, have been successfully applied to general image recognitions since their overwhelming performance at the 2012 ImageNet Large Scale Visual Recognition Challenge. Recently, such techniques have also been applied to various medical, including histopathological, images to assist the process of medical diagnosis. In some cases, deep learning-based algorithms have already outperformed experienced pathologists for recognition of histopathological images. However, pathological images differ from general images in some aspects, and thus, machine learning of histopathological images requires specialized learning methods. Moreover, many pathologists are skeptical about the ability of deep learning technology to accurately recognize histopathological images because what the learned neural network recognizes is often indecipherable to humans. In this review, we first introduce various applications incorporating machine learning developed to assist the process of pathologic diagnosis, and then describe machine learning problems related to histopathological image analysis, and review potential ways to solve these problems.

Unsupervised Domain Adaptation for Classification of Histopathology Whole-Slide Images

Computational image analysis is one means for evaluating digitized histopathology specimens that can increase the reproducibility and reliability with which cancer diagnoses are rendered while simultaneously providing insight as to the underlying mechanisms of disease onset and progression. A major challenge that is confronted when analyzing samples that have been prepared at disparate laboratories and institutions is that the algorithms used to assess the digitized specimens often exhibit heterogeneous staining characteristics because of slight differences in incubation times and the protocols used to prepare the samples. Unfortunately, such variations can render a prediction model learned from one batch of specimens ineffective for characterizing an ensemble originating from another site. In this work, we propose to adopt unsupervised domain adaptation to effectively transfer the discriminative knowledge obtained from any given source domain to the target domain without requiring any additional labeling or annotation of images at the target site. In this paper, our team investigates the use of two approaches for performing the adaptation: (1) color normalization and (2) adversarial training. The adversarial training strategy is implemented through the use of convolutional neural networks to find an invariant feature space and Siamese architecture within the target domain to add a regularization that is appropriate for the entire set of whole-slide images. The adversarial adaptation results in significant classification improvement compared with the baseline models under a wide range of experimental settings.

Adaptive color deconvolution for histological WSI normalization

BACKGROUND AND OBJECTIVE

Color consistency of histological images is significant for developing reliable computer-aided diagnosis (CAD) systems. However, the color appearance of digital histological images varies across different specimen preparations, staining, and scanning situations. This variability affects the diagnosis and decreases the accuracy of CAD approaches. It is important and challenging to develop effective color normalization methods for digital histological images.

METHODS

We proposed a novel adaptive color deconvolution (ACD) algorithm for stain separation and color normalization of hematoxylin-eosin-stained whole slide images (WSIs). To avoid artifacts and reduce the failure rate of normalization, multiple prior knowledges of staining are considered and embedded in the ACD model. To improve the capacity of color normalization for various WSIs, an integrated optimization is designed to simultaneously estimate the parameters of the stain separation and color normalization. The solving of ACD model and application of the proposed method involves only pixel-wise operation, which makes it very efficient and applicable to WSIs.

RESULTS

The proposed method was evaluated on four WSI-datasets including breast, lung and cervix cancers and was compared with 6 state-of-the-art methods. The proposed method achieved the most consistent performance in color normalization according to the quantitative metrics. Through a qualitative assessment for 500 WSIs, the failure rate of normalization was 0.4% and the structure and color artifacts were effectively avoided. Applied to CAD methods, the area under receiver operating characteristic curve for cancer image classification was improved from 0.842 to 0.914. The average time of solving the ACD model is 2.97 s.

CONCLUSIONS

The proposed ACD model has prone effective for color normalization of hematoxylin-eosin-stained WSIs in various color appearances. The model is robust and can be applied to WSIs containing different lesions. The proposed model can be efficiently solved and is effective to improve the performance of cancer image recognition, which is adequate for developing automatic CAD programs and systems based on WSIs.

Novel chromaticity similarity based color texture descriptor for digital pathology image analysis

Pathology images are color in nature due to the use of chemical staining in biopsy examination. Aware of the high color diagnosticity in pathology images, this work introduces a compact rotation-invariant texture descriptor, named quantized diagnostic counter-color pattern (QDCP), for digital pathology image understanding. On the basis of color similarity quantified by the inner product of unit-length color vectors, local counter-color textons are indexed first. Then the underlined distribution of QDCP indexes is estimated by an image-wise histogram. Since QDCP is computed based on color difference directly, it is robust to small color variation usually observed in pathology images. This study also discusses QDCP's extraction, parameter settings, and feature fusion techniques in a generic pathology image analysis pipeline, and introduces two more descriptors QDCP-LBP and QDCP/LBP. Experimentation on public pathology image sets suggests that the introduced color texture descriptors, especially QDCP-LBP, outperform prior color texture features in terms of strong descriptive power, low computational complexity, and high adaptability to different image sets.