Spatial Statistics for Segmenting Histological Structures in H&E Stained Tissue Images

A Proposal for a Modified Evaluation System of Tumor-Infiltrating Lymphocytes Using HE-Stained Sections in Breast Cancer

Tumor-infiltrating lymphocyte (TIL) scoring in tumor specimens has gained increasing attention in determining patients who are likely to benefit from immunotherapies. However, the histological evaluation methods of TILs in breast cancer remain limited. This study aimed to assess four components of lymphocytic reaction and overall lymphocytic score (L-score) used in colorectal cancer, investigate its association with clinicopathological factors, and examine the effect of TILs on postoperative mortality using 231 invasive breast cancers without neoadjuvant chemotherapy. Besides L-score, increasing modified L-score lacking peritumoral lymphocytic reaction was significantly associated with aggressive breast cancer phenotypes, including larger invasive size, higher tumor stage, higher Ki-67 labeling index, triple negative and HER2-enriched subtypes, and higher Nottingham histological grade. Importantly, modified L-score status but not L-score or TIL-Working Group (WG) score status was positively correlated with the disease-specific survival rate of the overall patients as well as the patients with luminal type or histological Grade III breast cancers. These results indicated that the modified L-score is a favorable method to comprehensively assess lymphocytic reaction to predict prognosis among patients with breast cancer, even compared with the currently used TIL-WG method, which may possess their potential integration into clinical practice.

Intrinsically explainable deep learning architecture for semantic segmentation of histological structures in heart tissue

BACKGROUND

Analysis of structures contained in tissue samples and the relevant contextual information is of utmost importance to histopathologists during diagnosis. Cardiac biopsies require in-depth analysis of the relationships between biological structures. Statistical measures are insufficient for determining a model's viability and applicability in the diagnostic process. A deeper understanding of predictions is necessary in order to support histopathologists.

METHODS

We propose a method for providing supporting information in the form of segmentation of histological structures to histopathologists based on these principles. The proposed method utilizes nuclei type and density information in addition to standard image input provided at two different zoom levels for the semantic segmentation of blood vessels, inflammation, and endocardium in heart tissue.

RESULTS

The proposed method was able to reach state-of-the-art segmentation results. The overall quality and viability of the predictions was qualitatively evaluated by two pathologists and a histotechnologist.

CONCLUSIONS

The decision process of the proposed deep learning model utilizes the provided information sources correctly and simulates the decision process of histopathologists via the usage of a custom-designed attention gate that provides a combination of spatial and encoder attention mechanisms. The implementation is available at https://github.com/mathali/IEDL-segmentation-of-heart-tissue.

Spatial modelling of the tumor microenvironment from multiplex immunofluorescence images: methods and applications

Spatial modelling methods have gained prominence with developments in high throughput imaging platforms. Multiplex immunofluorescence (mIF) provides the scope to examine interactions between tumor and immune compartment at single cell resolution using a panel of antibodies that can be chosen based on the cancer type or the clinical interest of the study. The markers can be used to identify the phenotypes and to examine cellular interactions at global and local scales. Several translational studies rely on key understanding of the tumor microenvironment (TME) to identify drivers of immune response in immunotherapy based clinical trials. To improve the success of ongoing trials, a number of retrospective approaches can be adopted to understand differences in response, recurrence and progression by examining the patient's TME from tissue samples obtained at baseline and at various time points along the treatment. The multiplex immunofluorescence (mIF) technique provides insight on patient specific cell populations and their relative spatial distribution as qualitative measures of a favorable treatment outcome. Spatial analysis of these images provides an understanding of the intratumoral heterogeneity and clustering among cell populations in the TME. A number of mathematical models, which establish clustering as a measure of deviation from complete spatial randomness, can be applied to the mIF images represented as spatial point patterns. These mathematical models, developed for landscape ecology and geographic information studies, can be applied to the TME after careful consideration of the tumor type (cold vs. hot) and the tumor immune landscape. The spatial modelling of mIF images can show observable engagement of T cells expressing immune checkpoint molecules and this can then be correlated with single-cell RNA sequencing data.

Advantages of manual and automatic computer-aided compared to traditional histopathological diagnosis of melanoma: A pilot study

BACKGROUND

Cutaneous malignant melanoma (CMM) accounts for the highest mortality rate among all skin cancers. Traditional histopathologic diagnosis may be limited by the pathologists' subjectivity. Second-opinion strategies and multidisciplinary consultations are usually performed to overcome this issue. An available solution in the future could be the use of automated solutions based on a computational algorithm that could help the pathologist in everyday practice. The aim of this pilot study was to investigate the potential diagnostic aid of a machine-based algorithm in the histopathologic diagnosis of CMM.

METHODS

We retrospectively examined excisional biopsies of 50 CMM and 20 benign congenital compound nevi. Hematoxylin and eosin (H&E) stained WSI were reviewed independently by two expert dermatopathologists. A fully automated pipeline for WSI processing to support the estimation and prioritization of the melanoma areas was developed.

RESULTS

The spatial distribution of the nuclei in the sample provided a multi-scale overview of the tumor. A global overview of the lesion's silhouette was achieved and, by increasing the magnification, the topological distribution of the nuclei and the most informative areas of interest for the CMM diagnosis were identified and highlighted. These silhouettes allow the histopathologist to discriminate between nevus and CMM with an accuracy of 96% without any extra information.

CONCLUSION

In this study we proposed an easy-to-use model that produces segmentations of CMM silhouettes at fine detail level.

Fast whole-slide cartography in colon cancer histology using superpixels and CNN classification

Purpose: Automatic outlining of different tissue types in digitized histological specimen provides a basis for follow-up analyses and can potentially guide subsequent medical decisions. The immense size of whole-slide-images (WSIs), however, poses a challenge in terms of computation time. In this regard, the analysis of nonoverlapping patches outperforms pixelwise segmentation approaches but still leaves room for optimization. Furthermore, the division into patches, regardless of the biological structures they contain, is a drawback due to the loss of local dependencies. Approach: We propose to subdivide the WSI into coherent regions prior to classification by grouping visually similar adjacent pixels into superpixels. Afterward, only a random subset of patches per superpixel is classified and patch labels are combined into a superpixel label. We propose a metric for identifying superpixels with an uncertain classification and evaluate two medical applications, namely tumor area and invasive margin estimation and tumor composition analysis. Results: The algorithm has been developed on 159 hand-annotated WSIs of colon resections and its performance is compared with an analysis without prior segmentation. The algorithm shows an average speed-up of 41% and an increase in accuracy from 93.8% to 95.7%. By assigning a rejection label to uncertain superpixels, we further increase the accuracy by 0.4%. While tumor area estimation shows high concordance to the annotated area, the analysis of tumor composition highlights limitations of our approach. Conclusion: By combining superpixel segmentation and patch classification, we designed a fast and accurate framework for whole-slide cartography that is AI-model agnostic and provides the basis for various medical endpoints.

Explainable AI (xAI) for Anatomic Pathology

Pathologists are adopting whole slide images (WSIs) for diagnosis, thanks to recent FDA approval of WSI systems as class II medical devices. In response to new market forces and recent technology advances outside of pathology, a new field of computational pathology has emerged that applies artificial intelligence (AI) and machine learning algorithms to WSIs. Computational pathology has great potential for augmenting pathologists' accuracy and efficiency, but there are important concerns regarding trust of AI due to the opaque, black-box nature of most AI algorithms. In addition, there is a lack of consensus on how pathologists should incorporate computational pathology systems into their workflow. To address these concerns, building computational pathology systems with explainable AI (xAI) mechanisms is a powerful and transparent alternative to black-box AI models. xAI can reveal underlying causes for its decisions; this is intended to promote safety and reliability of AI for critical tasks such as pathology diagnosis. This article outlines xAI enabled applications in anatomic pathology workflow that improves efficiency and accuracy of the practice. In addition, we describe HistoMapr-Breast, an initial xAI enabled software application for breast core biopsies. HistoMapr-Breast automatically previews breast core WSIs and recognizes the regions of interest to rapidly present the key diagnostic areas in an interactive and explainable manner. We anticipate xAI will ultimately serve pathologists as an interactive computational guide for computer-assisted primary diagnosis.

A computer-aided diagnosis system for differentiation and delineation of malignant regions on whole-slide prostate histopathology image using spatial statistics and multidimensional DenseNet

PURPOSE

Prostate cancer (PCa) is a major health concern in aging males, and proper management of the disease depends on accurately interpreting pathology specimens. However, reading prostatectomy histopathology slides, which is basically for staging, is usually time consuming and differs from reading small biopsy specimens, which is mainly used for diagnosis. Generally, each prostatectomy specimen generates tens of large tissue sections and for each section, the malignant region needs to be delineated to assess the amount of tumor and its burden. With the aim of reducing the workload of pathologists, in this study, we focus on developing a computer-aided diagnosis (CAD) system based on a densely connected convolutional neural network (DenseNet) for whole-slide histopathology images to outline the malignant regions.

METHODS

We use an efficient color normalization process based on ranklet transformation to automatically correct the intensity of the images. Additionally, we use spatial probability to segment the tissue structure regions for different tissue recognition patterns. Based on the segmentation, we incorporate a multidimensional structure into DenseNet to determine if a particular prostatic region is benign or malignant.

RESULTS

As demonstrated by the experimental results with a test set of 2,663 images from 32 whole-slide prostate histopathology images, our proposed system achieved 0.726, 0.6306, and 0.5209 in the average of the Dice coefficient, Jaccard similarity coefficient, and Boundary F1 score measures, respectively. Then, the accuracy, sensitivity, specificity, and the area under the ROC curve (AUC) of the proposed classification method were observed to be 95.0% (2544/2663), 96.7% (1210/1251), 93.9% (1334/1412), and 0.9831, respectively.

DISCUSSIONS

We provide a detailed discussion on how our proposed system demonstrates considerable improvement compared with similar methods considered in previous researches as well as how it can be used for delineating malignant regions.

Data on cut-edge for spatial clustering based on proximity graphs

Cluster analysis plays a significant role regarding automating such a knowledge discovery process in spatial data mining. A good clustering algorithm supports two essential conditions, namely high intra-cluster similarity and low inter-cluster similarity. Maximized intra-cluster/within-cluster similarity produces low distances between data points inside the same cluster. However, minimized inter-cluster/between-cluster similarity increases the distance between data points in different clusters by furthering them apart from each other. We previously presented a spatial clustering algorithm, abbreviated CutESC (Cut-Edge for Spatial Clustering) with a graph-based approach. The data presented in this article is related to and supportive to the research paper entitled “CutESC: Cutting edge spatial clustering technique based on proximity graphs” (Aksac et al., 2019) [1], where interpretation research data presented here is available. In this article, we share the parametric version of our algorithm named CutESC-P, the best parameter settings for the experiments, the additional analyses and some additional information related to the proposed algorithm (CutESC) in [1].

An automated method for segmentation and quantification of blood vessels in histology images

Alzheimer's disease (AD) is a chronic neuro-degenerative disease that adversely affect many people on a global scale. Despite different diagnostic and therapeutic treatment, there is no cure for AD. The brain is one of the most complex organ and researchers are still trying to understand so as to find a cure.

OBJECTIVE

To complement the efforts of clinical researchers engaged in research in alzheimer's disease, accurate segmentation and quantification of blood vessels in brain images is required.

METHOD

For robust segmentation of blood vessels even in the presence of colour variation, we introduce a fully automated morphological tool that can extract and quantify vessels from haematoxylin and diaminobenzidine stained histology brain image. The method, exploits saturation channel of stained image slides, ISODATA threshold method is applied to obtain a binary image. This helps in eliminating background and remaining with only blood vessels. A one-stage procedure that includes eliminating small artefacts is performed on the binary mask. The intensity of the image is transformed. Joining is performed to deal with fragmentation of intact blood vessels on the images, and artefactual appearance of the blood vessel structures. The artefactual fragments based on measured incoherence with neighbouring tissue are removed. The vessels are then labelled to facilitate quantification. Morphometric measurements are used during the vessel quantification assess both vessels with lumen and vessels without lumen. We have quantified the diameter of blood vessels.

RESULTS

The image processing technique is developed in close collaboration with neuroscientist experts to help clinician. We have evaluated our proposed approach qualitatively. The method was validated against their manual quantification results. Qualitative results show that the method can indeed segment the blood vessels in the presence of colour variations and artefacts. The quantitative method produces fairly better results.

Generative Adversarial Networks for Facilitating Stain-Independent Supervised and Unsupervised Segmentation: A Study on Kidney Histology

A major challenge in the field of segmentation in digital pathology is given by the high effort for manual data annotations in combination with many sources introducing variability in the image domain. This requires methods that are able to cope with variability without requiring to annotate a large amount of samples for each characteristic. In this paper, we develop approaches based on adversarial models for image-to-image translation relying on unpaired training. Specifically, we propose approaches for stain-independent supervised segmentation relying on image-to-image translation for obtaining an intermediate representation. Furthermore, we develop a fully-unsupervised segmentation approach exploiting image-to-image translation to convert from the image to the label domain. Finally, both approaches are combined to obtain optimum performance in unsupervised segmentation independent of the characteristics of the underlying stain. Experiments on patches showing kidney histology proof that stain-translation can be performed highly effectively and can be used for domain adaptation to obtain independence of the underlying stain. It is even capable of facilitating the underlying segmentation task, thereby boosting the accuracy if an appropriate intermediate stain is selected. Combining domain adaptation with unsupervised segmentation finally showed the most significant improvements.

Head and Neck Cancer Detection in Digitized Whole-Slide Histology Using Convolutional Neural Networks

Primary management for head and neck cancers, including squamous cell carcinoma (SCC), involves surgical resection with negative cancer margins. Pathologists guide surgeons during these operations by detecting cancer in histology slides made from the excised tissue. In this study, 381 digitized, histological whole-slide images (WSI) from 156 patients with head and neck cancer were used to train, validate, and test an inception-v4 convolutional neural network. The proposed method is able to detect and localize primary head and neck SCC on WSI with an AUC of 0.916 for patients in the SCC testing group and 0.954 for patients in the thyroid carcinoma testing group. Moreover, the proposed method is able to diagnose WSI with cancer versus normal slides with an AUC of 0.944 and 0.995 for the SCC and thyroid carcinoma testing groups, respectively. For comparison, we tested the proposed, diagnostic method on an open-source dataset of WSI from sentinel lymph nodes with breast cancer metastases, CAMELYON 2016, to obtain patch-based cancer localization and slide-level cancer diagnoses. The experimental design yields a robust method with potential to help create a tool to increase efficiency and accuracy of pathologists detecting head and neck cancers in histological images.

Color normalization of faded H&E-stained histological images using spectral matching

Histological samples stained with hematoxylin-eosin (H&E) are commonly used by pathologists in cancer diagnoses. However, the preparation, digitization, and storage of tissue samples can lead to color variations that produce poor performance when using histological image processing techniques. Thus, normalization methods have been proposed to adjust the color of the image. This can be achieved through the use of a spectral matching technique, where it is first necessary to estimate the H&E representation and the stain concentration in the image pixels by means of the RGB model. This study presents an estimation method for H&E stain representation for the normalization of faded histological samples. This application has been explored only to a limited extent in the literature, but has the capacity to expand the use of faded samples. To achieve this, the normalized images must have a coherent color representation of the H&E stain with no introduction of noise, which was realized by applying the methodology described in this proposal. The estimation method presented here aims to normalize histological samples with different degrees of fading using a combination of fuzzy theory and the Cuckoo search algorithm, and dictionary learning with an initialization method for optimization. In visual and quantitative comparisons of estimates of H&E stain representation from the literature, our proposed method achieved very good results, with a high feature similarity between the original and normalized images.

Digital Staining of High-Definition Fourier Transform Infrared (FT-IR) Images Using Deep Learning

Histological stains, such as hematoxylin and eosin (H&E), are routinely used in clinical diagnosis and research. While these labels offer a high degree of specificity, throughput is limited by the need for multiple samples. Traditional histology stains, such as immunohistochemical labels, also rely only on protein expression and cannot quantify small molecules and metabolites that may aid in diagnosis. Finally, chemical stains and dyes permanently alter the tissue, making downstream analysis impossible. Fourier transform infrared (FT-IR) spectroscopic imaging has shown promise for label-free characterization of important tissue phenotypes and can bypass the need for many chemical labels. Fourier transform infrared classification commonly leverages supervised learning, requiring human annotation that is tedious and prone to errors. One alternative is digital staining, which leverages machine learning to map IR spectra to a corresponding chemical stain. This replaces human annotation with computer-aided alignment. Previous work relies on alignment of adjacent serial tissue sections. Since the tissue samples are not identical at the cellular level, this technique cannot be applied to high-definition FT-IR images. In this paper, we demonstrate that cellular-level mapping can be accomplished using identical samples for both FT-IR and chemical labels. In addition, higher-resolution results can be achieved using a deep convolutional neural network that integrates spatial and spectral features.

Associations in tumor infiltrating lymphocytes between clinicopathological factors and clinical outcomes in estrogen receptor-positive/human epidermal growth factor receptor type 2 negative breast cancer

The value of assessing tumor infiltrating lymphocytes (TILs) in estrogen receptor (ER) positive/human epidermal growth factor receptor type 2 (HER2) negative breast cancer has yet to be determined. In the present study, a total of 184 cases with early distant recurrence detected within 5 years following the primary operation, 134 with late distant recurrence diagnosed following 5 years or longer and 321 controls without recurrence for >10 years following starting the initial treatment for ER-positive/HER2 negative breast cancer, registered in 9 institutions, were analyzed. The distributions of TILs and their clinical relevance were investigated. TIL distributions did not differ significantly among the early, late and no recurrence groups, employing a 30% cut-off point as a dichotomous variable. In those who had received adjuvant chemotherapy as well as endocrine therapy, a trend toward higher TIL proportions was detected when the early recurrence group was compared with the no recurrence group employing the 30% cut-off point (P=0.064). The TIL distributions were significantly associated with nodal metastasis (P=0.004), ER status (P=0.045), progesterone receptor (PgR) status (P=0.002), tumor grade (P=0.021), and the Ki67 labeling index (LI) (P=0.002) in the no recurrence group and with the Ki67 LI in the recurrence groups (P=0.002 in early recurrence group, P=0.023 in late recurrence group). High TIL distributions also predicted shorter survival time following the detection of recurrence (P=0.026). However, these prognostic interactions were not significant in multivariate analysis (P=0.200). The present retrospective study demonstrated no significant interaction between TIL proportions and the timing of recurrence. However, higher TIL proportions were observed in breast cancer patients with aggressive biological phenotypes, which tended to be more responsive to chemotherapy. The clinical relevance of stromal TILs for identifying patients who would likely benefit from additional therapies merits further investigation in a larger patient population.