Deep learning based Nucleus Classification in pancreas histological images

Quantitative multiplex immunohistochemistry reveals inter-patient lymphovascular and immune heterogeneity in primary cutaneous melanoma

Introduction

Quantitative, multiplexed imaging is revealing complex spatial relationships between phenotypically diverse tumor infiltrating leukocyte populations and their prognostic implications. The underlying mechanisms and tissue structures that determine leukocyte distribution within and around tumor nests, however, remain poorly understood. While presumed players in metastatic dissemination, new preclinical data demonstrates that blood and lymphatic vessels (lymphovasculature) also dictate leukocyte trafficking within tumor microenvironments and thereby impact anti-tumor immunity. Here we interrogate these relationships in primary human cutaneous melanoma.

Methods

We established a quantitative, multiplexed imaging platform to simultaneously detect immune infiltrates and tumor-associated vessels in formalin-fixed paraffin embedded patient samples. We performed a discovery, retrospective analysis of 28 treatment-naïve, primary cutaneous melanomas.

Results

Here we find that the lymphvasculature and immune infiltrate is heterogenous across patients in treatment naïve, primary melanoma. We categorized five lymphovascular subtypes that differ by functionality and morphology and mapped their localization in and around primary tumors. Interestingly, the localization of specific vessel subtypes, but not overall vessel density, significantly associated with the presence of lymphoid aggregates, regional progression, and intratumoral T cell infiltrates.

Discussion

We describe a quantitative platform to enable simultaneous lymphovascular and immune infiltrate analysis and map their spatial relationships in primary melanoma. Our data indicate that tumor-associated vessels exist in different states and that their localization may determine potential for metastasis or immune infiltration. This platform will support future efforts to map tumor-associated lymphovascular evolution across stage, assess its prognostic value, and stratify patients for adjuvant therapy.

Survey of Denoising, Segmentation and Classification of Pancreatic Cancer Imaging

BACKGROUND

Pancreatic cancer is one of the most serious problems that has taken many lives worldwide. The diagnostic procedure using the traditional approaches was manual by visually analyzing the large volumes of the dataset, making it time-consuming and prone to subjective errors. Hence the need for the computer-aided diagnosis system (CADs) emerged that comprises the machine and deep learning approaches for denoising, segmentation and classification of pancreatic cancer.

INTRODUCTION

There are different modalities used for the diagnosis of pancreatic cancer, such as Positron Emission Tomography/Computed Tomography (PET/CT), Magnetic Resonance Imaging (MRI), Multiparametric-MRI (Mp-MRI), Radiomics and Radio-genomics. Although these modalities gave remarkable results in diagnosis on the basis of different criteria. CT is the most commonly used modality that produces detailed and fine contrast images of internal organs of the body. However, it may also contain a certain amount of gaussian and rician noise that is necessary to be preprocessed before segmentation of the required region of interest (ROI) from the images and classification of cancer.

METHOD

This paper analyzes different methodologies used for the complete diagnosis of pancreatic cancer, including the denoising, segmentation and classification, along with the challenges and future scope for the diagnosis of pancreatic cancer.

RESULT

Various filters are used for denoising and image smoothening and filters as gaussian scale mixture process, non-local means, median filter, adaptive filter and average filter have been used more for better results.

CONCLUSION

In terms of segmentation, atlas based region-growing method proved to give better results as compared to the state of the art whereas, for the classification, deep learning approaches outperformed other methodologies to classify the images as cancerous and non- cancerous. These methodologies have proved that CAD systems have become a better solution to the ongoing research proposals for the detection of pancreatic cancer worldwide.

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding

Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

Applications of Deep Learning in Endocrine Neoplasms

Machine learning methods have been growing in prominence across all areas of medicine. In pathology, recent advances in deep learning (DL) have enabled computational analysis of histological samples, aiding in diagnosis and characterization in multiple disease areas. In cancer, and particularly endocrine cancer, DL approaches have been shown to be useful in tasks ranging from tumor grading to gene expression prediction. This review summarizes the current state of DL research in endocrine cancer histopathology with an emphasis on experimental design, significant findings, and key limitations.

Deciphering the Immune Complexity in Esophageal Adenocarcinoma and Pre-Cancerous Lesions With Sequential Multiplex Immunohistochemistry and Sparse Subspace Clustering Approach

Esophageal adenocarcinoma (EAC) develops from a chronic inflammatory environment across four stages: intestinal metaplasia, known as Barrett's esophagus, low- and high-grade dysplasia, and adenocarcinoma. Although the genomic characteristics of this progression have been well defined via large-scale DNA sequencing, the dynamics of various immune cell subsets and their spatial interactions in their tumor microenvironment remain unclear. Here, we applied a sequential multiplex immunohistochemistry (mIHC) platform with computational image analysis pipelines that allow for the detection of 10 biomarkers in one formalin-fixed paraffin-embedded (FFPE) tissue section. Using this platform and quantitative image analytics, we studied changes in the immune landscape during disease progression based on 40 normal and diseased areas from endoscopic mucosal resection specimens of chemotherapy treatment- naïve patients, including normal esophagus, metaplasia, low- and high-grade dysplasia, and adenocarcinoma. The results revealed a steady increase of FOXP3+ T regulatory cells and a CD163+ myelomonocytic cell subset. In parallel to the manual gating strategy applied for cell phenotyping, we also adopted a sparse subspace clustering (SSC) algorithm allowing the automated cell phenotyping of mIHC-based single-cell data. The algorithm successfully identified comparable cell types, along with significantly enriched FOXP3 T regulatory cells and CD163+ myelomonocytic cells as found in manual gating. In addition, SCC identified a new CSF1R+CD1C+ myeloid lineage, which not only was previously unknown in this disease but also increases with advancing disease stages. This study revealed immune dynamics in EAC progression and highlighted the potential application of a new multiplex imaging platform, combined with computational image analysis on routine clinical FFPE sections, to investigate complex immune populations in tumor ecosystems.

An omic and multidimensional spatial atlas from serial biopsies of an evolving metastatic breast cancer

Mechanisms of therapeutic resistance and vulnerability evolve in metastatic cancers as tumor cells and extrinsic microenvironmental influences change during treatment. To support the development of methods for identifying these mechanisms in individual people, here we present an omic and multidimensional spatial (OMS) atlas generated from four serial biopsies of an individual with metastatic breast cancer during 3.5 years of therapy. This resource links detailed, longitudinal clinical metadata that includes treatment times and doses, anatomic imaging, and blood-based response measurements to clinical and exploratory analyses, which includes comprehensive DNA, RNA, and protein profiles; images of multiplexed immunostaining; and 2- and 3-dimensional scanning electron micrographs. These data report aspects of heterogeneity and evolution of the cancer genome, signaling pathways, immune microenvironment, cellular composition and organization, and ultrastructure. We present illustrative examples of how integrative analyses of these data reveal potential mechanisms of response and resistance and suggest novel therapeutic vulnerabilities.

Epigenetic loss of heterogeneity from low to high grade localized prostate tumours

Identifying precise molecular subtypes attributable to specific stages of localized prostate cancer has proven difficult due to high levels of heterogeneity. Bulk assays represent a population-average, which mask the heterogeneity that exists at the single-cell level. In this work, we sequence the accessible chromatin regions of 14,424 single-cells from 18 flash-frozen prostate tumours. We observe shared chromatin features among low-grade prostate cancer cells are lost in high-grade tumours. Despite this loss, high-grade tumours exhibit an enrichment for FOXA1, HOXB13 and CDX2 transcription factor binding sites, indicating a shared trans-regulatory programme. We identify two unique genes encoding neuronal adhesion molecules that are highly accessible in high-grade prostate tumours. We show NRXN1 and NLGN1 expression in epithelial, endothelial, immune and neuronal cells in prostate cancer using cyclic immunofluorescence. Our results provide a deeper understanding of the active gene regulatory networks in primary prostate tumours, critical for molecular stratification of the disease.

Relevance of circulating hybrid cells as a non-invasive biomarker for myriad solid tumors

Metastatic progression defines the final stages of tumor evolution and underlies the majority of cancer-related deaths. The heterogeneity in disseminated tumor cell populations capable of seeding and growing in distant organ sites contributes to the development of treatment resistant disease. We recently reported the identification of a novel tumor-derived cell population, circulating hybrid cells (CHCs), harboring attributes from both macrophages and neoplastic cells, including functional characteristics important to metastatic spread. These disseminated hybrids outnumber conventionally defined circulating tumor cells (CTCs) in cancer patients. It is unknown if CHCs represent a generalized cancer mechanism for cell dissemination, or if this population is relevant to the metastatic cascade. Herein, we detect CHCs in the peripheral blood of patients with cancer in myriad disease sites encompassing epithelial and non-epithelial malignancies. Further, we demonstrate that in vivo-derived hybrid cells harbor tumor-initiating capacity in murine cancer models and that CHCs from human breast cancer patients express stem cell antigens, features consistent with the potential to seed and grow at metastatic sites. Finally, we reveal heterogeneity of CHC phenotypes reflect key tumor features, including oncogenic mutations and functional protein expression. Importantly, this novel population of disseminated neoplastic cells opens a new area in cancer biology and renewed opportunity for battling metastatic disease.

State of machine and deep learning in histopathological applications in digestive diseases

Machine learning (ML)- and deep learning (DL)-based imaging modalities have exhibited the capacity to handle extremely high dimensional data for a number of computer vision tasks. While these approaches have been applied to numerous data types, this capacity can be especially leveraged by application on histopathological images, which capture cellular and structural features with their high-resolution, microscopic perspectives. Already, these methodologies have demonstrated promising performance in a variety of applications like disease classification, cancer grading, structure and cellular localizations, and prognostic predictions. A wide range of pathologies requiring histopathological evaluation exist in gastroenterology and hepatology, indicating these as disciplines highly targetable for integration of these technologies. Gastroenterologists have also already been primed to consider the impact of these algorithms, as development of real-time endoscopic video analysis software has been an active and popular field of research. This heightened clinical awareness will likely be important for future integration of these methods and to drive interdisciplinary collaborations on emerging studies. To provide an overview on the application of these methodologies for gastrointestinal and hepatological histopathological slides, this review will discuss general ML and DL concepts, introduce recent and emerging literature using these methods, and cover challenges moving forward to further advance the field.

Automatic Pancreatic Ductal Adenocarcinoma Detection in Whole Slide Images Using Deep Convolutional Neural Networks

Pancreatic ductal adenocarcinoma (PDAC) is one of the deadliest cancer types worldwide, with the lowest 5-year survival rate among all kinds of cancers. Histopathology image analysis is considered a gold standard for PDAC detection and diagnosis. However, the manual diagnosis used in current clinical practice is a tedious and time-consuming task and diagnosis concordance can be low. With the development of digital imaging and machine learning, several scholars have proposed PDAC analysis approaches based on feature extraction methods that rely on field knowledge. However, feature-based classification methods are applicable only to a specific problem and lack versatility, so that the deep-learning method is becoming a vital alternative to feature extraction. This paper proposes the first deep convolutional neural network architecture for classifying and segmenting pancreatic histopathological images on a relatively large WSI dataset. Our automatic patch-level approach achieved 95.3% classification accuracy and the WSI-level approach achieved 100%. Additionally, we visualized the classification and segmentation outcomes of histopathological images to determine which areas of an image are more important for PDAC identification. Experimental results demonstrate that our proposed model can effectively diagnose PDAC using histopathological images, which illustrates the potential of this practical application.

SHIFT: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning

Spatially-resolved molecular profiling by immunostaining tissue sections is a key feature in cancer diagnosis, subtyping, and treatment, where it complements routine histopathological evaluation by clarifying tumor phenotypes. In this work, we present a deep learning-based method called speedy histological-to-immunofluorescent translation (SHIFT) which takes histologic images of hematoxylin and eosin (H&E)-stained tissue as input, then in near-real time returns inferred virtual immunofluorescence (IF) images that estimate the underlying distribution of the tumor cell marker pan-cytokeratin (panCK). To build a dataset suitable for learning this task, we developed a serial staining protocol which allows IF and H&E images from the same tissue to be spatially registered. We show that deep learning-extracted morphological feature representations of histological images can guide representative sample selection, which improved SHIFT generalizability in a small but heterogenous set of human pancreatic cancer samples. With validation in larger cohorts, SHIFT could serve as an efficient preliminary, auxiliary, or substitute for panCK IF by delivering virtual panCK IF images for a fraction of the cost and in a fraction of the time required by traditional IF.

A machine learning algorithm for simulating immunohistochemistry: development of SOX10 virtual IHC and evaluation on primarily melanocytic neoplasms

Immunohistochemistry (IHC) is a diagnostic technique used throughout pathology. A machine learning algorithm that could predict individual cell immunophenotype based on hematoxylin and eosin (H&E) staining would save money, time, and reduce tissue consumed. Prior approaches have lacked the spatial accuracy needed for cell-specific analytical tasks. Here IHC performed on destained H&E slides is used to create a neural network that is potentially capable of predicting individual cell immunophenotype. Twelve slides were stained with H&E and scanned to create digital whole slide images. The H&E slides were then destained, and stained with SOX10 IHC. The SOX10 IHC slides were scanned, and corresponding H&E and IHC digital images were registered. Color-thresholding and machine learning techniques were applied to the registered H&E and IHC images to segment 3,396,668 SOX10-negative cells and 306,166 SOX10-positive cells. The resulting segmentation was used to annotate the original H&E images, and a convolutional neural network was trained to predict SOX10 nuclear staining. Sixteen thousand three hundred and nine image patches were used to train the virtual IHC (vIHC) neural network, and 1,813 image patches were used to quantitatively evaluate it. The resulting vIHC neural network achieved an area under the curve of 0.9422 in a receiver operator characteristics analysis when sorting individual nuclei. The vIHC network was applied to additional images from clinical practice, and was evaluated qualitatively by a board-certified dermatopathologist. Further work is needed to make the process more efficient and accurate for clinical use. This proof-of-concept demonstrates the feasibility of creating neural network-driven vIHC assays.

Loss of myoepithelial calponin-1 characterizes high-risk ductal carcinoma in situ cases, which are further stratified by T cell composition

A hallmark of ductal carcinoma in situ (DCIS) progression is a loss of the surrounding ductal myoepithelium. However, whether compromise in myoepithelial differentiation, rather than overt cellular loss, can be used to predict the risk of DCIS progression is unknown. Here we address this question utilizing pure and mixed DCIS cases (N = 30) as surrogates for DCIS at low and high risk for progression, respectively. We used multiplex immunohistochemical staining to evaluate the relationship between myoepithelial cell differentiation and lymphoid immune cell types associated with poor prognostic DCIS. Our results show that myoepithelial calponin-1 discriminates between pure and mixed DCIS lesions better than histological subtype, presence of necrosis, or nuclear grade. Additionally, focal loss of myoepithelial cells associated with increased PD-1+CD8+ T cells, which suggests a link between the myoepithelium and immune surveillance. To identify associations between calponin-1 expression and immune response, we performed unsupervised hierarchical clustering of myoepithelial and immune cell biomarkers on 219 DCIS lesions from 30 cases. Notably, the majority of pure (low-risk) DCIS lesions clustered in a high calponin-1, T cell low group, whereas the majority of mixed (high-risk) DCIS lesions clustered in a low calponin-1, T cell high group, specifically with CD8+ and PD-1+CD8+ T cells. However, a subset of pure DCIS lesions had a similar calponin-1 and immune signature as the majority of mixed DCIS lesions, which have low calponin-1 and T cell enrichment-raising the possibility that these pure DCIS lesions might be at a high risk for progression.

RESTORE: Robust intEnSiTy nORmalization mEthod for multiplexed imaging

Recent advances in multiplexed imaging technologies promise to improve the understanding of the functional states of individual cells and the interactions between the cells in tissues. This often requires compilation of results from multiple samples. However, quantitative integration of information between samples is complicated by variations in staining intensity and background fluorescence that obscure biological variations. Failure to remove these unwanted artifacts will complicate downstream analysis and diminish the value of multiplexed imaging for clinical applications. Here, to compensate for unwanted variations, we automatically identify negative control cells for each marker within the same tissue and use their expression levels to infer background signal level. The intensity profile is normalized by the inferred level of the negative control cells to remove between-sample variation. Using a tissue microarray data and a pair of longitudinal biopsy samples, we demonstrated that the proposed approach can remove unwanted variations effectively and shows robust performance.

Chang et al. develop an analytical method called RESTORE to control for variations due to technical artifacts in multiplexed imaging. They test their method on a CycIF stained tissue microarray dataset and biopsies processed at different times. Their method can improve the applicability of imaging techniques in diagnostics and inference using unbiased clustering methods.

Robust Cell Detection and Segmentation for Image Cytometry Reveal Th17 Cell Heterogeneity

Image cytometry enables quantitative cell characterization with preserved tissue architecture; thus, it has been highlighted in the advancement of multiplex immunohistochemistry (IHC) and digital image analysis in the context of immune-based biomarker monitoring associated with cancer immunotherapy. However, one of the challenges in the current image cytometry methodology is a technical limitation in the segmentation of nuclei and cellular components particularly in heterogeneously stained cancer tissue images. To improve the detection and specificity of single-cell segmentation in hematoxylin-stained images (which can be utilized for recently reported 12-biomarker chromogenic sequential multiplex IHC), we adapted a segmentation algorithm previously developed for hematoxlin and eosin-stained images, where morphological features are extracted based on Gabor-filtering, followed by stacking of image pixels into n-dimensional feature space and unsupervised clustering of individual pixels. Our proposed method showed improved sensitivity and specificity in comparison with standard segmentation methods. Replacing previously proposed methods with our method in multiplex IHC/image cytometry analysis, we observed higher detection of cell lineages including relatively rare T_H 17 cells, further enabling sub-population analysis into T_H 1-like and T_H 2-like phenotypes based on T-bet and GATA3 expression. Interestingly, predominance of T_H 2-like T_H 17 cells was associated with human papilloma virus (HPV)-negative status of oropharyngeal squamous cell carcinoma of head and neck, known as a poor-prognostic subtype in comparison with HPV-positive status. Furthermore, T_H 2-like T_H 17 cells in HPV-negative head and neck cancer tissues were spatiotemporally correlated with CD66b⁺ granulocytes, presumably associated with an immunosuppressive microenvironment. Our cell segmentation method for multiplex IHC/image cytometry potentially contributes to in-depth immune profiling and spatial association, leading to further tissue-based biomarker exploration. © 2019 International Society for Advancement of Cytometry.

SHIFT: speedy histopathological-to-immunofluorescent translation of whole slide images using conditional generative adversarial networks

Multiplexed imaging such as multicolor immunofluorescence staining, multiplexed immunohistochemistry (mIHC) or cyclic immunofluorescence (cycIF) enables deep assessment of cellular complexity in situ and, in conjunction with standard histology stains like hematoxylin and eosin (H&E), can help to unravel the complex molecular relationships and spatial interdependencies that undergird disease states. However, these multiplexed imaging methods are costly and can degrade both tissue quality and antigenicity with each successive cycle of staining. In addition, computationally intensive image processing such as image registration across multiple channels is required. We have developed a novel method, speedy histopathological-to-immunofluorescent translation (SHIFT) of whole slide images (WSIs) using conditional generative adversarial networks (cGANs). This approach is rooted in the assumption that specific patterns captured in IF images by stains like DAPI, pan-cytokeratin (panCK), or α-smooth muscle actin ( α-SMA) are encoded in H&E images, such that a SHIFT model can learn useful feature representations or architectural patterns in the H&E stain that help generate relevant IF stain patterns. We demonstrate that the proposed method is capable of generating realistic tumor marker IF WSIs conditioned on corresponding H&E-stained WSIs with up to 94.5% accuracy in a matter of seconds. Thus, this method has the potential to not only improve our understanding of the mapping of histological and morphological profiles into protein expression profiles, but also greatly increase the e ciency of diagnostic and prognostic decision-making.