Correlating nuclear morphometric patterns with estrogen receptor status in breast cancer pathologic specimens

A review of mechanistic learning in mathematical oncology

Mechanistic learning refers to the synergistic combination of mechanistic mathematical modeling and data-driven machine or deep learning. This emerging field finds increasing applications in (mathematical) oncology. This review aims to capture the current state of the field and provides a perspective on how mechanistic learning may progress in the oncology domain. We highlight the synergistic potential of mechanistic learning and point out similarities and differences between purely data-driven and mechanistic approaches concerning model complexity, data requirements, outputs generated, and interpretability of the algorithms and their results. Four categories of mechanistic learning (sequential, parallel, extrinsic, intrinsic) of mechanistic learning are presented with specific examples. We discuss a range of techniques including physics-informed neural networks, surrogate model learning, and digital twins. Example applications address complex problems predominantly from the domain of oncology research such as longitudinal tumor response predictions or time-to-event modeling. As the field of mechanistic learning advances, we aim for this review and proposed categorization framework to foster additional collaboration between the data- and knowledge-driven modeling fields. Further collaboration will help address difficult issues in oncology such as limited data availability, requirements of model transparency, and complex input data which are embraced in a mechanistic learning framework.

Data-Efficient Computational Pathology Platform for Faster and Cheaper Breast Cancer Subtype Identifications: Development of a Deep Learning Model

BACKGROUND

Breast cancer subtyping is a crucial step in determining therapeutic options, but the molecular examination based on immunohistochemical staining is expensive and time-consuming. Deep learning opens up the possibility to predict the subtypes based on the morphological information from hematoxylin and eosin staining, a much cheaper and faster alternative. However, training the predictive model conventionally requires a large number of histology images, which is challenging to collect by a single institute.

OBJECTIVE

We aimed to develop a data-efficient computational pathology platform, 3DHistoNet, which is capable of learning from z-stacked histology images to accurately predict breast cancer subtypes with a small sample size.

METHODS

We retrospectively examined 401 cases of patients with primary breast carcinoma diagnosed between 2018 and 2020 at the Department of Pathology, National Cancer Center, South Korea. Pathology slides of the patients with breast carcinoma were prepared according to the standard protocols. Age, gender, histologic grade, hormone receptor (estrogen receptor [ER], progesterone receptor [PR], and androgen receptor [AR]) status, erb-B2 receptor tyrosine kinase 2 (HER2) status, and Ki-67 index were evaluated by reviewing medical charts and pathological records.

RESULTS

The area under the receiver operating characteristic curve and decision curve were analyzed to evaluate the performance of our 3DHistoNet platform for predicting the ER, PR, AR, HER2, and Ki67 subtype biomarkers with 5-fold cross-validation. We demonstrated that 3DHistoNet can predict all clinically important biomarkers (ER, PR, AR, HER2, and Ki67) with performance exceeding the conventional multiple instance learning models by a considerable margin (area under the receiver operating characteristic curve: 0.75-0.91 vs 0.67-0.8). We further showed that our z-stack histology scanning method can make up for insufficient training data sets without any additional cost incurred. Finally, 3DHistoNet offered an additional capability to generate attention maps that reveal correlations between Ki67 and histomorphological features, which renders the hematoxylin and eosin image in higher fidelity to the pathologist.

CONCLUSIONS

Our stand-alone, data-efficient pathology platform that can both generate z-stacked images and predict key biomarkers is an appealing tool for breast cancer diagnosis. Its development would encourage morphology-based diagnosis, which is faster, cheaper, and less error-prone compared to the protein quantification method based on immunohistochemical staining.

Next-Generation Morphometry for pathomics-data mining in histopathology

Pathology diagnostics relies on the assessment of morphology by trained experts, which remains subjective and qualitative. Here we developed a framework for large-scale histomorphometry (FLASH) performing deep learning-based semantic segmentation and subsequent large-scale extraction of interpretable, quantitative, morphometric features in non-tumour kidney histology. We use two internal and three external, multi-centre cohorts to analyse over 1000 kidney biopsies and nephrectomies. By associating morphometric features with clinical parameters, we confirm previous concepts and reveal unexpected relations. We show that the extracted features are independent predictors of long-term clinical outcomes in IgA-nephropathy. We introduce single-structure morphometric analysis by applying techniques from single-cell transcriptomics, identifying distinct glomerular populations and morphometric phenotypes along a trajectory of disease progression. Our study provides a concept for Next-generation Morphometry (NGM), enabling comprehensive quantitative pathology data mining, i.e., pathomics.

Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction

Breast cancer is one of the precarious conditions that affect women, and a substantive cure has not yet been discovered for it. With the advent of Artificial intelligence (AI), recently, deep learning techniques have been used effectively in breast cancer detection, facilitating early diagnosis and therefore increasing the chances of patients' survival. Compared to classical machine learning techniques, deep learning requires less human intervention for similar feature extraction. This study presents a systematic literature review on the deep learning-based methods for breast cancer detection that can guide practitioners and researchers in understanding the challenges and new trends in the field. Particularly, different deep learning-based methods for breast cancer detection are investigated, focusing on the genomics and histopathological imaging data. The study specifically adopts the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which offer a detailed analysis and synthesis of the published articles. Several studies were searched and gathered, and after the eligibility screening and quality evaluation, 98 articles were identified. The results of the review indicated that the Convolutional Neural Network (CNN) is the most accurate and extensively used model for breast cancer detection, and the accuracy metrics are the most popular method used for performance evaluation. Moreover, datasets utilized for breast cancer detection and the evaluation metrics are also studied. Finally, the challenges and future research direction in breast cancer detection based on deep learning models are also investigated to help researchers and practitioners acquire in-depth knowledge of and insight into the area.

Breast Cancer Dataset, Classification and Detection Using Deep Learning

Incorporating scientific research into clinical practice via clinical informatics, which includes genomics, proteomics, bioinformatics, and biostatistics, improves patients' treatment. Computational pathology is a growing subspecialty with the potential to integrate whole slide images, multi-omics data, and health informatics. Pathology and laboratory medicine are critical to diagnosing cancer. This work will review existing computational and digital pathology methods for breast cancer diagnosis with a special focus on deep learning. The paper starts by reviewing public datasets related to breast cancer diagnosis. Additionally, existing deep learning methods for breast cancer diagnosis are reviewed. The publicly available code repositories are introduced as well. The paper is closed by highlighting challenges and future works for deep learning-based diagnosis.

Multi-modality artificial intelligence in digital pathology

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

Weakly supervised learning on unannotated hematoxylin and eosin stained slides predicts BRAF mutation in thyroid cancer with high accuracy

Deep neural networks (DNNs) that predict mutational status from H&E slides of cancers can enable inexpensive and timely precision oncology. Although expert knowledge is reliable for annotating regions informative of malignancy and other known histological patterns (strong supervision), it is unreliable for identifying regions informative of mutational status. This poses a serious impediment to obtaining higher prognostic accuracy and discovering new knowledge of pathobiology. We used a weakly supervised learning technique to train a DNN to predict BRAF V600E mutational status, determined using DNA testing, in H&E stained images of thyroid cancer tissue without regional annotations. Our discovery cohort was a tissue microarray of only 85 patients from a single hospital. Yet, on a large independent external cohort of 444 patients from other hospitals, the trained model gave an AUC = 0.98 (95% CI: 0.97-1.00), which is much higher than the previously reported results for detecting any mutation using H&E by DNNs trained using strong supervision. We also developed a visualization technique that can automatically highlight regions the DNN found most informative for predicting mutational status. Our visualization is spatially granular and highly specific in highlighting regions with strong negative and positive regions and move us towards explainable artificial intelligence. Using t-tests, we confirmed that the proportions of follicular or papillary histology and oncocytic cytology, as noted for each patient by a pathologist who was blinded to the mutational status, were significantly different between mutated and wildtype patients. However, based solely on these features noted by the pathologist, a logistic regression classifier gave an average AUC = 0.78 in 5-fold CV, which is much lower than that obtained using the DNN. These results highlight the potential of weakly supervised learning for training DNN models for problems where the informative visual patterns and their locations are not known a priori. This article is protected by copyright. All rights reserved.

Computer Vision-Based Microcalcification Detection in Digital Mammograms Using Fully Connected Depthwise Separable Convolutional Neural Network

Microcalcification clusters in mammograms are one of the major signs of breast cancer. However, the detection of microcalcifications from mammograms is a challenging task for radiologists due to their tiny size and scattered location inside a denser breast composition. Automatic CAD systems need to predict breast cancer at the early stages to support clinical work. The intercluster gap, noise between individual MCs, and individual object’s location can affect the classification performance, which may reduce the true-positive rate. In this study, we propose a computer-vision-based FC-DSCNN CAD system for the detection of microcalcification clusters from mammograms and classification into malignant and benign classes. The computer vision method automatically controls the noise and background color contrast and directly detects the MC object from mammograms, which increases the classification performance of the neural network. The breast cancer classification framework has four steps: image preprocessing and augmentation, RGB to grayscale channel transformation, microcalcification region segmentation, and MC ROI classification using FC-DSCNN to predict malignant and benign cases. The proposed method was evaluated on 3568 DDSM and 2885 PINUM mammogram images with automatic feature extraction, obtaining a score of 0.97 with a 2.35 and 0.99 true-positive ratio with 2.45 false positives per image, respectively. Experimental results demonstrated that the performance of the proposed method remains higher than the traditional and previous approaches.

Deep Learning Image Analysis of Benign Breast Disease to Identify Subsequent Risk of Breast Cancer

Background

New biomarkers of risk may improve breast cancer (BC) risk prediction. We developed a computational pathology method to segment benign breast disease (BBD) whole slide images into epithelium, fibrous stroma, and fat. We applied our method to the BBD BC nested case-control study within the Nurses' Health Studies to assess whether computer-derived tissue composition or a morphometric signature was associated with subsequent risk of BC.

Methods

Tissue segmentation and nuclei detection deep-learning networks were established and applied to 3795 whole slide images from 293 cases who developed BC and 1132 controls who did not. Percentages of each tissue region were calculated, and 615 morphometric features were extracted. Elastic net regression was used to create a BC morphometric signature. Associations between BC risk factors and age-adjusted tissue composition among controls were assessed using analysis of covariance. Unconditional logistic regression, adjusting for the matching factors, BBD histological subtypes, parity, menopausal status, and body mass index evaluated the relationship between tissue composition and BC risk. All statistical tests were 2-sided.

Results

Among controls, direction of associations between BBD subtypes, parity, and number of births with breast composition varied by tissue region; select regions were associated with childhood body size, body mass index, age of menarche, and menopausal status (all P < .05). A higher proportion of epithelial tissue was associated with increased BC risk (odds ratio = 1.39, 95% confidence interval = 0.91 to 2.14, for highest vs lowest quartiles, P trend  = .047). No morphometric signature was associated with BC.

Conclusions

The amount of epithelial tissue may be incorporated into risk assessment models to improve BC risk prediction.

Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains

For newly diagnosed breast cancer, estrogen receptor status (ERS) is a key molecular marker used for prognosis and treatment decisions. During clinical management, ERS is determined by pathologists from immunohistochemistry (IHC) staining of biopsied tissue for the targeted receptor, which highlights the presence of cellular surface antigens. This is an expensive, time-consuming process which introduces discordance in results due to variability in IHC preparation and pathologist subjectivity. In contrast, hematoxylin and eosin (H&E) staining—which highlights cellular morphology—is quick, less expensive, and less variable in preparation. Here we show that machine learning can determine molecular marker status, as assessed by hormone receptors, directly from cellular morphology. We develop a multiple instance learning-based deep neural network that determines ERS from H&E-stained whole slide images (WSI). Our algorithm—trained strictly with WSI-level annotations—is accurate on a varied, multi-country dataset of 3,474 patients, achieving an area under the curve (AUC) of 0.92 for sensitivity and specificity. Our approach has the potential to augment clinicians’ capabilities in cancer prognosis and theragnosis by harnessing biological signals imperceptible to the human eye.

Determination of estrogen receptor status (ERS) in breast cancer tissue requires immunohistochemistry, which is sensitive to the vagaries of sample processing and the subjectivity of pathologists. Here the authors present a deep learning model that determines ERS from H&E stained tissue, which could improve oncology decisions in under-resourced settings.

A deep learning model to predict RNA-Seq expression of tumours from whole slide images

Deep learning methods for digital pathology analysis are an effective way to address multiple clinical questions, from diagnosis to prediction of treatment outcomes. These methods have also been used to predict gene mutations from pathology images, but no comprehensive evaluation of their potential for extracting molecular features from histology slides has yet been performed. We show that HE2RNA, a model based on the integration of multiple data modes, can be trained to systematically predict RNA-Seq profiles from whole-slide images alone, without expert annotation. Through its interpretable design, HE2RNA provides virtual spatialization of gene expression, as validated by CD3- and CD20-staining on an independent dataset. The transcriptomic representation learned by HE2RNA can also be transferred on other datasets, even of small size, to increase prediction performance for specific molecular phenotypes. We illustrate the use of this approach in clinical diagnosis purposes such as the identification of tumors with microsatellite instability.

RNA-sequencing of tumour tissue can provide important diagnostic and prognostic information but this is costly and not routinely performed in all clinical settings. Here, the authors show that whole slide histology slides—part of routine care—can be used to predict RNA-sequencing data and thus reduce the need for additional analyses.

Deep Learning to Estimate Human Epidermal Growth Factor Receptor 2 Status from Hematoxylin and Eosin-Stained Breast Tissue Images

Context:

Several therapeutically important mutations in cancers are economically detected using immunohistochemistry (IHC), which highlights the overexpression of specific antigens associated with the mutation. However, IHC panels can be imprecise and relatively expensive in low-income settings. On the other hand, although hematoxylin and eosin (H&E) staining used to visualize the general tissue morphology is a routine and low cost, it does not highlight any specific antigen or mutation.

Aims:

Using the human epidermal growth factor receptor 2 (HER2) mutation in breast cancer as an example, we strengthen the case for cost-effective detection and screening of overexpression of HER2 protein in H&E-stained tissue.

Settings and Design:

We use computational methods that reliably detect subtle morphological changes associated with the over-expression of mutation-specific proteins directly from H&E images.

Subjects and Methods:

We trained a classification pipeline to determine HER2 overexpression status of H&E stained whole slide images. Our training dataset was derived from a single hospital containing 26 (11 HER2+ and 15 HER2–) cases. We tested the classification pipeline on 26 (8 HER2+ and 18 HER2–) held-out cases from the same hospital and 45 independent cases (23 HER2+ and 22 HER2–) from the TCGA-BRCA cohort. The pipeline was composed of a stain separation module and three deep neural network modules in tandem for robustness and interpretability.

Statistical Analysis Used:

We evaluate our trained model through area under the curve (AUC)-receiver operating characteristic.

Results:

Our pipeline achieved an AUC of 0.82 (confidence interval [CI]: 0.65–0.98) on held-out cases and an AUC of 0.76 (CI: 0.61–0.89) on the independent dataset from TCGA. We also demonstrate the region-level correspondence of HER2 overexpression between a patient's IHC and H&E serial sections.

Conclusions:

Our work strengthens the case for automatically quantifying the overexpression of mutation-specific proteins in H&E-stained digital pathology, and it highlights the importance of multi-stage machine learning pipelines for added robustness and interpretability.

A Machine-learning Approach for the Assessment of the Proliferative Compartment of Solid Tumors on Hematoxylin-Eosin-Stained Sections

We introduce a machine learning-based analysis to predict the immunohistochemical (IHC) labeling index for the cell proliferation marker Ki67/MIB1 on cancer tissues based on morphometrical features extracted from hematoxylin and eosin (H&E)-stained formalin-fixed, paraffin-embedded (FFPE) tumor tissue samples. We provided a proof-of-concept prediction of the Ki67/MIB1 IHC positivity of cancer cells through the definition and quantitation of single nuclear features. In the first instance, we set our digital framework on Ki67/MIB1-stained OSCC (oral squamous cell carcinoma) tissue sample whole slide images, using QuPath as a working platform and its integrated algorithms, and we built a classifier in order to distinguish tumor and stroma classes and, within them, Ki67-positive and Ki67-negative cells; then, we sorted the morphometric features of tumor cells related to their Ki67 IHC status. Among the evaluated features, nuclear hematoxylin mean optical density (NHMOD) presented as the best one to distinguish Ki67/MIB1 positive from negative cells. We confirmed our findings in a single-cell level analysis of H&E staining on Ki67-immunostained/H&E-decolored tissue samples. Finally, we tested our digital framework on a case series of oral squamous cell carcinomas (OSCC), arranged in tissue microarrays; we selected two consecutive sections of each OSCC FFPE TMA (tissue microarray) block, respectively stained with H&E and immuno-stained for Ki67/MIB1. We automatically detected tumor cells in H&E slides and generated a “false color map” (FCM) based on NHMOD through the QuPath measurements map tool. FCM nearly coincided with the actual immunohistochemical result, allowing the prediction of Ki67/MIB1 positive cells in a direct visual fashion. Our proposed approach provides the pathologist with a fast method of identifying the proliferating compartment of the tumor through a quantitative assessment of the nuclear features on H&E slides, readily appreciable by visual inspection. Although this technique needs to be fine-tuned and tested on larger series of tumors, the digital analysis approach appears to be a promising tool to quickly forecast the tumor’s proliferation fraction directly on routinely H&E-stained digital sections.

Deep learned tissue "fingerprints" classify breast cancers by ER/PR/Her2 status from H&E images

Because histologic types are subjective and difficult to reproduce between pathologists, tissue morphology often takes a back seat to molecular testing for the selection of breast cancer treatments. This work explores whether a deep-learning algorithm can learn objective histologic H&E features that predict the clinical subtypes of breast cancer, as assessed by immunostaining for estrogen, progesterone, and Her2 receptors (ER/PR/Her2). Translating deep learning to this and related problems in histopathology presents a challenge due to the lack of large, well-annotated data sets, which are typically required for the algorithms to learn statistically significant discriminatory patterns. To overcome this limitation, we introduce the concept of “tissue fingerprints,” which leverages large, unannotated datasets in a label-free manner to learn H&E features that can distinguish one patient from another. The hypothesis is that training the algorithm to learn the morphological differences between patients will implicitly teach it about the biologic variation between them. Following this training internship, we used the features the network learned, which we call “fingerprints,” to predict ER, PR, and Her2 status in two datasets. Despite the discovery dataset being relatively small by the standards of the machine learning community (n = 939), fingerprints enabled the determination of ER, PR, and Her2 status from whole slide H&E images with 0.89 AUC (ER), 0.81 AUC (PR), and 0.79 AUC (Her2) on a large, independent test set (n = 2531). Tissue fingerprints are concise but meaningful histopathologic image representations that capture biological information and may enable machine learning algorithms that go beyond the traditional ER/PR/Her2 clinical groupings by directly predicting theragnosis.

Radiomics-based convolutional neural network for brain tumor segmentation on multiparametric magnetic resonance imaging

Accurate segmentation of gliomas on routine magnetic resonance image (MRI) scans plays an important role in disease diagnosis, prognosis, and patient treatment planning. We present a fully automated approach, radiomics-based convolutional neural network (RadCNN), for segmenting both high- and low-grade gliomas using multimodal MRI volumes (T1c, T2w, and FLAIR). RadCNN incorporates radiomic texture features (i.e., Haralick, Gabor, and Laws) within DeepMedic [a deep 3-D convolutional neural network (CNN) segmentation framework that uses image intensities; a top performing method in the BraTS 2016 challenge] to further augment the performance of brain tumor subcompartment segmentation. We first identify textural radiomic representations that best separate the different subcompartments [enhancing tumor (ET), whole tumor (WT), and tumor core (TC)] on the training set, and then feed these representations as inputs to the CNN classifier for prediction of different subcompartments. We hypothesize that textural radiomic representations of lesion subcompartments will enhance the separation of subcompartment boundaries, and hence providing these features as inputs to the deep CNN, over and above raw intensity values alone, will improve the subcompartment segmentation. Using a training set of N = 241 patients, validation set of N = 44 , and test set of N = 46 patients, RadCNN method achieved Dice similarity coefficient (DSC) scores of 0.71, 0.89, and 0.73 for ET, WT, and TC, respectively. Compared to the DeepMedic model, RadCNN showed improvement in DSC scores for both ET and WT and demonstrated comparable results in segmenting the TC. Similarly, smaller Hausdorff distance measures were obtained with RadCNN as compared to the DeepMedic model across all the subcompartments. Following the segmentation of the different subcompartments, we extracted a set of subcompartment specific radiomic descriptors that capture lesion disorder and assessed their ability in separating patients into different survival cohorts (short-, mid- and long-term survival) based on their overall survival from the date of baseline diagnosis. Using a multilinear regression approach, we achieved accuracies of 0.57, 0.63, and 0.45 for the training, validation, and test cases, respectively.

Higher order genomic organization and epigenetic control maintain cellular identity and prevent breast cancer

Cells establish and sustain structural and functional integrity of the genome to support cellular identity and prevent malignant transformation. In this review, we present a strategic overview of epigenetic regulatory mechanisms including histone modifications and higher order chromatin organization (HCO) that are perturbed in breast cancer onset and progression. Implications for dysfunctions that occur in hormone regulation, cell cycle control, and mitotic bookmarking in breast cancer are considered, with an emphasis on epithelial-to-mesenchymal transition and cancer stem cell activities. The architectural organization of regulatory machinery is addressed within the contexts of translating cancer-compromised genomic organization to advances in breast cancer risk assessment, diagnosis, prognosis, and identification of novel therapeutic targets with high specificity and minimal off target effects.