Deep learning-based tumor microenvironment analysis in colon adenocarcinoma histopathological whole-slide images

Deep Learning-Powered Whole Slide Image Analysis in Cancer Pathology

Pathology is the cornerstone of modern cancer care. With the advancement of precision oncology, the demand for histopathologic diagnosis and stratification of patients is increasing as personalized cancer therapy relies on accurate biomarker assessment. Recently, rapid development of whole slide imaging technology has enabled digitalization of traditional histologic slides at high resolution, holding promise to improve both the precision and efficiency of histopathologic evaluation. In particular, deep learning approaches, such as Convolutional Neural Network, Graph Convolutional Network, and Transformer, have shown great promise in enhancing the sensitivity and accuracy of whole slide image (WSI) analysis in cancer pathology because of their ability to handle high-dimensional and complex image data. The integration of deep learning models with WSIs enables us to explore and mine morphologic features beyond the visual perception of pathologists, which can help automate clinical diagnosis, assess histopathologic grade, predict clinical outcomes, and even discover novel morphologic biomarkers. In this review, we present a comprehensive framework for incorporating deep learning with WSIs, highlighting how deep learning-driven WSI analysis advances clinical tasks in cancer care. Furthermore, we critically discuss the opportunities and challenges of translating deep learning-based digital pathology into clinical practice, which should be considered to support personalized treatment of cancer patients.

Artificial Intelligence-Based Single-Cell Analysis as a Next-Generation Histologic Grading Approach in Colorectal Cancer: Prognostic Role and Tumor Biology Assessment

The management of colorectal carcinoma (CRC) relies on pathological interpretation. Digital pathology approaches allow for development of new potent artificial intelligence-based prognostic parameters. The study aimed to develop an artificial intelligence-based image analysis platform allowing fully automatized, quantitative, and explainable tumor microenvironment analysis and extraction of prognostic information from hematoxylin and eosin-stained whole-slide images of CRC patients. Three well--characterized, multi-institutional patient cohorts were included (patient n = 1438, whole-slide image n > 2400). The developed image analysis platform implements quality control and established algorithms to segment tissue and detect cell types. It enabled systematic analysis of immune infiltrate, assessing its prognostic relevance, intratumoral heterogeneity, and biological concepts across multiple survival end points. Analyzing single-cell types and their combinations reveals independent, prognostic parameters, highlighting significant intratumoral heterogeneity, especially in the biopsy setting, which must be accounted for. A key morphologic concept related to tumor control by the immune system is described, resulting in a capable, independent prognostic parameter (tumor "out of control"). Our findings have direct clinical implications and can be used as a foundation for updating the existing CRC grading systems.

ECMHA-PP: A Breast Cancer Prognosis Prediction Model Based on Energy-Constrained Multi-Head Self-Attention

PURPOSE

Breast cancer is a significant threat to women's health. Precise prognosis prediction for breast cancer can help doctors implement more rational treatment strategies. Artificial intelligence can assist doctors in decision-making and enhance prediction accuracy.

EXPERIMENTAL DESIGN

In this paper, a deep learning model ECMHA-PP (Energy Constrained Multi-Head Self-Attention based Prognosis Prediction) is proposed to predict the prognosis of breast cancer. ECMHA-PP utilizes patients' clinical data and extracts features through a cross-position mix and a channel mix multi-layer perceptron. Then, it incorporates an energy-constrained multi-head self-attention layer to improve feature extraction capability. The source code of ECMHA-PP has been hosted on GitHub and is available at https://github.com/xiaoliu166370/ECMHA-PP.

RESULTS

To evaluate our proposed method, prognostic prediction experiments were performed on the METABRIC dataset, yielding outstanding results with an average accuracy of 93.0% and an average area under the curve of 0.974. To further validate the model's performance, we conducted tests on another independent dataset, BRCA, achieving an accuracy of 87.6%.

CONCLUSIONS AND CLINICAL RELEVANCE

In comparison with other widely used advanced methods, ECMHA-PP demonstrated higher comprehensive performance, making it a reliable prognostic prediction model for breast cancer. Given its robust feature extraction and prediction capabilities.

NACNet: A histology context-aware transformer graph convolution network for predicting treatment response to neoadjuvant chemotherapy in Triple Negative Breast Cancer

Neoadjuvant chemotherapy (NAC) response prediction for triple negative breast cancer (TNBC) patients is a challenging task clinically as it requires understanding complex histology interactions within the tumor microenvironment (TME). Digital whole slide images (WSIs) capture detailed tissue information, but their giga-pixel size necessitates computational methods based on multiple instance learning, which typically analyze small, isolated image tiles without the spatial context of the TME. To address this limitation and incorporate TME spatial histology interactions in predicting NAC response for TNBC patients, we developed a histology context-aware transformer graph convolution network (NACNet). Our deep learning method identifies the histopathological labels on individual image tiles from WSIs, constructs a spatial TME graph, and represents each node with features derived from tissue texture and social network analysis. It predicts NAC response using a transformer graph convolution network model enhanced with graph isomorphism network layers. We evaluate our method with WSIs of a cohort of TNBC patient (N=105) and compared its performance with multiple state-of-the-art machine learning and deep learning models, including both graph and non-graph approaches. Our NACNet achieves 90.0% accuracy, 96.0% sensitivity, 88.0% specificity, and an AUC of 0.82, through eight-fold cross-validation, outperforming baseline models. These comprehensive experimental results suggest that NACNet holds strong potential for stratifying TNBC patients by NAC response, thereby helping to prevent overtreatment, improve patient quality of life, reduce treatment cost, and enhance clinical outcomes, marking an important advancement toward personalized breast cancer treatment.

Signal enhancement ratio of multi-phase contrast-enhanced MRI: an imaging biomarker for survival in pancreatic adenocarcinoma

OBJECTIVES

To evaluate signal enhancement ratio (SER) for tissue characterization and prognosis stratification in pancreatic adenocarcinoma (PDAC), with quantitative histopathological analysis (QHA) as the reference standard.

METHODS

This retrospective study included 277 PDAC patients who underwent multi-phase contrast-enhanced (CE) MRI and whole-slide imaging (WSI) from three centers (2015-2021). SER is defined as (SIlt - SIpre)/(SIea - SIpre), where SIpre, SIea, and SIlt represent the signal intensity of the tumor in pre-contrast, early-, and late post-contrast images, respectively. Deep-learning algorithms were implemented to quantify the stroma, epithelium, and lumen of PDAC on WSIs. Correlation, regression, and Bland-Altman analyses were utilized to investigate the associations between SER and QHA. The prognostic significance of SER on overall survival (OS) was evaluated using Cox regression analysis and Kaplan-Meier curves.

RESULTS

The internal dataset comprised 159 patients, which was further divided into training, validation, and internal test datasets (n = 60, 41, and 58, respectively). Sixty-five and 53 patients were included in two external test datasets. Excluding lumen, SER demonstrated significant correlations with stroma (r = 0.29-0.74, all p < 0.001) and epithelium (r = -0.23 to -0.71, all p < 0.001) across a wide post-injection time window (range, 25-300 s). Bland-Altman analysis revealed a small bias between SER and QHA for quantifying stroma/epithelium in individual training, validation (all within ± 2%), and three test datasets (all within ± 4%). Moreover, SER-predicted low stromal proportion was independently associated with worse OS (HR = 1.84 (1.17-2.91), p = 0.009) in training and validation datasets, which remained significant across three combined test datasets (HR = 1.73 (1.25-2.41), p = 0.001).

CONCLUSION

SER of multi-phase CE-MRI allows for tissue characterization and prognosis stratification in PDAC.

CLINICAL RELEVANCE STATEMENT

The signal enhancement ratio of multi-phase CE-MRI can serve as a novel imaging biomarker for characterizing tissue composition and holds the potential for improving patient stratification and therapy in PDAC.

KEY POINTS

Imaging biomarkers are needed to better characterize tumor tissue in pancreatic adenocarcinoma. Signal enhancement ratio (SER)-predicted stromal/epithelial proportion showed good agreement with histopathology measurements across three distinct centers. Signal enhancement ratio (SER)-predicted stromal proportion was demonstrated to be an independent prognostic factor for OS in PDAC.

Novel Artificial Intelligence Combining Convolutional Neural Network and Support Vector Machine to Predict Colorectal Cancer Prognosis and Mutational Signatures From Hematoxylin and Eosin Images

Reducing recurrence following radical resection of colon cancer without overtreatment or undertreatment remains a challenge. Postoperative adjuvant chemotherapy (Adj) is currently administered based solely on pathologic TNM stage. However, prognosis can vary significantly among patients with the same disease stage. Therefore, novel classification systems in addition to the TNM are necessary to inform decision-making regarding postoperative treatment strategies, especially stage II and III disease, and minimize overtreatment and undertreatment with Adj. We developed a prognostic prediction system for colorectal cancer using a combined convolutional neural network and support vector machine approach to extract features from hematoxylin and eosin staining images. We combined the TNM and our artificial intelligence (AI)-based classification system into a modified TNM-AI classification system with high discriminative power for recurrence-free survival. Furthermore, the cancer cell population recognized by this system as low risk of recurrence exhibited the mutational signature SBS87 as a genetic phenotype. The novel AI-based classification system developed here is expected to play an important role in prognostic prediction and personalized treatment selection in oncology.

Transformaer-based model for lung adenocarcinoma subtypes

BACKGROUND

Lung cancer has the highest morbidity and mortality rate among all types of cancer. Histological subtypes serve as crucial markers for the development of lung cancer and possess significant clinical values for cancer diagnosis, prognosis, and prediction of treatment responses. However, existing studies only dichotomize normal and cancerous tissues, failing to capture the unique characteristics of tissue sections and cancer types.

PURPOSE

Therefore, we have pioneered the classification of lung adenocarcinoma (LAD) cancer tissues into five subtypes (acinar, lepidic, micropapillary, papillary, and solid) based on section data in whole-slide image sections. In addition, a novel model called HybridNet was designed to improve the classification performance.

METHODS

HybridNet primarily consists of two interactive streams: a Transformer and a convolutional neural network (CNN). The Transformer stream captures rich global representations using a self-attention mechanism, while the CNN stream extracts local semantic features to optimize image details. Specifically, during the dual-stream parallelism, the feature maps of the Transformer stream as weights are weighted and summed with those of the CNN stream backbone; at the end of the parallelism, the respective final features are concatenated to obtain more discriminative semantic information.

RESULTS

Experimental results on a private dataset of LAD showed that HybridNet achieved 95.12% classification accuracy, and the accuracy of five histological subtypes (acinar, lepidic, micropapillary, papillary, and solid) reached 94.5%, 97.1%, 94%, 91%, and 99% respectively; the experimental results on the public BreakHis dataset show that HybridNet achieves the best results in three evaluation metrics: accuracy, recall and F1-score, with 92.40%, 90.63%, and 91.43%, respectively.

CONCLUSIONS

The process of classifying LAD into five subtypes assists pathologists in selecting appropriate treatments and enables them to predict tumor mutation burden (TMB) and analyze the spatial distribution of immune checkpoint proteins based on this and other clinical data. In addition, the proposed HybridNet fuses CNN and Transformer information several times and is able to improve the accuracy of subtype classification, and also shows satisfactory performance on public datasets with some generalization ability.

From Pixels to Prognosis: A Survey on AI-Driven Cancer Patient Survival Prediction Using Digital Histology Images

Survival analysis is an integral part of medical statistics that is extensively utilized to establish prognostic indices for mortality or disease recurrence, assess treatment efficacy, and tailor effective treatment plans. The identification of prognostic biomarkers capable of predicting patient survival is a primary objective in the field of cancer research. With the recent integration of digital histology images into routine clinical practice, a plethora of Artificial Intelligence (AI)-based methods for digital pathology has emerged in scholarly literature, facilitating patient survival prediction. These methods have demonstrated remarkable proficiency in analyzing and interpreting whole slide images, yielding results comparable to those of expert pathologists. The complexity of AI-driven techniques is magnified by the distinctive characteristics of digital histology images, including their gigapixel size and diverse tissue appearances. Consequently, advanced patch-based methods are employed to effectively extract features that correlate with patient survival. These computational methods significantly enhance survival prediction accuracy and augment prognostic capabilities in cancer patients. The review discusses the methodologies employed in the literature, their performance metrics, ongoing challenges, and potential solutions for future advancements. This paper explains survival analysis and feature extraction methods for analyzing cancer patients. It also compiles essential acronyms related to cancer precision medicine. Furthermore, it is noteworthy that this is the inaugural review paper in the field. The target audience for this interdisciplinary review comprises AI practitioners, medical statisticians, and progressive oncologists who are enthusiastic about translating AI-driven solutions into clinical practice. We expect this comprehensive review article to guide future research directions in the field of cancer research.

Combined expert-in-the-loop-random forest multiclass segmentation U-net based artificial intelligence model: evaluation of non-small cell lung cancer in fibrotic and non-fibrotic microenvironments

BACKGROUND

The tumor microenvironment (TME) plays a key role in lung cancer initiation, proliferation, invasion, and metastasis. Artificial intelligence (AI) methods could potentially accelerate TME analysis. The aims of this study were to (1) assess the feasibility of using hematoxylin and eosin (H&E)-stained whole slide images (WSI) to develop an AI model for evaluating the TME and (2) to characterize the TME of adenocarcinoma (ADCA) and squamous cell carcinoma (SCCA) in fibrotic and non-fibrotic lung.

METHODS

The cohort was derived from chest CT scans of patients presenting with lung neoplasms, with and without background fibrosis. WSI images were generated from slides of all 76 available pathology cases with ADCA (n = 53) or SCCA (n = 23) in fibrotic (n = 47) or non-fibrotic (n = 29) lung. Detailed ground-truth annotations, including of stroma (i.e., fibrosis, vessels, inflammation), necrosis and background, were performed on WSI and optimized via an expert-in-the-loop (EITL) iterative procedure using a lightweight [random forest (RF)] classifier. A convolution neural network (CNN)-based model was used to achieve tissue-level multiclass segmentation. The model was trained on 25 annotated WSI from 13 cases of ADCA and SCCA within and without fibrosis and then applied to the 76-case cohort. The TME analysis included tumor stroma ratio (TSR), tumor fibrosis ratio (TFR), tumor inflammation ratio (TIR), tumor vessel ratio (TVR), tumor necrosis ratio (TNR), and tumor background ratio (TBR).

RESULTS

The model's overall classification for precision, sensitivity, and F1-score were 94%, 90%, and 91%, respectively. Statistically significant differences were noted in TSR (p = 0.041) and TFR (p = 0.001) between fibrotic and non-fibrotic ADCA. Within fibrotic lung, statistically significant differences were present in TFR (p = 0.039), TIR (p = 0.003), TVR (p = 0.041), TNR (p = 0.0003), and TBR (p = 0.020) between ADCA and SCCA.

CONCLUSION

The combined EITL-RF CNN model using only H&E WSI can facilitate multiclass evaluation and quantification of the TME. There are significant differences in the TME of ADCA and SCCA present within or without background fibrosis. Future studies are needed to determine the significance of TME on prognosis and treatment.

Image-based artificial intelligence for the prediction of pathological complete response to neoadjuvant chemoradiotherapy in patients with rectal cancer: a systematic review and meta-analysis

OBJECTIVE

Artificial intelligence (AI) holds enormous potential for noninvasively identifying patients with rectal cancer who could achieve pathological complete response (pCR) following neoadjuvant chemoradiotherapy (nCRT). We aimed to conduct a meta-analysis to summarize the diagnostic performance of image-based AI models for predicting pCR to nCRT in patients with rectal cancer.

METHODS

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A literature search of PubMed, Embase, Cochrane Library, and Web of Science was performed from inception to July 29, 2023. Studies that developed or utilized AI models for predicting pCR to nCRT in rectal cancer from medical images were included. The Quality Assessment of Diagnostic Accuracy Studies-AI was used to appraise the methodological quality of the studies. The bivariate random-effects model was used to summarize the individual sensitivities, specificities, and areas-under-the-curve (AUCs). Subgroup and meta-regression analyses were conducted to identify potential sources of heterogeneity. Protocol for this study was registered with PROSPERO (CRD42022382374).

RESULTS

Thirty-four studies (9933 patients) were identified. Pooled estimates of sensitivity, specificity, and AUC of AI models for pCR prediction were 82% (95% CI: 76-87%), 84% (95% CI: 79-88%), and 90% (95% CI: 87-92%), respectively. Higher specificity was seen for the Asian population, low risk of bias, and deep-learning, compared with the non-Asian population, high risk of bias, and radiomics (all P < 0.05). Single-center had a higher sensitivity than multi-center (P = 0.001). The retrospective design had lower sensitivity (P = 0.012) but higher specificity (P < 0.001) than the prospective design. MRI showed higher sensitivity (P = 0.001) but lower specificity (P = 0.044) than non-MRI. The sensitivity and specificity of internal validation were higher than those of external validation (both P = 0.005).

CONCLUSIONS

Image-based AI models exhibited favorable performance for predicting pCR to nCRT in rectal cancer. However, further clinical trials are warranted to verify the findings.

From pixels to patient care: deep learning-enabled pathomics signature offers precise outcome predictions for immunotherapy in esophageal squamous cell cancer

BACKGROUND

Immunotherapy has significantly improved survival of esophageal squamous cell cancer (ESCC) patients, however the clinical benefit was limited to only a small portion of patients. This study aimed to perform a deep learning signature based on H&E-stained pathological specimens to accurately predict the clinical benefit of PD-1 inhibitors in ESCC patients.

METHODS

ESCC patients receiving PD-1 inhibitors from Shandong Cancer Hospital were included. WSI images of H&E-stained histological specimens of included patients were collected, and randomly divided into training (70%) and validation (30%) sets. The labels of images were defined by the progression-free survival (PFS) with the interval of 4 months. The pretrained ViT model was used for patch-level model training, and all patches were projected into probabilities after linear classifier. Then the most predictive patches were passed to RNN for final patient-level prediction to construct ESCC-pathomics signature (ESCC-PS). Accuracy rate and survival analysis were performed to evaluate the performance of ViT-RNN survival model in validation cohort.

RESULTS

163 ESCC patients receiving PD-1 inhibitors were included for model training. There were 486,188 patches of 1024*1024 pixels from 324 WSI images of H&E-stained histological specimens after image pre-processing. There were 120 patients with 227 images in training cohort and 43 patients with 97 images in validation cohort, with balanced baseline characteristics between two groups. The ESCC-PS achieved an accuracy of 84.5% in the validation cohort, and could distinguish patients into three risk groups with the median PFS of 2.6, 4.5 and 12.9 months (P < 0.001). The multivariate cox analysis revealed ESCC-PS could act as an independent predictor of survival from PD-1 inhibitors (P < 0.001). A combined signature incorporating ESCC-PS and expression of PD-L1 shows significantly improved accuracy in outcome prediction of PD-1 inhibitors compared to ESCC-PS and PD-L1 anlone, with the area under curve value of 0.904, 0.924, 0.610 for 6-month PFS and C-index of 0.814, 0.806, 0.601, respectively.

CONCLUSIONS

The outcome supervised pathomics signature based on deep learning has the potential to enable superior prognostic stratification of ESCC patients receiving PD-1 inhibitors, which convert the images pixels to an effective and labour-saving tool to optimize clinical management of ESCC patients.

AI-powered interpretable imaging phenotypes noninvasively characterize tumor microenvironment associated with diverse molecular signatures and survival in breast cancer

BACKGROUND AND OBJECTIVES

Tumor microenvironment (TME) is a determining factor in decision-making and personalized treatment for breast cancer, which is highly intra-tumor heterogeneous (ITH). However, the noninvasive imaging phenotypes of TME are poorly understood, even invasive genotypes have been largely known in breast cancer.

METHODS

Here, we develop an artificial intelligence (AI)-driven approach for noninvasively characterizing TME by integrating the predictive power of deep learning with the explainability of human-interpretable imaging phenotypes (IMPs) derived from 4D dynamic imaging (DCE-MRI) of 342 breast tumors linked to genomic and clinical data, which connect cancer phenotypes to genotypes. An unsupervised dual-attention deep graph clustering model (DGCLM) is developed to divide bulk tumor into multiple spatially segregated and phenotypically consistent subclusters. The IMPs ranging from spatial heterogeneity to kinetic heterogeneity are leveraged to capture architecture, interaction, and proximity between intratumoral subclusters.

RESULTS

We demonstrate that our IMPs correlate with well-known markers of TME and also can predict distinct molecular signatures, including expression of hormone receptor, epithelial growth factor receptor and immune checkpoint proteins, with the performance of accuracy, reliability and transparency superior to recent state-of-the-art radiomics and 'black-box' deep learning methods. Moreover, prognostic value is confirmed by survival analysis accounting for IMPs.

CONCLUSIONS

Our approach provides an interpretable, quantitative, and comprehensive perspective to characterize TME in a noninvasive and clinically relevant manner.

Treatment response prediction of neoadjuvant chemotherapy for rectal cancer by deep learning of colonoscopy images

In current clinical practice, several treatment methods, including neoadjuvant therapy, are being developed to improve overall survival or local recurrence rates for locally advanced rectal cancer. The response to neoadjuvant therapy is usually evaluated using imaging data collected before and after preoperative treatment or postsurgical pathological diagnosis. However, there is a need to accurately predict the response to preoperative treatment before treatment is administered. The present study used a deep learning network to examine colonoscopy images and construct a model to predict the response of rectal cancer to neoadjuvant chemotherapy. A total of 53 patients who underwent preoperative chemotherapy followed by radical resection for advanced rectal cancer at the Osaka University Hospital between January 2011 and August 2019 were retrospectively analyzed. A convolutional neural network model was constructed using 403 images from 43 patients as the learning set. The diagnostic accuracy of the deep learning model was evaluated using 84 images from 10 patients as the validation set. The model demonstrated a sensitivity, specificity, accuracy, positive predictive value and area under the curve of 77.6% (38/49), 62.9% (22/33), 71.4% (60/84), 74.5% (38/51) and 0.713, respectively, in predicting a poor response to neoadjuvant therapy. Overall, deep learning of colonoscopy images may contribute to an accurate prediction of the response of rectal cancer to neoadjuvant chemotherapy.

Colon histology slide classification with deep-learning framework using individual and fused features

Cancer occurrence rates are gradually rising in the population, which reasons a heavy diagnostic burden globally. The rate of colorectal (bowel) cancer (CC) is gradually rising, and is currently listed as the third most common cancer globally. Therefore, early screening and treatments with a recommended clinical protocol are necessary to trat cancer. The proposed research aim of this paper to develop a Deep-Learning Framework (DLF) to classify the colon histology slides into normal/cancer classes using deep-learning-based features. The stages of the framework include the following: (ⅰ) Image collection, resizing, and pre-processing; (ⅱ) Deep-Features (DF) extraction with a chosen scheme; (ⅲ) Binary classification with a 5-fold cross-validation; and (ⅳ) Verification of the clinical significance. This work classifies the considered image database using the follwing: (ⅰ) Individual DF, (ⅱ) Fused DF, and (ⅲ) Ensemble DF. The achieved results are separately verified using binary classifiers. The proposed work considered 4000 (2000 normal and 2000 cancer) histology slides for the examination. The result of this research confirms that the fused DF helps to achieve a detection accuracy of 99% with the K-Nearest Neighbor (KNN) classifier. In contrast, the individual and ensemble DF provide classification accuracies of 93.25 and 97.25%, respectively.

Recent Advancements in Deep Learning Using Whole Slide Imaging for Cancer Prognosis

This review furnishes an exhaustive analysis of the latest advancements in deep learning techniques applied to whole slide images (WSIs) in the context of cancer prognosis, focusing specifically on publications from 2019 through 2023. The swiftly maturing field of deep learning, in combination with the burgeoning availability of WSIs, manifests significant potential in revolutionizing the predictive modeling of cancer prognosis. In light of the swift evolution and profound complexity of the field, it is essential to systematically review contemporary methodologies and critically appraise their ramifications. This review elucidates the prevailing landscape of this intersection, cataloging major developments, evaluating their strengths and weaknesses, and providing discerning insights into prospective directions. In this paper, a comprehensive overview of the field aims to be presented, which can serve as a critical resource for researchers and clinicians, ultimately enhancing the quality of cancer care outcomes. This review's findings accentuate the need for ongoing scrutiny of recent studies in this rapidly progressing field to discern patterns, understand breakthroughs, and navigate future research trajectories.

CoADS: Cross attention based dual-space graph network for survival prediction of lung cancer using whole slide images

BACKGROUND AND OBJECTIVE

Accurate overall survival (OS) prediction for lung cancer patients is of great significance, which can help classify patients into different risk groups to benefit from personalized treatment. Histopathology slides are considered the gold standard for cancer diagnosis and prognosis, and many algorithms have been proposed to predict the OS risk. Most methods rely on selecting key patches or morphological phenotypes from whole slide images (WSIs). However, OS prediction using the existing methods exhibits limited accuracy and remains challenging.

METHODS

In this paper, we propose a novel cross-attention-based dual-space graph convolutional neural network model (CoADS). To facilitate the improvement of survival prediction, we fully take into account the heterogeneity of tumor sections from different perspectives. CoADS utilizes the information from both physical and latent spaces. With the guidance of cross-attention, both the spatial proximity in physical space and the feature similarity in latent space between different patches from WSIs are integrated effectively.

RESULTS

We evaluated our approach on two large lung cancer datasets of 1044 patients. The extensive experimental results demonstrated that the proposed model outperforms state-of-the-art methods with the highest concordance index.

CONCLUSIONS

The qualitative and quantitative results show that the proposed method is more powerful for identifying the pathology features associated with prognosis. Furthermore, the proposed framework can be extended to other pathological images for predicting OS or other prognosis indicators, and thus delivering individualized treatment.

A bibliometric analysis of worldwide cancer research using machine learning methods

With the progress and development of computer technology, applying machine learning methods to cancer research has become an important research field. To analyze the most recent research status and trends, main research topics, topic evolutions, research collaborations, and potential directions of this research field, this study conducts a bibliometric analysis on 6206 research articles worldwide collected from PubMed between 2011 and 2021 concerning cancer research using machine learning methods. Python is used as a tool for bibliometric analysis, Gephi is used for social network analysis, and the Latent Dirichlet Allocation model is used for topic modeling. The trend analysis of articles not only reflects the innovative research at the intersection of machine learning and cancer but also demonstrates its vigorous development and increasing impacts. In terms of journals, Nature Communications is the most influential journal and Scientific Reports is the most prolific one. The United States and Harvard University have contributed the most to cancer research using machine learning methods. As for the research topic, "Support Vector Machine," "classification," and "deep learning" have been the core focuses of the research field. Findings are helpful for scholars and related practitioners to better understand the development status and trends of cancer research using machine learning methods, as well as to have a deeper understanding of research hotspots.

Deep learning for multi-class semantic segmentation enables colorectal cancer detection and classification in digital pathology images

In colorectal cancer (CRC), artificial intelligence (AI) can alleviate the laborious task of characterization and reporting on resected biopsies, including polyps, the numbers of which are increasing as a result of CRC population screening programs ongoing in many countries all around the globe. Here, we present an approach to address two major challenges in the automated assessment of CRC histopathology whole-slide images. We present an AI-based method to segment multiple ([Formula: see text]) tissue compartments in the H &E-stained whole-slide image, which provides a different, more perceptible picture of tissue morphology and composition. We test and compare a panel of state-of-the-art loss functions available for segmentation models, and provide indications about their use in histopathology image segmentation, based on the analysis of (a) a multi-centric cohort of CRC cases from five medical centers in the Netherlands and Germany, and (b) two publicly available datasets on segmentation in CRC. We used the best performing AI model as the basis for a computer-aided diagnosis system that classifies colon biopsies into four main categories that are relevant pathologically. We report the performance of this system on an independent cohort of more than 1000 patients. The results show that with a good segmentation network as a base, a tool can be developed which can support pathologists in the risk stratification of colorectal cancer patients, among other possible uses. We have made the segmentation model available for research use on https://grand-challenge.org/algorithms/colon-tissue-segmentation/ .

The promise and challenge of spatial omics in dissecting tumour microenvironment and the role of AI

Growing evidence supports the critical role of tumour microenvironment (TME) in tumour progression, metastases, and treatment response. However, the in-situ interplay among various TME components, particularly between immune and tumour cells, are largely unknown, hindering our understanding of how tumour progresses and responds to treatment. While mainstream single-cell omics techniques allow deep, single-cell phenotyping, they lack crucial spatial information for in-situ cell-cell interaction analysis. On the other hand, tissue-based approaches such as hematoxylin and eosin and chromogenic immunohistochemistry staining can preserve the spatial information of TME components but are limited by their low-content staining. High-content spatial profiling technologies, termed spatial omics, have greatly advanced in the past decades to overcome these limitations. These technologies continue to emerge to include more molecular features (RNAs and/or proteins) and to enhance spatial resolution, opening new opportunities for discovering novel biological knowledge, biomarkers, and therapeutic targets. These advancements also spur the need for novel computational methods to mine useful TME insights from the increasing data complexity confounded by high molecular features and spatial resolution. In this review, we present state-of-the-art spatial omics technologies, their applications, major strengths, and limitations as well as the role of artificial intelligence (AI) in TME studies.

Prognostic prediction based on histopathologic features of tumor microenvironment in colorectal cancer

Purpose

To automatically quantify colorectal tumor microenvironment (TME) in hematoxylin and eosin stained whole slide images (WSIs), and to develop a TME signature for prognostic prediction in colorectal cancer (CRC).

Methods

A deep learning model based on VGG19 architecture and transfer learning strategy was trained to recognize nine different tissue types in whole slide images of patients with CRC. Seven of the nine tissue types were defined as TME components besides background and debris. Then 13 TME features were calculated based on the areas of TME components. A total of 562 patients with gene expression data, survival information and WSIs were collected from The Cancer Genome Atlas project for further analysis. A TME signature for prognostic prediction was developed and validated using Cox regression method. A prognostic prediction model combined the TME signature and clinical variables was also established. At last, gene-set enrichment analysis was performed to identify the significant TME signature associated pathways by querying Gene Ontology database and Kyoto Encyclopedia of Genes and Genomes database.

Results

The deep learning model achieved an accuracy of 94.2% for tissue type recognition. The developed TME signature was found significantly associated to progression-free survival. The clinical combined model achieved a concordance index of 0.714. Gene-set enrichment analysis revealed the TME signature associated genes were enriched in neuroactive ligand-receptor interaction pathway.

Conclusion

The TME signature was proved to be a prognostic factor and the associated biologic pathways would be beneficial to a better understanding of TME in CRC patients.

GraphLSurv: A scalable survival prediction network with adaptive and sparse structure learning for histopathological whole-slide images

BACKGROUND AND OBJECTIVE

Predicting patients' survival from gigapixel Whole-Slide Images (WSIs) has always been a challenging task. To learn effective WSI representations for survival prediction, existing deep learning methods have explored utilizing graphs to describe the complex structure inner WSIs, where graph node is respective to WSI patch. However, these graphs are often densely-connected or static, leading to some redundant or missing patch correlations. Moreover, these methods cannot be directly scaled to the very-large WSI with more than 10,000 patches. To address these, this paper proposes a scalable graph convolution network, GraphLSurv, which can efficiently learn adaptive and sparse structures to better characterize WSIs for survival prediction.

METHODS

GraphLSurv has three highlights in methodology: (1) it generates adaptive and sparse structures for patches so that latent patch correlations could be captured and adjusted dynamically according to prediction tasks; (2) based on the generated structure and a given graph, GraphLSurv further aggregates local microenvironmental cues into a non-local embedding using the proposed hybrid message passing network; (3) to make this network suitable for very large-scale graphs, it adopts an anchor-based technique to reduce theorical computation complexity.

RESULTS

The experiments on 2268 WSIs show that GraphLSurv achieves a concordance-index of 0.66132 and 0.68348, with an improvement of 3.79% and 3.41% compared to existing methods, on NLST and TCGA-BRCA, respectively.

CONCLUSIONS

GraphLSurv could often perform better than previous methods, which suggests that GraphLSurv could provide an important and effective means for WSI survival prediction. Moreover, this work empirically shows that adaptive and sparse structures could be more suitable than static or dense ones for modeling WSIs.

Histopathological diagnosis of colon cancer using micro-FTIR hyperspectral imaging and deep learning

BACKGROUND AND OBJECTIVE

Current studies based on digital biopsy images have achieved satisfactory results in detecting colon cancer despite their limited visual spectral range. Such methods may be less accurate when applied to samples taken from the tumor margin region or to samples containing multiple diagnoses. In contrast with the traditional computer vision approach, micro-FTIR hyperspectral images quantify the tissue-light interaction on a histochemical level and characterize different tissue pathologies, as they present a unique spectral signature. Therefore, this paper investigates the possibility of using hyperspectral images acquired over micro-FTIR absorbance spectroscopy to characterize healthy, inflammatory, and tumor colon tissues.

METHODS

The proposed method consists of modeling hyperspectral data into a voxel format to detect the patterns of each voxel using fully connected deep neural network. A web-based computer-aided diagnosis tool for inference is also provided.

RESULTS

Our experiments were performed using the K-fold cross-validation protocol in an intrapatient approach and achieved an overall accuracy of 99% using a deep neural network and 96% using a linear support vector machine. Through the experiments, we noticed the high performance of the method in characterizing such tissues using deep learning and hyperspectral images, indicating that the infrared spectrum contains relevant information and can be used to assist pathologists during the diagnostic process.

Leveraging Marine Predators Algorithm with Deep Learning for Lung and Colon Cancer Diagnosis

Cancer is a deadly disease caused by various biochemical abnormalities and genetic diseases. Colon and lung cancer have developed as two major causes of disability and death in human beings. The histopathological detection of these malignancies is a vital element in determining the optimal solution. Timely and initial diagnosis of the sickness on either front diminishes the possibility of death. Deep learning (DL) and machine learning (ML) methods are used to hasten such cancer recognition, allowing the research community to examine more patients in a much shorter period and at a less cost. This study introduces a marine predator's algorithm with deep learning as a lung and colon cancer classification (MPADL-LC3) technique. The presented MPADL-LC3 technique aims to properly discriminate different types of lung and colon cancer on histopathological images. To accomplish this, the MPADL-LC3 technique employs CLAHE-based contrast enhancement as a pre-processing step. In addition, the MPADL-LC3 technique applies MobileNet to derive feature vector generation. Meanwhile, the MPADL-LC3 technique employs MPA as a hyperparameter optimizer. Furthermore, deep belief networks (DBN) can be applied for lung and color classification. The simulation values of the MPADL-LC3 technique were examined on benchmark datasets. The comparison study highlighted the enhanced outcomes of the MPADL-LC3 system in terms of different measures.

Deep learning-based morphological feature analysis and the prognostic association study in colon adenocarcinoma histopathological images

Colorectal cancer (CRC) is now the third most common malignancy to cause mortality worldwide, and its prognosis is of great importance. Recent CRC prognostic prediction studies mainly focused on biomarkers, radiometric images, and end-to-end deep learning methods, while only a few works paid attention to exploring the relationship between the quantitative morphological features of patients' tissue slides and their prognosis. However, existing few works in this area suffered from the drawback of choosing the cells randomly from the whole slides, which contain the non-tumor region that lakes information about prognosis. In addition, the existing works, which tried to demonstrate their biological interpretability using patients' transcriptome data, failed to show the biological meaning closely related to cancer. In this study, we proposed and evaluated a prognostic model using morphological features of cells in the tumor region. The features were first extracted by the software CellProfiler from the tumor region selected by Eff-Unet deep learning model. Features from different regions were then averaged for each patient as their representative, and the Lasso-Cox model was used to select the prognosis-related features. The prognostic prediction model was at last constructed using the selected prognosis-related features and was evaluated through KM estimate and cross-validation. In terms of biological meaning, Gene Ontology (GO) enrichment analysis of the expressed genes that correlated with the prognostically significant features was performed to show the biological interpretability of our model.With the help of tumor segmentation, our model achieved better statistical significance and better biological interpretability compared to the results without tumor segmentation. Statistically, the Kaplan Meier (KM) estimate of our model showed that the model using features in the tumor region has a higher C-index, a lower p-value, and a better performance on cross-validation than the model without tumor segmentation. In addition, revealing the pathway of the immune escape and the spread of the tumor, the model with tumor segmentation demonstrated a biological meaning much more related to cancer immunobiology than the model without tumor segmentation. Our prognostic prediction model using quantitive morphological features from tumor regions was almost as good as the TNM tumor staging system as they had a close C-index, and our model can be combined with the TNM tumor stage system to make a better prognostic prediction. And to the best of our knowledge, the biological mechanisms in our study were the most relevant to the immune mechanism of cancer compared to the previous studies.

Accurate Diagnosis and Survival Prediction of Bladder Cancer Using Deep Learning on Histological Slides

(1) Background: Early diagnosis and treatment are essential to reduce the mortality rate of bladder cancer (BLCA). We aimed to develop deep learning (DL)-based weakly supervised models for the diagnosis of BLCA and prediction of overall survival (OS) in muscle-invasive bladder cancer (MIBC) patients using whole slide digitized histological images (WSIs). (2) Methods: Diagnostic and prognostic models were developed using 926 WSIs of 412 BLCA patients from The Cancer Genome Atlas cohort. We collected 250 WSIs of 150 BLCA patients from the Renmin Hospital of Wuhan University cohort for external validation of the models. Two DL models were developed: a BLCA diagnostic model (named BlcaMIL) and an MIBC prognostic model (named MibcMLP). (3) Results: The BlcaMIL model identified BLCA with accuracy 0.987 in the external validation set, comparable to that of expert uropathologists and outperforming a junior pathologist. The C-index values for the MibcMLP model on the internal and external validation sets were 0.631 and 0.622, respectively. The risk score predicted by MibcMLP was a strong predictor independent of existing clinical or histopathologic indicators, as demonstrated by univariate Cox (HR = 2.390, p < 0.0001) and multivariate Cox (HR = 2.414, p < 0.0001) analyses. The interpretability of DL models can help in the analysis of critical regions associated with tumors to enrich the information obtained from WSIs. Furthermore, the expression of six genes (ANAPC7, MAPKAPK5, COX19, LINC01106, AL161431.1 and MYO16-AS1) was significantly associated with MibcMLP-predicted risk scores, revealing possible potential biological correlations. (4) Conclusions: Our study developed DL models for accurately diagnosing BLCA and predicting OS in MIBC patients, which will help promote the precise pathological diagnosis of BLCA and risk stratification of MIBC to improve clinical treatment decisions.

Application of Artificial Intelligence in Pathology: Trends and Challenges

Given the recent success of artificial intelligence (AI) in computer vision applications, many pathologists anticipate that AI will be able to assist them in a variety of digital pathology tasks. Simultaneously, tremendous advancements in deep learning have enabled a synergy with artificial intelligence (AI), allowing for image-based diagnosis on the background of digital pathology. There are efforts for developing AI-based tools to save pathologists time and eliminate errors. Here, we describe the elements in the development of computational pathology (CPATH), its applicability to AI development, and the challenges it faces, such as algorithm validation and interpretability, computing systems, reimbursement, ethics, and regulations. Furthermore, we present an overview of novel AI-based approaches that could be integrated into pathology laboratory workflows.

Digital image analysis provides robust tissue microenvironment-based prognosticators in patients with stage I-IV colorectal cancer

In patients with colorectal cancer (CRC), a promising marker is tumor-stroma ratio (TSR). Quantification issues highlight the importance of precise assessment that might be solved by artificial intelligence-based digital image analysis systems. Some alternatives have been offered so far, although these platforms are either proprietary developments or require additional programming skills. Our aim was to validate a user-friendly, commercially available software running in everyday computational environment to improve TSR assessment and also to compare the prognostic value of assessing TSR in 3 distinct regions of interests, like hotspot, invasive front, and whole tumor. Furthermore, we compared the prognostic power of TSR with the newly suggested carcinoma percentage (CP) and carcinoma-stroma percentage (CSP). Slides of 185 patients with stage I-IV CRC with clinical follow-up data were scanned and evaluated by a senior pathologist. A machine learning-based digital pathology software was trained to recognize tumoral and stromal compartments. The aforementioned parameters were evaluated in the hotspot, invasive front, and whole tumor area, both visually and by machine learning. Patients were classified based on TSR, CP, and CSP values. On multivariate analysis, TSR-hotspot was found to be an independent prognostic factor of overall survival (hazard ratio for TSR-hotspotsoftware: 2.005 [95% confidence interval (CI): 1.146-3.507], P = .011, for TSR-hotspotvisual: 1.781 [CI: 1.060-2.992], P = .029). Also, TSR was an independent predictor for distant metastasis and local relapse in most settings. Generally, software performance was comparable to visual evaluation and delivered reliable prognostication in more settings also with CP and CSP values. This study presents that software-assisted evaluation is a robust prognosticator. Our approach used a less sophisticated and thus easily accessible software without the aid of a convolutional neural network; however, it was still effective enough to deliver reliable prognostic information.

PPsNet: An improved deep learning model for microsatellite instability high prediction in colorectal cancer from whole slide images

BACKGROUND AND OBJECTIVE

Recent studies have shown that colorectal cancer (CRC) patients with microsatellite instability high (MSI-H) are more likely to benefit from immunotherapy. However, current MSI testing methods are not available for all patients due to the lack of available equipment and trained personnel, as well as the high cost of the assay. Here, we developed an improved deep learning model to predict MSI-H in CRC from whole slide images (WSIs).

METHODS

We established the MSI-H prediction model based on two stages: tumor detection and MSI classification. Previous works applied fine-tuning strategy directly for tumor detection, but ignoring the challenge of vanishing gradient due to the large number of convolutional layers. We added auxiliary classifiers to intermediate layers of pre-trained models to help propagate gradients back through in an effective manner. To predict MSI status, we constructed a pair-wise learning model with a synergic network, named parameter partial sharing network (PPsNet), where partial parameters are shared among two deep convolutional neural networks (DCNNs). The proposed PPsNet contained fewer parameters and reduced the problem of intra-class variation and inter-class similarity. We validated the proposed model on a holdout test set and two external test sets.

RESULTS

144 H&E-stained WSIs from 144 CRC patients (81 cases with MSI-H and 63 cases with MSI-L/MSS) were collected retrospectively from three hospitals. The experimental results indicate that deep supervision based fine-tuning almost outperforms training from scratch and utilizing fine-tuning directly. The proposed PPsNet always achieves better accuracy and area under the receiver operating characteristic curve (AUC) than other solutions with four different neural network architectures on validation. The proposed method finally achieves obvious improvements than other state-of-the-art methods on the validation dataset with an accuracy of 87.28% and AUC of 94.29%.

CONCLUSIONS

The proposed method can obviously increase model performance and our model yields better performance than other methods. Additionally, this work also demonstrates the feasibility of MSI-H prediction using digital pathology images based on deep learning in the Asian population. It is hoped that this model could serve as an auxiliary tool to identify CRC patients with MSI-H more time-saving and efficiently.

PregGAN: A prognosis prediction model for breast cancer based on conditional generative adversarial networks

BACKGROUND AND OBJECTIVE

Generative adversarial network (GAN) is able to learn from a set of training data and generate new data with the same characteristics as the training data. Based on the characteristics of GAN, this paper developed its capability as a tool of disease prognosis prediction, and proposed a prognostic model PregGAN based on conditional generative adversarial network (CGAN).

METHODS

The idea of PregGAN is to generate the prognosis prediction results based on the clinical data of patients. PregGAN added the clinical data as conditions to the training process. Conditions were used as the input to the generator along with noises. The generator synthesized new samples using the noises vectors and the conditions. In order to solve the mode collapse problem during PregGAN training, Wasserstein distance and gradient penalty strategy were used to make the training process more stable.

RESULTS

In the prognosis prediction experiments using the METABRIC breast cancer dataset, PregGAN achieved good results, with the average accurate (ACC) of 90.6% and the average AUC (area under curve) of 0.946.

CONCLUSIONS

Experimental results show that PregGAN is a reliable prognosis predictive model for breast cancer. Due to the strong ability of probability distribution learning, PregGAN can also be used for the prognosis prediction of other diseases.

Computational pathology in ovarian cancer

Histopathologic evaluations of tissue sections are key to diagnosing and managing ovarian cancer. Pathologists empirically assess and integrate visual information, such as cellular density, nuclear atypia, mitotic figures, architectural growth patterns, and higher-order patterns, to determine the tumor type and grade, which guides oncologists in selecting appropriate treatment options. Latent data embedded in pathology slides can be extracted using computational imaging. Computers can analyze digital slide images to simultaneously quantify thousands of features, some of which are visible with a manual microscope, such as nuclear size and shape, while others, such as entropy, eccentricity, and fractal dimensions, are quantitatively beyond the grasp of the human mind. Applications of artificial intelligence and machine learning tools to interpret digital image data provide new opportunities to explore and quantify the spatial organization of tissues, cells, and subcellular structures. In comparison to genomic, epigenomic, transcriptomic, and proteomic patterns, morphologic and spatial patterns are expected to be more informative as quantitative biomarkers of complex and dynamic tumor biology. As computational pathology is not limited to visual data, nuanced subvisual alterations that occur in the seemingly “normal” pre-cancer microenvironment could facilitate research in early cancer detection and prevention. Currently, efforts to maximize the utility of computational pathology are focused on integrating image data with other -omics platforms that lack spatial information, thereby providing a new way to relate the molecular, spatial, and microenvironmental characteristics of cancer. Despite a dire need for improvements in ovarian cancer prevention, early detection, and treatment, the ovarian cancer field has lagged behind other cancers in the application of computational pathology. The intent of this review is to encourage ovarian cancer research teams to apply existing and/or develop additional tools in computational pathology for ovarian cancer and actively contribute to advancing this important field.

Fine-Tuned DenseNet-169 for Breast Cancer Metastasis Prediction Using FastAI and 1-Cycle Policy

Lymph node metastasis in breast cancer may be accurately predicted using a DenseNet-169 model. However, the current system for identifying metastases in a lymph node is manual and tedious. A pathologist well-versed with the process of detection and characterization of lymph nodes goes through hours investigating histological slides. Furthermore, because of the massive size of most whole-slide images (WSI), it is wise to divide a slide into batches of small image patches and apply methods independently on each patch. The present work introduces a novel method for the automated diagnosis and detection of metastases from whole slide images using the Fast AI framework and the 1-cycle policy. Additionally, it compares this new approach to previous methods. The proposed model has surpassed other state-of-art methods with more than 97.4% accuracy. In addition, a mobile application is developed for prompt and quick response. It collects user information and models to diagnose metastases present in the early stages of cancer. These results indicate that the suggested model may assist general practitioners in accurately analyzing breast cancer situations, hence preventing future complications and mortality. With digital image processing, histopathologic interpretation and diagnostic accuracy have improved considerably.

Deep Learning on Histopathological Images for Colorectal Cancer Diagnosis: A Systematic Review

Colorectal cancer (CRC) is the second most common cancer in women and the third most common in men, with an increasing incidence. Pathology diagnosis complemented with prognostic and predictive biomarker information is the first step for personalized treatment. The increased diagnostic load in the pathology laboratory, combined with the reported intra- and inter-variability in the assessment of biomarkers, has prompted the quest for reliable machine-based methods to be incorporated into the routine practice. Recently, Artificial Intelligence (AI) has made significant progress in the medical field, showing potential for clinical applications. Herein, we aim to systematically review the current research on AI in CRC image analysis. In histopathology, algorithms based on Deep Learning (DL) have the potential to assist in diagnosis, predict clinically relevant molecular phenotypes and microsatellite instability, identify histological features related to prognosis and correlated to metastasis, and assess the specific components of the tumor microenvironment.

Artificial Intelligence-based Tumor Segmentation in Mouse Models of Lung Adenocarcinoma

BACKGROUND

Mouse models are highly effective for studying the pathophysiology of lung adenocarcinoma and evaluating new treatment strategies. Treatment efficacy is primarily determined by the total tumor burden measured on excised tumor specimens. The measurement process is time-consuming and prone to human errors. To address this issue, we developed a novel deep learning model to segment lung tumor foci on digitally scanned hematoxylin and eosin (H&E) histology slides.

METHODS

Digital slides of 239 mice from 9 experimental cohorts were split into training (n=137), validation (n=37), and testing cohorts (n=65). Image patches of 500×500 pixels were extracted from 5× and 10× magnifications, along with binary masks of expert annotations representing ground-truth tumor regions. Deep learning models utilizing DeepLabV3+ and UNet architectures were trained for binary segmentation of tumor foci under varying stain normalization conditions. The performance of algorithm segmentation was assessed by Dice Coefficient, and detection was evaluated by sensitivity and positive-predictive value (PPV).

RESULTS

The best model on patch-based validation was DeepLabV3+ using a Resnet-50 backbone, which achieved Dice 0.890 and 0.873 on validation and testing cohort, respectively. This result corresponded to 91.3 Sensitivity and 51.0 PPV in the validation cohort and 93.7 Sensitivity and 51.4 PPV in the testing cohort. False positives could be reduced 10-fold with thresholding artificial intelligence (AI) predicted output by area, without negative impact on Dice Coefficient. Evaluation at various stain normalization strategies did not demonstrate improvement from the baseline model.

CONCLUSIONS

A robust AI-based algorithm for detecting and segmenting lung tumor foci in the pre-clinical mouse models was developed. The output of this algorithm is compatible with open-source software that researchers commonly use.

Deep Neural Network-Aided Histopathological Analysis of Myocardial Injury

Deep neural networks have become the mainstream approach for analyzing and interpreting histology images. In this study, we established and validated an interpretable DNN model to assess endomyocardial biopsy (EMB) data of patients with myocardial injury. Deep learning models were used to extract features and classify EMB histopathological images of heart failure cases diagnosed with either ischemic cardiomyopathy or idiopathic dilated cardiomyopathy and non-failing cases (organ donors without a history of heart failure). We utilized the gradient-weighted class activation mapping (Grad-CAM) technique to emphasize injured regions, providing an entry point to assess the dominant morphology in the process of a comprehensive evaluation. To visualize clustered regions of interest (ROI), we utilized uniform manifold approximation and projection (UMAP) embedding for dimension reduction. We further implemented a multi-model ensemble mechanism to improve the quantitative metric (area under the receiver operating characteristic curve, AUC) to 0.985 and 0.992 on ROI-level and case-level, respectively, outperforming the achievement of 0.971 ± 0.017 and 0.981 ± 0.020 based on the sub-models. Collectively, this new methodology provides a robust and interpretive framework to explore local histopathological patterns, facilitating the automatic and high-throughput quantification of cardiac EMB analysis.

Deep embeddings and logistic regression for rapid active learning in histopathological images

BACKGROUND AND OBJECTIVE

Recognizing different tissue components is one of the most fundamental and essential works in digital pathology. Current methods are often based on convolutional neural networks (CNNs), which need numerous annotated samples for training. Creating large-scale histopathological datasets is labor-intensive, where interactive data annotation is a potential solution.

METHODS

We propose DELR (Deep Embedding-based Logistic Regression) to enable rapid model training and inference for histopathological image analysis. DELR utilizes a pretrained CNN to encode images as compact embeddings with low computational cost. The embeddings are then used to train a Logistic Regression model efficiently. We implemented DELR in an active learning framework, and validated it on three histopathological problems (binary, 4-category, and 8-category classification challenge for lung, breast, and colorectal cancer, respectively). We also investigated the influence of active learning strategy and type of the encoder.

RESULTS

On all the three datasets, DELR can achieve an area under curve (AUC) metric higher than 0.95 with only 100 image patches per class. Although its AUC is slightly lower than a fine-tuned CNN counterpart, DELR can be 536, 316, and 1481 times faster after pre-encoding. Moreover, DELR is proved to be compatible with a variety of active learning strategies and encoders.

CONCLUSIONS

DELR can achieve comparable accuracy to CNN with rapid running speed. These advantages make it a potential solution for real-time interactive data annotation.

Next-Generation Digital Histopathology of the Tumor Microenvironment

Progress in cancer research is substantially dependent on innovative technologies that permit a concerted analysis of the tumor microenvironment and the cellular phenotypes resulting from somatic mutations and post-translational modifications. In view of a large number of genes, multiplied by differential splicing as well as post-translational protein modifications, the ability to identify and quantify the actual phenotypes of individual cell populations in situ, i.e., in their tissue environment, has become a prerequisite for understanding tumorigenesis and cancer progression. The need for quantitative analyses has led to a renaissance of optical instruments and imaging techniques. With the emergence of precision medicine, automated analysis of a constantly increasing number of cellular markers and their measurement in spatial context have become increasingly necessary to understand the molecular mechanisms that lead to different pathways of disease progression in individual patients. In this review, we summarize the joint effort that academia and industry have undertaken to establish methods and protocols for molecular profiling and immunophenotyping of cancer tissues for next-generation digital histopathology-which is characterized by the use of whole-slide imaging (brightfield, widefield fluorescence, confocal, multispectral, and/or multiplexing technologies) combined with state-of-the-art image cytometry and advanced methods for machine and deep learning.