Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning

Automated exploitation of deep learning for cancer patient stratification across multiple types

MOTIVATION

Recent frameworks based on deep learning have been developed to identify cancer subtypes from high-throughput gene expression profiles. Unfortunately, the performance of deep learning is highly dependent on its neural network architectures which are often hand-crafted with expertise in deep neural networks, meanwhile, the optimization and adjustment of the network are usually costly and time consuming.

RESULTS

To address such limitations, we proposed a fully automated deep neural architecture search model for diagnosing consensus molecular subtypes from gene expression data (DNAS). The proposed model uses ant colony algorithm, one of the heuristic swarm intelligence algorithms, to search and optimize neural network architecture, and it can automatically find the optimal deep learning model architecture for cancer diagnosis in its search space. We validated DNAS on eight colorectal cancer datasets, achieving the average accuracy of 95.48%, the average specificity of 98.07%, and the average sensitivity of 96.24%, respectively. Without the loss of generality, we investigated the general applicability of DNAS further on other cancer types from different platforms including lung cancer and breast cancer, and DNAS achieved an area under the curve of 95% and 96%, respectively. In addition, we conducted gene ontology enrichment and pathological analysis to reveal interesting insights into cancer subtype identification and characterization across multiple cancer types.

AVAILABILITY AND IMPLEMENTATION

The source code and data can be downloaded from https://github.com/userd113/DNAS-main. And the web server of DNAS is publicly accessible at 119.45.145.120:5001.

Comparison of KRAS gene in circulating tumor DNA levels vs histological grading of colorectal cancer patients through liquid biopsy

Background

To determine KRAS gene in circulating tumor DNA in comparison with histological grading through liquid biopsy in colorectal cancer patients.

Methods

This dual-centered cross-sectional study included 73 diagnosed patients of colorectal cancer at different grading levels [Grade I, well differentiated (n = 7, 9.5%); Grade II, moderately differentiated (n = 14,18.9%); and Grade III, poorly differentiated (n = 52, 70%)]. Blood was collected, and plasma was separated. ctDNA was extracted, using magnetic bead-based technique (MagMAX Cell-Free DNA kit). KRAS gene was quantified through qPCR. STRING database was used to find KRAS interactomes.

Results

Mean threshold cycle (CT value) of KRAS gene in Grade III samples showed significantly higher (P = 0.001) levels of ctDNA (2.7 ± 1.14) compared with Grade II and Grade I (3.1 ± 0.68, 2.3 ± 0.60), respectively. Grading characterization showed that rectal cancer (n = 22, 42.3%) with Grade III (68.8%) was more prevalent than colon and sigmoid cancer (n = 19, 36.5%, n = 11, 21%, respectively). STRING database showed 10 functional genes interacting with KRAS expressed as gene/proteins.

Conclusion

Liquid biopsy can be used to detect ctDNA in plasma of CRC patients and enabled to detect the KRAS gene by qPCR. The technique being less invasive and cost-effective is convenient for multiple biopsies in different cancers.

Machine learning-enhanced insights into sphingolipid-based prognostication: revealing the immunological landscape and predictive proficiency for immunomotherapy and chemotherapy responses in pancreatic carcinoma

Background: With a poor prognosis for affected individuals, pancreatic adenocarcinoma (PAAD) is known as a complicated and diverse illness. Immunocytes have become essential elements in the development of PAAD. Notably, sphingolipid metabolism has a dual function in the development of tumors and the invasion of the immune system. Despite these implications, research on the predictive ability of sphingolipid variables for PAAD prognosis is strikingly lacking, and it is yet unclear how they can affect PAAD immunotherapy and targeted pharmacotherapy. Methods: The investigation process included SPG detection while also being pertinent to the prognosis for PAAD. Both the analytical capability of CIBERSORT and the prognostic capability of the pRRophetic R package were used to evaluate the immunological environments of the various HCC subtypes. In addition, CCK-8 experiments on PAAD cell lines were carried out to confirm the accuracy of drug sensitivity estimates. The results of these trials, which also evaluated cell survival and migratory patterns, confirmed the usefulness of sphingolipid-associated genes (SPGs). Results: As a result of this thorough investigation, 32 SPGs were identified, each of which had a measurable influence on the dynamics of overall survival. This collection of genes served as the conceptual framework for the development of a prognostic model, which was carefully assembled from 10 chosen genes. It should be noted that this grouping of patients into cohorts with high and low risk was a sign of different immune profiles and therapy responses. The increased abundance of SPGs was identified as a possible sign of inadequate responses to immune-based treatment approaches. The careful CCK-8 testing carried out on PAAD cell lines was of the highest importance for providing clear confirmation of drug sensitivity estimates. Conclusion: The significance of Sphingolipid metabolism in the complex web of PAAD development is brought home by this study. The novel risk model, built on the complexity of sphingolipid-associated genes, advances our understanding of PAAD and offers doctors a powerful tool for developing personalised treatment plans that are specifically suited to the unique characteristics of each patient.

Leveraging Attention-Based Convolutional Neural Networks for Meningioma Classification in Computational Histopathology

Convolutional neural networks (CNNs) are becoming increasingly valuable tools for advanced computational histopathology, promoting precision medicine through exceptional visual decoding abilities. Meningiomas, the most prevalent primary intracranial tumors, necessitate accurate grading and classification for informed clinical decision-making. Recently, DNA methylation-based molecular classification of meningiomas has proven to be more effective in predicting tumor recurrence than traditional histopathological methods. However, DNA methylation profiling is expensive, labor-intensive, and not widely accessible. Consequently, a digital histology-based prediction of DNA methylation classes would be advantageous, complementing molecular classification. In this study, we developed and rigorously assessed an attention-based multiple-instance deep neural network for predicting meningioma methylation classes using tumor methylome data from 142 (+51) patients and corresponding hematoxylin-eosin-stained histological sections. Pairwise analysis of sample cohorts from three meningioma methylation classes demonstrated high accuracy in two combinations. The performance of our approach was validated using an independent set of 51 meningioma patient samples. Importantly, attention map visualization revealed that the algorithm primarily focuses on tumor regions deemed significant by neuropathologists, offering insights into the decision-making process of the CNN. Our findings highlight the capacity of CNNs to effectively harness phenotypic information from histological sections through computerized images for precision medicine. Notably, this study is the first demonstration of predicting clinically relevant DNA methylome information using computer vision applied to standard histopathology. The introduced AI framework holds great potential in supporting, augmenting, and expediting meningioma classification in the future.

Deep learning-derived spatial organization features on histology images predicts prognosis in colorectal liver metastasis patients after hepatectomy

Histopathological images of colorectal liver metastases (CRLM) contain rich morphometric information that may predict patients' outcomes. However, to our knowledge, no study has reported any practical deep learning framework based on the histology images of CRLM, and their direct association with prognosis remains largely unknown. In this study, we developed a deep learning-based framework for fully automated tissue classification and quantification of clinically relevant spatial organization features (SOFs) in H&E-stained images of CRLM. The SOFs based risk-scoring system demonstrated a strong and robust prognostic value that is independent of the current clinical risk score (CRS) system in independent clinical cohorts. Our framework enables fully automated tissue classification of H&E images of CRLM, which could significantly reduce assessment subjectivity and the workload of pathologists. The risk-scoring system provides a time- and cost-efficient tool to assist clinical decision-making for patients with CRLM, which could potentially be implemented in clinical practice.

Artificial Intelligence Reveals Distinct Prognostic Subgroups of Muscle-Invasive Bladder Cancer on Histology Images

Muscle-invasive bladder cancer (MIBC) is a highly heterogeneous and costly disease with significant morbidity and mortality. Understanding tumor histopathology leads to tailored therapies and improved outcomes. In this study, we employed a weakly supervised learning and neural architecture search to develop a data-driven scoring system. This system aimed to capture prognostic histopathological patterns observed in H&E-stained whole-slide images. We constructed and externally validated our scoring system using multi-institutional datasets with 653 whole-slide images. Additionally, we explored the association between our scoring system, seven histopathological features, and 126 molecular signatures. Through our analysis, we identified two distinct risk groups with varying prognoses, reflecting inherent differences in histopathological and molecular subtypes. The adjusted hazard ratio for overall mortality was 1.46 (95% CI 1.05-2.02; z: 2.23; p = 0.03), thus identifying two prognostic subgroups in high-grade MIBC. Furthermore, we observed an association between our novel digital biomarker and the squamous phenotype, subtypes of miRNA, mRNA, long non-coding RNA, DNA hypomethylation, and several gene mutations, including FGFR3 in MIBC. Our findings underscore the risk of confounding bias when reducing the complex biological and clinical behavior of tumors to a single mutation. Histopathological changes can only be fully captured through comprehensive multi-omics profiles. The introduction of our scoring system has the potential to enhance daily clinical decision making for MIBC. It facilitates shared decision making by offering comprehensive and precise risk stratification, treatment planning, and cost-effective preselection for expensive molecular characterization.

Colorectal Cancer Organoid-Stroma Biobank Allows Subtype-Specific Assessment of Individualized Therapy Responses

In colorectal cancers, the tumor microenvironment plays a key role in prognosis and therapy efficacy. Patient-derived tumor organoids (PDTO) show enormous potential for preclinical testing; however, cultured tumor cells lose important characteristics, including the consensus molecular subtypes (CMS). To better reflect the cellular heterogeneity, we established the colorectal cancer organoid-stroma biobank of matched PDTOs and cancer-associated fibroblasts (CAF) from 30 patients. Context-specific phenotyping showed that xenotransplantation or coculture with CAFs improves the transcriptomic fidelity and instructs subtype-specific stromal gene expression. Furthermore, functional profiling in coculture exposed CMS4-specific therapeutic resistance to gefitinib and SN-38 and prognostic expression signatures. Chemogenomic library screening identified patient- and therapy-dependent mechanisms of stromal resistance including MET as a common target. Our results demonstrate that colorectal cancer phenotypes are encrypted in the cancer epithelium in a plastic fashion that strongly depends on the context. Consequently, CAFs are essential for a faithful representation of molecular subtypes and therapy responses ex vivo.

SIGNIFICANCE

Systematic characterization of the organoid-stroma biobank provides a resource for context dependency in colorectal cancer. We demonstrate a colorectal cancer subtype memory of PDTOs that is independent of specific driver mutations. Our data underscore the importance of functional profiling in cocultures for improved preclinical testing and identification of stromal resistance mechanisms. This article is featured in Selected Articles from This Issue, p. 2109.

Improved accuracy in colorectal cancer tissue decomposition through refinement of established deep learning solutions

Hematoxylin and eosin-stained biopsy slides are regularly available for colorectal cancer patients. These slides are often not used to define objective biomarkers for patient stratification and treatment selection. Standard biomarkers often pertain to costly and slow genetic tests. However, recent work has shown that relevant biomarkers can be extracted from these images using convolutional neural networks (CNNs). The CNN-based biomarkers predicted colorectal cancer patient outcomes comparably to gold standards. Extracting CNN-biomarkers is fast, automatic, and of minimal cost. CNN-based biomarkers rely on the ability of CNNs to recognize distinct tissue types from microscope whole slide images. The quality of these biomarkers (coined 'Deep Stroma') depends on the accuracy of CNNs in decomposing all relevant tissue classes. Improving tissue decomposition accuracy is essential for improving the prognostic potential of CNN-biomarkers. In this study, we implemented a novel training strategy to refine an established CNN model, which then surpassed all previous solutions . We obtained a 95.6% average accuracy in the external test set and 99.5% in the internal test set. Our approach reduced errors in biomarker-relevant classes, such as Lymphocytes, and was the first to include interpretability methods. These methods were used to better apprehend our model's limitations and capabilities.

Identification and verification of a prognostic signature based on a miRNA-mRNA interaction pattern in colon adenocarcinoma

The expression characteristics of non-coding RNA (ncRNA) in colon adenocarcinoma (COAD) are involved in regulating various biological processes. To achieve these functions, ncRNA and a member of the Argonaute protein family form an RNA-induced silencing complex (RISC). The RISC is directed by ncRNA, especially microRNA (miRNA), to bind the target complementary mRNAs and regulate their expression by interfering with mRNA cleavage, degradation, or translation. However, how to identify potential miRNA biomarkers and therapeutic targets remains unclear. Here, we performed differential gene screening based on The Cancer Genome Atlas dataset and annotated meaningful differential genes to enrich related biological processes and regulatory cancer pathways. According to the overlap between the screened differential mRNAs and differential miRNAs, a prognosis model based on a least absolute shrinkage and selection operator-based Cox proportional hazards regression analysis can be established to obtain better prognosis characteristics. To further explore the therapeutic potential of miRNA as a target of mRNA intervention, we conducted an immunohistochemical analysis and evaluated the expression level in the tissue microarray of 100 colorectal cancer patients. The results demonstrated that the expression level of POU4F1, DNASE1L2, and WDR72 in the signature was significantly upregulated in COAD and correlated with poor prognosis. Establishing a prognostic signature based on miRNA target genes will help elucidate the molecular pathogenesis of COAD and provide novel potential targets for RNA therapy.

Deep learning exploration of single-cell and spatially resolved cancer transcriptomics to unravel tumour heterogeneity

Tumour heterogeneity is one of the critical confounding aspects in decoding tumour growth. Malignant cells display variations in their gene transcription profiles and mutation spectra even when originating from a single progenitor cell. Single-cell and spatial transcriptomics sequencing have recently emerged as key technologies for unravelling tumour heterogeneity. Single-cell sequencing promotes individual cell-type identification through transcriptome-wide gene expression measurements of each cell. Spatial transcriptomics facilitates identification of cell-cell interactions and the structural organization of heterogeneous cells within a tumour tissue through associating spatial RNA abundance of cells at distinct spots in the tissue section. However, extracting features and analyzing single-cell and spatial transcriptomics data poses challenges. Single-cell transcriptome data is extremely noisy and its sparse nature and dropouts can lead to misinterpretation of gene expression and the misclassification of cell types. Deep learning predictive power can overcome data challenges, provide high-resolution analysis and enhance precision oncology applications that involve early cancer prognosis, diagnosis, patient survival estimation and anti-cancer therapy planning. In this paper, we provide a background to and review of the recent progress of deep learning frameworks to investigate tumour heterogeneity using both single-cell and spatial transcriptomics data types.

Machine learning in computational histopathology: Challenges and opportunities

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response

In recent years, the application of advanced analytics, especially artificial intelligence (AI), to digital H&E images, and other histological image types, has begun to radically change how histological images are used in the clinic. Alongside the recognition that the tumour microenvironment (TME) has a profound impact on tumour phenotype, the technical development of highly multiplexed immunofluorescence platforms has enhanced the biological complexity that can be captured in the TME with high precision. AI has an increasingly powerful role in the recognition and quantitation of image features and the association of such features with clinically important outcomes, as occurs in distinct stages in conventional machine learning. Deep-learning algorithms are able to elucidate TME patterns inherent in the input data with minimum levels of human intelligence and, hence, have the potential to achieve clinically relevant predictions and discovery of important TME features. Furthermore, the diverse repertoire of deep-learning algorithms able to interrogate TME patterns extends beyond convolutional neural networks to include attention-based models, graph neural networks, and multimodal models. To date, AI models have largely been evaluated retrospectively, outside the well-established rigour of prospective clinical trials, in part because traditional clinical trial methodology may not always be suitable for the assessment of AI technology. However, to enable digital pathology-based advanced analytics to meaningfully impact clinical care, specific measures of 'added benefit' to the current standard of care and validation in a prospective setting are important. This will need to be accompanied by adequate measures of explainability and interpretability. Despite such challenges, the combination of expanding datasets, increased computational power, and the possibility of integration of pre-clinical experimental insights into model development means there is exciting potential for the future progress of these AI applications. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Overcoming the challenges to implementation of artificial intelligence in pathology

Pathologists worldwide are facing remarkable challenges with increasing workloads and lack of time to provide consistently high-quality patient care. The application of artificial intelligence (AI) to digital whole-slide images has the potential of democratizing the access to expert pathology and affordable biomarkers by supporting pathologists in the provision of timely and accurate diagnosis as well as supporting oncologists by directly extracting prognostic and predictive biomarkers from tissue slides. The long-awaited adoption of AI in pathology, however, has not materialized, and the transformation of pathology is happening at a much slower pace than that observed in other fields (eg, radiology). Here, we provide a critical summary of the developments in digital and computational pathology in the last 10 years, outline key hurdles and ways to overcome them, and provide a perspective for AI-supported precision oncology in the future.

Artificial Intelligence Enables Quantitative Assessment of Ulcerative Colitis Histology

Ulcerative colitis is a chronic inflammatory bowel disease that is characterized by a relapsing and remitting course. Assessment of disease activity critically informs treatment decisions. In addition to endoscopic remission, histologic remission is emerging as a treatment target and a key factor in the evaluation of disease activity and therapeutic efficacy. However, manual pathologist evaluation is semiquantitative and limited in granularity. Machine learning approaches are increasingly being developed to aid pathologists in accurate and reproducible scoring of histology, enabling precise quantitation of clinically relevant features. Here, we report the development and validation of convolutional neural network models that quantify histologic features pertinent to ulcerative colitis disease activity, directly from hematoxylin and eosin-stained whole slide images. Tissue and cell model predictions were used to generate quantitative human-interpretable features to fully characterize the histology samples. Tissue and cell predictions showed comparable agreement to pathologist annotations, and the extracted slide-level human-interpretable features demonstrated strong correlations with disease severity and pathologist-assigned Nancy histological index scores. Moreover, using a random forest classifier based on 13 human-interpretable features derived from the tissue and cell models, we were able to accurately predict Nancy histological index scores, with a weighted kappa (κ = 0.91) and Spearman correlation (⍴ = 0.89, P < .001) when compared with pathologist consensus Nancy histological index scores. We were also able to predict histologic remission, based on the absence of neutrophil extravasation, with a high accuracy of 0.97. This work demonstrates the potential of computer vision to enable a standardized and robust assessment of ulcerative colitis histopathology for translational research and improved evaluation of disease activity and prognosis.

Recent application of artificial intelligence on histopathologic image-based prediction of gene mutation in solid cancers

PURPOSE

Evaluation of genetic mutations in cancers is important because distinct mutational profiles help determine individualized drug therapy. However, molecular analyses are not routinely performed in all cancers because they are expensive, time-consuming and not universally available. Artificial intelligence (AI) has shown the potential to determine a wide range of genetic mutations on histologic image analysis. Here, we assessed the status of mutation prediction AI models on histologic images by a systematic review.

METHODS

A literature search using the MEDLINE, Embase and Cochrane databases was conducted in August 2021. The articles were shortlisted by titles and abstracts. After a full-text review, publication trends, study characteristic analysis and comparison of performance metrics were performed.

RESULTS

Twenty-four studies were found mostly from developed countries, and their number is increasing. The major targets were gastrointestinal, genitourinary, gynecological, lung and head and neck cancers. Most studies used the Cancer Genome Atlas, with a few using an in-house dataset. The area under the curve of some of the cancer driver gene mutations in particular organs was satisfactory, such as 0.92 of BRAF in thyroid cancers and 0.79 of EGFR in lung cancers, whereas the average of all gene mutations was 0.64, which is still suboptimal.

CONCLUSION

AI has the potential to predict gene mutations on histologic images with appropriate caution. Further validation with larger datasets is still required before AI models can be used in clinical practice to predict gene mutations.

Consensus Molecular Subtypes as Biomarkers of Fluorouracil and Folinic Acid Maintenance Therapy With or Without Panitumumab in RAS Wild-Type Metastatic Colorectal Cancer (PanaMa, AIO KRK 0212)

PURPOSE

Consensus molecular subtypes (CMSs) were evaluated as prognostic and predictive biomarkers of patients with RAS wild-type metastatic colorectal cancer (mCRC) receiving fluorouracil and folinic acid (FU/FA) with or without panitumumab (Pmab) after Pmab + mFOLFOX6 induction within the randomized phase II PanaMa trial.

METHODS

CMSs were determined in the safety set (ie, patients that received induction) and full analysis set (FAS; ie, randomly assigned patients who received maintenance) and correlated with median progression-free survival (PFS) and overall survival (OS) since the start of induction or maintenance treatment and objective response rates (ORRs). Hazard ratios (HRs) and 95% CI were calculated by univariate/multivariate Cox regression analyses.

RESULTS

Of 377 patients of the safety set, 296 (78.5%) had available CMS data: CMS1/2/3/4: 29 (9.8%)/122 (41.2%)/33 (11.2%)/112 (37.8%) and unclassifiable: 17 (5.7%). The CMSs were prognostic biomarkers in terms of PFS (P < .0001), OS (P < .0001), and ORR (P = .02) since the start of induction treatment. In FAS patients (n = 196), with CMS2/4 tumors, the addition of Pmab to FU/FA maintenance therapy was associated with longer PFS (CMS2: HR, 0.58 [95% CI, 0.36 to 0.95], P = .03; CMS4: HR, 0.63 [95% CI, 0.38 to 1.03], P = .07) and OS (CMS2: HR, 0.88 [95% CI, 0.52 to 1.52], P = .66; CMS4: HR, 0.54 [95% CI, 0.30 to 0.96], P = .04). The CMS interacted significantly with treatment in terms of PFS (CMS2 v CMS1/3: P = .02; CMS4 v CMS1/3: P = .03) and OS (CMS2 v CMS1/3: P = .03; CMS4 v CMS1/3: P < .001).

CONCLUSION

The CMS had a prognostic impact on PFS, OS, and ORR in RAS wild-type mCRC. In PanaMa, Pmab + FU/FA maintenance was associated with beneficial outcomes in CMS2/4, whereas no benefit was observed in CMS1/3 tumors.

Prognostic signatures of sphingolipids: Understanding the immune landscape and predictive role in immunotherapy response and outcomes of hepatocellular carcinoma

Background

Hepatocellular carcinoma (HCC) is a complex disease with a poor outlook for patients in advanced stages. Immune cells play an important role in the progression of HCC. The metabolism of sphingolipids functions in both tumor growth and immune infiltration. However, little research has focused on using sphingolipid factors to predict HCC prognosis. This study aimed to identify the key sphingolipids genes (SPGs) in HCC and develop a reliable prognostic model based on these genes.

Methods

The TCGA, GEO, and ICGC datasets were grouped using SPGs obtained from the InnateDB portal. A prognostic gene signature was created by applying LASSO-Cox analysis and evaluating it with Cox regression. The validity of the signature was verified using ICGC and GEO datasets. The tumor microenvironment (TME) was examined using ESTIMATE and CIBERSORT, and potential therapeutic targets were identified through machine learning. Single-cell sequencing was used to examine the distribution of signature genes in cells within the TME. Cell viability and migration were tested to confirm the role of the key SPGs.

Results

We identified 28 SPGs that have an impact on survival. Using clinicopathological features and 6 genes, we developed a nomogram for HCC. The high- and low-risk groups were found to have distinct immune characteristics and response to drugs. Unlike CD8 T cells, M0 and M2 macrophages were found to be highly infiltrated in the TME of the high-risk subgroup. High levels of SPGs were found to be a good indicator of response to immunotherapy. In cell function experiments, SMPD2 and CSTA were found to enhance survival and migration of Huh7 cells, while silencing these genes increased the sensitivity of Huh7 cells to lapatinib.

Conclusion

The study presents a six-gene signature and a nomogram that can aid clinicians in choosing personalized treatments for HCC patients. Furthermore, it uncovers the connection between sphingolipid-related genes and the immune microenvironment, offering a novel approach for immunotherapy. By focusing on crucial sphingolipid genes like SMPD2 and CSTA, the efficacy of anti-tumor therapy can be increased in HCC cells.

Deep learning in digital pathology for personalized treatment plans of cancer patients

Over the past decade, many new cancer treatments have been developed and made available to patients. However, in most cases, these treatments only benefit a specific subgroup of patients, making the selection of treatment for a specific patient an essential but challenging task for oncologists. Although some biomarkers were found to associate with treatment response, manual assessment is time-consuming and subjective. With the rapid developments and expanded implementation of artificial intelligence (AI) in digital pathology, many biomarkers can be quantified automatically from histopathology images. This approach allows for a more efficient and objective assessment of biomarkers, aiding oncologists in formulating personalized treatment plans for cancer patients. This review presents an overview and summary of the recent studies on biomarker quantification and treatment response prediction using hematoxylin-eosin (H&E) stained pathology images. These studies have shown that an AI-based digital pathology approach can be practical and will become increasingly important in improving the selection of cancer treatments for patients.

Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology

Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.

The Applications of Artificial Intelligence in Digestive System Neoplasms: A Review

Importance

Digestive system neoplasms (DSNs) are the leading cause of cancer-related mortality with a 5-year survival rate of less than 20%. Subjective evaluation of medical images including endoscopic images, whole slide images, computed tomography images, and magnetic resonance images plays a vital role in the clinical practice of DSNs, but with limited performance and increased workload of radiologists or pathologists. The application of artificial intelligence (AI) in medical image analysis holds promise to augment the visual interpretation of medical images, which could not only automate the complicated evaluation process but also convert medical images into quantitative imaging features that associated with tumor heterogeneity.

Highlights

We briefly introduce the methodology of AI for medical image analysis and then review its clinical applications including clinical auxiliary diagnosis, assessment of treatment response, and prognosis prediction on 4 typical DSNs including esophageal cancer, gastric cancer, colorectal cancer, and hepatocellular carcinoma.

Conclusion

AI technology has great potential in supporting the clinical diagnosis and treatment decision-making of DSNs. Several technical issues should be overcome before its application into clinical practice of DSNs.

Interpretable deep learning model to predict the molecular classification of endometrial cancer from haematoxylin and eosin-stained whole-slide images: a combined analysis of the PORTEC randomised trials and clinical cohorts

BACKGROUND

Endometrial cancer can be molecularly classified into POLE^mut, mismatch repair deficient (MMRd), p53 abnormal (p53abn), and no specific molecular profile (NSMP) subgroups. We aimed to develop an interpretable deep learning pipeline for whole-slide-image-based prediction of the four molecular classes in endometrial cancer (im4MEC), to identify morpho-molecular correlates, and to refine prognostication.

METHODS

This combined analysis included diagnostic haematoxylin and eosin-stained slides and molecular and clinicopathological data from 2028 patients with intermediate-to-high-risk endometrial cancer from the PORTEC-1 (n=466), PORTEC-2 (n=375), and PORTEC-3 (n=393) randomised trials and the TransPORTEC pilot study (n=110), the Medisch Spectrum Twente cohort (n=242), a case series of patients with POLE^mut endometrial cancer in the Leiden Endometrial Cancer Repository (n=47), and The Cancer Genome Atlas-Uterine Corpus Endometrial Carcinoma cohort (n=395). PORTEC-3 was held out as an independent test set and a four-fold cross validation was performed. Performance was measured with the macro and class-wise area under the receiver operating characteristic curve (AUROC). Whole-slide images were segmented into tiles of 360 μm resized to 224 × 224 pixels. im4MEC was trained to learn tile-level morphological features with self-supervised learning and to molecularly classify whole-slide images with an attention mechanism. The top 20 tiles with the highest attention scores were reviewed to identify morpho-molecular correlates. Predictions of a nuclear classification deep learning model serve to derive interpretable morphological features. We analysed 5-year recurrence-free survival and explored prognostic refinement by molecular class using the Kaplan-Meier method.

FINDINGS

im4MEC attained macro-average AUROCs of 0·874 (95% CI 0·856-0·893) on four-fold cross-validation and 0·876 on the independent test set. The class-wise AUROCs were 0·849 for POLE^mut (n=51), 0·844 for MMRd (n=134), 0·883 for NSMP (n=120), and 0·928 for p53abn (n=88). POLE^mut and MMRd tiles had a high density of lymphocytes, p53abn tiles had strong nuclear atypia, and the morphology of POLE^mut and MMRd endometrial cancer overlapped. im4MEC highlighted a low tumour-to-stroma ratio as a potentially novel characteristic feature of the NSMP class. 5-year recurrence-free survival was significantly different between im4MEC predicted molecular classes in PORTEC-3 (log-rank p<0·0001). The ten patients with aggressive p53abn endometrial cancer that was predicted as MMRd showed inflammatory morphology and appeared to have a better prognosis than patients with correctly predicted p53abn endometrial cancer (p=0·30). The four patients with NSMP endometrial cancer that was predicted as p53abn showed higher nuclear atypia and appeared to have a worse prognosis than patients with correctly predicted NSMP (p=0·13). Patients with MMRd endometrial cancer predicted as POLE^mut had an excellent prognosis, as do those with true POLE^mut endometrial cancer.

INTERPRETATION

We present the first interpretable deep learning model, im4MEC, for haematoxylin and eosin-based prediction of molecular endometrial cancer classification. im4MEC robustly identified morpho-molecular correlates and could enable further prognostic refinement of patients with endometrial cancer.

FUNDING

The Hanarth Foundation, the Promedica Foundation, and the Swiss Federal Institutes of Technology.

Multimodal investigation of bladder cancer data based on computed tomography, whole slide imaging, and transcriptomics

Background

Multimodal analysis has shown great potential in the diagnosis and management of cancer. This study aimed to determine the multimodal data associations between radiological, pathologic, and molecular characteristics in bladder cancer.

Methods

A retrospective study of computed tomography (CT), pathologic slice, and RNA sequencing data from 127 consecutive adult patients in China who underwent bladder surgery and were pathologically diagnosed with bladder cancer was conducted. A total of 200 radiological and 1,029 pathologic features were extracted by radiomics and pathomics. Multimodal associations analysis and structural equation modeling were used to measure the cross-modal associations and structural relationships between CT and pathologic slice. A convolutional neural network was constructed for molecular subtyping based on multimodal imaging features. Class activation maps were used to examine the feature contribution in model decision-making. Cox regression and Kaplan-Meier survival analysis were used to explore the relevance of multimodal features to the prognosis of patients with bladder cancer.

Results

A total of 77 densely associated blocks of feature pairs were identified between CT and whole slide images. The largest cross-modal associated block reflected the tumor-grade properties. A significant relation was found between pathological features and molecular subtypes (β=0.396; P<0.001). High-grade bladder cancer showed heterogeneity of significance across different scales and higher disorders at the microscopic level. The fused radiological and pathologic features achieved higher accuracy (area under the curve: 0.89; 95% CI: 0.75-1.0) than the unimodal method. Thirteen prognosis-related features from CT and whole slide images were identified.

Conclusions

Our work demonstrated the associations between CT, pathologic slices, and molecular signatures, and the potential to use multimodal data analysis in related clinical applications. Multimodal data analysis showed the potential of cross-inference of modal data and had higher diagnostic accuracy than the unimodal method.

Facts and Hopes on the Use of Artificial Intelligence for Predictive Immunotherapy Biomarkers in Cancer

Immunotherapy by immune checkpoint inhibitors has become a standard treatment strategy for many types of solid tumors. However, the majority of patients with cancer will not respond, and predicting response to this therapy is still a challenge. Artificial intelligence (AI) methods can extract meaningful information from complex data, such as image data. In clinical routine, radiology or histopathology images are ubiquitously available. AI has been used to predict the response to immunotherapy from radiology or histopathology images, either directly or indirectly via surrogate markers. While none of these methods are currently used in clinical routine, academic and commercial developments are pointing toward potential clinical adoption in the near future. Here, we summarize the state of the art in AI-based image biomarkers for immunotherapy response based on radiology and histopathology images. We point out limitations, caveats, and pitfalls, including biases, generalizability, and explainability, which are relevant for researchers and health care providers alike, and outline key clinical use cases of this new class of predictive biomarkers.

Application of Deep Learning in Cancer Prognosis Prediction Model

As an important branch of artificial intelligence and machine learning, deep learning (DL) has been widely used in various aspects of cancer auxiliary diagnosis, among which cancer prognosis is the most important part. High-accuracy cancer prognosis is beneficial to the clinical management of patients with cancer. Compared with other methods, DL models can significantly improve the accuracy of prediction. Therefore, this article is a systematic review of the latest research on DL in cancer prognosis prediction. First, the data type, construction process, and performance evaluation index of the DL model are introduced in detail. Then, the current mainstream baseline DL cancer prognosis prediction models, namely, deep neural networks, convolutional neural networks, deep belief networks, deep residual networks, and vision transformers, including network architectures, the latest application in cancer prognosis, and their respective characteristics, are discussed. Next, some key factors that affect the predictive performance of the model and common performance enhancement techniques are listed. Finally, the limitations of the DL cancer prognosis prediction model in clinical practice are summarized, and the future research direction is prospected. This article could provide relevant researchers with a comprehensive understanding of DL cancer prognostic models and is expected to promote the research progress of cancer prognosis prediction.

Artificial intelligence applications in predicting the behavior of gastrointestinal cancers in pathology

Recent research has provided a wealth of data supporting the application of artificial intelligence (AI)-based applications in routine pathology practice. Indeed, it is clear that these methods can significantly support an accurate and rapid diagnosis by eliminating errors, increasing reliability, and improving workflow. In addition, the effectiveness of AI in the pathological evaluation of prognostic parameters associated with behavior, course, and treatment in many types of tumors has also been noted. Regarding gastrointestinal system (GIS) cancers, the contribution of AI methods to pathological diagnosis has been investigated in many studies. On the other hand, studies focusing on AI applications in evaluating parameters to determine tumor behavior are relatively few. For this purpose, the potential of AI models has been studied over a broad spectrum, from tumor subtyping to the identification of new digital biomarkers. The capacity of AI to infer genetic alterations of cancer tissues from digital slides has been demonstrated. Although current data suggest the merit of AI-based approaches in assessing tumor behavior in GIS cancers, a wide range of challenges still need to be solved, from laboratory infrastructure to improving the robustness of algorithms, before incorporating AI applications into real-life GIS pathology practice. This review aims to present data from AI applications in evaluating pathological parameters related to the behavior of GIS cancer with an overview of the opportunities and challenges encountered in implementing AI in pathology.

Quantitative Pathologic Analysis of Digitized Images of Colorectal Carcinoma Improves Prediction of Recurrence-Free Survival

BACKGROUND & AIMS

To examine whether quantitative pathologic analysis of digitized hematoxylin and eosin slides of colorectal carcinoma (CRC) correlates with clinicopathologic features, molecular alterations, and prognosis.

METHODS

A quantitative segmentation algorithm (QuantCRC) was applied to 6468 digitized hematoxylin and eosin slides of CRCs. Fifteen parameters were recorded from each image and tested for associations with clinicopathologic features and molecular alterations. A prognostic model was developed to predict recurrence-free survival using data from the internal cohort (n = 1928) and validated on an internal test (n = 483) and external cohort (n = 938).

RESULTS

There were significant differences in QuantCRC according to stage, histologic subtype, grade, venous/lymphatic/perineural invasion, tumor budding, CD8 immunohistochemistry, mismatch repair status, KRAS mutation, BRAF mutation, and CpG methylation. A prognostic model incorporating stage, mismatch repair, and QuantCRC resulted in a Harrell's concordance (c)-index of 0.714 (95% confidence interval [CI], 0.702-0.724) in the internal test and 0.744 (95% CI, 0.741-0.754) in the external cohort. Removing QuantCRC from the model reduced the c-index to 0.679 (95% CI, 0.673-0.694) in the external cohort. Prognostic risk groups were identified, which provided a hazard ratio of 2.24 (95% CI, 1.33-3.87, P = .004) for low vs high-risk stage III CRCs and 2.36 (95% CI, 1.07-5.20, P = .03) for low vs high-risk stage II CRCs, in the external cohort after adjusting for established risk factors. The predicted median 36-month recurrence rate for high-risk stage III CRCs was 32.7% vs 13.4% for low-risk stage III and 15.8% for high-risk stage II vs 5.4% for low-risk stage II CRCs.

CONCLUSIONS

QuantCRC provides a powerful adjunct to routine pathologic reporting of CRC. A prognostic model using QuantCRC improves prediction of recurrence-free survival.

Optimization of deep learning models for the prediction of gene mutations using unsupervised clustering

Deep learning models are increasingly being used to interpret whole-slide images (WSIs) in digital pathology and to predict genetic mutations. Currently, it is commonly assumed that tumor regions have most of the predictive power. However, it is reasonable to assume that other tissues from the tumor microenvironment may also provide important predictive information. In this paper, we propose an unsupervised clustering-based multiple-instance deep learning model for the prediction of genetic mutations using WSIs of three cancer types obtained from The Cancer Genome Atlas. Our proposed model facilitates the identification of spatial regions related to specific gene mutations and exclusion of patches that lack predictive information through the use of unsupervised clustering. This results in a more accurate prediction of gene mutations when compared with models using all image patches on WSIs and two recently published algorithms for all three different cancer types evaluated in this study. In addition, our study validates the hypothesis that the prediction of gene mutations solely based on tumor regions on WSI slides may not always provide the best performance. Other tissue types in the tumor microenvironment could provide a better prediction ability than tumor tissues alone. These results highlight the heterogeneity in the tumor microenvironment and the importance of identification of predictive image patches in digital pathology prediction tasks.

Deep learning model to predict Epstein-Barr virus associated gastric cancer in histology

The detection of Epstein-Barr virus (EBV) in gastric cancer patients is crucial for clinical decision making, as it is related with specific treatment responses and prognoses. Despite its importance, the limited medical resources preclude universal EBV testing. Herein, we propose a deep learning-based EBV prediction method from H&E-stained whole-slide images (WSI). Our model was developed using 319 H&E stained WSI (26 EBV positive; TCGA dataset) from the Cancer Genome Atlas, and 108 WSI (8 EBV positive; ISH dataset) from an independent institution. Our deep learning model, EBVNet consists of two sequential components: a tumor classifier and an EBV classifier. We visualized the learned representation by the classifiers using UMAP. We externally validated the model using 60 additional WSI (7 being EBV positive; HGH dataset). We compared the model's performance with those of four pathologists. EBVNet achieved an AUPRC of 0.65, whereas the four pathologists yielded a mean AUPRC of 0.41. Moreover, EBVNet achieved an negative predictive value, sensitivity, specificity, precision, and F1-score of 0.98, 0.86, 0.92, 0.60, and 0.71, respectively. Our proposed model is expected to contribute to prescreen patients for confirmatory testing, potentially to save test-related cost and labor.

Phenotypic plasticity and genetic control in colorectal cancer evolution

Genetic and epigenetic variation, together with transcriptional plasticity, contribute to intratumour heterogeneity1. The interplay of these biological processes and their respective contributions to tumour evolution remain unknown. Here we show that intratumour genetic ancestry only infrequently affects gene expression traits and subclonal evolution in colorectal cancer (CRC). Using spatially resolved paired whole-genome and transcriptome sequencing, we find that the majority of intratumour variation in gene expression is not strongly heritable but rather 'plastic'. Somatic expression quantitative trait loci analysis identified a number of putative genetic controls of expression by cis-acting coding and non-coding mutations, the majority of which were clonal within a tumour, alongside frequent structural alterations. Consistently, computational inference on the spatial patterning of tumour phylogenies finds that a considerable proportion of CRCs did not show evidence of subclonal selection, with only a subset of putative genetic drivers associated with subclone expansions. Spatial intermixing of clones is common, with some tumours growing exponentially and others only at the periphery. Together, our data suggest that most genetic intratumour variation in CRC has no major phenotypic consequence and that transcriptional plasticity is, instead, widespread within a tumour.

Correlation of Immunological and Histopathological Features with Gene Expression-Based Classifiers in Colon Cancer Patients

The purpose of this study was to evaluate the association between four distinct histopathological features: (1) tumor infiltrating lymphocytes, (2) mucinous differentiation, (3) tumor-stroma ratio, plus (4) tumor budding and two gene expression-based classifiers—(1) consensus molecular subtypes (CMS) plus (2) colorectal cancer intrinsic subtypes (CRIS). All four histopathological features were retrospectively scored on hematoxylin and eosin sections of the most invasive part of the primary tumor in 218 stage II and III colon cancer patients from two independent cohorts (AMC-AJCC-90 and AC-ICAM). RNA-based CMS and CRIS assignments were independently obtained for all patients. Contingency tables were constructed and a χ2 test was used to test for statistical significance. Odds ratios with 95% confidence intervals were calculated. The presence of tumor infiltrating lymphocytes and a mucinous phenotype (>50% mucinous surface area) were strongly correlated with CMS1 (p < 0.001 and p = 0.008) and CRIS-A (p = 0.006 and p < 0.001). The presence of mucus (≥ 10%) was associated with CMS3: mucus was present in 64.1% of all CMS3 tumors (p < 0.001). Although a clear association between tumor-stroma ratio and CMS4 was established in this study (p = 0.006), still 32 out of 61 (52.5%) CMS4 tumors were scored as stroma-low, indicating that CMS4 tumors cannot be identified solely based on stromal content. Higher budding counts were seen in CMS4 and CRIS-B tumors (p = 0.045 and p = 0.046). No other associations of the measured parameters were seen for any of the other CRIS subtypes. Our analysis revealed clear associations between histopathologic features and CMS or CRIS subtypes. However, identification of distinct molecular subtypes solely based on histopathology proved to be infeasible. Combining both molecular and morphologic features could potentially improve patient stratification.

Use of Personalized Biomarkers in Metastatic Colorectal Cancer and the Impact of AI

Colorectal cancer is a major cause of cancer-related death worldwide and is correlated with genetic and epigenetic alterations in the colonic epithelium. Genetic changes play a major role in the pathophysiology of colorectal cancer through the development of gene mutations, but recent research has shown an important role for epigenetic alterations. In this review, we try to describe the current knowledge about epigenetic alterations, including DNA methylation and histone modifications, as well as the role of non-coding RNAs as epigenetic regulators and the prognostic and predictive biomarkers in metastatic colorectal disease that can allow increases in the effectiveness of treatments. Additionally, the intestinal microbiota's composition can be an important biomarker for the response to strategies based on the immunotherapy of CRC. The identification of biomarkers in mCRC can be enhanced by developing artificial intelligence programs. We present the actual models that implement AI technology as a bridge connecting ncRNAs with tumors and conducted some experiments to improve the quality of the model used as well as the speed of the model that provides answers to users. In order to carry out this task, we implemented six algorithms: the naive Bayes classifier, the random forest classifier, the decision tree classifier, gradient boosted trees, logistic regression and SVM.

Digital image analysis provides robust tissue microenvironment-based prognosticators in patients with stage I-IV colorectal cancer

In patients with colorectal cancer (CRC), a promising marker is tumor-stroma ratio (TSR). Quantification issues highlight the importance of precise assessment that might be solved by artificial intelligence-based digital image analysis systems. Some alternatives have been offered so far, although these platforms are either proprietary developments or require additional programming skills. Our aim was to validate a user-friendly, commercially available software running in everyday computational environment to improve TSR assessment and also to compare the prognostic value of assessing TSR in 3 distinct regions of interests, like hotspot, invasive front, and whole tumor. Furthermore, we compared the prognostic power of TSR with the newly suggested carcinoma percentage (CP) and carcinoma-stroma percentage (CSP). Slides of 185 patients with stage I-IV CRC with clinical follow-up data were scanned and evaluated by a senior pathologist. A machine learning-based digital pathology software was trained to recognize tumoral and stromal compartments. The aforementioned parameters were evaluated in the hotspot, invasive front, and whole tumor area, both visually and by machine learning. Patients were classified based on TSR, CP, and CSP values. On multivariate analysis, TSR-hotspot was found to be an independent prognostic factor of overall survival (hazard ratio for TSR-hotspotsoftware: 2.005 [95% confidence interval (CI): 1.146-3.507], P = .011, for TSR-hotspotvisual: 1.781 [CI: 1.060-2.992], P = .029). Also, TSR was an independent predictor for distant metastasis and local relapse in most settings. Generally, software performance was comparable to visual evaluation and delivered reliable prognostication in more settings also with CP and CSP values. This study presents that software-assisted evaluation is a robust prognosticator. Our approach used a less sophisticated and thus easily accessible software without the aid of a convolutional neural network; however, it was still effective enough to deliver reliable prognostic information.

Deep learning-based prediction of molecular cancer biomarkers from tissue slides: A new tool for precision oncology

Molecular tests are necessary to stratify cancer patients for targeted therapy. However, high cost and technical barriers limit the application of these tests, hindering optimal treatment. Recently, deep learning (DL) has been applied to predict molecular test results from digitized images of tissue slides. Furthermore, treatment response and prognosis can be predicted from tissue slides using DL. In this review, we summarized DL-based studies regarding the prediction of genetic mutation, microsatellite instability, tumor mutational burden, molecular subtypes, gene expression, treatment response, and prognosis directly from hematoxylin- and eosin-stained tissue slides. Although performance needs to be improved, these studies clearly demonstrated the feasibility of DL-based prediction of key molecular features in cancer tissues. With the accumulation of data and technical advances, the performance of the DL system could be improved in the near future. Therefore, we expect that DL could provide cost- and time-effective alternative tools for patient stratification in the era of precision oncology.

Multi-modality artificial intelligence in digital pathology

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

Consensus molecular subtype 4 (CMS4)-targeted therapy in primary colon cancer: A proof-of-concept study

Background

Mesenchymal Consensus Molecular Subtype 4 (CMS4) colon cancer is associated with poor prognosis and therapy resistance. In this proof-of-concept study, we assessed whether a rationally chosen drug could mitigate the distinguishing molecular features of primary CMS4 colon cancer.

Methods

In the ImPACCT trial, informed consent was obtained for molecular subtyping at initial diagnosis of colon cancer using a validated RT-qPCR CMS4-test on three biopsies per tumor (Phase-1, n=69 patients), and for neoadjuvant CMS4-targeting therapy with imatinib (Phase-2, n=5). Pre- and post-treatment tumor biopsies were analyzed by RNA-sequencing and immunohistochemistry. Imatinib-induced gene expression changes were associated with molecular subtypes and survival in an independent cohort of 3232 primary colon cancer.

Results

The CMS4-test classified 52/172 biopsies as CMS4 (30%). Five patients consented to imatinib treatment prior to surgery, yielding 15 pre- and 15 post-treatment samples for molecular analysis. Imatinib treatment caused significant suppression of mesenchymal genes and upregulation of genes encoding epithelial junctions. The gene expression changes induced by imatinib were associated with improved survival and a shift from CMS4 to CMS2.

Conclusion

Imatinib may have value as a CMS-switching drug in primary colon cancer and induces a gene expression program that is associated with improved survival.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Complement downregulation promotes an inflammatory signature that renders colorectal cancer susceptible to immunotherapy

BACKGROUND AND AIMS

The role of inflammatory immune responses in colorectal cancer (CRC) development and response to therapy is a matter of intense debate. While inflammation is a known driver of CRC, inflammatory immune infiltrates are a positive prognostic factor in CRC and predispose to response to immune checkpoint blockade (ICB) therapy. Unfortunately, over 85% of CRC cases are primarily unresponsive to ICB due to the absence of an immune infiltrate, and even the cases that show an initial immune infiltration can become refractory to ICB. The identification of therapy supportive immune responses in the field has been partially hindered by the sparsity of suitable mouse models to recapitulate the human disease. In this study, we aimed to understand how the dysregulation of the complement anaphylatoxin C3a receptor (C3aR), observed in subsets of patients with CRC, affects the immune responses, the development of CRC, and response to ICB therapy.

METHODS

We use a comprehensive approach encompassing analysis of publicly available human CRC datasets, inflammation-driven and newly generated spontaneous mouse models of CRC, and multiplatform high-dimensional analysis of immune responses using microbiota sequencing, RNA sequencing, and mass cytometry.

RESULTS

We found that patients' regulation of the complement C3aR is associated with epigenetic modifications. Specifically, downregulation of C3ar1 in human CRC promotes a tumor microenvironment characterized by the accumulation of innate and adaptive immune cells that support antitumor immunity. In addition, in vivo studies in our newly generated mouse model revealed that the lack of C3a in the colon activates a microbiota-mediated proinflammatory program which promotes the development of tumors with an immune signature that renders them responsive to the ICB therapy.

CONCLUSIONS

Our findings reveal that C3aR may act as a previously unrecognized checkpoint to enhance antitumor immunity in CRC. C3aR can thus be exploited to overcome ICB resistance in a larger group of patients with CRC.

The evolving role of morphology in endometrial cancer diagnostics: From histopathology and molecular testing towards integrative data analysis by deep learning

Endometrial cancer (EC) diagnostics is evolving into a system in which molecular aspects are increasingly important. The traditional histological subtype-driven classification has shifted to a molecular-based classification that stratifies EC into DNA polymerase epsilon mutated (POLEmut), mismatch repair deficient (MMRd), and p53 abnormal (p53abn), and the remaining EC as no specific molecular profile (NSMP). The molecular EC classification has been implemented in the World Health Organization 2020 classification and the 2021 European treatment guidelines, as it serves as a better basis for patient management. As a result, the integration of the molecular class with histopathological variables has become a critical focus of recent EC research. Pathologists have observed and described several morphological characteristics in association with specific genomic alterations, but these appear insufficient to accurately classify patients according to molecular subgroups. This requires pathologists to rely on molecular ancillary tests in routine workup. In this new era, it has become increasingly challenging to assign clinically relevant weights to histological and molecular features on an individual patient basis. Deep learning (DL) technology opens new options for the integrative analysis of multi-modal image and molecular datasets with clinical outcomes. Proof-of-concept studies in other cancers showed promising accuracy in predicting molecular alterations from H&E-stained tumor slide images. This suggests that some morphological characteristics that are associated with molecular alterations could be identified in EC, too, expanding the current understanding of the molecular-driven EC classification. Here in this review, we report the morphological characteristics of the molecular EC classification currently identified in the literature. Given the new challenges in EC diagnostics, this review discusses, therefore, the potential supportive role that DL could have, by providing an outlook on all relevant studies using DL on histopathology images in various cancer types with a focus on EC. Finally, we touch upon how DL might shape the management of future EC patients.

Current Developments of Artificial Intelligence in Digital Pathology and Its Future Clinical Applications in Gastrointestinal Cancers

The implementation of DP will revolutionize current practice by providing pathologists with additional tools and algorithms to improve workflow. Furthermore, DP will open up opportunities for development of AI-based tools for more precise and reproducible diagnosis through computational pathology. One of the key features of AI is its capability to generate perceptions and recognize patterns beyond the human senses. Thus, the incorporation of AI into DP can reveal additional morphological features and information. At the current rate of AI development and adoption of DP, the interest in computational pathology is expected to rise in tandem. There have already been promising developments related to AI-based solutions in prostate cancer detection; however, in the GI tract, development of more sophisticated algorithms is required to facilitate histological assessment of GI specimens for early and accurate diagnosis. In this review, we aim to provide an overview of the current histological practices in AP laboratories with respect to challenges faced in image preprocessing, present the existing AI-based algorithms, discuss their limitations and present clinical insight with respect to the application of AI in early detection and diagnosis of GI cancer.

Consensus Molecular Subtypes Efficiently Classify Gastric Adenocarcinomas and Predict the Response to Anti-PD-1 Immunotherapy

Background: Gastric adenocarcinoma (GAC) is highly heterogeneous and closely related to colorectal cancer (CRC) both molecularly and functionally. GAC is currently subtyped using a system developed by TCGA. However, with the emergence of immunotherapies, this system has failed to identify suitable treatment candidates. Methods: Consensus molecular subtypes (CMSs) developed for CRC were used for molecular subtyping in GAC based on public expression cohorts, including TCGA, ACRG, and a cohort of GAC patients treated with the programmed cell death 1 (PD-1) inhibitor pembrolizumab. All aspects of each subtype, including clinical outcome, molecular characteristics, oncogenic pathway activity, and the response to immunotherapy, were fully explored. Results: CMS classification was efficiently applied to GAC. CMS4, characterized by EMT activation, stromal invasion, angiogenesis, and the worst clinical outcomes (median OS 24.2 months), was the predominant subtype (38.8%~44.3%) and an independent prognostic indicator that outperformed classical TCGA subtyping. CMS1 (20.9%~21.5%) displayed hypermutation, low SCNV, immune activation, and best clinical outcomes (median OS > 120 months). CMS3 (17.95%~25.7%) was characterized by overactive metabolism, KRAS mutation, and intermediate outcomes (median OS 85.6 months). CMS2 (14.6%~16.3%) was enriched for WNT and MYC activation, differentiated epithelial characteristics, APC mutation, lack of ARID1A, and intermediate outcomes (median OS 48.7 months). Notably, CMS1 was strongly correlated with immunotherapy biomarkers and favorable for the anti-PD-1 drug pembrolizumab, whereas CMS4 was poorly responsive but became more sensitive after EMT-based stratification. Conclusions: Our study reveals the practical utility of CMS classification for GAC to improve clinical outcomes and identify candidates who will respond to immunotherapy.

Improving feature extraction from histopathological images through a fine-tuning ImageNet model

Background

Due to lack of annotated pathological images, transfer learning has been the predominant approach in the field of digital pathology. Pre-trained neural networks based on ImageNet database are often used to extract "off-the-shelf" features, achieving great success in predicting tissue types, molecular features, and clinical outcomes, etc. We hypothesize that fine-tuning the pre-trained models using histopathological images could further improve feature extraction, and downstream prediction performance.

Methods

We used 100 000 annotated H&E image patches for colorectal cancer (CRC) to fine-tune a pre-trained Xception model via a 2-step approach. The features extracted from fine-tuned Xception (FTX-2048) model and Image-pretrained (IMGNET-2048) model were compared through: (1) tissue classification for H&E images from CRC, same image type that was used for fine-tuning; (2) prediction of immune-related gene expression, and (3) gene mutations for lung adenocarcinoma (LUAD). Five-fold cross validation was used for model performance evaluation. Each experiment was repeated 50 times.

Findings

The extracted features from the fine-tuned FTX-2048 exhibited significantly higher accuracy (98.4%) for predicting tissue types of CRC compared to the "off-the-shelf" features directly from Xception based on ImageNet database (96.4%) (P value = 2.2 × 10-6). Particularly, FTX-2048 markedly improved the accuracy for stroma from 87% to 94%. Similarly, features from FTX-2048 boosted the prediction of transcriptomic expression of immune-related genes in LUAD. For the genes that had significant relationships with image features (P < 0.05, n = 171), the features from the fine-tuned model improved the prediction for the majority of the genes (139; 81%). In addition, features from FTX-2048 improved prediction of mutation for 5 out of 9 most frequently mutated genes (STK11, TP53, LRP1B, NF1, and FAT1) in LUAD.

Conclusions

We proved the concept that fine-tuning the pretrained ImageNet neural networks with histopathology images can produce higher quality features and better prediction performance for not only the same-cancer tissue classification where similar images from the same cancer are used for fine-tuning, but also cross-cancer prediction for gene expression and mutation at patient level.

Deep-learning-based classification of desmoplastic reaction on H&E predicts poor prognosis in oesophageal squamous cell carcinoma

AIMS

Desmoplastic reaction (DR) categorisation has been shown to be a promising prognostic factor in oesophageal squamous cell carcinoma (ESCC). The usual DR evaluation is performed using semiquantitative scores, which can be subjective. This study aimed to investigate whether a deep-learning classifier could be used for DR classification. We further assessed the prognostic significance of the deep-learning classifier and compared it to that of manual DR reporting and other pathological factors currently used in the clinic.

METHODS AND RESULTS

From 222 surgically resected ESCC cases, 31 randomly selected haematoxylin-eosin-digitised whole slides of patients with immature DR were used to train and develop a deep-learning classifier. The classifier was trained for 89 370 iterations. The accuracy of the deep-learning classifier was assessed to 30 unseen cases, and the results revealed a Dice coefficient score of 0.81. For survival analysis, the classifier was then applied to the entire cohort of patients, which was split into a training (n = 156) and a test (n = 66) cohort. The automated DR classification had a higher prognostic significance for disease-specific survival than the manually classified DR in both the training and test cohorts. In addition, the automated DR classification outperformed the prognostic accuracy of the gold-standard factors of tumour depth and lymph node metastasis.

CONCLUSIONS

This study demonstrated that DR can be objectively and quantitatively assessed in ESCC using a deep-learning classifier and that automatically classed DR has a higher prognostic significance than manual DR and other features currently used in the clinic.

Immune Response in Colorectal Carcinoma: A Review of Its Significance as a Predictive and Prognostic Biomarker

Colorectal carcinoma is a leading cause of cancer-related death worldwide. There is significant prognostic heterogeneity in stage II and III tumours, necessitating the development of new biomarkers to better identify patients at risk of disease progression. Recently, the tumour immune environment, particularly the type and quantity of T lymphocytes, has been shown to be a useful biomarker in predicting prognosis for patients with colorectal carcinoma. In this review, the significance of the immune response in colorectal carcinoma, including its influence on prognosis and response to therapy, will be detailed.

Towards a national strategy for digital pathology in Switzerland

Precision medicine is entering a new era of digital diagnostics; the availability of integrated digital pathology (DP) and structured clinical datasets has the potential to become a key catalyst for biomedical research, education and business development. In Europe, national programs for sharing of this data will be crucial for the development, testing, and validation of machine learning-enabled tools supporting clinical decision-making. Here, the Swiss Digital Pathology Consortium (SDiPath) discusses the creation of a Swiss Digital Pathology Infrastructure (SDPI), which aims to develop a unified national DP network bringing together the Swiss Personalized Health Network (SPHN) with Swiss university hospitals and subsequent inclusion of cantonal and private institutions. This effort builds on existing developments for the national implementation of structured pathology reporting. Opening this national infrastructure and data to international researchers in a sequential rollout phase can enable the large-scale integration of health data and pooling of resources for research purposes and clinical trials. Therefore, the concept of a SDPI directly synergizes with the priorities of the European Commission communication on the digital transformation of healthcare on an international level, and with the aims of the Swiss State Secretariat for Economic Affairs (SECO) for advancing research and innovation in the digitalization domain. SDPI directly addresses the needs of existing national and international research programs in neoplastic and non-neoplastic diseases by providing unprecedented access to well-curated clinicopathological datasets for the development and implementation of novel integrative methods for analysis of clinical outcomes and treatment response. In conclusion, a SDPI would facilitate and strengthen inter-institutional collaboration in technology, clinical development, business and research at a national and international scale, promoting improved patient care via precision medicine.

Recent Applications of Artificial Intelligence from Histopathologic Image-Based Prediction of Microsatellite Instability in Solid Cancers: A Systematic Review

Cancers with high microsatellite instability (MSI-H) have a better prognosis and respond well to immunotherapy. However, MSI is not tested in all cancers because of the additional costs and time of diagnosis. Therefore, artificial intelligence (AI)-based models have been recently developed to evaluate MSI from whole slide images (WSIs). Here, we aimed to assess the current state of AI application to predict MSI based on WSIs analysis in MSI-related cancers and suggest a better study design for future studies. Studies were searched in online databases and screened by reference type, and only the full texts of eligible studies were reviewed. The included 14 studies were published between 2018 and 2021, and most of the publications were from developed countries. The commonly used dataset is The Cancer Genome Atlas dataset. Colorectal cancer (CRC) was the most common type of cancer studied, followed by endometrial, gastric, and ovarian cancers. The AI models have shown the potential to predict MSI with the highest AUC of 0.93 in the case of CRC. The relatively limited scale of datasets and lack of external validation were the limitations of most studies. Future studies with larger datasets are required to implicate AI models in routine diagnostic practice for MSI prediction.

Spatially resolved transcriptomics provide a new method for cancer research

A major feature of cancer is the heterogeneity, both intratumoral and intertumoral. Traditional single-cell techniques have given us a comprehensive understanding of the biological characteristics of individual tumor cells, but the lack of spatial context of the transcriptome has limited the study of cell-to-cell interaction patterns and hindered further exploration of tumor heterogeneity. In recent years, the advent of spatially resolved transcriptomics (SRT) technology has made possible the multidimensional analysis of the tumor microenvironment in the context of intact tissues. Different SRT methods are applicable to different working ranges due to different working principles. In this paper, we review the advantages and disadvantages of various current SRT methods and the overall idea of applying these techniques to oncology studies, hoping to help researchers find breakthroughs. Finally, we discussed the future direction of SRT technology, and deeper investigation into the complex mechanisms of tumor development from different perspectives through multi-omics fusion, paving the way for precisely targeted tumor therapy.

Multi-Size Deep Learning Based Preoperative Computed Tomography Signature for Prognosis Prediction of Colorectal Cancer

Background: Preoperative and postoperative evaluation of colorectal cancer (CRC) patients is crucial for subsequent treatment guidance. Our study aims to provide a timely and rapid assessment of the prognosis of CRC patients with deep learning according to non-invasive preoperative computed tomography (CT) and explore the underlying biological explanations.

Methods: A total of 808 CRC patients with preoperative CT (development cohort: n = 426, validation cohort: n = 382) were enrolled in our study. We proposed a novel end-to-end Multi-Size Convolutional Neural Network (MSCNN) to predict the risk of CRC recurrence with CT images (CT signature). The prognostic performance of CT signature was evaluated by Kaplan-Meier curve. An integrated nomogram was constructed to improve the clinical utility of CT signature by combining with other clinicopathologic factors. Further visualization and correlation analysis for CT deep features with paired gene expression profiles were performed to reveal the molecular characteristics of CRC tumors learned by MSCNN in radiographic imaging.

Results: The Kaplan-Meier analysis showed that CT signature was a significant prognostic factor for CRC disease-free survival (DFS) prediction [development cohort: hazard ratio (HR): 50.7, 95% CI: 28.4–90.6, p &lt; 0.001; validation cohort: HR: 2.04, 95% CI: 1.44–2.89, p &lt; 0.001]. Multivariable analysis confirmed the independence prognostic value of CT signature (development cohort: HR: 30.7, 95% CI: 19.8–69.3, p &lt; 0.001; validation cohort: HR: 1.83, 95% CI: 1.19–2.83, p = 0.006). Dimension reduction and visualization of CT deep features demonstrated a high correlation with the prognosis of CRC patients. Functional pathway analysis further indicated that CRC patients with high CT signature presented down-regulation of several immunology pathways. Correlation analysis found that CT deep features were mainly associated with activation of metabolic and proliferative pathways.

Conclusions: Our deep learning based preoperative CT signature can effectively predict prognosis of CRC patients. Integration analysis of multi-omic data revealed that some molecular characteristics of CRC tumor can be captured by deep learning in CT images.

Developing image analysis methods for digital pathology

The potential to use quantitative image analysis and artificial intelligence is one of the driving forces behind digital pathology. However, despite novel image analysis methods for pathology being described across many publications, few become widely adopted and many are not applied in more than a single study. The explanation is often straightforward: software implementing the method is simply not available, or is too complex, incomplete, or dataset‐dependent for others to use. The result is a disconnect between what seems already possible in digital pathology based upon the literature, and what actually is possible for anyone wishing to apply it using currently available software. This review begins by introducing the main approaches and techniques involved in analysing pathology images. I then examine the practical challenges inherent in taking algorithms beyond proof‐of‐concept, from both a user and developer perspective. I describe the need for a collaborative and multidisciplinary approach to developing and validating meaningful new algorithms, and argue that openness, implementation, and usability deserve more attention among digital pathology researchers. The review ends with a discussion about how digital pathology could benefit from interacting with and learning from the wider bioimage analysis community, particularly with regard to sharing data, software, and ideas. © 2022 The Author. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Swarm learning for decentralized artificial intelligence in cancer histopathology

Artificial intelligence (AI) can predict the presence of molecular alterations directly from routine histopathology slides. However, training robust AI systems requires large datasets for which data collection faces practical, ethical and legal obstacles. These obstacles could be overcome with swarm learning (SL), in which partners jointly train AI models while avoiding data transfer and monopolistic data governance. Here, we demonstrate the successful use of SL in large, multicentric datasets of gigapixel histopathology images from over 5,000 patients. We show that AI models trained using SL can predict BRAF mutational status and microsatellite instability directly from hematoxylin and eosin (H&E)-stained pathology slides of colorectal cancer. We trained AI models on three patient cohorts from Northern Ireland, Germany and the United States, and validated the prediction performance in two independent datasets from the United Kingdom. Our data show that SL-trained AI models outperform most locally trained models, and perform on par with models that are trained on the merged datasets. In addition, we show that SL-based AI models are data efficient. In the future, SL can be used to train distributed AI models for any histopathology image analysis task, eliminating the need for data transfer.

A decentralized, privacy-preserving machine learning framework used to train a clinically relevant AI system identifies actionable molecular alterations in patients with colorectal cancer by use of routine histopathology slides collected in real-world settings.