Pan-cancer integrative histology-genomic analysis via multimodal deep learning

Closing the gap in the clinical adoption of computational pathology: a standardized, open-source framework to integrate deep-learning models into the laboratory information system

BACKGROUND

Digital pathology (DP) has revolutionized cancer diagnostics and enabled the development of deep-learning (DL) models aimed at supporting pathologists in their daily work and improving patient care. However, the clinical adoption of such models remains challenging. Here, we describe a proof-of-concept framework that, leveraging Health Level 7 (HL7) standard and open-source DP resources, allows a seamless integration of both publicly available and custom developed DL models in the clinical workflow.

METHODS

Development and testing of the framework were carried out in a fully digitized Italian pathology department. A Python-based server-client architecture was implemented to interconnect through HL7 messaging the anatomic pathology laboratory information system (AP-LIS) with an external artificial intelligence-based decision support system (AI-DSS) containing 16 pre-trained DL models. Open-source toolboxes for DL model deployment were used to run DL model inference, and QuPath was used to provide an intuitive visualization of model predictions as colored heatmaps.

RESULTS

A default deployment mode runs continuously in the background as each new slide is digitized, choosing the correct DL model(s) on the basis of the tissue type and staining. In addition, pathologists can initiate the analysis on-demand by selecting a specific DL model from the virtual slide tray. In both cases, the AP-LIS transmits an HL7 message to the AI-DSS, which processes the message, runs DL model inference, and creates the appropriate visualization style for the employed classification model. The AI-DSS transmits model inference results to the AP-LIS, where pathologists can visualize the output in QuPath and/or directly as slide description in the virtual slide tray.

CONCLUSIONS

Taken together, the developed integration framework through the use of the HL7 standard and freely available DP resources offers a standardized, portable, and open-source solution that lays the groundwork for the future widespread adoption of DL models in pathology diagnostics.

Multimodal integration of longitudinal noninvasive diagnostics for survival prediction in immunotherapy using deep learning

OBJECTIVES

Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for personalized treatment approaches.

MATERIALS AND METHODS

In this study, we developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network, building upon a combination of recent successful developments. We integrated pre- and on-treatment blood measurements, prescribed medications, and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality at 3, 6, 9, and 12 months. Different variants of our extended MMTSimTA network were implemented and compared to baseline methods, incorporating intermediate and late fusion-based integration methods.

RESULTS

The strongest prognostic performance was demonstrated using a variant of the MMTSimTA model with area under the curves of 0.84 ± 0.04, 0.83 ± 0.02, 0.82 ± 0.02, 0.81 ± 0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively.

DISCUSSION

Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction.

CONCLUSION

Our study demonstrates that multimodal longitudinal integration of noninvasive data using deep learning may offer a promising approach for personalized prognostication in immunotherapy-treated cancer patients.

Multimodal Explainable Artificial Intelligence for Prognostic Stratification of Glioblastoma Patients

Glioblastoma is the most common and aggressive malignant adult tumor of the central nervous system, with a grim prognosis and heterogeneous morphologic and molecular profiles. Since the adoption of the current standard-of-care treatment in 2005, no substantial prognostic improvement has been noticed. In this study, we seek the identification of prognostically relevant glioblastoma characteristics from routinely acquired hematoxylin and eosin-stained whole slide images (WSI) and clinical data, that integrated via advanced computational methods could yield improved patient prognostic stratification and hence optimize clinical decision making and patient management. The proposed WSI analysis capitalizes on a comprehensive curation of apparent artifactual content and an interpretability mechanism via a weakly supervised attention-based multiple- instance learning approach that further utilizes clustering to constrain the search space. Patterns automatically identified by our approach as of high prognostic value classify each WSI as representative of short or long survivors. Further assessments of the prognostic relevance of the associated clinical patient data are performed both in isolation and in an integrated manner, using XGBoost and shapley additive explanations (SHAP). The multimodal integration of WSI with clinical data yields enhanced stratification performance when compared with using either one of the modalities. Identifying tumor morphological and clinical patterns associated with short and long survival will enable the clinical neuropathologist to provide additional relevant prognostic information to the treating team and suggest avenues of biological investigation for further understanding and potentially treating glioblastoma.

End-to-end prognostication in pancreatic cancer by multimodal deep learning: a retrospective, multicenter study

OBJECTIVES

Pancreatic cancer treatment plans involving surgery and/or chemotherapy are highly dependent on disease stage. However, current staging systems are ineffective and poorly correlated with survival outcomes. We investigate how artificial intelligence (AI) can enhance prognostic accuracy in pancreatic cancer by integrating multiple data sources.

MATERIALS AND METHODS

Patients with histopathology and/or radiology/follow-up confirmed pancreatic ductal adenocarcinoma (PDAC) from a Dutch center (2004-2023) were included in the development cohort. Two additional PDAC cohorts from a Dutch and Spanish center were used for external validation. Prognostic models including clinical variables, contrast-enhanced CT images, and a combination of both were developed to predict high-risk short-term survival. All models were trained using five-fold cross-validation and assessed by the area under the time-dependent receiver operating characteristic curve (AUC).

RESULTS

The models were developed on 401 patients (203 females, 198 males, median survival (OS) = 347 days, IQR: 171-585), with 98 (24.4%) short-term survivors (OS < 230 days) and 303 (75.6%) long-term survivors. The external validation cohorts included 361 patients (165 females, 138 males, median OS = 404 days, IQR: 173-736), with 110 (30.5%) short-term survivors and 251 (69.5%) longer survivors. The best AUC for predicting short vs. long-term survival was achieved with the multi-modal model (AUC = 0.637 (95% CI: 0.500-0.774)) in the internal validation set. External validation showed AUCs of 0.571 (95% CI: 0.453-0.689) and 0.675 (95% CI: 0.593-0.757).

CONCLUSION

Multimodal AI can predict long vs. short-term survival in PDAC patients, showing potential as a prognostic tool in clinical decision-making.

KEY POINTS

Question Prognostic tools for pancreatic ductal adenocarcinoma (PDAC) remain limited, with TNM staging offering suboptimal accuracy in predicting patient survival outcomes. Findings The multimodal AI model demonstrated improved prognostic performance over TNM and unimodal models for predicting short- and long-term survival in PDAC patients. Clinical relevance Multimodal AI provides enhanced prognostic accuracy compared to current staging systems, potentially improving clinical decision-making and personalized management strategies for PDAC patients.

Multimodal fusion model for prognostic prediction and radiotherapy response assessment in head and neck squamous cell carcinoma

Accurate prediction of prognosis and postoperative radiotherapy response is critical for personalized treatment in head and neck squamous cell carcinoma (HNSCC). We developed a multimodal deep learning model (MDLM) integrating computed tomography, whole-slide images, and clinical features from 1087 HNSCC patients across multiple centers. The MDLM exhibited good performance in predicting overall survival (OS) and disease-free survival in external test cohorts. Additionally, the MDLM outperformed unimodal models. Patients with a high-risk score who underwent postoperative radiotherapy exhibited prolonged OS compared to those who did not (P = 0.016), whereas no significant improvement in OS was observed among patients with a low-risk score (P = 0.898). Biological exploration indicated that the model may be related to changes in the cytochrome P450 metabolic pathway, tumor microenvironment, and myeloid-derived cell subpopulations. Overall, the MDLM effectively predicts prognosis and postoperative radiotherapy response, offering a promising tool for personalized HNSCC therapy.

Deep learning radiopathomics based on pretreatment MRI and whole slide images for predicting over survival in locally advanced nasopharyngeal carcinoma

PURPOSE

To develop an integrative radiopathomic model based on deep learning to predict overall survival (OS) in locally advanced nasopharyngeal carcinoma (LANPC) patients.

MATERIALS AND METHODS

A cohort of 343 LANPC patients with pretreatment MRI and whole slide image (WSI) were randomly divided into training (n = 202), validation (n = 91), and external test (n = 50) sets. For WSIs, a self-attention mechanism was employed to assess the significance of different patches for the prognostic task, aggregating them into a WSI-level representation. For MRI, a multilayer perceptron was used to encode the extracted radiomic features, resulting in an MRI-level representation. These were combined in a multimodal fusion model to produce prognostic predictions. Model performances were evaluated using the concordance index (C-index), and Kaplan-Meier curves were employed for risk stratification. To enhance model interpretability, attention-based and Integrated Gradients techniques were applied to explain how WSIs and MRI features contribute to prognosis predictions.

RESULTS

The radiopathomics model achieved high predictive accuracy in predicting the OS, with a C-index of 0.755 (95 % CI: 0.673-0.838) and 0.744 (95 % CI: 0.623-0.808) in the training and validation sets, respectively, outperforming single-modality models (radiomic signature: 0.636, 95 % CI: 0.584-0.688; deep pathomic signature: 0.736, 95 % CI: 0.684-0.810). In the external test, similar findings were observed for the predictive performance of the radiopathomics, radiomic signature, and deep pathomic signature, with their C-indices being 0.735, 0.626, and 0.660 respectively. The radiopathomics model effectively stratified patients into high- and low-risk groups (P < 0.001). Additionally, attention heatmaps revealed that high-attention regions corresponded with tumor areas in both risk groups.

CONCLUSIO

n: The radiopathomics model holds promise for predicting clinical outcomes in LANPC patients, offering a potential tool for improving clinical decision-making.

Sex differences in bladder cancer: understanding biological and clinical implications

Bladder cancer (BC) remains a significant global health concern, with substantial sex and racial disparities in incidence, progression, and outcomes. BC is the sixth most common cancer among males and the seventeenth most common among females worldwide. Over 90% of BC cases are urothelial carcinoma (UC) with high degrees of pathological heterogeneity. Molecular subtyping of BC has also revealed distinct luminal, basal, and neuroendocrine subtypes, each with unique genetic and immune signatures. Emerging research uncovers the biasing effects of the sex hormones with androgens increasing BC risk through both tumor cell intrinsic and extrinsic mechanisms. The sex chromosomes, including both the X and Y chromosomes, also contribute to the sex differences in BC. The effect of sex chromosome is both independent from and synergistic with the effects of sex hormones. Loss of the Y chromosome is frequently observed in BC patients, while an extra copy of the X chromosome confers better protection against BC in females than in males. Advent of advanced technologies such as multiomics and artificial intelligence will likely further improve the understanding of sex differences in BC, which may ultimately lead to personalized preventative and treatment strategies depending on the biological sex of patients. This review delves into the impacts of biology of sex on BC, emphasizing the importance of further research into sex-specific biology to improve cancer prevention and care.

Evaluating an information theoretic approach for selecting multimodal data fusion methods

OBJECTIVE

Interest has grown in combining radiology, pathology, genomic, and clinical data to improve the accuracy of diagnostic and prognostic predictions toward precision health. However, most existing works choose their datasets and modeling approaches empirically and in an ad hoc manner. A prior study proposed four partial information decomposition (PID)-based metrics to provide a theoretical understanding of multimodal data interactions: redundancy, uniqueness of each modality, and synergy. However, these metrics have only been evaluated in a limited collection of biomedical data, and the existing work does not elucidate the effect of parameter selection when calculating the PID metrics. In this work, we evaluate PID metrics on a wider range of biomedical data, including clinical, radiology, pathology, and genomic data, and propose potential improvements to the PID metrics.

METHODS

We apply the PID metrics to seven different modality pairs across four distinct cohorts (datasets). We compare and interpret trends in the resulting PID metrics and downstream model performance in these multimodal cohorts. The downstream tasks being evaluated include predicting the prognosis (either overall survival or recurrence) of patients with non-small cell lung cancer, prostate cancer, and glioblastoma.

RESULTS

We found that, while PID metrics are informative, solely relying on these metrics to decide on a fusion approach does not always yield a machine learning model with optimal performance. Of the seven different modality pairs, three had poor (0%), three had moderate (66%-89%), and only one had perfect (100%) consistency between the PID values and model performance. We propose two improvements to the PID metrics (determining the optimal parameters and uncertainty estimation) and identified areas where PID metrics could be further improved.

CONCLUSION

The current PID metrics are not accurate enough for estimating the multimodal data interactions and need to be improved before they can serve as a reliable tool. We propose improvements and provide suggestions for future work. Code: https://github.com/zhtyolivia/pid-multimodal.

Cross-modal alignment and contrastive learning for enhanced cancer survival prediction

BACKGROUND AND OBJECTIVE

Integrating multimodal data, such as pathology images and genomics, is crucial for understanding cancer heterogeneity, personalized treatment complexity, and enhancing survival prediction. However, most current prognostic methods are limited to a single domain of histopathology or genomics, inevitably reducing their potential for accurate patient outcome prediction. Despite advancements in the concurrent analysis of pathology and genomic data, existing approaches inadequately address the intricate intermodal relationships.

METHODS

This paper introduces the CPathomic method for multimodal data-based survival prediction. By leveraging whole slide pathology images to guide local pathological features, the method effectively mitigates significant intermodal differences through a cross-modal representational contrastive learning module. Furthermore, it facilitates interactive learning between different modalities through cross-modal and gated attention modules.

RESULTS

The extensive experiments on five public TCGA datasets demonstrate that CPathomic framework effectively bridges modality gaps, consistently outperforming alternative multimodal survival prediction methods.

CONCLUSION

The model we propose, CPathomic, unveils the potential of contrastive learning and cross-modal attention in the representation and fusion of multimodal data, enhancing the performance of patient survival prediction.

Comprehensive analysis of SQOR involvement in ferroptosis resistance of pancreatic ductal adenocarcinoma in hypoxic environments

Introduction

Pancreatic ductal adenocarcinoma (PDAC) exhibits higher hypoxia level than most solid tumors, and the presence of intratumoral hypoxia is associated with a poor prognosis. However, the identification of hypoxia levels based on pathological images, and the mechanisms regulating ferroptosis resistance, remain to be elucidated. The objective of this study was to construct a deep learning model to evaluate the hypoxia characteristics of PDAC and to explore the role of Sulfide quinone oxidoreductase (SQOR) in hypoxia-mediated ferroptosis resistance.

Methods

Multi-omics data were integrated to analyze the correlation between hypoxia score of PDAC, SQOR expression and prognosis, and ferroptosis resistance level. A deep learning model of Whole Slide Images (WSIs) were constructed to predict the hypoxia level of patients. In vitro hypoxia cell models, SQOR knockdown experiments and nude mouse xenograft models were used to verify the regulatory function of SQOR on ferroptosis.

Results

PDAC exhibited significantly higher hypoxia levels than normal tissues, correlating with reduced overall survival in patients. In slide level, our deep learning model can effectively identify PDAC hypoxia levels with good performance. SQOR was upregulated in tumor tissues and positively associated with both hypoxia score and ferroptosis resistance. SQOR promotes the malignant progression of PDAC in hypoxic environment by enhancing the resistance of tumor cells to ferroptosis. SQOR knockdown resulted in decreased cell viability, decreased migration ability and increased MDA level under hypoxic Ersatin induced conditions. Furthermore, SQOR inhibitor in combination with ferroptosis inducer has the potential to inhibit tumor growth in vivo in a synergistic manner.

Discussion

This study has established a hypoxia detection model of PDAC based on WSIs, providing a new tool for clinical evaluation. The study revealed a new mechanism of SQOR mediating ferroptosis resistance under hypoxia and provided a basis for targeted therapy.

Interpretable Multimodal Fusion Model for Bridged Histology and Genomics Survival Prediction in Pan-Cancer

Understanding the prognosis of cancer patients is crucial for enabling precise diagnosis and treatment by clinical practitioners. Multimodal fusion models based on artificial intelligence (AI) offer a comprehensive depiction of the tumor heterogeneity landscape, facilitating more accurate predictions of cancer patient prognosis. However, in the real-world, the lack of complete multimodal data from patients often hinders the practical clinical utility of such models. To address this limitation, an interpretable bridged multimodal fusion model is developed that combines histopathology, genomics, and transcriptomics. This model assists clinical practitioners in achieving more precise prognosis predictions, particularly when patients lack corresponding molecular features. The predictive capabilities of the model are validated across 12 cancer types, achieving optimal performance in both complete and missing modalities. The work highlights the promise of developing a clinically applicable medical multimodal fusion model. This not only aids in reducing the healthcare burden on cancer patients but also provides improved assistance for clinical practitioners in precise diagnosis and treatment.

Bridging Domain Gaps in Computational Pathology: A Comparative Study of Adaptation Strategies

Due to the high variability in Hematoxylin and Eosin (H&E)-stained Whole Slide Images (WSIs), hidden stratification, and batch effects, generalizing beyond the training distribution is one of the main challenges in Deep Learning (DL) for Computational Pathology (CPath). But although DL depends on large volumes of diverse and annotated data, it is common to have a significant number of annotated samples from one or multiple source distributions, and another partially annotated or unlabeled dataset representing a target distribution for which we want to generalize, the so-called Domain Adaptation (DA). In this work, we focus on the task of generalizing from a single source distribution to a target domain. As it is still not clear which domain adaptation strategy is best suited for CPath, we evaluate three different DA strategies, namely FixMatch, CycleGAN, and a self-supervised feature extractor, and show that DA is still a challenge in CPath.

Multimodal radiopathological integration for prognosis and prediction of adjuvant chemotherapy benefit in resectable lung adenocarcinoma: A multicentre study

Lung adenocarcinoma (LUAD) has a heterogeneous prognosis and controversial postoperative treatment protocols. We aim to develop and validate a multimodal analysis framework that integrates CT images with H&E-stained whole-slide images (WSIs) to enhance risk stratification and predict adjuvant chemotherapy benefit in LUAD patients. We retrospectively collected data from 1039 resectable LUAD patients (stage I-III) across four centres, forming a training dataset (n = 303), two testing datasets (n = 197 and n = 228) for survival analysis, and a feature testing dataset (n = 311) for interpretability analysis. We extracted 487 tumour/peritumour radiomics features from CT images and 783 multiscale pathomics features from WSIs, characterising the shape of tumour (CT) and cancer nuclei (WSIs), as well as the intensity and texture of tumour/peritumour regions (CT) and tumour regions/epithelium/stroma (WSIs). A survival support vector machine (SVM) was employed to establish a radiopathomics signature using the optimal set of multimodal features, including 2 tumour radiomics features, 3 peritumour radiomics features, and 4 nuclei heterogeneity pathomics features. The radiopathomics signature outperformed both radiomics and pathomics signatures in predicting disease-free survival (DFS) (C-index: training dataset, 0.744 vs. 0.734 and 0.692; testing dataset 1, 0.719 vs. 0.701 and 0.638; testing dataset 2, 0.711 vs. 0.689 and 0.684), demonstrating greater robustness compared to the state-of-the-art deep learning integration approaches. It provided additional prognostic information beyond clinical risk factors (C-index of clinical plus radiopathomics vs. clinical models: training dataset, 0.763 vs. 0.676; testing dataset 1, 0.739 vs. 0.676; testing dataset 2, 0.711 vs. 0.699, p < 0.001). Compared to low-risk patients categorised by the radiopathomics signature, high-risk patients achieved comparable DFS when receiving adjuvant chemotherapy (training dataset, HR = 1.53, 95 % CI 0.85-2.73, p = 0.153; testing dataset 1 and 2, HR = 1.62, 95 % CI 0.92-2.85, p = 0.096), but had significantly worse DFS when only observed after surgery (training dataset, HR = 4.46, 95 % CI 2.82-7.05, p < 0.001; testing datasets 1 and 2, HR = 3.52, 95 % CI 2.26-5.49, p < 0.001), indicating the predictive value of the radiopathomics signature for adjuvant chemotherapy benefit (interaction p < 0.05). Further interpretability analysis revealed that the radiopathomics signature was associated with various prognostic/treatment-related biomarkers, including differentiation, immune phenotypes, and EGFR status. The multimodal integration framework offered a cost-effective approach for LUAD characterisation by leveraging complementary information from radiological and histopathological imaging. The radiopathomics signature demonstrated robust prognostic capabilities, providing valuable insights for postoperative treatment decisions.

Artificial intelligence entering the pathology arena in oncology: current applications and future perspectives

BACKGROUND

Artificial intelligence (AI) is rapidly transforming the fields of pathology and oncology, offering novel opportunities for advancing diagnosis, prognosis, and treatment of cancer.

METHODS

Through a systematic review-based approach, the representatives from the European Society for Medical Oncology (ESMO) Precision Oncology Working Group (POWG) and international experts identified studies in pathology and oncology that applied AI-based algorithms for tumour diagnosis, molecular biomarker detection, and cancer prognosis assessment. These findings were synthesised to provide a comprehensive overview of current AI applications and future directions in cancer pathology.

RESULTS

The integration of AI tools in digital pathology is markedly improving the accuracy and efficiency of image analysis, allowing for automated tumour detection and classification, identification of prognostic molecular biomarkers, and prediction of treatment response and patient outcomes. Several barriers for the adoption of AI in clinical workflows, such as data availability, explainability, and regulatory considerations, still persist. There are currently no prognostic or predictive AI-based biomarkers supported by level IA or IB evidence. The ongoing advancements in AI algorithms, particularly foundation models, generalist models and transformer-based deep learning, offer immense promise for the future of cancer research and care. AI is also facilitating the integration of multi-omics data, leading to more precise patient stratification and personalised treatment strategies.

CONCLUSIONS

The application of AI in pathology is poised to not only enhance the accuracy and efficiency of cancer diagnosis and prognosis but also facilitate the development of personalised treatment strategies. Although barriers to implementation remain, ongoing research and development in this field coupled with addressing ethical and regulatory considerations will likely lead to a future where AI plays an integral role in cancer management and precision medicine. The continued evolution and adoption of AI in pathology and oncology are anticipated to reshape the landscape of cancer care, heralding a new era of precision medicine and improved patient outcomes.

Leveraging TME features and multi-omics data with an advanced deep learning framework for improved Cancer survival prediction

Glioma, a malignant intracranial tumor with high invasiveness and heterogeneity, significantly impacts patient survival. This study integrates multi-omics data to improve prognostic prediction and identify therapeutic targets. Using single-cell data from glioblastoma (GBM) and low-grade glioma (LGG) samples, we identified 55 distinct cell states via the EcoTyper framework, validated for stability and prognostic impact in an independent cohort. We constructed multi-omics datasets of 620 samples, integrating transcriptomic, copy number variation (CNV), somatic mutation (MUT), Microbe (MIC), EcoTyper result data. A scRNA-seq enhanced Self-Normalizing Network-based glioma prognosis model achieved a C-index of 0.822 (training) and 0.817 (test), with AUC values of 0.867, 0.876, and 0.844 at 1, 3, and 5 years in the training set, and 0.820, 0.947, and 0.936 in the test set. Gradient attribution analysis enhanced the interpretability of the model and identified key molecular markers. The classification into high- and low-risk groups was validated as an independent prognostic factor. HDAC inhibitors are proposed as potential treatments. This study demonstrates the potential of integrating scRNA-seq and multi-omics data for robust glioma prognosis and clinical decision-making support.

Foundation model-driven multimodal prognostic prediction in patients undergoing primary surgery for high-grade serous ovarian cancer

High-grade serous ovarian cancer (HGSOC) presents challenges in prognostic prediction. This study aimed to develop a universal foundation model-driven multimodal model (FoMu model) to assess the prognosis of HGSOC patients. We conducted a retrospective cohort study involving 712 eligible patients across four centers, collecting clinical, MRI, and hematoxylin and eosin (H&E)-stained whole slide images (WSIs) data. Pre-trained radiological and pathological foundation models were employed for feature precoding. Subsequently, we introduced unimodal and cross-modal adaptive aggregation networks to comprehensively model the features derived from each modality. Our findings revealed that both unimodal and cross-modal FoMu models exhibited superior and stable predictive capabilities for overall survival (OS) and progression-free survival (PFS). In summary, our study successfully developed a FoMu model that effectively integrates multimodal data to assess the prognoses of HGSOC patients, highlighting its potential for improving individualized patient management and clinical decision-making in future applications.

Perspectives on cancer therapy-synthetic lethal precision medicine strategies, molecular mechanisms, therapeutic targets and current technical challenges

In recent years, synthetic lethality has become an important theme in the field of targeted cancer therapy. Synthetic lethality refers to simultaneous defects in two or more genes leading to cell death, whereas defects in any single gene do not lead to cell death. Taking advantage of the genetic vulnerability that exists within cancer cells, it theoretically has no negative impact on healthy cells and has fewer side effects than non-specific chemotherapy. Currently, targeted cancer therapies focus on inhibiting key pathways in cancer. However, it has been found that over-activation of oncogenic-related signaling pathways can also induce cancer cell death, which is a major breakthrough in the new field of targeted therapies. In this review, we summarize the conventional gene targets in synthetic lethality (PARP, ATR, ATM, WEE1, PRMT) and provide an in-depth analysis of their latest potential mechanisms. We explore the impact of over-activation of pathways such as PI3K/AKT, MAPK, and WNT on cancer cell survival, and present the technical challenges of current research. Important theoretical foundations and insights are provided for the application of synthetic lethal strategies in cancer therapy, as well as future research directions.

New horizons at the interface of artificial intelligence and translational cancer research

Artificial intelligence (AI) is increasingly being utilized in cancer research as a computational strategy for analyzing multiomics datasets. Advances in single-cell and spatial profiling technologies have contributed significantly to our understanding of tumor biology, and AI methodologies are now being applied to accelerate translational efforts, including target discovery, biomarker identification, patient stratification, and therapeutic response prediction. Despite these advancements, the integration of AI into clinical workflows remains limited, presenting both challenges and opportunities. This review discusses AI applications in multiomics analysis and translational oncology, emphasizing their role in advancing biological discoveries and informing clinical decision-making. Key areas of focus include cellular heterogeneity, tumor microenvironment interactions, and AI-aided diagnostics. Challenges such as reproducibility, interpretability of AI models, and clinical integration are explored, with attention to strategies for addressing these hurdles. Together, these developments underscore the potential of AI and multiomics to enhance precision oncology and contribute to advancements in cancer care.

A concept-based interpretable model for the diagnosis of choroid neoplasias using multimodal data

Diagnosing rare diseases remains a critical challenge in clinical practice, often requiring specialist expertise. Despite the promising potential of machine learning, the scarcity of data on rare diseases and the need for interpretable, reliable artificial intelligence (AI) models complicates development. This study introduces a multimodal concept-based interpretable model tailored to distinguish uveal melanoma (0.4-0.6 per million in Asians) from hemangioma and metastatic carcinoma following the clinical practice. We collected a comprehensive dataset on Asians to date on choroid neoplasm imaging with radiological reports, encompassing over 750 patients from 2013 to 2019. Our model integrates domain expert insights from radiological reports and differentiates between three types of choroidal tumors, achieving an F1 score of 0.91. This performance not only matches senior ophthalmologists but also improves the diagnostic accuracy of less experienced clinicians by 42%. The results underscore the potential of interpretable AI to enhance rare disease diagnosis and pave the way for future advancements in medical AI.

Weakly supervised multi-modal contrastive learning framework for predicting the HER2 scores in breast cancer

Human epidermal growth factor receptor 2 (HER2) is an important biomarker for prognosis and prediction of treatment response in breast cancer (BC). HER2 scoring is typically evaluated by pathologist microscopic observation on immunohistochemistry (IHC) images, which is labor-intensive and results in observational biases among different pathologists. Most existing methods generally use hand-crafted features or deep learning models in unimodal (hematoxylin and eosin (H&E) or IHC) to predict HER2 scores through supervised or weakly supervised learning. Consequently, the information from different modalities is not effectively integrated into feature learning which can help improve HER2 scoring performance. In this paper, we propose a novel weakly supervised multi-modal contrastive learning (WSMCL) framework to predict the HER2 scores in BC at the whole slide image (WSI) level. It aims to leverage multi-modal (H&E and IHC) joint learning under the weak supervision of WSI label to achieve the HER2 score prediction. Specifically, the patch features within H&E and IHC WSIs are respectively extracted and then the multi-head self-attention (MHSA) is used to explore the global dependencies of the patches within each modality. The patch features corresponding to top-k and bottom-k attention scores generated by MHSA in each modality are selected as the candidates for multi-modal joint learning. Particularly, a multi-modal attentive contrastive learning (MACL) module is designed to guarantee the semantic alignment of the candidate features from different modalities. Extensive experiments demonstrate the proposed WSMCL has the better HER2 scoring performance and outperforms the state-of-the-art methods. The code is available at https://github.com/HFUT-miaLab/WSMCL.

Role of Radiomics-based Multiomics Panel in the Microenvironment and Prognosis of Hepatocellular Carcinoma

Hepatocellular carcinoma (HCC) is the most prevalent form of liver tumor, characterized by restricted therapeutic options and typically low long-term survival rates. Recently, immunotherapy has revolutionized HCC treatment, making the tumor microenvironment (TME) a research focus. Radiomics is increasingly crucial in HCC clinical decisions, offering advanced tools for TME characterization and prognosis assessment. Meanwhile, advancements in pathomics provide new insights into HCC's comprehensive traits and details. Advancements in genomics and transcriptomics enable the integration of radiomics and pathomics with genetic data to better understand HCC heterogeneity and its microenvironment, aiding prognostic assessments. This review provides a comprehensive overview of pivotal radiomics studies focused on TME prediction, underscoring the synergistic effects of integrating multiomics approaches for TME analysis and HCC outcome prediction. It critically examines the challenges and opportunities inherent in multiomics research, emphasizing its substantial significance in both research and clinical contexts.

The current landscape of artificial intelligence in computational histopathology for cancer diagnosis

Artificial intelligence (AI) marks a frontier in histopathologic analysis shift towards the clinic, becoming a mainstream choice to interpret histological images. Surveying studies assessing AI applications in histopathology from 2013 to 2024, we review key methods (including supervised, unsupervised, weakly supervised and transfer learning) in deep learning-based pattern recognition in computational histopathology for diagnostic and prognostic purposes. Deep learning methods also showed utility in identifying a wide range of genetic mutations and standard pathology biomarkers from routine histology. This survey of 41 primary studies also encompasses key regions of AI applicability in histopathology in a multi-cancer review while marking prospects to introduce AI into the clinical setting with key examples including Swarm Learning and Data Fusion.

TDMFS: Tucker decomposition multimodal fusion model for pan-cancer survival prediction

Integrated analysis of multimodal data offers a more comprehensive view for cancer survival prediction, yet it faces challenges like computational intensity, overfitting, and challenges in achieving a unified representation due to data heterogeneity. To address the above issues, the first Tucker decomposition multimodal fusion model was hereby proposed for pan-cancer survival prediction (TDMFS). The model employed Tucker decomposition to limit complex tensor parameters during fusion, achieving deep modality integration with reduced computational cost and lower overfitting risk. The individual modality-specific representations were then fully exploited by signal modulation mechanisms in a bilinear pooling decomposition to serve as complementary information for the deep fusion representation. Furthermore, the performance of TDMFS was evaluated using a 5-fold cross-validation method with two modal data, gene expression (GeneExpr), and copy number variation (CNV), for 33 cancers from The Cancer Genome Atlas (TCGA) database. The experiments demonstrated that the proposed TDMFS model achieved an average C-index of 0.757 across 33 cancer datasets, with a C-index exceeding 0.80 on 10 of these datasets. Survival curves for both high and low risk patients plotted on 27 cancer datasets were statistically significant. The TDMFS model demonstrated superior performance in survival prediction, outperforming models like LinearSum and Multimodal Factorisation Higher Order Pooling, making it a valuable asset for advancing clinical cancer research.

Current methods in explainable artificial intelligence and future prospects for integrative physiology

Explainable artificial intelligence (XAI) is gaining importance in physiological research, where artificial intelligence is now used as an analytical and predictive tool for many medical research questions. The primary goal of XAI is to make AI models understandable for human decision-makers. This can be achieved in particular through providing inherently interpretable AI methods or by making opaque models and their outputs transparent using post hoc explanations. This review introduces XAI core topics and provides a selective overview of current XAI methods in physiology. It further illustrates solved and discusses open challenges in XAI research using existing practical examples from the medical field. The article gives an outlook on two possible future prospects: (1) using XAI methods to provide trustworthy AI for integrative physiological research and (2) integrating physiological expertise about human explanation into XAI method development for useful and beneficial human-AI partnerships.

Multimodal data integration in early-stage breast cancer

The use of biomarkers in breast cancer has significantly improved patient outcomes through targeted therapies, such as hormone therapy anti-Her2 therapy and CDK4/6 or PARP inhibitors. However, existing knowledge does not fully encompass the diverse nature of breast cancer, particularly in triple-negative tumors. The integration of multi-omics and multimodal data has the potential to provide new insights into biological processes, to improve breast cancer patient stratification, enhance prognosis and response prediction, and identify new biomarkers. This review presents a comprehensive overview of the state-of-the-art multimodal (including molecular and image) data integration algorithms developed and with applicability to breast cancer stratification, prognosis, or biomarker identification. We examined the primary challenges and opportunities of these multimodal data integration algorithms, including their advantages, limitations, and critical considerations for future research. We aimed to describe models that are not only academically and preclinically relevant, but also applicable to clinical settings.

M4: Multi-proxy multi-gate mixture of experts network for multiple instance learning in histopathology image analysis

Multiple instance learning (MIL) has been successfully applied for whole slide images (WSIs) analysis in computational pathology, enabling a wide range of prediction tasks from tumor subtyping to inferring genetic mutations and multi-omics biomarkers. However, existing MIL methods predominantly focus on single-task learning, resulting in not only overall low efficiency but also the overlook of inter-task relatedness. To address these issues, we proposed an adapted architecture of Multi-gate Mixture-of-experts with Multi-proxy for Multiple instance learning (M4), and applied this framework for simultaneous prediction of multiple genetic mutations from WSIs. The proposed M4 model has two main innovations: (1) adopting a multi-gate mixture-of-experts strategy for multiple genetic mutation simultaneous prediction on a single WSI; (2) introducing a multi-proxy CNN construction on the expert and gate networks to effectively and efficiently capture patch-patch interactions from WSI. Our model achieved significant improvements across five tested TCGA datasets in comparison to current state-of-the-art single-task methods. The code is available at: https://github.com/Bigyehahaha/M4.

Future of Artificial Intelligence-Machine Learning Trends in Pathology and Medicine

Artificial intelligence (AI) and machine learning (ML) are transforming the field of medicine. Health care organizations are now starting to establish management strategies for integrating such platforms (AI-ML toolsets) that leverage the computational power of advanced algorithms to analyze data and to provide better insights that ultimately translate to enhanced clinical decision-making and improved patient outcomes. Emerging AI-ML platforms and trends in pathology and medicine are reshaping the field by offering innovative solutions to enhance diagnostic accuracy, operational workflows, clinical decision support, and clinical outcomes. These tools are also increasingly valuable in pathology research in which they contribute to automated image analysis, biomarker discovery, drug development, clinical trials, and productive analytics. Other related trends include the adoption of ML operations for managing models in clinical settings, the application of multimodal and multiagent AI to utilize diverse data sources, expedited translational research, and virtualized education for training and simulation. As the final chapter of our AI educational series, this review article delves into the current adoption, future directions, and transformative potential of AI-ML platforms in pathology and medicine, discussing their applications, benefits, challenges, and future perspectives.

Self-interactive learning: Fusion and evolution of multi-scale histomorphology features for molecular traits prediction in computational pathology

Predicting disease-related molecular traits from histomorphology brings great opportunities for precision medicine. Despite the rich information present in histopathological images, extracting fine-grained molecular features from standard whole slide images (WSI) is non-trivial. The task is further complicated by the lack of annotations for subtyping and contextual histomorphological features that might span multiple scales. This work proposes a novel multiple-instance learning (MIL) framework capable of WSI-based cancer morpho-molecular subtyping by fusion of different-scale features. Our method, debuting as Inter-MIL, follows a weakly-supervised scheme. It enables the training of the patch-level encoder for WSI in a task-aware optimisation procedure, a step normally not modelled in most existing MIL-based WSI analysis frameworks. We demonstrate that optimising the patch-level encoder is crucial to achieving high-quality fine-grained and tissue-level subtyping results and offers a significant improvement over task-agnostic encoders. Our approach deploys a pseudo-label propagation strategy to update the patch encoder iteratively, allowing discriminative subtype features to be learned. This mechanism also empowers extracting fine-grained attention within image tiles (the small patches), a task largely ignored in most existing weakly supervised-based frameworks. With Inter-MIL, we carried out four challenging cancer molecular subtyping tasks in the context of ovarian, colorectal, lung, and breast cancer. Extensive evaluation results show that Inter-MIL is a robust framework for cancer morpho-molecular subtyping with superior performance compared to several recently proposed methods, in small dataset scenarios where the number of available training slides is less than 100. The iterative optimisation mechanism of Inter-MIL significantly improves the quality of the image features learned by the patch embedded and generally directs the attention map to areas that better align with experts' interpretation, leading to the identification of more reliable histopathology biomarkers. Moreover, an external validation cohort is used to verify the robustness of Inter-MIL on molecular trait prediction.

Artificial intelligence in gastrointestinal cancer research: Image learning advances and applications

With the rapid advancement of artificial intelligence (AI) technologies, including deep learning, large language models, and neural networks, these methodologies are increasingly being developed and integrated into cancer research. Gastrointestinal tumors are characterized by complexity and heterogeneity, posing significant challenges for early detection, diagnostic accuracy, and the development of personalized treatment strategies. The application of AI in digestive oncology has demonstrated its transformative potential. AI not only alleviates the diagnostic burden on clinicians, but it improves tumor screening sensitivity, specificity, and accuracy. Additionally, AI aids the detection of biomarkers such as microsatellite instability and mismatch repair, supports intraoperative assessments of tumor invasion depth, predicts treatment responses, and facilitates the design of personalized treatment plans to potentially significantly enhance patient outcomes. Moreover, the integration of AI with multiomics analyses and imaging technologies has led to substantial advancements in foundational research on the tumor microenvironment. This review highlights the progress of AI in gastrointestinal oncology over the past 5 years with focus on early tumor screening, diagnosis, molecular marker identification, treatment planning, and prognosis predictions. We also explored the potential of AI to enhance medical imaging analyses to aid tumor detection and characterization as well as its role in automating and refining histopathological assessments.

Peptide hydrogel platform encapsulating manganese ions and high-density lipoprotein nanoparticle-mimicking nanovaccines for the prevention and treatment of gastric cancer

BACKGROUND

Advanced gastric cancer remains a significant global health challenge, with limited therapeutic options available. In contrast, immunotherapy have emerged as promising alternatives, offering greater potency in treating advanced gastric cancer. However, the development of novel and efficient immunotherapeutic strategy is crucial to enhance the body's immune response against gastric cancer.

METHODS

This study developed a single-injection peptide hydrogel-based nanovaccine therapy for gastric cancer treatment. The therapy utilizes a RADA32 peptide hydrogel, which is sensitive to metal ion concentration, to encapsulate manganese ions and HPPS nanovaccines. The HPPS nanovaccines contain antigen peptide and CpG-ODN, designed to activate both the toll-like receptor 9 (TLR9) and cGAS-STING signaling pathways in antigen-presenting cells. This design aims to facilitate a stable and sustained release of the nanovaccine, thereby enhancing the body's effective recognition and response to antigens.

RESULTS

The efficacy of the system was confirmed using the model antigen OVA and the gastric cancer-specific antigen MG7-related peptide. The results demonstrated that the nanovaccine effectively activated the immune response, leading to enhanced recognition and response to the antigens. This activation of both TLR9 and cGAS-STING pathways in antigen-presenting cells was crucial for the observed immune response, highlighting the potential of this approach to stimulate a robust and sustained immune response against gastric cancer.

CONCLUSIONS

This study presents a novel strategy for clinical anti-tumor vaccine administration, offering a promising approach for the prevention and treatment of gastric cancer. The single-injection peptide hydrogel-based nanovaccine system provides a convenient and effective method to enhance the body's immune response against gastric cancer. This approach could potentially be expanded to other types of cancer, providing a versatile platform for cancer immunotherapy.

Pan-cancer analysis shapes the understanding of cancer biology and medicine

Advances in multi-omics datasets and analytical methods have revolutionized cancer research, offering a comprehensive, pan-cancer perspective. Pan-cancer studies identify shared mechanisms and unique traits across different cancer types, which are reshaping diagnostic and treatment strategies. However, continued innovation is required to refine these approaches and deepen our understanding of cancer biology and medicine. This review summarized key findings from pan-cancer research and  explored their potential to drive future advancements in oncology.

Predicting response to neoadjuvant chemotherapy in muscle-invasive bladder cancer via interpretable multimodal deep learning

Building accurate prediction models and identifying predictive biomarkers for treatment response in Muscle-Invasive Bladder Cancer (MIBC) are essential for improving patient survival but remain challenging due to tumor heterogeneity, despite numerous related studies. To address this unmet need, we developed an interpretable Graph-based Multimodal Late Fusion (GMLF) deep learning framework. Integrating histopathology and cell type data from standard H&E images with gene expression profiles derived from RNA sequencing from the SWOG S1314-COXEN clinical trial (ClinicalTrials.gov NCT02177695 2014-06-25), GMLF uncovered new histopathological, cellular, and molecular determinants of response to neoadjuvant chemotherapy. Specifically, we identified key gene signatures that drive the predictive power of our model, including alterations in TP63, CCL5, and DCN. Our discovery can optimize treatment strategies for patients with MIBC, e.g., improving clinical outcomes, avoiding unnecessary treatment, and ultimately, bladder preservation. Additionally, our approach could be used to uncover predictors for other cancers.

The artificial intelligence revolution in gastric cancer management: clinical applications

Nowadays, gastric cancer has become a significant issue in the global cancer burden, and its impact cannot be ignored. The rapid development of artificial intelligence technology is attempting to address this situation, aiming to change the clinical management landscape of gastric cancer fundamentally. In this transformative change, machine learning and deep learning, as two core technologies, play a pivotal role, bringing unprecedented innovations and breakthroughs in the diagnosis, treatment, and prognosis evaluation of gastric cancer. This article comprehensively reviews the latest research status and application of artificial intelligence algorithms in gastric cancer, covering multiple dimensions such as image recognition, pathological analysis, personalized treatment, and prognosis risk assessment. These applications not only significantly improve the sensitivity of gastric cancer risk monitoring, the accuracy of diagnosis, and the precision of survival prognosis but also provide robust data support and a scientific basis for clinical decision-making. The integration of artificial intelligence, from optimizing the diagnosis process and enhancing diagnostic efficiency to promoting the practice of precision medicine, demonstrates its promising prospects for reshaping the treatment model of gastric cancer. Although most of the current AI-based models have not been widely used in clinical practice, with the continuous deepening and expansion of precision medicine, we have reason to believe that a new era of AI-driven gastric cancer care is approaching.

Benchmarking pathology foundation models for non-neoplastic pathology in the placenta

Machine learning (ML) applications within diagnostic histopathology have been extremely successful. While many successful models have been built using general-purpose models trained largely on everyday objects, there is a recent trend toward pathology-specific foundation models, trained using histopathology images. Pathology foundation models show strong performance on cancer detection and subtyping, grading, and predicting molecular diagnoses. However, we have noticed lacunae in the testing of foundation models. Nearly all the benchmarks used to test them are focused on cancer. Neoplasia is an important pathologic mechanism and key concern in much of clinical pathology, but it represents one of many pathologic bases of disease. Non-neoplastic pathology dominates findings in the placenta, a critical organ in human development, as well as a specimen commonly encountered in clinical practice. Very little to none of the data used in training pathology foundation models is placenta. Thus, placental pathology is doubly out of distribution, representing a useful challenge for foundation models. We developed benchmarks for estimation of gestational age, classifying normal tissue, identifying inflammation in the umbilical cord and membranes, and in classification of macroscopic lesions including villous infarction, intervillous thrombus, and perivillous fibrin deposition. We tested 5 pathology foundation models and 4 non-pathology models for each benchmark in tasks including zero-shot K-nearest neighbor classification and regression, content-based image retrieval, supervised regression, and whole-slide attention-based multiple instance learning. In each task, the best performing model was a pathology foundation model. However, the gap between pathology and non-pathology models was diminished in tasks related to inflammation or those in which a supervised task was performed using model embeddings. Performance was comparable among pathology foundation models. Among non-pathology models, ResNet consistently performed worse, while models from the present decade showed better performance. Future work could examine the impact of incorporating placental data into foundation model training.

Prediction and analysis of tumor infiltrating lymphocytes across 28 cancers by TILScout using deep learning

The density of tumor-infiltrating lymphocytes (TILs) serves as a valuable indicator for predicting anti-tumor responses, but its broad impact across various types of cancers remains underexplored. We introduce TILScout, a pan-cancer deep-learning approach to compute patch-level TIL scores from whole slide images (WSIs). TILScout achieved accuracies of 0.9787 and 0.9628, and AUCs of 0.9988 and 0.9934 in classifying WSI patches into three categories-TIL-positive, TIL-negative, and other/necrotic-on validation and independent test sets, respectively, surpassing previous studies. The biological significance of TILScout-derived TIL scores across 28 cancers was validated through comprehensive functional and correlational analyses. A consistent decrease in TIL scores with an increase in cancer stage provides direct evidence that the lower TIL content may stimulate cancer progression. Additionally, TIL scores correlated with immune checkpoint gene expression and genomic variation in common cancer driver genes. Our comprehensive pan-cancer survey highlights the critical prognostic significance of TILs within the tumor microenvironment.

A deep-learning model to predict the completeness of cytoreductive surgery in colorectal cancer with peritoneal metastasis☆

BACKGROUND

Colorectal cancer (CRC) with peritoneal metastasis (PM) is associated with poor prognosis. The Peritoneal Cancer Index (PCI) is used to evaluate the extent of PM and to select Cytoreductive Surgery (CRS). However, PCI score is not accurate to guide patient's selection for CRS.

OBJECTIVE

We have developed a novel AI framework of decoupling feature alignment and fusion (DeAF) by deep learning to aid selection of PM patients and predict surgical completeness of CRS.

METHODS

186 CRC patients with PM recruited from four tertiary hospitals were enrolled. In the training cohort, deep learning was used to train the DeAF model using Simsiam algorithms by contrast CT images and then fuse clinicopathological parameters to increase performance. The accuracy, sensitivity, specificity, and AUC by ROC were evaluated both in the internal validation cohort and three external cohorts.

RESULTS

The DeAF model demonstrated a robust accuracy to predict the completeness of CRS with AUC of 0.9 (95 % CI: 0.793-1.000) in internal validation cohort. The model can guide selection of suitable patients and predict potential benefits from CRS. The high predictive performance in predicting CRS completeness were validated in three external cohorts with AUC values of 0.906(95 % CI: 0.812-1.000), 0.960(95 % CI: 0.885-1.000), and 0.933 (95 % CI: 0.791-1.000), respectively.

CONCLUSION

The novel DeAF framework can aid surgeons to select suitable PM patients for CRS and predict the completeness of CRS. The model can change surgical decision-making and provide potential benefits for PM patients.

Self-supervised learning reveals clinically relevant histomorphological patterns for therapeutic strategies in colon cancer

Self-supervised learning (SSL) automates the extraction and interpretation of histopathology features on unannotated hematoxylin-eosin-stained whole slide images (WSIs). We train an SSL Barlow Twins encoder on 435 colon adenocarcinoma WSIs from The Cancer Genome Atlas to extract features from small image patches (tiles). Leiden community detection groups tiles into histomorphological phenotype clusters (HPCs). HPC reproducibility and predictive ability for overall survival are confirmed in an independent clinical trial (N = 1213 WSIs). This unbiased atlas results in 47 HPCs displaying unique and shared clinically significant histomorphological traits, highlighting tissue type, quantity, and architecture, especially in the context of tumor stroma. Through in-depth analyses of these HPCs, including immune landscape and gene set enrichment analyses, and associations to clinical outcomes, we shine light on the factors influencing survival and responses to treatments of standard adjuvant chemotherapy and experimental therapies. Further exploration of HPCs may unveil additional insights and aid decision-making and personalized treatments for colon cancer patients.

Prognostic and predictive values of a multimodal nomogram incorporating tumor and peritumor morphology with immune status in resectable lung adenocarcinoma

BACKGROUND

Current prognostic and predictive biomarkers for lung adenocarcinoma (LUAD) predominantly rely on unimodal approaches, limiting their characterization ability. There is an urgent need for a comprehensive and accurate biomarker to guide individualized adjuvant therapy decisions.

METHODS

In this retrospective study, data from patients with resectable LUAD (stage I-III) were collected from two hospitals and a publicly available dataset, forming a training dataset (n=223), a validation dataset (n=95), a testing dataset (n=449), and the non-small cell lung cancer (NSCLC) Radiogenomics dataset (n=59). Tumor and peritumor scores were constructed from preoperative CT radiomics features (shape/intensity/texture). An immune score was derived from the density of tumor-infiltrating lymphocytes (TILs) within the cancer epithelium and stroma on hematoxylin and eosin-stained whole-slide images. A clinical score was constructed based on clinicopathological risk factors. A Cox regression model was employed to integrate these scores, thereby constructing a multimodal nomogram to predict disease-free survival (DFS). The adjuvant chemotherapy benefit rate was subsequently calculated based on this nomogram.

RESULTS

The multimodal nomogram outperformed each of the unimodal scores in predicting DFS, with a C-index of 0.769 (vs 0.634-0.731) in the training dataset, 0.730 (vs 0.548-0.713) in the validation dataset, and 0.751 (vs 0.660-0.692) in the testing dataset. It was independently associated with DFS after adjusting for other clinicopathological risk factors (training dataset: HR=3.02, p<0.001; validation dataset: HR=2.33, p<0.001; testing dataset: HR=2.03, p=0.001). The adjuvant chemotherapy benefit rate effectively distinguished between patients benefiting from adjuvant chemotherapy and those from observation alone (interaction p<0.001). Furthermore, the high-/low-risk groups defined by the multimodal nomogram provided refined stratification of candidates for adjuvant chemotherapy identified by current guidelines (p<0.001). Gene set enrichment analyses using the NSCLC Radiogenomics dataset revealed associations between tumor/peritumor scores and pathways involved in epithelial-mesenchymal transition, angiogenesis, IL6-JAK-STAT3 signaling, and reactive oxidative species.

CONCLUSION

The multimodal nomogram, which incorporates tumor and peritumor morphology with anti-tumor immune response, provides superior prognostic accuracy compared with unimodal scores. Its defined adjuvant chemotherapy benefit rates can inform individualized adjuvant therapy decisions.

Dual-stream cross-modal fusion alignment network for survival analysis

Survival prediction serves as a pivotal component in precision oncology, enabling the optimization of treatment strategies through mortality risk assessment. While the integration of histopathological images and genomic profiles offers enhanced potential for patient stratification, existing methodologies are constrained by two fundamental limitations: (i) insufficient attention to fine-grained local features in favor of global representations, and (ii) suboptimal cross-modal fusion strategies that either neglect intrinsic correlations or discard modality-specific information. To address these challenges, we propose DSCASurv, a novel cross-modal fusion alignment framework designed to explore and integrate intrinsic correlations across multimodal data, thereby improving the accuracy of survival prediction. Specifically, DSCASurv leverages the local feature extraction capabilities of convolutional layers and the long-range dependency modeling of scanning state space models to extract intra-modal representations, while generating cross-modal representations through dual parallel mixer architectures. A cross-modal attention module functions as a bridge for inter-modal information exchange and complementary information transfer. The framework ultimately integrates all intra-modal representations to generate survival predictions by enhancing and recalibrating complementary information. Extensive experiments on five benchmark cancer datasets demonstrate the superior performance of our approach compared to existing methods.

Deep learning-driven survival prediction in pan-cancer studies by integrating multimodal histology-genomic data

Accurate cancer prognosis is essential for personalized clinical management, guiding treatment strategies and predicting patient survival. Conventional methods, which depend on the subjective evaluation of histopathological features, exhibit significant inter-observer variability and limited predictive power. To overcome these limitations, we developed cross-attention transformer-based multimodal fusion network (CATfusion), a deep learning framework that integrates multimodal histology-genomic data for comprehensive cancer survival prediction. By employing self-supervised learning strategy with TabAE for feature extraction and utilizing cross-attention mechanisms to fuse diverse data types, including mRNA-seq, miRNA-seq, copy number variation, DNA methylation variation, mutation data, and histopathological images. By successfully integrating this multi-tiered patient information, CATfusion has become an advanced survival prediction model to utilize the most diverse data types across various cancer types. CATfusion's architecture, which includes a bidirectional multimodal attention mechanism and self-attention block, is adept at synchronizing the learning and integration of representations from various modalities. CATfusion achieves superior predictive performance over traditional and unimodal models, as demonstrated by enhanced C-index and survival area under the curve scores. The model's high accuracy in stratifying patients into distinct risk groups is a boon for personalized medicine, enabling tailored treatment plans. Moreover, CATfusion's interpretability, enabled by attention-based visualization, offers insights into the biological underpinnings of cancer prognosis, underscoring its potential as a transformative tool in oncology.

Multimodal histopathologic models stratify hormone receptor-positive early breast cancer

The Oncotype DX® Recurrence Score (RS) is an assay for hormone receptor-positive early breast cancer with extensively validated predictive and prognostic value. However, its cost and lag time have limited global adoption, and previous attempts to estimate it using clinicopathologic variables have had limited success. To address this, we assembled 6172 cases across three institutions and developed Orpheus, a multimodal deep learning tool to infer the RS from H&E whole-slide images. Our model identifies TAILORx high-risk cases (RS > 25) with an area under the curve (AUC) of 0.89, compared to a leading clinicopathologic nomogram with 0.73. Furthermore, in patients with RS ≤ 25, Orpheus ascertains risk of metastatic recurrence more accurately than the RS itself (0.75 vs 0.49 mean time-dependent AUC). These findings have the potential to guide adjuvant therapy for high-risk cases and tailor surveillance for patients at elevated metastatic recurrence risk.

Artificial intelligence-based biomarkers for treatment decisions in oncology

The development of new therapeutic strategies such as immune checkpoint inhibitors (ICIs) and targeted therapies has increased the complexity of the treatment landscape for solid tumors. At the current rate of annual FDA approvals, the potential treatment options could increase by tenfold over the next 5 years. The cost of personalized medicine technologies limits its accessibility, thus increasing socioeconomic disparities in the treated population. In this review we describe artificial intelligence (AI)-based solutions - including deep learning (DL) methods for routine medical imaging and large language models (LLMs) for electronic health records (EHRs) - to support cancer treatment decisions with cost-effective biomarkers. We address the current limitations of these technologies and propose the next steps towards their adoption in routine clinical practice.

Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models

Training machine-learning models with synthetically generated data can alleviate the problem of data scarcity when acquiring diverse and sufficiently large datasets is costly and challenging. Here we show that cascaded diffusion models can be used to synthesize realistic whole-slide image tiles from latent representations of RNA-sequencing data from human tumours. Alterations in gene expression affected the composition of cell types in the generated synthetic image tiles, which accurately preserved the distribution of cell types and maintained the cell fraction observed in bulk RNA-sequencing data, as we show for lung adenocarcinoma, kidney renal papillary cell carcinoma, cervical squamous cell carcinoma, colon adenocarcinoma and glioblastoma. Machine-learning models pretrained with the generated synthetic data performed better than models trained from scratch. Synthetic data may accelerate the development of machine-learning models in scarce-data settings and allow for the imputation of missing data modalities.

Brain tumour histopathology through the lens of deep learning: A systematic review

PROBLEM

Machine learning (ML)/Deep learning (DL) techniques have been evolving to solve more complex diseases, but it has been used relatively little in Glioblastoma (GBM) histopathological studies, which could benefit greatly due to the disease's complex pathogenesis.

AIM

Conduct a systematic review to investigate how ML/DL techniques have influenced the progression of brain tumour histopathological research, particularly in GBM.

METHODS

54 eligible studies were collected from the PubMed and ScienceDirect databases, and their information about the types of brain tumour/s used, types of -omics data used with histopathological data, origins of the data, types of ML/DL and its training and evaluation methodologies, and the ML/DL task it was set to perform in the study were extracted to inform us of trends in GBM-related ML/DL-based research.

RESULTS

Only 8 GBM-related studies in the eligible utilised ML/DL methodologies to gain deeper insights into GBM pathogenesis by contextualising histological data with -omics data. However, we report that these studies have been published more recently. The most popular ML/DL models used in GBM-related research are the SVM classifier and ResNet-based CNN architecture. Still, a considerable number of studies failed to state training and evaluative methodologies clearly.

CONCLUSION

There is a growing trend towards using ML/DL approaches to uncover relationships between biological and histopathological data to bring new insights into GBM, thus pushing GBM research forward. Much work still needs to be done to properly report the ML/DL methodologies to showcase the models' robustness and generalizability and ensure the models are reproducible.

Hallmarks of artificial intelligence contributions to precision oncology

The integration of artificial intelligence (AI) into oncology promises to revolutionize cancer care. In this Review, we discuss ten AI hallmarks in precision oncology, organized into three groups: (1) cancer prevention and diagnosis, encompassing cancer screening, detection and profiling; (2) optimizing current treatments, including patient outcome prediction, treatment planning and monitoring, clinical trial design and matching, and developing response biomarkers; and (3) advancing new treatments by identifying treatment combinations, discovering cancer vulnerabilities and designing drugs. We also survey AI applications in interventional clinical trials and address key challenges to broader clinical adoption of AI: data quality and quantity, model accuracy, clinical relevance and patient benefit, proposing actionable solutions for each.

A comprehensive analysis of the prognostic value, expression characteristics and immune correlation of MKI67 in cancers

Background

nuclear-associated antigen Ki67 (Ki67) emerges as a clinically practical biomarker for proliferation assessment among many cancer types. However, the definite prognostic value of Ki67 against a specific cancer type has remained vague. This study aims to perform a comprehensive pan-cancer analysis of the prognosis value of Ki67 across various cancer types.

Methods

This study explored the expression, prognostic value, and tumor-infiltrating immune of MKI67 in the TCGA database by pan-cancer, and then performed immunohistochemical, correlation analysis and prognostic analysis using 10028 patients of the top 10 cancer patients in China we collected. The correlation between MKI67 expression and survival outcome, clinical features, MSI, TMB, and tumor-infiltrating immune cells by TCGA database, xCell, and TIMER algorithms.

Results

MKI67 expression was significantly upregulated across varied cancer types verified by datasets. We found MKI67 expression was significantly associated with poor prognosis in LUADLUSC, LIHC, and BRCA patients, but good prognosis in COADREAD and READ patients via Kaplan-Meier survival analysis using 10028 patients collected. These results of our validation were generally consistent with TCGA database except BRCA, COADREAD and READ. Meanwhile, upregulation of MKI67 elevates the degree of immune infiltration of several immune cell subtypes, such as functional T cells, CD4+ T cells, and CD8+ T cells, as well as, MKI67 was related to Cell cycle, Oocyte meiosis, p53 and other pathways.

Conclusion

Our comprehensive analysis may supply useful guidance on MKI67 applicability across various cancer types. These observed results contribute to the promise of MKI67 in a realistic clinical setting and improve the outcomes of cancer patients.

Systematic inference of super-resolution cell spatial profiles from histology images

Inferring cell spatial profiles from histology images is critical for cancer diagnosis and treatment in clinical settings. In this study, we report a weakly-supervised deep-learning method, HistoCell, to directly infer super-resolution cell spatial profiles consisting of cell types, cell states and their spatial network from histology images at the single-nucleus-level. Benchmark analysis demonstrates that HistoCell robustly achieves state-of-the-art performance in terms of cell type/states prediction solely from histology images across multiple cancer tissues. HistoCell can significantly enhance the deconvolution accuracy for the spatial transcriptomics data and enable accurate annotation of subtle cancer tissue architectures. Moreover, HistoCell is applied to de novo discovery of clinically relevant spatial organization indicators, including prognosis and drug response biomarkers, across diverse cancer types. HistoCell also enable image-based screening of cell populations that drives phenotype of interest, and is applied to discover the cell population and corresponding spatial organization indicators associated with gastric malignant transformation risk. Overall, HistoCell emerges as a powerful and versatile tool for cancer studies in histology image-only cohorts.

The puzzling Spitz tumours: is artificial intelligence the key to their understanding?

Since their first description in 1948, Spitz tumours remain one of the most challenging diagnostic entities in dermatopathology due to their complex histological features and ambiguous clinical behaviour. In recent years, artificial intelligence (AI) solutions have demonstrated significant potential across a wide range of medical applications, including computational pathology, for decision-making in diagnosis, along with promising advances in prognosis and tumour classification. However, the application of AI to Spitz tumours remains relatively underexplored, with few studies addressing this field. Yet in this evolving technological landscape, could AI provide the insights needed to help resolve the diagnostic uncertainties surrounding Spitz tumours? How could this technology be leveraged to bridge the gap between histopathological uncertainty and clinical accuracy? This review aims to provide an overview of the current state of AI applications in Spitz tumour analysis, identify existing research gaps, and propose future directions to optimize the use of AI in understanding and diagnosing these complex tumours.

Multimodal integration using a machine learning approach facilitates risk stratification in HR+/HER2- breast cancer

Hormone receptor-positive (HR+)/human epidermal growth factor receptor 2-negative (HER2-) breast cancer is the most common type of breast cancer, with continuous recurrence remaining an important clinical issue. Current relapse predictive models in HR+/HER2- breast cancer patients still have limitations. The integration of multidimensional data represents a promising alternative for predicting relapse. In this study, we leverage our multi-omics cohort comprising 579 HR+/HER2- breast cancer patients (200 patients with complete data across 7 modalities) and develop a machine-learning-based model, namely CIMPTGV, which integrates clinical information, immunohistochemistry, metabolomics, pathomics, transcriptomics, genomics, and copy number variations to predict recurrence risk of HR+/HER2- breast cancer. This model achieves concordance indices (C-indices) of 0.871 and 0.869 in the train and test sets, respectively. The risk population predicted by the CIMPTGV model encompasses those identified by single-modality models. Feature analysis reveals that synergistic and complementary effects exist in different modalities. Simultaneously, we develop a simplified model with a mean area under the curve (AUC) of 0.840, presenting a useful approach for clinical applications.

Current Advancements in Digital Neuropathology and Machine Learning for the Study of Neurodegenerative Diseases

Computational neurodegenerative neuropathology represents a transformative approach in the analysis and understanding of neurodegenerative diseases through utilization of whole slide images (WSIs) and advanced machine learning/artificial intelligence (ML/AI) techniques. This review explores the emerging field of computational neurodegenerative neuropathology, emphasizing its potential to enhance neuropathologic assessment, diagnosis, and research. Recent advancements in ML/AI technologies have significantly affected image-based medical fields, including anatomic pathology, by automating disease staging, identifying novel morphologic biomarkers, and uncovering new clinical insights via multi-modal AI approaches. Despite its promise, the field faces several challenges, including limited expert annotations, slide scanning inaccessibility, inter-institutional variability, and the complexities of sharing large WSI data sets. This review discusses the importance of improving deep learning model accuracy and efficiency for better interpretation of neuropathologic data. It highlights the potential of unsupervised learning to identify patterns in unannotated data. Furthermore, the development of explainable AI models is crucial for experimental neuropathology. By addressing these challenges and leveraging cutting-edge AI techniques, computational neurodegenerative neuropathology has the potential to revolutionize the field and significantly advance our understanding of disease.