A deep learning model to predict RNA-Seq expression of tumours from whole slide images

Artificial intelligence and whole slide imaging, a new tool for the microsatellite instability prediction in colorectal cancer: Friend or foe?

Colorectal cancer (CRC) is the third most common and second most deadly cancer worldwide. Despite advances in screening and treatment, CRC is heterogeneous and the response to therapy varies significantly, limiting personalized treatment options. Certain molecular biomarkers, including microsatellite instability (MSI), are critical in planning personalized treatment, although only a subset of patients may benefit. Currently, the primary methods for assessing MSI status include immunohistochemistry (IHC) for DNA mismatch repair proteins (MMRs), polymerase chain reaction (PCR)-based molecular testing, or next-generation sequencing (NGS). However, these techniques have limitations, are expensive and time-consuming, and often result in inter-method inconsistencies. Deficient mismatch repair (dMMR) or high microsatellite instability (MSI-H) are critical predictive biomarkers of response to immune checkpoint inhibitor (ICI) therapy and MSI testing is recommended to identify patients who may benefit. There is a pressing need for a more robust, reliable, and cost-effective approach that accurately assesses MSI status. Recent advances in computational pathology, in particular the development of technologies that digitally scan whole slide images (WSI) at high resolution, as well as new approaches to artificial intelligence (AI) in medicine, are increasingly gaining ground. This review aims to provide an overview of the latest findings on WSI and advances in AI methods for predicting MSI status, summarize their applications in CRC, and discuss their strengths and limitations in daily clinical practice.

MRI-based multimodal AI model enables prediction of recurrence risk and adjuvant therapy in breast cancer

Timely intervention and improved prognosis for breast cancer patients rely on early metastasis risk detection and accurate treatment predictions. This study introduces an advanced multimodal MRI and AI-driven 3D deep learning model, termed the 3D-MMR-model, designed to predict recurrence risk in non-metastatic breast cancer patients. We conducted a multicenter study involving 1199 non-metastatic breast cancer patients from four institutions in China, with comprehensive MRI and clinical data retrospectively collected. Our model employed multimodal-data fusion, utilizing contrast-enhanced T1-weighted imaging (T1 + C) and T2-weighted imaging (T2WI) volumes, processed through a modified 3D-UNet for tumor segmentation and a DenseNet121-based architecture for disease-free survival (DFS) prediction. Additionally, we performed RNA-seq analysis to delve further into the relationship between concentrated hotspots within the tumor region and the tumor microenvironment. The 3D-MR-model demonstrated superior predictive performance, with time-dependent ROC analysis yielding AUC values of 0.90, 0.89, and 0.88 for 2-, 3-, and 4-year DFS predictions, respectively, in the training cohort. External validation cohorts corroborated these findings, highlighting the model's robustness across diverse clinical settings. Integration of clinicopathological features further enhanced the model's accuracy, with a multimodal approach significantly improving risk stratification and decision-making in clinical practice. Visualization techniques provided insights into the decision-making process, correlating predictions with tumor microenvironment characteristics. In summary, the 3D-MMR-model represents a significant advancement in breast cancer prognosis, combining cutting-edge AI technology with multimodal imaging to deliver precise and clinically relevant predictions of recurrence risk. This innovative approach holds promise for enhancing patient outcomes and guiding individualized treatment plans in breast cancer care.

Artificial intelligence in predicting EGFR mutations from whole slide images in lung Cancer: A systematic review and Meta-Analysis

BACKGROUND

Epidermal growth factor receptor (EGFR) mutations play a pivotal role in guiding targeted therapy for lung cancer, making their accurate detection essential for personalized treatment. Recently, artificial intelligence (AI) has emerged as a promising tool for identifying EGFR mutation status from digital pathology images. This systematic review and meta-analysis evaluate the diagnostic accuracy of AI models in predicting EGFR mutations from whole slide images (WSIs) in lung cancer patients.

METHODS

A comprehensive search was conducted across four databases (EMBASE, PubMed, Web of Science, and Scopus) for studies published up to June 20th, 2024. Studies employing AI algorithms, including machine learning and deep learning techniques, to predict EGFR mutations from digital pathology images were included. The risk of bias and applicability concerns were assessed using the QUADAS-AI tool. Diagnostic accuracy metrics such as sensitivity, specificity, and the Area Under the Curve (AUC) were extracted. Random-effects models were applied to synthesize the AI model performance. This study is registered with PROSPERO (CRD42024570496).

RESULTS

Out of 1,828 identified studies, 16 met the inclusion criteria, with 4 eligible for meta-analysis. The pooled results demonstrated that AI algorithms achieved an AUC of 0.756 (95% CI: 0.669-0.824), a sensitivity of 66.3% (95% CI: X-Y), and a specificity of 68.1% (95% CI: X-Y). Subgroup analyses revealed that AI models using in-house datasets, surgical resection samples, the ResNet architecture, and tumor-focused regions exhibited improved predictive performance.

CONCLUSION

AI models exhibit potential for non-invasive prediction of EGFR mutations in lung cancer patients using WSIs, although current accuracy and precision warrant further refinement. Future research should aim to enhance AI algorithms, validate findings on larger datasets, and integrate these tools into clinical workflows to optimize lung cancer management.

Unravelling tumour spatiotemporal heterogeneity using spatial multimodal data

Analysing the genome, epigenome, transcriptome, proteome, and metabolome within the spatial context of cells has transformed our understanding of tumour spatiotemporal heterogeneity. Advances in spatial multi-omics technologies now reveal complex molecular interactions shaping cellular behaviour and tissue dynamics. This review highlights key technologies and computational methods that have advanced spatial domain identification and their pseudo-relations, as well as inference of intra- and inter-cellular molecular networks that drive disease progression. We also discuss strategies to address major challenges, including data sparsity, high-dimensionality, scalability, and heterogeneity. Furthermore, we outline how spatial multi-omics enables novel insights into disease mechanisms, advancing precision medicine and informing targeted therapies. KEY POINTS: Advancements in spatial multi-omics facilitate our understanding of tumour spatiotemporal heterogeneity. AI-driven multimodal models uncover complex molecular interactions that underlie cellular behaviours and tissue dynamics. Combining multi-omics technologies and AI-enabled bioinformatics tools helps predict critical disease stages, such as pre-cancer, advancing precision medicine, and informing targeted therapeutic strategies.

Artificial intelligence entering the pathology arena in oncology: current applications and future perspectives

BACKGROUND

Artificial intelligence (AI) is rapidly transforming the fields of pathology and oncology, offering novel opportunities for advancing diagnosis, prognosis, and treatment of cancer.

METHODS

Through a systematic review-based approach, the representatives from the European Society for Medical Oncology (ESMO) Precision Oncology Working Group (POWG) and international experts identified studies in pathology and oncology that applied AI-based algorithms for tumour diagnosis, molecular biomarker detection, and cancer prognosis assessment. These findings were synthesised to provide a comprehensive overview of current AI applications and future directions in cancer pathology.

RESULTS

The integration of AI tools in digital pathology is markedly improving the accuracy and efficiency of image analysis, allowing for automated tumour detection and classification, identification of prognostic molecular biomarkers, and prediction of treatment response and patient outcomes. Several barriers for the adoption of AI in clinical workflows, such as data availability, explainability, and regulatory considerations, still persist. There are currently no prognostic or predictive AI-based biomarkers supported by level IA or IB evidence. The ongoing advancements in AI algorithms, particularly foundation models, generalist models and transformer-based deep learning, offer immense promise for the future of cancer research and care. AI is also facilitating the integration of multi-omics data, leading to more precise patient stratification and personalised treatment strategies.

CONCLUSIONS

The application of AI in pathology is poised to not only enhance the accuracy and efficiency of cancer diagnosis and prognosis but also facilitate the development of personalised treatment strategies. Although barriers to implementation remain, ongoing research and development in this field coupled with addressing ethical and regulatory considerations will likely lead to a future where AI plays an integral role in cancer management and precision medicine. The continued evolution and adoption of AI in pathology and oncology are anticipated to reshape the landscape of cancer care, heralding a new era of precision medicine and improved patient outcomes.

Histology-Based Virtual RNA Inference Identifies Pathways Associated with Metastasis Risk in Colorectal Cancer

Colorectal cancer (CRC) remains a major health concern, with over 150,000 new diagnoses and more than 50,000 deaths annually in the United States, underscoring an urgent need for improved screening, prognostication, disease management, and therapeutic approaches. The tumor microenvironment (TME)-comprising cancerous and immune cells interacting within the tumor's spatial architecture-plays a critical role in disease progression and treatment outcomes, reinforcing its importance as a prognostic marker for metastasis and recurrence risk. However, traditional methods for TME characterization, such as bulk transcriptomics and multiplex protein assays, lack sufficient spatial resolution. Although spatial transcriptomics (ST) allows for the high-resolution mapping of whole transcriptomes at near-cellular resolution, current ST technologies (e.g., Visium, Xenium) are limited by high costs, low throughput, and issues with reproducibility, preventing their widespread application in large-scale molecular epidemiology studies. In this study, we refined and implemented Virtual RNA Inference (VRI) to derive ST-level molecular information directly from hematoxylin and eosin (H&E)-stained tissue images. Our VRI models were trained on the largest matched CRC ST dataset to date, comprising 45 patients and more than 300,000 Visium spots from primary tumors. Using state-of-the-art architectures (UNI, ResNet-50, ViT, and VMamba), we achieved a median Spearman's correlation coefficient of 0.546 between predicted and measured spot-level expression. As validation, VRI-derived gene signatures linked to specific tissue regions (tumor, interface, submucosa, stroma, serosa, muscularis, inflammation) showed strong concordance with signatures generated via direct ST, and VRI performed accurately in estimating cell-type proportions spatially from H&E slides. In an expanded CRC cohort controlling for tumor invasiveness and clinical factors, we further identified VRI-derived gene signatures significantly associated with key prognostic outcomes, including metastasis status. Although certain tumor-related pathways are not fully captured by histology alone, our findings highlight the ability of VRI to infer a wide range of "histology-associated" biological pathways at near-cellular resolution without requiring ST profiling. Future efforts will extend this framework to expand TME phenotyping from standard H&E tissue images, with the potential to accelerate translational CRC research at scale.

Multimodal fusion of radio-pathology and proteogenomics identify integrated glioma subtypes with prognostic and therapeutic opportunities

Integrating multimodal data can uncover causal features hidden in single-modality analyses, offering a comprehensive understanding of disease complexity. This study introduces a multimodal fusion subtyping (MOFS) framework that integrates radiological, pathological, genomic, transcriptomic, and proteomic data from 122 patients with IDH-wildtype adult glioma, identifying three subtypes: MOFS1 (proneural) with favorable prognosis, elevated neurodevelopmental activity, and abundant neurocyte infiltration; MOFS2 (proliferative) with the worst prognosis, superior proliferative activity, and genome instability; MOFS3 (TME-rich) with intermediate prognosis, abundant immune and stromal components, and sensitive to anti-PD-1 immunotherapy. STRAP emerges as a prognostic biomarker and potential therapeutic target for MOFS2, associated with its proliferative phenotype. Stromal infiltration in MOFS3 serves as a crucial prognostic indicator, allowing for further prognostic stratification. Additionally, we develop a deep neural network (DNN) classifier based on radiological features to further enhance the clinical translatability, providing a non-invasive tool for predicting MOFS subtypes. Overall, these findings highlight the potential of multimodal fusion in improving the classification, prognostic accuracy, and precision therapy of IDH-wildtype glioma, offering an avenue for personalized management.

Defining a 'cells to society' research framework for appendiceal tumours

Tumours of the appendix - a vestigial digestive organ attached to the colon - are rare. Although we estimate that around 3,000 new appendiceal cancer cases are diagnosed annually in the USA, the challenges of accurately diagnosing and identifying this tumour type suggest that this number may underestimate true population incidence. In the current absence of disease-specific screening and diagnostic imaging modalities, or well-established risk factors, the incidental discovery of appendix tumours is often prompted by acute presentations mimicking appendicitis or when the tumour has already spread into the abdominal cavity - wherein the potential misclassification of appendiceal tumours as malignancies of the colon and ovaries also increases. Notwithstanding these diagnostic difficulties, our understanding of appendix carcinogenesis has advanced in recent years. However, there persist considerable challenges to accelerating the pace of research discoveries towards the path to improved treatments and cures for patients with this group of orphan malignancies. The premise of this Expert Recommendation article is to discuss the current state of the field, to delineate unique challenges for the study of appendiceal tumours, and to propose key priority research areas that will deliver a more complete picture of appendix carcinogenesis and metastasis. The Appendix Cancer Pseudomyxoma Peritonei (ACPMP) Research Foundation Scientific Think Tank delivered a consensus of core research priorities for appendiceal tumours that are poised to be ground-breaking and transformative for scientific discovery and innovation. On the basis of these six research areas, here, we define the first 'cells to society' research framework for appendix tumours.

The Role of Artificial Intelligence and Big Data for Gastrointestinal Disease

Artificial intelligence (AI) is a rapidly evolving presence in all fields and industries, with the ability to both improve quality and reduce the burden of human effort. Gastroenterology is a field with a focus on diagnostic techniques and procedures, and AI and big data have established and growing roles to play. Alongside these opportunities are challenges, which will evolve in parallel.

Integrating Deep Learning Models with Genome-Wide Association Study-Based Identification Enhanced Phenotype Predictions in Group A Streptococcus

Group A Streptococcus (GAS) is a major pathogen with diverse clinical outcomes linked to its genetic variability, making accurate phenotype prediction essential. While previous studies have identified many GAS-associated genetic factors, translating these findings into predictive models remains challenging due to data complexity. The current study aimed to integrate deep learning models with genome-wide association study-derived genetic variants to predict pathogenic phenotypes in GAS. We evaluated the performance of several deep neural network models, including CNN, ResNet18, LSTM, and their ensemble approach in predicting GAS phenotypes. It was found that the ensemble model consistently achieved the highest prediction accuracy across phenotypes. Models trained on the full 4722-genotype set outperformed those trained on a reduced 175-genotype set, underscoring the importance of comprehensive variant data in capturing complex genotype-phenotype interactions. Performance changes in the reduced 175-genotype set compared to the full-set genotype scenarios revealed the impact of data dimensionality on model effectiveness, with CNN remaining robust, while ResNet18 and LSTM underperformed. Our findings emphasized the potential of deep learning in phenotype prediction and the critical role of data-model compatibility.

Histopathology based AI model predicts anti-angiogenic therapy response in renal cancer clinical trial

Anti-angiogenic (AA) therapy is a cornerstone of metastatic clear cell renal cell carcinoma (ccRCC) treatment, but not everyone responds, and predictive biomarkers are lacking. CD31, a marker of vasculature, is insufficient, and the Angioscore, an RNA-based angiogenesis quantification method, is costly, associated with delays, difficult to standardize, and does not account for tumor heterogeneity. Here, we developed an interpretable deep learning (DL) model that predicts the Angioscore directly from ubiquitous histopathology slides yielding a visual vascular network (H&E DL Angio). H&E DL Angio achieves a strong correlation with the Angioscore across multiple cohorts (spearman correlations of 0.77 and 0.73). Using this approach, we found that angiogenesis inversely correlates with grade and stage and is associated with driver mutation status. Importantly, DL Angio expediently predicts AA response in both a real-world and IMmotion150 trial cohorts, out-performing CD31, and closely approximating the Angioscore (c-index 0.66 vs 0.67) at a fraction of the cost.

Deep learning-driven survival prediction in pan-cancer studies by integrating multimodal histology-genomic data

Accurate cancer prognosis is essential for personalized clinical management, guiding treatment strategies and predicting patient survival. Conventional methods, which depend on the subjective evaluation of histopathological features, exhibit significant inter-observer variability and limited predictive power. To overcome these limitations, we developed cross-attention transformer-based multimodal fusion network (CATfusion), a deep learning framework that integrates multimodal histology-genomic data for comprehensive cancer survival prediction. By employing self-supervised learning strategy with TabAE for feature extraction and utilizing cross-attention mechanisms to fuse diverse data types, including mRNA-seq, miRNA-seq, copy number variation, DNA methylation variation, mutation data, and histopathological images. By successfully integrating this multi-tiered patient information, CATfusion has become an advanced survival prediction model to utilize the most diverse data types across various cancer types. CATfusion's architecture, which includes a bidirectional multimodal attention mechanism and self-attention block, is adept at synchronizing the learning and integration of representations from various modalities. CATfusion achieves superior predictive performance over traditional and unimodal models, as demonstrated by enhanced C-index and survival area under the curve scores. The model's high accuracy in stratifying patients into distinct risk groups is a boon for personalized medicine, enabling tailored treatment plans. Moreover, CATfusion's interpretability, enabled by attention-based visualization, offers insights into the biological underpinnings of cancer prognosis, underscoring its potential as a transformative tool in oncology.

Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models

Training machine-learning models with synthetically generated data can alleviate the problem of data scarcity when acquiring diverse and sufficiently large datasets is costly and challenging. Here we show that cascaded diffusion models can be used to synthesize realistic whole-slide image tiles from latent representations of RNA-sequencing data from human tumours. Alterations in gene expression affected the composition of cell types in the generated synthetic image tiles, which accurately preserved the distribution of cell types and maintained the cell fraction observed in bulk RNA-sequencing data, as we show for lung adenocarcinoma, kidney renal papillary cell carcinoma, cervical squamous cell carcinoma, colon adenocarcinoma and glioblastoma. Machine-learning models pretrained with the generated synthetic data performed better than models trained from scratch. Synthetic data may accelerate the development of machine-learning models in scarce-data settings and allow for the imputation of missing data modalities.

Differentiating Separate Primary Lung Adenocarcinomas From Intrapulmonary Metastases With Emphasis on Pathological and Molecular Considerations: Recommendations From the International Association for the Study of Lung Cancer Pathology Committee

INTRODUCTION

With the implementation of low-dose computed tomography screening, multiple pulmonary tumor nodules are diagnosed with increasing frequency and the selection of surgical treatments versus systemic therapies has become challenging on a daily basis in clinical practice. In the presence of multiple carcinomas, especially adenocarcinomas, pathologically determined to be of pulmonary origin, the distinction between separate primary lung carcinomas (SPLCs) and intrapulmonary metastases (IPMs) is important for staging, management, and prognostication.

METHODS

We systemically reviewed various means that aid in the differentiation between SPLCs and IPMs explored by histopathologic evaluation and molecular profiling, the latter includes DNA microsatellite analysis, array comparative genomic hybridization, TP53 and oncogenic driver mutation testing and, more recently, with promising effectiveness, next-generation sequencing comprising small- or large-scale multi-gene panels.

RESULTS

Comprehensive histologic evaluation may suffice to differentiate between SPLCs and IPMs. Nevertheless, molecular profiling using larger-scale next-generation sequencing typically provides superior discriminatory power, allowing for more accurate classification. On the basis of the literature review and expert opinions, we proposed a combined four-step histologic and molecular classification algorithm for addressing multiple pulmonary tumor nodules of adenocarcinoma histology that encourages a multidisciplinary approach. It is also noteworthy that new technologies combining machine learning and digital pathology may develop into valuable diagnostic tools for distinguishing SPLCs from IPMs in the future.

CONCLUSIONS

Although histopathologic evaluation is often adequate to differentiate SPLCs from IPMs, molecular profiling should be performed when possible, especially in cases with tumors exhibiting similar morphology. This manuscript summarized the previous efforts in resolving the current challenges and highlighted the recent progress in the differentiation methods and algorithms used in categorizing multiple lung adenocarcinomas into SPLCs or IPMs, which are becoming more and more critical in precision lung cancer management.

Scaling up spatial transcriptomics for large-sized tissues: uncovering cellular-level tissue architecture beyond conventional platforms with iSCALE

Recent advances in spatial transcriptomics (ST) technologies have transformed our ability to profile gene expression while retaining the crucial spatial context within tissues. However, existing ST platforms suffer from high costs, long turnaround times, low resolution, limited gene coverage, and small tissue capture areas, which hinder their broad applications. Here we present iSCALE, a method that predicts super-resolution gene expression and automatically annotates cellular-level tissue architecture for large-sized tissues that exceed the capture areas of standard ST platforms. The accuracy of iSCALE were validated by comprehensive evaluations, involving benchmarking experiments, immunohistochemistry staining, and manual annotation by pathologists. When applied to multiple sclerosis human brain samples, iSCALE uncovered lesion associated cellular characteristics that were undetectable by conventional ST experiments. Our results demonstrate iSCALE's utility in analyzing large-sized tissues with automatic and unbiased tissue annotation, inferring cell type composition, and pinpointing regions of interest for features not discernible through human visual assessment.

Harnessing AI for enhanced evidence-based laboratory medicine (EBLM)

The integration of artificial intelligence (AI) into laboratory medicine, is revolutionizing diagnostic accuracy, operational efficiency, and personalized patient care. AI technologies(machine learning, natural language processing and computer vision) advance evidence-based laboratory medicine (EBLM) by automating and optimizing critical processes(formulating clinical questions, conducting literature searches, appraising evidence, and developing clinical guidelines). These reduce the time for systematic reviews, ensuring consistency in appraisal, and enabling real-time updates to guidelines. AI supports personalized medicine by analyzing large datasets, genetic information and electronic health records (EHRs), to tailor diagnostic and treatment plans to patient profiles. Predictive analytics enhance outcomes by leveraging historical data and ongoing monitoring to predict responses and optimize care pathways. Despite the transformative potential, there are challenges. The accuracy, transparency, and explainability of AI algorithms is critical for gaining trust and ensuring ethical deployment. Integration into existing clinical workflows requires collaboration between AI developers and users to ensure seamless user-friendly adoption. Ethical considerations, such as privacy,data security, and algorithmic bias, must also be addressed to mitigate risks and ensure equitable healthcare delivery. Regulatory frameworks, eg. The EU AI Regulation, emphasize transparency, data governance, and human oversight, particularly for high-risk AI systems. The economic and operational benefits are cost savings, improved diagnostic precision, and enhanced patient outcomes. Future trends (federated learning and self-supervised learning), will enhance the scalability and applicability of AI in EBLM, paving the way for a new era of precision medicine. AI in EBLM has the potential to transform healthcare delivery, improve patient outcomes, and advance personalized/precision medicine.

Machine learning to dissect perturbations in complex cellular systems

Understanding the responses of biological systems to various perturbations, such as genetic, chemical, or environmental challenges, is essential for reconstructing causal network models. Emerging single-cell technologies have become instrumental in elucidating cell states and phenotypes and they have been used in combination with genetic screening. Recent advances in machine learning and artificial intelligence architectures have stimulated the development of computational tools for modeling perturbations and the response to compounds. This study outlined core principles underpinning perturbation analysis and discussed the methodologies and analytical frameworks used to decode drug and genetic perturbation responses, complex multicellular interactions, and network dynamics. The current tools used for various applications were overviewed. These developments hold great promise for improving drug development and personalized medicine. Foundation models and perturbation cell and tissue atlases offer immense potential for advancing our understanding of cellular behavior and disease mechanisms.

Systematic inference of super-resolution cell spatial profiles from histology images

Inferring cell spatial profiles from histology images is critical for cancer diagnosis and treatment in clinical settings. In this study, we report a weakly-supervised deep-learning method, HistoCell, to directly infer super-resolution cell spatial profiles consisting of cell types, cell states and their spatial network from histology images at the single-nucleus-level. Benchmark analysis demonstrates that HistoCell robustly achieves state-of-the-art performance in terms of cell type/states prediction solely from histology images across multiple cancer tissues. HistoCell can significantly enhance the deconvolution accuracy for the spatial transcriptomics data and enable accurate annotation of subtle cancer tissue architectures. Moreover, HistoCell is applied to de novo discovery of clinically relevant spatial organization indicators, including prognosis and drug response biomarkers, across diverse cancer types. HistoCell also enable image-based screening of cell populations that drives phenotype of interest, and is applied to discover the cell population and corresponding spatial organization indicators associated with gastric malignant transformation risk. Overall, HistoCell emerges as a powerful and versatile tool for cancer studies in histology image-only cohorts.

Benchmarking the translational potential of spatial gene expression prediction from histology

Spatial transcriptomics has enabled the quantification of gene expression at spatial coordinates across a tissue, offering crucial insights into molecular underpinnings of diseases. In light of this, several methods predicting spatial gene expression from paired histology images have provided the opportunity to enhance the utility of obtainable and cost-effective haematoxylin-and-eosin-stained histology images. To this end, we conduct a comprehensive benchmarking study encompassing eleven methods for predicting spatial gene expression with histology images. These methods are reproduced and evaluated using five Spatially Resolved Transcriptomics datasets, followed by external validation using The Cancer Genome Atlas data. Our evaluation incorporates diverse metrics which capture the performance of predicted gene expression, model generalisability, translational potential, usability and computational efficiency of each method. Our findings demonstrate the capacity of the methods to predict spatial gene expression from histology and highlight areas that can be addressed to support the advancement of this emerging field.

Development of a deep learning system for predicting biochemical recurrence in prostate cancer

BACKGROUND

Biochemical recurrence (BCR) occurs in 20%-40% of men with prostate cancer (PCa) who undergo radical prostatectomy. Predicting which patients will experience BCR in advance helps in formulating more targeted prostatectomy procedures. However, current preoperative recurrence prediction mainly relies on the use of the Gleason grading system, which omits within-grade morphological patterns and subtle histopathological features, leaving a significant amount of prognostic potential unexplored.

METHODS

We collected and selected a total of 1585 prostate biopsy images with tumor regions from 317 patients (5 Whole Slide Images per patient) to develop a deep learning system for predicting BCR of PCa before prostatectomy. The Inception_v3 neural network was employed to train and test models developed from patch-level images. The multiple instance learning method was used to extract whole slide image-level features. Finally, patient-level artificial intelligence models were developed by integrating deep learning -generated pathology features with several machine learning algorithms.

RESULTS

The BCR prediction system demonstrated great performance in the testing cohort (AUC = 0.911, 95% Confidence Interval: 0.840-0.982) and showed the potential to produce favorable clinical benefits according to Decision Curve Analyses. Increasing the number of WSIs for each patient improves the performance of the prediction system. Additionally, the study explores the correlation between deep learning -generated features and pathological findings, emphasizing the interpretative potential of artificial intelligence models in pathology.

CONCLUSIONS

Deep learning system can use biopsy samples to predict the risk of BCR in PCa, thereby formulating targeted treatment strategies.

Inferring multi-slice spatially resolved gene expression from H&E-stained histology images with STMCL

Spatial transcriptomics has significantly advanced the measurement of spatial gene expression in the field of biology. However, the high cost of ST limits its application in large-scale studies. Using deep learning to predict spatial gene expression from H&E-stained histology images offers a more cost-effective alternative, but existing methods fail to fully leverage the multimodal information provided by Spatial transcriptomics and pathology images. In response, this paper proposes STMCL, a novel multimodal contrastive learning framework. STMCL integrates multimodal information, including histology images, gene expression features of spots, and their locations, to accurately infer spatial gene expression profiles. We tested four different types of multi-slice spatial transcriptomics datasets generated by the 10X Genomics platform. The results indicate that STMCL has advantages over baseline methods in predicting spatial gene expression profiles. Furthermore, STMCL is capable of capturing cancer-specific highly expressed genes and preserving gene expression patterns while maintaining the original spatial structure of gene expression. Our code is available at https://github.com/wenwenmin/STMCL.

Exploring the landscape of urinary tract melanomas: A review for pathologists and clinicians

Melanomas originating within the urinary tract represent a rare and clinically challenging subset of malignancies. Despite extensive research on cutaneous melanomas, urinary tract melanomas remain relatively unexplored, presenting diagnostic dilemmas and limited treatment consensus. In this comprehensive review, we synthesize current knowledge on the epidemiology, risk factors, clinical presentation, histopathological characteristics, and treatment strategies specific to this disease. Enhancing clinical awareness, refining diagnostic approaches, and exploring novel therapeutic interventions hold promise for improving outcomes in this challenging malignancy subset.

Context aware machine learning techniques for brain tumor classification and detection - A review

Background

Machine learning has tremendous potential in acute medical care, particularly in the field of precise medical diagnosis, prediction, and classification of brain tumors. Malignant gliomas, due to their aggressive growth and dismal prognosis, stand out among various brain tumor types. Recent advancements in understanding the genetic abnormalities that underlie these tumors have shed light on their histo-pathological and biological characteristics, which support in better classification and prognosis.

Objectives

This review aims to predict gene alterations and establish structured correlations among various tumor types, extending the prediction of genetic mutations and structures using the latest machine learning techniques. Specifically, it focuses on multi-modalities of Magnetic Resonance Imaging (MRI) and histopathology, utilizing Convolutional Neural Networks (CNN) for image processing and analysis.

Methods

The review encompasses the most recent developments in MRI, and histology image processing methods across multiple tumor classes, including Glioma, Meningioma, Pituitary, Oligodendroglioma, and Astrocytoma. It identifies challenges in tumor classification, segmentation, datasets, and modalities, employing various neural network architectures. A competitive analysis assesses the performance of CNN. Furthermore it also implies K-MEANS clustering to predict Genetic structure, Genes Clusters prediction and Molecular Alteration of various types and grades of tumors e.g. Glioma, Meningioma, Pituitary, Oligodendroglioma, and Astrocytoma.

Results

CNN and KNN structures, with their ability to extract highlights in image-based information, prove effective in tumor classification and segmentation, surmounting challenges in image analysis. Competitive analysis reveals that CNN and outperform others algorithms on publicly available datasets, suggesting their potential for precise tumor diagnosis and treatment planning.

Conclusion

Machine learning, especially through CNN and SVM algorithms, demonstrates significant potential in the accurate diagnosis and classification of brain tumors based on imaging and histo-pathological data. Further advancements in this area hold promise for improving the accuracy and efficiency of intra-operative tumor diagnosis and treatment.

Deep learning uncovers histological patterns of YAP1/TEAD activity related to disease aggressiveness in cancer patients

Over the last decade, Hippo signaling has emerged as a major tumor-suppressing pathway. Its dysregulation is associated with abnormal expression of YAP1 and TEAD-family genes. Recent works have highlighted the role of YAP1/TEAD activity in several cancers and its potential therapeutic implications. Therefore, identifying patients with a dysregulated Hippo pathway is key to enhancing treatment impact. Although recent studies have derived RNA-seq-based signatures, there remains a need for a reproducible and cost-effective method to measure the pathway activation. In recent years, deep learning applied to histology slides have emerged as an effective way to predict molecular information from a data modality available in clinical routine. Here, we trained models to predict YAP1/TEAD activity from H&E-stained histology slides in multiple cancers. The robustness of our approach was assessed in seven independent validation cohorts. Finally, we showed that histological markers of disease aggressiveness were associated with dysfunctional Hippo signaling.

Artificial intelligence applied to 'omics data in liver disease: towards a personalised approach for diagnosis, prognosis and treatment

Advancements in omics technologies and artificial intelligence (AI) methodologies are fuelling our progress towards personalised diagnosis, prognosis and treatment strategies in hepatology. This review provides a comprehensive overview of the current landscape of AI methods used for analysis of omics data in liver diseases. We present an overview of the prevalence of different omics levels across various liver diseases, as well as categorise the AI methodology used across the studies. Specifically, we highlight the predominance of transcriptomic and genomic profiling and the relatively sparse exploration of other levels such as the proteome and methylome, which represent untapped potential for novel insights. Publicly available database initiatives such as The Cancer Genome Atlas and The International Cancer Genome Consortium have paved the way for advancements in the diagnosis and treatment of hepatocellular carcinoma. However, the same availability of large omics datasets remains limited for other liver diseases. Furthermore, the application of sophisticated AI methods to handle the complexities of multiomics datasets requires substantial data to train and validate the models and faces challenges in achieving bias-free results with clinical utility. Strategies to address the paucity of data and capitalise on opportunities are discussed. Given the substantial global burden of chronic liver diseases, it is imperative that multicentre collaborations be established to generate large-scale omics data for early disease recognition and intervention. Exploring advanced AI methods is also necessary to maximise the potential of these datasets and improve early detection and personalised treatment strategies.

Advances in the understanding and therapeutic manipulation of cancer immune responsiveness: a Society for Immunotherapy of Cancer (SITC) review

Cancer immunotherapy-including immune checkpoint inhibition (ICI) and adoptive cell therapy (ACT)-has become a standard, potentially curative treatment for a subset of advanced solid and liquid tumors. However, most patients with cancer do not benefit from the rapidly evolving improvements in the understanding of principal mechanisms determining cancer immune responsiveness (CIR); including patient-specific genetically determined and acquired factors, as well as intrinsic cancer cell biology. Though CIR is multifactorial, fundamental concepts are emerging that should be considered for the design of novel therapeutic strategies and related clinical studies. Recent advancements as well as novel approaches to address the limitations of current treatments are discussed here, with a specific focus on ICI and ACT.

Computer-assisted diagnosis to improve diagnostic pathology: A review

ABSTRACT

With an increasing demand for accuracy and efficiency in diagnostic pathology, computer-assisted diagnosis (CAD) emerges as a prominent and transformative solution. This review aims to explore the practical applications, implications, strengths, and weaknesses of CAD applied to diagnostic pathology. A comprehensive literature search was conducted to include English-language studies focusing on CAD tools, digital pathology, and Artificial intelligence (AI) applications in pathology. The review underscores the transformative potential of CAD tools in pathology, particularly in streamlining diagnostic processes, reducing turnaround times, and augmenting diagnostic accuracy. It emphasizes the strides made in digital pathology, the integration of AI, and the promising prospects for prognostic biomarker discovery using computational methods. Additionally, ethical considerations regarding data privacy, equity, and trust in AI deployment are examined. CAD has the potential to revolutionize diagnostic pathology. The insights gleaned from this review offer a panoramic view of recent advancements. Ultimately, this review aims to guide future research, influence clinical practice, and inform policy-making by elucidating the promising horizons and potential pitfalls of integrating CAD tools in pathology.

Deep learning methods for improving the accuracy and efficiency of pathological image analysis

This study presents a novel integration of two advanced deep learning models, U-Net and EfficientNetV2, to achieve high-precision segmentation and rapid classification of pathological images. A key innovation is the development of a new heatmap generation algorithm, which leverages meticulous image preprocessing, data enhancement strategies, ensemble learning, attention mechanisms, and deep feature fusion techniques. This algorithm not only produces highly accurate and interpretatively rich heatmaps but also significantly improves the accuracy and efficiency of pathological image analysis. Unlike existing methods, our approach integrates these advanced techniques into a cohesive framework, enhancing its ability to reveal critical features in pathological images. Rigorous experimental validation demonstrated that our algorithm excels in key performance indicators such as accuracy, recall rate, and processing speed, underscoring its potential for broader applications in pathological image analysis and beyond.

From whole-slide image to biomarker prediction: end-to-end weakly supervised deep learning in computational pathology

Hematoxylin- and eosin-stained whole-slide images (WSIs) are the foundation of diagnosis of cancer. In recent years, development of deep learning-based methods in computational pathology has enabled the prediction of biomarkers directly from WSIs. However, accurately linking tissue phenotype to biomarkers at scale remains a crucial challenge for democratizing complex biomarkers in precision oncology. This protocol describes a practical workflow for solid tumor associative modeling in pathology (STAMP), enabling prediction of biomarkers directly from WSIs by using deep learning. The STAMP workflow is biomarker agnostic and allows for genetic and clinicopathologic tabular data to be included as an additional input, together with histopathology images. The protocol consists of five main stages that have been successfully applied to various research problems: formal problem definition, data preprocessing, modeling, evaluation and clinical translation. The STAMP workflow differentiates itself through its focus on serving as a collaborative framework that can be used by clinicians and engineers alike for setting up research projects in the field of computational pathology. As an example task, we applied STAMP to the prediction of microsatellite instability (MSI) status in colorectal cancer, showing accurate performance for the identification of tumors high in MSI. Moreover, we provide an open-source code base, which has been deployed at several hospitals across the globe to set up computational pathology workflows. The STAMP workflow requires one workday of hands-on computational execution and basic command line knowledge.

Linking transcriptome and morphology in bone cells at cellular resolution with generative AI

Recent advancements in deep learning (DL) have revolutionized the capability of artificial intelligence (AI) by enabling the analysis of large-scale, complex datasets that are difficult for humans to interpret. However, large amounts of high-quality data are required to train such generative AI models successfully. With the rapid commercialization of single-cell sequencing and spatial transcriptomics platforms, the field is increasingly producing large-scale datasets such as histological images, single-cell molecular data, and spatial transcriptomic data. These molecular and morphological datasets parallel the multimodal text and image data used to train highly successful generative AI models for natural language processing and computer vision. Thus, these emerging data types offer great potential to train generative AI models that uncover intricate biological processes of bone cells at a cellular level. In this Perspective, we summarize the progress and prospects of generative AI applied to these datasets and their potential applications to bone research. In particular, we highlight three AI applications: predicting cell differentiation dynamics, linking molecular and morphological features, and predicting cellular responses to perturbations. To make generative AI models beneficial for bone research, important issues, such as technical biases in bone single-cell datasets, lack of profiling of important bone cell types, and lack of spatial information, needs to be addressed. Realizing the potential of generative AI for bone biology will also likely require generating large-scale, high-quality cellular-resolution spatial transcriptomics datasets, improving the sensitivity of current spatial transcriptomics datasets, and thorough experimental validation of model predictions.

Machine learning methods for histopathological image analysis: Updates in 2024

The combination of artificial intelligence and digital pathology has emerged as a transformative force in healthcare and biomedical research. As an update to our 2018 review, this review presents comprehensive analysis of machine learning applications in histopathological image analysis, with focus on the developments since 2018. We highlight significant advances that have expanded the technical capabilities and practical applications of computational pathology. The review examines progress in addressing key challenges in the field as follows: processing of gigapixel whole slide images, insufficient labeled data, multidimensional analysis, domain shifts across institutions, and interpretability of machine learning models. We evaluate emerging trends, such as foundation models and multimodal integration, that are reshaping the field. Overall, our review highlights the potential of machine learning in enhancing both routine pathological analysis and scientific discovery in pathology. By providing this comprehensive overview, this review aims to guide researchers and clinicians in understanding the current state of the pathology image analysis field and its future trajectory.

Predicting the tumor microenvironment composition and immunotherapy response in non-small cell lung cancer from digital histopathology images

Immune checkpoint inhibitors (ICI) have become integral to treatment of non-small cell lung cancer (NSCLC). However, reliable biomarkers predictive of immunotherapy efficacy are limited. Here, we introduce HistoTME, a novel weakly supervised deep learning approach to infer the tumor microenvironment (TME) composition directly from histopathology images of NSCLC patients. We show that HistoTME accurately predicts the expression of 30 distinct cell type-specific molecular signatures directly from whole slide images, achieving an average Pearson correlation of 0.5 with the ground truth on independent tumor cohorts. Furthermore, we find that HistoTME-predicted microenvironment signatures and their underlying interactions improve prognostication of lung cancer patients receiving immunotherapy, achieving an AUROC of 0.75 [95% CI: 0.61-0.88] for predicting treatment responses following first-line ICI treatment, utilizing an external clinical cohort of 652 patients. Collectively, HistoTME presents an effective approach for interrogating the TME and predicting ICI response, complementing PD-L1 expression, and bringing us closer to personalized immuno-oncology.

Prediction of gene expression-based breast cancer proliferation scores from histopathology whole slide images using deep learning

BACKGROUND

In breast cancer, several gene expression assays have been developed to provide a more personalised treatment. This study focuses on the prediction of two molecular proliferation signatures: an 11-gene proliferation score and the MKI67 proliferation marker gene. The aim was to assess whether these could be predicted from digital whole slide images (WSIs) using deep learning models.

METHODS

WSIs and RNA-sequencing data from 819 invasive breast cancer patients were included for training, and models were evaluated on an internal test set of 172 cases as well as on 997 cases from a fully independent external test set. Two deep Convolutional Neural Network (CNN) models were optimised using WSIs and gene expression readouts from RNA-sequencing data of either the proliferation signature or the proliferation marker, and assessed using Spearman correlation (r). Prognostic performance was assessed through Cox proportional hazard modelling, estimating hazard ratios (HR).

RESULTS

Optimised CNNs successfully predicted the proliferation score and proliferation marker on the unseen internal test set (ρ = 0.691(p < 0.001) with R2 = 0.438, and ρ = 0.564 (p < 0.001) with R2 = 0.251 respectively) and on the external test set (ρ = 0.502 (p < 0.001) with R2 = 0.319, and ρ = 0.403 (p < 0.001) with R2 = 0.222 respectively). Patients with a high proliferation score or marker were significantly associated with a higher risk of recurrence or death in the external test set (HR = 1.65 (95% CI: 1.05-2.61) and HR = 1.84 (95% CI: 1.17-2.89), respectively).

CONCLUSIONS

The results from this study suggest that gene expression levels of proliferation scores can be predicted directly from breast cancer morphology in WSIs using CNNs and that the predictions provide prognostic information that could be used in research as well as in the clinical setting.

Artificial intelligence in surgical pathology - Where do we stand, where do we go?

Surgical and neuropathologists continuously search for new and disease-specific features, such as independent predictors of tumor prognosis or determinants of tumor entities and sub-entities. This is a task where artificial intelligence (AI)/machine learning (ML) systems could significantly contribute to help with tumor outcome prediction and the search for new diagnostic or treatment stratification biomarkers. AI systems are increasingly integrated into routine pathology workflows to improve accuracy, reproducibility, productivity and to reveal difficult-to-see features in complicated histological slides, including the quantification of important markers for tumor grading and staging. In this article, we review the infrastructure needed to facilitate digital and computational pathology. We address the barriers for its full deployment in the clinical setting and describe the use of AI in intraoperative or postoperative settings were frozen or formalin-fixed, paraffin-embedded materials are used. We also summarize quality assessment issues of slide digitization, new spatial biology approaches, and the determination of specific gene-expression from whole slide images. Finally, we highlight new innovative and future technologies, such as large language models, optical biopsies, and mass spectrometry imaging.

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Prediction of Tumor-Associated Macrophages and Immunotherapy Benefits Using Weakly Supervised Contrastive Learning in Breast Cancer Pathology Images

The efficacy of immune checkpoint inhibitors is significantly influenced by the tumor immune microenvironment (TIME). RNA sequencing of tumor tissue can offer valuable insights into TIME, but its high cost and long turnaround time seriously restrict its utility in routine clinical examinations. Several recent studies have suggested that ultrahigh-resolution pathology images can infer cellular and molecular characteristics. However, few study pay attention to the quantitative estimation of various tumor infiltration immune cells from pathology images. In this paper, we integrated contrastive learning and weakly supervised learning to infer tumor-associated macrophages and potential immunotherapy benefit from whole slide images (WSIs) of H &E stained pathological sections. We split the high-resolution WSIs into tiles and then apply contrastive learning to extract features of each tile. After aggregating the features at the tile level, we employ weak supervisory signals to fine-tune the encoder for various downstream tasks. Comprehensive experiments on two independent breast cancer cohorts and spatial transcriptomics data demonstrate that the computational pathological features accurately predict the proportion of tumor-infiltrating immune cells, particularly the infiltration level of macrophages, as well as the immune subtypes and potential immunotherapy benefit. These findings demonstrate that our model effectively captures pathological features beyond human vision, establishing a mapping relationship between cellular compositions and histological morphology, thus expanding the clinical applications of digital pathology images.

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Artificial Intelligence Applications in Lymphoma Diagnosis and Management: Opportunities, Challenges, and Future Directions

Lymphoma, a heterogeneous group of blood cancers, presents significant diagnostic and therapeutic challenges due to its complex subtypes and variable clinical outcomes. Artificial intelligence (AI) has emerged as a promising tool to enhance the accuracy and efficiency of lymphoma pathology. This review explores the potential of AI in lymphoma diagnosis, classification, prognosis prediction, and treatment planning, as well as addressing the challenges and future directions in this rapidly evolving field.

Digital profiling of gene expression from histology images with linearized attention

Cancer is a heterogeneous disease requiring costly genetic profiling for better understanding and management. Recent advances in deep learning have enabled cost-effective predictions of genetic alterations from whole slide images (WSIs). While transformers have driven significant progress in non-medical domains, their application to WSIs lags behind due to high model complexity and limited dataset sizes. Here, we introduce SEQUOIA, a linearized transformer model that predicts cancer transcriptomic profiles from WSIs. SEQUOIA is developed using 7584 tumor samples across 16 cancer types, with its generalization capacity validated on two independent cohorts comprising 1368 tumors. Accurately predicted genes are associated with key cancer processes, including inflammatory response, cell cycles and metabolism. Further, we demonstrate the value of SEQUOIA in stratifying the risk of breast cancer recurrence and in resolving spatial gene expression at loco-regional levels. SEQUOIA hence deciphers clinically relevant information from WSIs, opening avenues for personalized cancer management.

HistoSPACE: Histology-inspired spatial transcriptome prediction and characterization engine

Spatial transcriptomics (ST) enables the visualization of gene expression within the context of tissue morphology. This emerging discipline has the potential to serve as a foundation for developing tools to design precision medicines. However, due to the higher costs and expertise required for such experiments, its translation into a regular clinical practice might be challenging. Despite implementing modern deep learning to enhance information obtained from histological images using AI, efforts have been constrained by limitations in the diversity of information. In this paper, we developed a model, HistoSPACE, that explores the diversity of histological images available with ST data to extract molecular insights from tissue images. Further, our approach allows us to link the predicted expression with disease pathology. Our proposed study built an image encoder derived from a universal image autoencoder. This image encoder was connected to convolution blocks to build the final model. It was further fine-tuned with the help of ST-Data. The number of model parameters is small and requires lesser system memory and relatively lesser training time. Making it lightweight in comparison to traditional histological models. Our developed model demonstrates significant efficiency compared to contemporary algorithms, revealing a correlation of 0.56 in leave-one-out cross-validation. Finally, its robustness was validated through an independent dataset, showing similar prediction with predefined disease pathology. Our code is available at https://github.com/samrat-lab/HistoSPACE.

Development and Validation of an AI-Driven System for Automatic Literature Analysis and Molecular Regulatory Network Construction

Decoding gene regulatory networks is essential for understanding the mechanisms underlying many complex diseases. GENET is developed, an automated system designed to extract and visualize extensive molecular relationships from published biomedical literature. Using natural language processing, entities and relations are identified from a randomly selected set of 1788 scientific articles, and visualized in a filterable knowledge graph. The performance of GENET is evaluated and compared with existing methods. The named entity recognition model has achieved an overall precision of 94.23% (4835/5131; 93.56-94.84%), recall of 97.72% (4835/4948; 97.27-98.10%), and an F1 score of 95.94%. The relation extraction model has demonstrated an overall precision of 91.63% (2593/2830; 90.55-92.59%), recall of 89.17% (2593/2908; 87.99-90.25%), and an F1 score of 90.38%. GENET significantly outperforms existing methods in extracting molecular relationships (P < 0.001). Additionally, GENET has successfully predicted WNT family member 4 regulates insulin-like growth factor 2 via signal transducer and activator of transcription 3 in colon cancer. With RNA sequencing data and multiple immunofluorescence, the authenticity of this prediction is validated, supporting the promising feasibility of GENET.

SG-Fusion: A swin-transformer and graph convolution-based multi-modal deep neural network for glioma prognosis

The integration of morphological attributes extracted from histopathological images and genomic data holds significant importance in advancing tumor diagnosis, prognosis, and grading. Histopathological images are acquired through microscopic examination of tissue slices, providing valuable insights into cellular structures and pathological features. On the other hand, genomic data provides information about tumor gene expression and functionality. The fusion of these two distinct data types is crucial for gaining a more comprehensive understanding of tumor characteristics and progression. In the past, many studies relied on single-modal approaches for tumor diagnosis. However, these approaches had limitations as they were unable to fully harness the information from multiple data sources. To address these limitations, researchers have turned to multi-modal methods that concurrently leverage both histopathological images and genomic data. These methods better capture the multifaceted nature of tumors and enhance diagnostic accuracy. Nonetheless, existing multi-modal methods have, to some extent, oversimplified the extraction processes for both modalities and the fusion process. In this study, we presented a dual-branch neural network, namely SG-Fusion. Specifically, for the histopathological modality, we utilize the Swin-Transformer structure to capture both local and global features and incorporate contrastive learning to encourage the model to discern commonalities and differences in the representation space. For the genomic modality, we developed a graph convolutional network based on gene functional and expression level similarities. Additionally, our model integrates a cross-attention module to enhance information interaction and employs divergence-based regularization to enhance the model's generalization performance. Validation conducted on glioma datasets from the Cancer Genome Atlas unequivocally demonstrates that our SG-Fusion model outperforms both single-modal methods and existing multi-modal approaches in both survival analysis and tumor grading.

Applications of artificial intelligence in digital pathology for gastric cancer

Gastric cancer is one of the most common cancers and is one of the leading causes of cancer-related deaths in worldwide. Early diagnosis and treatment are essential for a positive outcome. The integration of artificial intelligence in the pathology field is increasingly widespread, including histopathological images analysis. In recent years, the application of digital pathology technology emerged as a potential solution to enhance the understanding and management of gastric cancer. Through sophisticated image analysis algorithms, artificial intelligence technologies facilitate the accuracy and sensitivity of gastric cancer diagnosis and treatment and personalized therapeutic strategies. This review aims to evaluate the current landscape and future potential of artificial intelligence in transforming gastric cancer pathology, so as to provide ideas for future research.

Spatial oncology: Translating contextual biology to the clinic

Microscopic examination of cells in their tissue context has been the driving force behind diagnostic histopathology over the past two centuries. Recently, the rise of advanced molecular biomarkers identified through single cell profiling has increased our understanding of cellular heterogeneity in cancer but have yet to significantly impact clinical care. Spatial technologies integrating molecular profiling with microenvironmental features are poised to bridge this translational gap by providing critical in situ context for understanding cellular interactions and organization. Here, we review how spatial tools have been used to study tumor ecosystems and their clinical applications. We detail findings in cell-cell interactions, microenvironment composition, and tissue remodeling for immune evasion and therapeutic resistance. Additionally, we highlight the emerging role of multi-omic spatial profiling for characterizing clinically relevant features including perineural invasion, tertiary lymphoid structures, and the tumor-stroma interface. Finally, we explore strategies for clinical integration and their augmentation of therapeutic and diagnostic approaches.

Prediction of homologous recombination deficiency from routine histology with attention-based multiple instance learning in nine different tumor types

BACKGROUND

Homologous recombination deficiency (HRD) is recognized as a pan-cancer predictive biomarker that potentially indicates who could benefit from treatment with PARP inhibitors (PARPi). Despite its clinical significance, HRD testing is highly complex. Here, we investigated in a proof-of-concept study whether Deep Learning (DL) can predict HRD status solely based on routine hematoxylin & eosin (H&E) histology images across nine different cancer types.

METHODS

We developed a deep learning pipeline with attention-weighted multiple instance learning (attMIL) to predict HRD status from histology images. As part of our approach, we calculated a genomic scar HRD score by combining loss of heterozygosity (LOH), telomeric allelic imbalance (TAI), and large-scale state transitions (LST) from whole genome sequencing (WGS) data of n = 5209 patients across two independent cohorts. The model's effectiveness was evaluated using the area under the receiver operating characteristic curve (AUROC), focusing on its accuracy in predicting genomic HRD against a clinically recognized cutoff value.

RESULTS

Our study demonstrated the predictability of genomic HRD status in endometrial, pancreatic, and lung cancers reaching cross-validated AUROCs of 0.79, 0.58, and 0.66, respectively. These predictions generalized well to an external cohort, with AUROCs of 0.93, 0.81, and 0.73. Moreover, a breast cancer-trained image-based HRD classifier yielded an AUROC of 0.78 in the internal validation cohort and was able to predict HRD in endometrial, prostate, and pancreatic cancer with AUROCs of 0.87, 0.84, and 0.67, indicating that a shared HRD-like phenotype occurs across these tumor entities.

CONCLUSIONS

This study establishes that HRD can be directly predicted from H&E slides using attMIL, demonstrating its applicability across nine different tumor types.

The ACROBAT 2022 challenge: Automatic registration of breast cancer tissue

The alignment of tissue between histopathological whole-slide-images (WSI) is crucial for research and clinical applications. Advances in computing, deep learning, and availability of large WSI datasets have revolutionised WSI analysis. Therefore, the current state-of-the-art in WSI registration is unclear. To address this, we conducted the ACROBAT challenge, based on the largest WSI registration dataset to date, including 4,212 WSIs from 1,152 breast cancer patients. The challenge objective was to align WSIs of tissue that was stained with routine diagnostic immunohistochemistry to its H&E-stained counterpart. We compare the performance of eight WSI registration algorithms, including an investigation of the impact of different WSI properties and clinical covariates. We find that conceptually distinct WSI registration methods can lead to highly accurate registration performances and identify covariates that impact performances across methods. These results provide a comparison of the performance of current WSI registration methods and guide researchers in selecting and developing methods.

The digital revolution in veterinary pathology

For the past two centuries, the use of traditional light microscopy to examine tissues to make diagnoses has remained relatively unchanged. While the fundamental concept of tissue slide analysis has stayed the same, our interaction with the microscope is undergoing significant changes. Digital pathology (DP) has gained momentum in veterinary science and is on the verge of becoming a vital tool in diagnostics, research and education. Many diagnostic laboratories have incorporated DP as a critical part of their workflows. Innovations in DP and whole slide image technology have made telediagnosis (the process of transmitting digital clinical data using telecommunication networks for distant diagnosis) more accessible, leading to improved patient care through streamlining of workflows and greater accessibility of second opinions. The integration of machine learning and artificial intelligence and human-in-the-loop protocols for DP workflows will further the development of computer-aided diagnosis and prognostic tools. Despite its present weaknesses, DP will progressively aid veterinary clinicians and pathologists in delivering more accurate and reliable diagnoses. Consistent incorporation of DP frontline advancements into routine veterinary diagnostic pipelines will assist in improving current tools and help prepare pathologists for the progression of digitalization in the field.

An initial game-theoretic assessment of enhanced tissue preparation and imaging protocols for improved deep learning inference of spatial transcriptomics from tissue morphology

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale.

Gene expression prediction from histology images via hypergraph neural networks

Spatial transcriptomics reveals the spatial distribution of genes in complex tissues, providing crucial insights into biological processes, disease mechanisms, and drug development. The prediction of gene expression based on cost-effective histology images is a promising yet challenging field of research. Existing methods for gene prediction from histology images exhibit two major limitations. First, they ignore the intricate relationship between cell morphological information and gene expression. Second, these methods do not fully utilize the different latent stages of features extracted from the images. To address these limitations, we propose a novel hypergraph neural network model, HGGEP, to predict gene expressions from histology images. HGGEP includes a gradient enhancement module to enhance the model's perception of cell morphological information. A lightweight backbone network extracts multiple latent stage features from the image, followed by attention mechanisms to refine the representation of features at each latent stage and capture their relations with nearby features. To explore higher-order associations among multiple latent stage features, we stack them and feed into the hypergraph to establish associations among features at different scales. Experimental results on multiple datasets from disease samples including cancers and tumor disease, demonstrate the superior performance of our HGGEP model than existing methods.

Multimodal contrastive learning for spatial gene expression prediction using histology images

In recent years, the advent of spatial transcriptomics (ST) technology has unlocked unprecedented opportunities for delving into the complexities of gene expression patterns within intricate biological systems. Despite its transformative potential, the prohibitive cost of ST technology remains a significant barrier to its widespread adoption in large-scale studies. An alternative, more cost-effective strategy involves employing artificial intelligence to predict gene expression levels using readily accessible whole-slide images stained with Hematoxylin and Eosin (H&E). However, existing methods have yet to fully capitalize on multimodal information provided by H&E images and ST data with spatial location. In this paper, we propose mclSTExp, a multimodal contrastive learning with Transformer and Densenet-121 encoder for Spatial Transcriptomics Expression prediction. We conceptualize each spot as a "word", integrating its intrinsic features with spatial context through the self-attention mechanism of a Transformer encoder. This integration is further enriched by incorporating image features via contrastive learning, thereby enhancing the predictive capability of our model. We conducted an extensive evaluation of highly variable genes in two breast cancer datasets and a skin squamous cell carcinoma dataset, and the results demonstrate that mclSTExp exhibits superior performance in predicting spatial gene expression. Moreover, mclSTExp has shown promise in interpreting cancer-specific overexpressed genes, elucidating immune-related genes, and identifying specialized spatial domains annotated by pathologists. Our source code is available at https://github.com/shizhiceng/mclSTExp.