Pan-cancer integrative histology-genomic analysis via multimodal deep learning

Translation of AI into oncology clinical practice

Artificial intelligence (AI) is a transformative technology that is capturing popular imagination and can revolutionize biomedicine. AI and machine learning (ML) algorithms have the potential to break through existing barriers in oncology research and practice such as automating workflow processes, personalizing care, and reducing healthcare disparities. Emerging applications of AI/ML in the literature include screening and early detection of cancer, disease diagnosis, response prediction, prognosis, and accelerated drug discovery. Despite this excitement, only few AI/ML models have been properly validated and fewer have become regulated products for routine clinical use. In this review, we highlight the main challenges impeding AI/ML clinical translation. We present different clinical use cases from the domains of radiology, radiation oncology, immunotherapy, and drug discovery in oncology. We dissect the unique challenges and opportunities associated with each of these cases. Finally, we summarize the general requirements for successful AI/ML implementation in the clinic, highlighting specific examples and points of emphasis including the importance of multidisciplinary collaboration of stakeholders, role of domain experts in AI augmentation, transparency of AI/ML models, and the establishment of a comprehensive quality assurance program to mitigate risks of training bias and data drifts, all culminating toward safer and more beneficial AI/ML applications in oncology labs and clinics.

Standardizing analysis of intra-tumoral heterogeneity with computational pathology

Many malignant cancers like glioblastoma are highly adaptive diseases that dynamically change their regional biology to survive and thrive under diverse microenvironmental and therapeutic pressures. While the concept of intra-tumoral heterogeneity has become a major paradigm in cancer research and care, systematic approaches to assess and document bio-variation in cancer are still in their infancy. Here we discuss existing approaches and challenges to documenting intra-tumoral heterogeneity and emerging computational approaches that leverage artificial intelligence to begin to overcome these limitations. We propose how these emerging techniques can be coupled with a diversity of molecular tools to address intra-tumoral heterogeneity more systematically in research and in practice, especially across larger specimens and longitudinal analyses. Systematic documentation and characterization of heterogeneity across entire tumor specimens and their longitudinal evolution has the potential to improve our understanding and treatment of cancer.

Automatic Segmentation with Deep Learning in Radiotherapy

This review provides a formal overview of current automatic segmentation studies that use deep learning in radiotherapy. It covers 807 published papers and includes multiple cancer sites, image types (CT/MRI/PET), and segmentation methods. We collect key statistics about the papers to uncover commonalities, trends, and methods, and identify areas where more research might be needed. Moreover, we analyzed the corpus by posing explicit questions aimed at providing high-quality and actionable insights, including: "What should researchers think about when starting a segmentation study?", "How can research practices in medical image segmentation be improved?", "What is missing from the current corpus?", and more. This allowed us to provide practical guidelines on how to conduct a good segmentation study in today's competitive environment that will be useful for future research within the field, regardless of the specific radiotherapeutic subfield. To aid in our analysis, we used the large language model ChatGPT to condense information.

Machine learning in computational histopathology: Challenges and opportunities

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Novel tools for early diagnosis and precision treatment based on artificial intelligence

Lung cancer has the highest mortality rate among all cancers in the world. Hence, early diagnosis and personalized treatment plans are crucial to improving its 5-year survival rate. Chest computed tomography (CT) serves as an essential tool for lung cancer screening, and pathology images are the gold standard for lung cancer diagnosis. However, medical image evaluation relies on manual labor and suffers from missed diagnosis or misdiagnosis, and physician heterogeneity. The rapid development of artificial intelligence (AI) has brought a whole novel opportunity for medical task processing, demonstrating the potential for clinical application in lung cancer diagnosis and treatment. AI technologies, including machine learning and deep learning, have been deployed extensively for lung nodule detection, benign and malignant classification, and subtype identification based on CT images. Furthermore, AI plays a role in the non-invasive prediction of genetic mutations and molecular status to provide the optimal treatment regimen, and applies to the assessment of therapeutic efficacy and prognosis of lung cancer patients, enabling precision medicine to become a reality. Meanwhile, histology-based AI models assist pathologists in typing, molecular characterization, and prognosis prediction to enhance the efficiency of diagnosis and treatment. However, the leap to extensive clinical application still faces various challenges, such as data sharing, standardized label acquisition, clinical application regulation, and multimodal integration. Nevertheless, AI holds promising potential in the field of lung cancer to improve cancer care.

Computational textural mapping harmonises sampling variation and reveals multidimensional histopathological fingerprints

BACKGROUND

Technical factors can bias H&E digital slides potentially compromising computational histopathology studies. Here, we hypothesised that sample quality and sampling variation can introduce even greater and undocumented technical fallacy.

METHODS

Using The Cancer Genome Atlas (TCGA) clear-cell renal cell carcinoma (ccRCC) as a model disease, we annotated ~78,000 image tiles and trained deep learning models to detect histological textures and lymphocyte infiltration at the tumour core and its surrounding margin and correlated these with clinical, immunological, genomic, and transcriptomic profiles.

RESULTS

The models reached 95% validation accuracy for classifying textures and 95% for lymphocyte infiltration enabling reliable profiling of ccRCC samples. We validated the lymphocyte-per-texture distributions in the Helsinki dataset (n = 64). Texture analysis indicated constitutive sampling bias by TCGA clinical centres and technically suboptimal samples. We demonstrate how computational texture mapping (CTM) can abrogate these issues by normalising textural variance. CTM-harmonised histopathological architecture resonated with both expected associations and novel molecular fingerprints. For instance, tumour fibrosis associated with histological grade, epithelial-to-mesenchymal transition, low mutation burden and metastasis.

CONCLUSIONS

This study highlights texture-based standardisation to resolve technical bias in computational histopathology and understand the molecular basis of tissue architecture. All code, data and models are released as a community resource.

Prognostic and clinicopathological role of RACK1 for cancer patients: a systematic review and meta-analysis

Background

The receptor for activated C kinase 1 (RACK1) expression is associated with clinicopathological characteristics and the prognosis of various cancers; however, the conclusions are controversial. As a result, this study aimed to explore the clinicopathological and prognostic values of RACK1 expression in patients with cancer.

Methodology

PubMed, Embase, Web of Science, Cochrane Library, and Scopus were comprehensively explored from their inception to April 20, 2023, for selecting studies on the clinicopathological and prognostic role of RACK1 in patients with cancer that met the criteria for inclusion in this review. Pooled hazard ratios (HRs) and 95% confidence intervals (CIs) were used to assess the prognosis-predictive value of RACK1 expression, while pooled odds ratios (ORs) and 95% CIs were used to evaluate the correlation between RACK1 expression and the clinicopathological characteristics of patients with cancer. The quality of the included studies was evaluated using the Newcastle-Ottawa Scale.

Results

Twenty-two studies (13 on prognosis and 20 on clinicopathological characteristics) were included in this systematic review and meta-analysis. The findings indicated that high RACK1 expression was significantly associated with poor overall survival (HR = 1.62; 95% CI, 1.13-2.33; P = 0.009; I2 = 89%) and reversely correlated with disease-free survival/recurrence-free survival (HR = 1.87; 95% CI, 1.22-2.88; P = 0.004; I2 = 0%). Furthermore, increased RACK1 expression was significantly associated with lymphatic invasion/N+ stage (OR = 1.74; 95% CI, 1.04-2.90; P = 0.04; I2 = 79%) of tumors.

Conclusions

RACK1 may be a global predictive marker of poor prognosis in patients with cancer and unfavorable clinicopathological characteristics. However, further clinical studies are required to validate these findings.

An overview and a roadmap for artificial intelligence in hematology and oncology

BACKGROUND

Artificial intelligence (AI) is influencing our society on many levels and has broad implications for the future practice of hematology and oncology. However, for many medical professionals and researchers, it often remains unclear what AI can and cannot do, and what are promising areas for a sensible application of AI in hematology and oncology. Finally, the limits and perils of using AI in oncology are not obvious to many healthcare professionals.

METHODS

In this article, we provide an expert-based consensus statement by the joint Working Group on "Artificial Intelligence in Hematology and Oncology" by the German Society of Hematology and Oncology (DGHO), the German Association for Medical Informatics, Biometry and Epidemiology (GMDS), and the Special Interest Group Digital Health of the German Informatics Society (GI). We provide a conceptual framework for AI in hematology and oncology.

RESULTS

First, we propose a technological definition, which we deliberately set in a narrow frame to mainly include the technical developments of the last ten years. Second, we present a taxonomy of clinically relevant AI systems, structured according to the type of clinical data they are used to analyze. Third, we show an overview of potential applications, including clinical, research, and educational environments with a focus on hematology and oncology.

CONCLUSION

Thus, this article provides a point of reference for hematologists and oncologists, and at the same time sets forth a framework for the further development and clinical deployment of AI in hematology and oncology in the future.

Unleashing the potential of AI for pathology: challenges and recommendations

Computational pathology is currently witnessing a surge in the development of AI techniques, offering promise for achieving breakthroughs and significantly impacting the practices of pathology and oncology. These AI methods bring with them the potential to revolutionize diagnostic pipelines as well as treatment planning and overall patient care. Numerous peer-reviewed studies reporting remarkable performance across diverse tasks serve as a testimony to the potential of AI in the field. However, widespread adoption of these methods in clinical and pre-clinical settings still remains a challenge. In this review article, we present a detailed analysis of the major obstacles encountered during the development of effective models and their deployment in practice. We aim to provide readers with an overview of the latest developments, assist them with insights into identifying some specific challenges that may require resolution, and suggest recommendations and potential future research directions. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response

In recent years, the application of advanced analytics, especially artificial intelligence (AI), to digital H&E images, and other histological image types, has begun to radically change how histological images are used in the clinic. Alongside the recognition that the tumour microenvironment (TME) has a profound impact on tumour phenotype, the technical development of highly multiplexed immunofluorescence platforms has enhanced the biological complexity that can be captured in the TME with high precision. AI has an increasingly powerful role in the recognition and quantitation of image features and the association of such features with clinically important outcomes, as occurs in distinct stages in conventional machine learning. Deep-learning algorithms are able to elucidate TME patterns inherent in the input data with minimum levels of human intelligence and, hence, have the potential to achieve clinically relevant predictions and discovery of important TME features. Furthermore, the diverse repertoire of deep-learning algorithms able to interrogate TME patterns extends beyond convolutional neural networks to include attention-based models, graph neural networks, and multimodal models. To date, AI models have largely been evaluated retrospectively, outside the well-established rigour of prospective clinical trials, in part because traditional clinical trial methodology may not always be suitable for the assessment of AI technology. However, to enable digital pathology-based advanced analytics to meaningfully impact clinical care, specific measures of 'added benefit' to the current standard of care and validation in a prospective setting are important. This will need to be accompanied by adequate measures of explainability and interpretability. Despite such challenges, the combination of expanding datasets, increased computational power, and the possibility of integration of pre-clinical experimental insights into model development means there is exciting potential for the future progress of these AI applications. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

RNA-to-image multi-cancer synthesis using cascaded diffusion models

Data scarcity presents a significant obstacle in the field of biomedicine, where acquiring diverse and sufficient datasets can be costly and challenging. Synthetic data generation offers a potential solution to this problem by expanding dataset sizes, thereby enabling the training of more robust and generalizable machine learning models. Although previous studies have explored synthetic data generation for cancer diagnosis, they have predominantly focused on single modality settings, such as whole-slide image tiles or RNA-Seq data. To bridge this gap, we propose a novel approach, RNA-Cascaded-Diffusion-Model or RNA-CDM, for performing RNA-to-image synthesis in a multi-cancer context, drawing inspiration from successful text-to-image synthesis models used in natural images. In our approach, we employ a variational auto-encoder to reduce the dimensionality of a patient's gene expression profile, effectively distinguishing between different types of cancer. Subsequently, we employ a cascaded diffusion model to synthesize realistic whole-slide image tiles using the latent representation derived from the patient's RNA-Seq data. Our results demonstrate that the generated tiles accurately preserve the distribution of cell types observed in real-world data, with state-of-the-art cell identification models successfully detecting important cell types in the synthetic samples. Furthermore, we illustrate that the synthetic tiles maintain the cell fraction observed in bulk RNA-Seq data and that modifications in gene expression affect the composition of cell types in the synthetic tiles. Next, we utilize the synthetic data generated by RNA-CDM to pretrain machine learning models and observe improved performance compared to training from scratch. Our study emphasizes the potential usefulness of synthetic data in developing machine learning models in sarce-data settings, while also highlighting the possibility of imputing missing data modalities by leveraging the available information. In conclusion, our proposed RNA-CDM approach for synthetic data generation in biomedicine, particularly in the context of cancer diagnosis, offers a novel and promising solution to address data scarcity. By generating synthetic data that aligns with real-world distributions and leveraging it to pretrain machine learning models, we contribute to the development of robust clinical decision support systems and potential advancements in precision medicine.

Precision diagnosis and staging of TDP-43 proteinopathies: harnessing the power of artificial intelligence

This scientific commentary refers to ‘Data-driven neuropathological staging and subtyping of TDP-43 proteinopathies’ by Young et al. (https://doi.org/10.1093/brain/awad145).

Artificial Intelligence in the Pathology of Gastric Cancer

Recent advances in artificial intelligence (AI) have provided novel tools for rapid and precise pathologic diagnosis. The introduction of digital pathology has enabled the acquisition of scanned slide images that are essential for the application of AI. The application of AI for improved pathologic diagnosis includes the error-free detection of potentially negligible lesions, such as a minute focus of metastatic tumor cells in lymph nodes, the accurate diagnosis of potentially controversial histologic findings, such as very well-differentiated carcinomas mimicking normal epithelial tissues, and the pathological subtyping of the cancers. Additionally, the utilization of AI algorithms enables the precise decision of the score of immunohistochemical markers for targeted therapies, such as human epidermal growth factor receptor 2 and programmed death-ligand 1. Studies have revealed that AI assistance can reduce the discordance of interpretation between pathologists and more accurately predict clinical outcomes. Several approaches have been employed to develop novel biomarkers from histologic images using AI. Moreover, AI-assisted analysis of the cancer microenvironment showed that the distribution of tumor-infiltrating lymphocytes was related to the response to the immune checkpoint inhibitor therapy, emphasizing its value as a biomarker. As numerous studies have demonstrated the significance of AI-assisted interpretation and biomarker development, the AI-based approach will advance diagnostic pathology.

Machine learning with multimodal data for COVID-19

In response to the unprecedented global healthcare crisis of the COVID-19 pandemic, the scientific community has joined forces to tackle the challenges and prepare for future pandemics. Multiple modalities of data have been investigated to understand the nature of COVID-19. In this paper, MIDRC investigators present an overview of the state-of-the-art development of multimodal machine learning for COVID-19 and model assessment considerations for future studies. We begin with a discussion of the lessons learned from radiogenomic studies for cancer diagnosis. We then summarize the multi-modality COVID-19 data investigated in the literature including symptoms and other clinical data, laboratory tests, imaging, pathology, physiology, and other omics data. Publicly available multimodal COVID-19 data provided by MIDRC and other sources are summarized. After an overview of machine learning developments using multimodal data for COVID-19, we present our perspectives on the future development of multimodal machine learning models for COVID-19.

A Weakly Supervised Deep Learning Model and Human-Machine Fusion for Accurate Grading of Renal Cell Carcinoma from Histopathology Slides

(1) Background: The Fuhrman grading (FG) system is widely used in the management of clear cell renal cell carcinoma (ccRCC). However, it is affected by observer variability and irreproducibility in clinical practice. We aimed to use a deep learning multi-class model called SSL-CLAM to assist in diagnosing the FG status of ccRCC patients using digitized whole slide images (WSIs). (2) Methods: We recruited 504 eligible ccRCC patients from The Cancer Genome Atlas (TCGA) cohort and obtained 708 hematoxylin and eosin-stained WSIs for the development and internal validation of the SSL-CLAM model. Additionally, we obtained 445 WSIs from 188 ccRCC eligible patients in the Clinical Proteomic Tumor Analysis Consortium (CPTAC) cohort as an independent external validation set. A human-machine fusion approach was used to validate the added value of the SSL-CLAM model for pathologists. (3) Results: The SSL-CLAM model successfully diagnosed the five FG statuses (Grade-0, 1, 2, 3, and 4) of ccRCC, and achieved AUCs of 0.917 and 0.887 on the internal and external validation sets, respectively, outperforming a junior pathologist. For the normal/tumor classification (Grade-0, Grade-1/2/3/4) task, the SSL-CLAM model yielded AUCs close to 1 on both the internal and external validation sets. The SSL-CLAM model achieved a better performance for the two-tiered FG (Grade-0, Grade-1/2, and Grade-3/4) task, with AUCs of 0.936 and 0.915 on the internal and external validation sets, respectively. The human-machine diagnostic performance was superior to that of the SSL-CLAM model, showing promising prospects. In addition, the high-attention regions of the SSL-CLAM model showed that with an increasing FG status, the cell nuclei in the tumor region become larger, with irregular contours and increased cellular pleomorphism. (4) Conclusions: Our findings support the feasibility of using deep learning and human-machine fusion methods for FG classification on WSIs from ccRCC patients, which may assist pathologists in making diagnostic decisions.

Hierarchical discriminative learning improves visual representations of biomedical microscopy

Learning high-quality, self-supervised, visual representations is essential to advance the role of computer vision in biomedical microscopy and clinical medicine. Previous work has focused on self-supervised representation learning (SSL) methods developed for instance discrimination and applied them directly to image patches, or fields-of-view, sampled from gigapixel whole-slide images (WSIs) used for cancer diagnosis. However, this strategy is limited because it (1) assumes patches from the same patient are independent, (2) neglects the patient-slide-patch hierarchy of clinical biomedical microscopy, and (3) requires strong data augmentations that can degrade downstream performance. Importantly, sampled patches from WSIs of a patient's tumor are a diverse set of image examples that capture the same underlying cancer diagnosis. This motivated HiDisc, a data-driven method that leverages the inherent patient-slide-patch hierarchy of clinical biomedical microscopy to define a hierarchical discriminative learning task that implicitly learns features of the underlying diagnosis. HiDisc uses a self-supervised contrastive learning framework in which positive patch pairs are defined based on a common ancestry in the data hierarchy, and a unified patch, slide, and patient discriminative learning objective is used for visual SSL. We benchmark HiDisc visual representations on two vision tasks using two biomedical microscopy datasets, and demonstrate that (1) HiDisc pretraining outperforms current state-of-the-art self-supervised pretraining methods for cancer diagnosis and genetic mutation prediction, and (2) HiDisc learns high-quality visual representations using natural patch diversity without strong data augmentations.

Algorithmic fairness in artificial intelligence for medicine and healthcare

In healthcare, the development and deployment of insufficiently fair systems of artificial intelligence (AI) can undermine the delivery of equitable care. Assessments of AI models stratified across subpopulations have revealed inequalities in how patients are diagnosed, treated and billed. In this Perspective, we outline fairness in machine learning through the lens of healthcare, and discuss how algorithmic biases (in data acquisition, genetic variation and intra-observer labelling variability, in particular) arise in clinical workflows and the resulting healthcare disparities. We also review emerging technology for mitigating biases via disentanglement, federated learning and model explainability, and their role in the development of AI-based software as a medical device.

Quantifying Intratumoral Heterogeneity and Immunoarchitecture Generated In-Silico by a Spatial Quantitative Systems Pharmacology Model

Spatial heterogeneity is a hallmark of cancer. Tumor heterogeneity can vary with time and location. The tumor microenvironment (TME) encompasses various cell types and their interactions that impart response to therapies. Therefore, a quantitative evaluation of tumor heterogeneity is crucial for the development of effective treatments. Different approaches, such as multiregional sequencing, spatial transcriptomics, analysis of autopsy samples, and longitudinal analysis of biopsy samples, can be used to analyze the intratumoral heterogeneity (ITH) and temporal evolution and to reveal the mechanisms of therapeutic response. However, because of the limitations of these data and the uncertainty associated with the time points of sample collection, having a complete understanding of intratumoral heterogeneity role is challenging. Here, we used a hybrid model that integrates a whole-patient compartmental quantitative-systems-pharmacology (QSP) model with a spatial agent-based model (ABM) describing the TME; we applied four spatial metrics to quantify model-simulated intratumoral heterogeneity and classified the TME immunoarchitecture for representative cases of effective and ineffective anti-PD-1 therapy. The four metrics, adopted from computational digital pathology, included mixing score, average neighbor frequency, Shannon's entropy and area under the curve (AUC) of the G-cross function. A fifth non-spatial metric was used to supplement the analysis, which was the ratio of the number of cancer cells to immune cells. These metrics were utilized to classify the TME as "cold", "compartmentalized" and "mixed", which were related to treatment efficacy. The trends in these metrics for effective and ineffective treatments are in qualitative agreement with the clinical literature, indicating that compartmentalized immunoarchitecture is likely to result in more efficacious treatment outcomes.

Histopathological bladder cancer gene mutation prediction with hierarchical deep multiple-instance learning

Gene mutation detection is usually carried out by molecular biological methods, which is expensive and has a long-time cycle. In contrast, pathological images are ubiquitous. If clinically significant gene mutations can be predicted only through pathological images, it will greatly promote the widespread use of gene mutation detection in clinical practice. However, current gene mutation prediction methods based on pathological images are ineffective because of the inability to identify mutated regions in gigapixel Whole Slide Image (WSI). To address this challenge, hereby we propose a carefully designed framework for WSI-based gene mutation prediction, which consists of three parts. (i) The first part of cancerous area segmentation, based on supervised learning, quickly filters out a large number of non-mutated regions; (ii) the second part of cancerous patch clustering, based on the representations derived from contrastive learning, ensures the comprehensiveness of patch selection; and (iii) the third part of mutation classification, based on the proposed hierarchical deep multi-instance learning method (HDMIL), ensures that sufficient patches are considered and inaccurate selections are ignored. In addition, benefiting from a two-stage attention mechanism in HDMIL, the patches that are highly correlated with gene mutations can be identified. This interpretability can help a pathologist to analyze the correlation between gene mutation and histopathological morphology. Experimental results demonstrate that the proposed gene mutation prediction framework significantly outperforms the state-of-the-art methods. In the TCGA bladder cancer dataset, five clinically relevant gene mutations are well predicted.

Generalizable biomarker prediction from cancer pathology slides with self-supervised deep learning: A retrospective multi-centric study

Deep learning (DL) can predict microsatellite instability (MSI) from routine histopathology slides of colorectal cancer (CRC). However, it is unclear whether DL can also predict other biomarkers with high performance and whether DL predictions generalize to external patient populations. Here, we acquire CRC tissue samples from two large multi-centric studies. We systematically compare six different state-of-the-art DL architectures to predict biomarkers from pathology slides, including MSI and mutations in BRAF, KRAS, NRAS, and PIK3CA. Using a large external validation cohort to provide a realistic evaluation setting, we show that models using self-supervised, attention-based multiple-instance learning consistently outperform previous approaches while offering explainable visualizations of the indicative regions and morphologies. While the prediction of MSI and BRAF mutations reaches a clinical-grade performance, mutation prediction of PIK3CA, KRAS, and NRAS was clinically insufficient.

Transcriptome Meta-Analysis of Triple-Negative Breast Cancer Response to Neoadjuvant Chemotherapy

Triple-negative breast cancer (TNBC) is a heterogeneous disease with varying responses to neoadjuvant chemotherapy (NAC). The identification of biomarkers to predict NAC response and inform personalized treatment strategies is essential. In this study, we conducted large-scale gene expression meta-analyses to identify genes associated with NAC response and survival outcomes. The results showed that immune, cell cycle/mitotic, and RNA splicing-related pathways were significantly associated with favorable clinical outcomes. Furthermore, we integrated and divided the gene association results from NAC response and survival outcomes into four quadrants, which provided more insights into potential NAC response mechanisms and biomarker discovery.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

GraphLSurv: A scalable survival prediction network with adaptive and sparse structure learning for histopathological whole-slide images

BACKGROUND AND OBJECTIVE

Predicting patients' survival from gigapixel Whole-Slide Images (WSIs) has always been a challenging task. To learn effective WSI representations for survival prediction, existing deep learning methods have explored utilizing graphs to describe the complex structure inner WSIs, where graph node is respective to WSI patch. However, these graphs are often densely-connected or static, leading to some redundant or missing patch correlations. Moreover, these methods cannot be directly scaled to the very-large WSI with more than 10,000 patches. To address these, this paper proposes a scalable graph convolution network, GraphLSurv, which can efficiently learn adaptive and sparse structures to better characterize WSIs for survival prediction.

METHODS

GraphLSurv has three highlights in methodology: (1) it generates adaptive and sparse structures for patches so that latent patch correlations could be captured and adjusted dynamically according to prediction tasks; (2) based on the generated structure and a given graph, GraphLSurv further aggregates local microenvironmental cues into a non-local embedding using the proposed hybrid message passing network; (3) to make this network suitable for very large-scale graphs, it adopts an anchor-based technique to reduce theorical computation complexity.

RESULTS

The experiments on 2268 WSIs show that GraphLSurv achieves a concordance-index of 0.66132 and 0.68348, with an improvement of 3.79% and 3.41% compared to existing methods, on NLST and TCGA-BRCA, respectively.

CONCLUSIONS

GraphLSurv could often perform better than previous methods, which suggests that GraphLSurv could provide an important and effective means for WSI survival prediction. Moreover, this work empirically shows that adaptive and sparse structures could be more suitable than static or dense ones for modeling WSIs.

Multimodal data fusion for cancer biomarker discovery with deep learning

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology

The histopathological phenotype of tumors reflects the underlying genetic makeup. Deep learning can predict genetic alterations from pathology slides, but it is unclear how well these predictions generalize to external datasets. We performed a systematic study on Deep-Learning-based prediction of genetic alterations from histology, using two large datasets of multiple tumor types. We show that an analysis pipeline that integrates self-supervised feature extraction and attention-based multiple instance learning achieves a robust predictability and generalizability.

Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology

Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.

Global Epidemiology and Genetics of Hepatocellular Carcinoma

Hepatocellular carcinoma (HCC) is one of the leading cancers worldwide. Classically, HCC develops in genetically susceptible individuals who are exposed to risk factors, especially in the presence of liver cirrhosis. Significant temporal and geographic variations exist for HCC and its etiologies. Over time, the burden of HCC has shifted from the low-moderate to the high sociodemographic index regions, reflecting the transition from viral to nonviral causes. Geographically, the hepatitis viruses predominate as the causes of HCC in Asia and Africa. Although there are genetic conditions that confer increased risk for HCC, these diagnoses are rarely recognized outside North America and Europe. In this review, we will evaluate the epidemiologic trends and risk factors of HCC, and discuss the genetics of HCC, including monogenic diseases, single-nucleotide polymorphisms, gut microbiome, and somatic mutations.

Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer

Although it has long been known that the immune cell composition has a strong prognostic and predictive value in colorectal cancer (CRC), scoring systems such as the immunoscore (IS) or quantification of intraepithelial lymphocytes are only slowly being adopted into clinical routine use and have their limitations. To address this we established and evaluated a multistain deep learning model (MSDLM) utilizing artificial intelligence (AI) to determine the AImmunoscore (AIS) in more than 1,000 patients with CRC. Our model had high prognostic capabilities and outperformed other clinical, molecular and immune cell-based parameters. It could also be used to predict the response to neoadjuvant therapy in patients with rectal cancer. Using an explainable AI approach, we confirmed that the MSDLM's decisions were based on established cellular patterns of anti-tumor immunity. Hence, the AIS could provide clinicians with a valuable decision-making tool based on the tumor immune microenvironment.

Exploring prognostic indicators in the pathological images of ovarian cancer based on a deep survival network

Background: Tumor pathology can assess patient prognosis based on a morphological deviation of tumor tissue from normal. Digitizing whole slide images (WSIs) of tissue enables the use of deep learning (DL) techniques in pathology, which may shed light on prognostic indicators of cancers, and avoid biases introduced by human experience. Purpose: We aim to explore new prognostic indicators of ovarian cancer (OC) patients using the DL framework on WSIs, and provide a valuable approach for OC risk stratification. Methods: We obtained the TCGA-OV dataset from the NIH Genomic Data Commons Data Portal database. The preprocessing of the dataset was comprised of three stages: 1) The WSIs and corresponding clinical data were paired and filtered based on a unique patient ID; 2) a weakly-supervised CLAM WSI-analysis tool was exploited to segment regions of interest; 3) the pre-trained model ResNet50 on ImageNet was employed to extract feature tensors. We proposed an attention-based network to predict a hazard score for each case. Furthermore, all cases were divided into a high-risk score group and a low-risk one according to the median as the threshold value. The multi-omics data of OC patients were used to assess the potential applications of the risk score. Finally, a nomogram based on risk scores and age features was established. Results: A total of 90 WSIs were processed, extracted, and fed into the attention-based network. The mean value of the resulting C-index was 0.5789 (0.5096-0.6053), and the resulting p-value was 0.00845. Moreover, the risk score showed a better prediction ability in the HRD + subgroup. Conclusion: Our deep learning framework is a promising method for searching WSIs, and providing a valuable clinical means for prognosis.

Multi-omics Uncovering Different Faces of Clear Cell Ovarian Cancer

The diagnosis of clear cell ovarian cancer relies on expert histopathology review. Further characterization from deep genomic and transcriptomic analyses can identify different subgroups. International collaboration is required to define the clinical impact and therapy opportunities in these specific subclassifications. See related article by Bolton et al., p. 4947.

Human and machine: Better at pathology together?

Meaningful integration of artificial intelligence (AI) will transform the application of "big data" for patient care, diagnosis, and research. In this issue of Cancer Cell, Chen et al. describe a transparent system to integrate histopathology and molecular data to predict outcomes and identify novel biomarkers in cancer.