Two-stage Cox-nnet: biologically interpretable neural-network model for prognosis prediction and its application in liver cancer survival using histopathology and transcriptomic data

MIF: Multi-Shot Interactive Fusion Model for Cancer Survival Prediction Using Pathological Image and Genomic Data

Accurate cancer survival prediction is crucial for oncologists to determine therapeutic plan, which directly influences the treatment efficacy and survival outcome of patient. Recently, multimodal fusion-based prognostic methods have demonstrated effectiveness for survival prediction by fusing diverse cancer-related data from different medical modalities, e.g., pathological images and genomic data. However, these works still face significant challenges. First, most approaches attempt multimodal fusion by simple one-shot fusion strategy, which is insufficient to explore complex interactions underlying in highly disparate multimodal data. Second, current methods for investigating multimodal interactions face the capability-efficiency dilemma, which is the difficult balance between powerful modeling capability and applicable computational efficiency, thus impeding effective multimodal fusion. In this study, to encounter these challenges, we propose an innovative multi-shot interactive fusion method named MIF for precise survival prediction by utilizing pathological and genomic data. Particularly, a novel multi-shot fusion framework is introduced to promote multimodal fusion by decomposing it into successive fusing stages, thus delicately integrating modalities in a progressive way. Moreover, to address the capacity-efficiency dilemma, various affinity-based interactive modules are introduced to synergize the multi-shot framework. Specifically, by harnessing comprehensive affinity information as guidance for mining interactions, the proposed interactive modules can efficiently generate low-dimensional discriminative multimodal representations. Extensive experiments on different cancer datasets unravel that our method not only successfully achieves state-of-the-art performance by performing effective multimodal fusion, but also possesses high computational efficiency compared to existing survival prediction methods.

Predicting interval from diagnosis to delivery in preeclampsia using electronic health records

Preeclampsia is a major cause of maternal and perinatal mortality with no known cure. Delivery timing is critical to balancing maternal and fetal risks. We develop and externally validate PEDeliveryTime, a class of clinically informative models which resulted from deep-learning models, to predict the time from PE diagnosis to delivery using electronic health records. We build the models on 1533 PE cases from the University of Michigan and validate it on 2172 preeclampsia cases from the University of Florida. PEDeliveryTime full model contains only 12 features yet achieves high c-index of 0.79 and 0.74 on the Michigan and Florida data set respectively. For the early-onset preeclampsia subset, the full model reaches 0.76 and 0.67 on the Michigan and Florida test sets. Collectively, these models perform an early assessment of delivery urgency and might help to better prioritize medical resources.

Cox-Sage: enhancing Cox proportional hazards model with interpretable graph neural networks for cancer prognosis

High-throughput sequencing technologies have facilitated a deeper exploration of prognostic biomarkers. While many deep learning (DL) methods primarily focus on feature extraction or employ simplistic fully connected layers within prognostic modules, the interpretability of DL-extracted features can be challenging. To address these challenges, we propose an interpretable cancer prognosis model called Cox-Sage. Specifically, we first propose an algorithm to construct a patient similarity graph from heterogeneous clinical data, and then extract protein-coding genes from the patient's gene expression data to embed them as features into the graph nodes. We utilize multilayer graph convolution to model proportional hazards pattern and introduce a mathematical method to clearly explain the meaning of our model's parameters. Based on this approach, we propose two metrics for measuring gene importance from different perspectives: mean hazard ratio and reciprocal of the mean hazard ratio. These metrics can be used to discover two types of important genes: genes whose low expression levels are associated with high cancer prognosis risk, and genes whose high expression levels are associated with high cancer prognosis risk. We conducted experiments on seven datasets from TCGA, and our model achieved superior prognostic performance compared with some state-of-the-art methods. As a primary research, we performed prognostic biomarker discovery on the LIHC (Liver Hepatocellular Carcinoma) dataset. Our code and dataset can be found at https://github.com/beeeginner/Cox-sage.

Characterization of Non-Monotonic Relationships between Tumor Mutational Burden and Clinical Outcomes

Potential clinical biomarkers are often assessed with Cox regressions or their ability to differentiate two groups of patients based on a single cutoff. However, both of these approaches assume a monotonic relationship between the potential biomarker and survival. Tumor mutational burden (TMB) is currently being studied as a predictive biomarker for immunotherapy, and a single cutoff is often used to divide patients. In this study, we introduce a two-cutoff approach that allows splitting of patients when a non-monotonic relationship is present and explore the use of neural networks to model more complex relationships of TMB to outcome data. Using real-world data, we find that while in most cases the true relationship between TMB and survival appears monotonic, that is not always the case and researchers should be made aware of this possibility.

SIGNIFICANCE

When a non-monotonic relationship to survival is present it is not possible to divide patients by a single value of a predictor. Neural networks allow for complex transformations and can be used to correctly split patients when a non-monotonic relationship is present.

Development of a deep learning model for predicting recurrence of hepatocellular carcinoma after liver transplantation

Background

Liver transplantation (LT) is one of the main curative treatments for hepatocellular carcinoma (HCC). Milan criteria has long been applied to candidate LT patients with HCC. However, the application of Milan criteria failed to precisely predict patients at risk of recurrence. As a result, we aimed to establish and validate a deep learning model comparing with Milan criteria and better guide post-LT treatment.

Methods

A total of 356 HCC patients who received LT with complete follow-up data were evaluated. The entire cohort was randomly divided into training set (n = 286) and validation set (n = 70). Multi-layer-perceptron model provided by pycox library was first used to construct the recurrence prediction model. Then tabular neural network (TabNet) that combines elements of deep learning and tabular data processing techniques was utilized to compare with Milan criteria and verify the performance of the model we proposed.

Results

Patients with larger tumor size over 7 cm, poorer differentiation of tumor grade and multiple tumor numbers were first classified as high risk of recurrence. We trained a classification model with TabNet and our proposed model performed better than the Milan criteria in terms of accuracy (0.95 vs. 0.86, p < 0.05). In addition, our model showed better performance results with improved AUC, NRI and hazard ratio, proving the robustness of the model.

Conclusion

A prognostic model had been proposed based on the use of TabNet on various parameters from HCC patients. The model performed well in post-LT recurrence prediction and the identification of high-risk subgroups.

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.

Deep learning and transfer learning identify breast cancer survival subtypes from single-cell imaging data

BACKGROUND

Single-cell multiplex imaging data have provided new insights into disease subtypes and prognoses recently. However, quantitative models that explicitly capture single-cell resolution cell-cell interaction features to predict patient survival at a population scale are currently missing.

METHODS

We quantified hundreds of single-cell resolution cell-cell interaction features through neighborhood calculation, in addition to cellular phenotypes. We applied these features to a neural-network-based Cox-nnet survival model to identify survival-associated features. We used non-negative matrix factorization (NMF) to identify patient survival subtypes. We identified atypical subpopulations of triple-negative breast cancer (TNBC) patients with moderate prognosis and Luminal A patients with poor prognosis and validated these subpopulations by label transferring using the UNION-COM method.

RESULTS

The neural-network-based Cox-nnet survival model using all cellular phenotype and cell-cell interaction features is highly predictive of patient survival in the test data (Concordance Index > 0.8). We identify seven survival subtypes using the top survival features, presenting distinct profiles of epithelial, immune, and fibroblast cells and their interactions. We reveal atypical subpopulations of TNBC patients with moderate prognosis (marked by GATA3 over-expression) and Luminal A patients with poor prognosis (marked by KRT6 and ACTA2 over-expression and CDH1 under-expression). These atypical subpopulations are validated in TCGA-BRCA and METABRIC datasets.

CONCLUSIONS

This work provides an approach to bridge single-cell level information toward population-level survival prediction.

Deep-learning and transfer learning identify new breast cancer survival subtypes from single-cell imaging data

Quantitative models that explicitly capture single-cell resolution cell-cell interaction features to predict patient survival at population scale are currently missing. Here, we computationally extracted hundreds of features describing single-cell based cell-cell interactions and cellular phenotypes from a large, published cohort of cyto-images of breast cancer patients. We applied these features to a neural-network based Cox-nnet survival model and obtained high accuracy in predicting patient survival in test data (Concordance Index > 0.8). We identified seven survival subtypes using the top survival features, which present distinct profiles of epithelial, immune, fibroblast cells, and their interactions. We identified atypical subpopulations of TNBC patients with moderate prognosis (marked by GATA3 over-expression) and Luminal A patients with poor prognosis (marked by KRT6 and ACTA2 over-expression and CDH1 under-expression). These atypical subpopulations are validated in TCGA-BRCA and METABRIC datasets. This work provides important guidelines on bridging single-cell level information towards population-level survival prediction.

STATEMENT OF TRANSLATIONAL RELEVANCE

Our findings from a breast cancer population cohort demonstrate the clinical utility of using the single-cell level imaging mass cytometry (IMC) data as a new type of patient prognosis prediction marker. Not only did the prognosis prediction achieve high accuracy with a Concordance index score greater than 0.8, it also enabled the discovery of seven survival subtypes that are more distinguishable than the molecular subtypes. These new subtypes present distinct profiles of epithelial, immune, fibroblast cells, and their interactions. Most importantly, this study identified and validated atypical subpopulations of TNBC patients with moderate prognosis (GATA3 over-expression) and Luminal A patients with poor prognosis (KRT6 and ACTA2 over-expression and CDH1 under-expression), using multiple large breast cancer cohorts.

Multimodal data fusion for cancer biomarker discovery with deep learning

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Multimodal deep learning to predict prognosis in adult and pediatric brain tumors

BACKGROUND

The introduction of deep learning in both imaging and genomics has significantly advanced the analysis of biomedical data. For complex diseases such as cancer, different data modalities may reveal different disease characteristics, and the integration of imaging with genomic data has the potential to unravel additional information than when using these data sources in isolation. Here, we propose a DL framework that combines these two modalities with the aim to predict brain tumor prognosis.

METHODS

Using two separate glioma cohorts of 783 adults and 305 pediatric patients we developed a DL framework that can fuse histopathology images with gene expression profiles. Three strategies for data fusion were implemented and compared: early, late, and joint fusion. Additional validation of the adult glioma models was done on an independent cohort of 97 adult patients.

RESULTS

Here we show that the developed multimodal data models achieve better prediction results compared to the single data models, but also lead to the identification of more relevant biological pathways. When testing our adult models on a third brain tumor dataset, we show our multimodal framework is able to generalize and performs better on new data from different cohorts. Leveraging the concept of transfer learning, we demonstrate how our pediatric multimodal models can be used to predict prognosis for two more rare (less available samples) pediatric brain tumors.

CONCLUSIONS

Our study illustrates that a multimodal data fusion approach can be successfully implemented and customized to model clinical outcome of adult and pediatric brain tumors.

Deep learning-based morphological feature analysis and the prognostic association study in colon adenocarcinoma histopathological images

Colorectal cancer (CRC) is now the third most common malignancy to cause mortality worldwide, and its prognosis is of great importance. Recent CRC prognostic prediction studies mainly focused on biomarkers, radiometric images, and end-to-end deep learning methods, while only a few works paid attention to exploring the relationship between the quantitative morphological features of patients' tissue slides and their prognosis. However, existing few works in this area suffered from the drawback of choosing the cells randomly from the whole slides, which contain the non-tumor region that lakes information about prognosis. In addition, the existing works, which tried to demonstrate their biological interpretability using patients' transcriptome data, failed to show the biological meaning closely related to cancer. In this study, we proposed and evaluated a prognostic model using morphological features of cells in the tumor region. The features were first extracted by the software CellProfiler from the tumor region selected by Eff-Unet deep learning model. Features from different regions were then averaged for each patient as their representative, and the Lasso-Cox model was used to select the prognosis-related features. The prognostic prediction model was at last constructed using the selected prognosis-related features and was evaluated through KM estimate and cross-validation. In terms of biological meaning, Gene Ontology (GO) enrichment analysis of the expressed genes that correlated with the prognostically significant features was performed to show the biological interpretability of our model.With the help of tumor segmentation, our model achieved better statistical significance and better biological interpretability compared to the results without tumor segmentation. Statistically, the Kaplan Meier (KM) estimate of our model showed that the model using features in the tumor region has a higher C-index, a lower p-value, and a better performance on cross-validation than the model without tumor segmentation. In addition, revealing the pathway of the immune escape and the spread of the tumor, the model with tumor segmentation demonstrated a biological meaning much more related to cancer immunobiology than the model without tumor segmentation. Our prognostic prediction model using quantitive morphological features from tumor regions was almost as good as the TNM tumor staging system as they had a close C-index, and our model can be combined with the TNM tumor stage system to make a better prognostic prediction. And to the best of our knowledge, the biological mechanisms in our study were the most relevant to the immune mechanism of cancer compared to the previous studies.

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

MOTIVATION

Accurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances in survival prediction methods suggest that integrating complementary information from different modalities, e.g. histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause the so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning and therefore pose a great challenge to further improvements of cancer survival prediction.

RESULTS

To solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data.

AVAILABILITY AND IMPLEMENTATION

CAMR is freely available at https://github.com/wxq-ustc/CAMR.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.