Multimodal deep learning for biomedical data fusion: a review

Label-free urinary protein detection through machine learning analysis of single droplet evaporation patterns

BACKGROUND

Chronic kidney disease (CKD) is a major global public health issue, with a steadily increasing incidence. Urinary protein detection serves as a crucial indicator for the diagnosis, monitoring and management of CKD. However, current methods for urinary protein measurement, such as urine dipstick tests, colorimetric assays, and 24-h total urine protein analysis, have certain limitations that restrict their routine application in CKD screening and follow-up. Therefore, there is an urgent need for a simple, convenient, and rapid diagnostic approach for kidney function assessment.

RESULTS

In this study, we have developed and validated a novel method for urine protein quantification based on dried droplet morphology analysis. Our approach demonstrates robust performance across a wide range of protein concentrations and is resilient to common interfering substances and variations in sample processing. It's worth noting that while our method shows excellent agreement with the colorimetric assay, it offers several potential advantages. These include reduced sample volume requirements, simplified sample preparation, and rapid analysis time. These factors could make our method particularly suitable for point-of-care testing or resource-limited settings where traditional laboratory infrastructure may be unavailable.

SIGNIFICANCE

The novel method for protein quantification in urine based on the morphology of a dried droplet uses only one drop of urine specimen. Combined with machine learning models, by identifying protein content in urine droplet drying patterns without the need for staining or antibody binding, may provide a more convenient alternative to current techniques for the assessment of proteinuria. Simple, low-cost, and fast, the system can be used as a powerful tool for CKD surveillance at the point of care.

Innovative multistep and synchronous soft sensing prediction of COD and NH3 in WWTPs via multimodal data and multiple attention mechanisms

Accurate prediction of Chemical Oxygen Demand (COD) and ammonia nitrogen (NH₃) is crucial for maintaining stable and effective wastewater treatment processes. Traditional methods rely on costly, high-maintenance sensors, limiting their application in resource-limited wastewater treatment plants. Soft sensing methods provide an alternative by reducing dependence on costly sensors. However, existing approaches cannot perform multitarget and multistep predictions, limiting their practical applicability. This study introduced a novel triple attention-enhanced encoder-decoder temporal convolutional network (TAED-TCN) to address this problem. The model used multimodal inputs, including easily accessible water quality parameters and wastewater surface images, for multistep and synchronous prediction of COD and NH₃. When it was validated with real-world sequencing batch reactor wastewater data, the model demonstrated superior multistep prediction performance. Specifically, the R² for 1-h predictions of COD and NH₃ was over 26.03 % and 20.51 % higher than the baseline model, respectively. By incorporating multiple attention mechanisms (feature, temporal, and cross-attention), TAED-TCN effectively captured essential features, model nonlinear relationships, and identified long-term dependencies, thus enabled consistent multitarget prediction results even under abnormal conditions. Additionally, economic analysis revealed that TAED-TCN could reduce COD and NH₃ monitoring costs by 79 % over the equipment life cycle. This study offers a cost-effective solution for water quality prediction, enhancing the operational efficiency of wastewater management.

Multimodal fusion architectures for Alzheimer's disease diagnosis: An experimental study

OBJECTIVE

In the attempt of early diagnosis of Alzheimer's Disease, varying forms of medical records of multiple modalities are gathered to seize the interaction of multiple factors. However, the heterogeneity of multimodal data brings a challenge. Hence, the role of artificial intelligence comes into play to provide the medical practitioner assistance in making diagnosis and prognosis. In order to be adopted as a clinical decision support system, interpretable or explainable model is important for healthcare professionals to trust the results. This study assessed various popular machine learning models under two multimodal fusion architectures to find the best combination in terms of both predictive performance and interpretability.

METHODS

Two architectures, early and late, also known as feature- and decision-level fusion were chosen for multinomial classification task. On top of the commonly used simple concatenation, this study employed weighted and hybrid weighted concatenation to fuse features within and across modalities under the two fusion structures. To test the efficacy of each model pipeline, the assessment was done according to their distinct foundations on which the models were built and each of their advantages was recognized. Classification metrics were unified and visualized into a pentagon to compare the overall performance of each pipeline. In addition, interpretability analysis was provided to quantify the importance of each modality and feature recognized by each model.

RESULTS

The potential characteristics of each type of pipelines in terms of prediction accuracy and ability to capture the relevant markers of each cognitive state were uncovered. In this particular healthcare application, the tree-based and linear models were the top 2 choices. Coupled with early and late fusion structure with weighted concatenation, reaching the balanced accuracy of 0.920 and 0.912, consecutively. The top 5 most important features revealed belong to Cognitive Test Scores and Neuropsychological Battery of Test modalities.

CONCLUSION

This work contributes as medical applications of artificial intelligence evaluation to aid practitioners in understanding the capability of different fusion architectures with different classifiers in getting to know the use of machine learning in clinical setting. With accurate classification, early detection of Mild Cognitive Impairment and Alzheimer's Disease can be achieved.

Multimodal malware classification using proposed ensemble deep neural network framework

In the contemporary technological world, fortifying cybersecurity defense against dynamic threat landscapes is imperative. Malware detectors play a critical role in this endeavor, utilizing various techniques such as statistical analysis, static and dynamic analysis, and machine learning (ML) to compare signatures and identify threats. Deep learning (DL) aids in accurately classifying complex malware features. The cross-domain research in data fusion strives to integrate information from multiple sources to augment reliability and minimize errors in detecting sophisticated cyber threats. This collaborative approach is the least addressed and pivotal for protecting against the advancing environment of modern malware attacks. This study presents a state-of-the-art malware analysis framework that employs a multimodal approach by integrating malware images and numeric features for effective malware classification. The experiments are performed sequentially, encompassing data preprocessing, feature selection using Neighbourhood Component Analysis (NCA), and dataset balancing with Synthetic Minority Over-sampling Technique (SMOTE). Subsequently, the late fusion technique is utilized for multimodal classification by employing Random Under Sampling and Boosting (RUSBoost) and the proposed ensemble deep neural network. The RUSBoost technique involves random undersampling and adaptive boosting to moderate bias toward majority classes while improving minority class (malware) detection. Multimodal Late fusion experimental results (95.36%) of RUSBoost (numeric) and the proposed model (imagery) outperform the standalone prevailing results for imagery (95.02%) and numeric (93.36%) data. The effectiveness of the proposed model is verified through the evaluation metrics such as Recall (86.5%), F1-score (85.0%), and Precision (79.9%). The multimodal late fusion of numeric and visual data makes the model more robust in detecting diverse malware variants. The experimental outcomes demonstrate that multimodal analysis may efficiently increase the identification strength and accuracy, especially when majority vote and bagging are employed for late fusion.

Multiple deep learning models based on MRI images in discriminating glioblastoma from solitary brain metastases: a multicentre study

OBJECTIVE

Development of a deep learning model for accurate preoperative identification of glioblastoma and solitary brain metastases by combining multi-centre and multi-sequence magnetic resonance images and comparison of the performance of different deep learning models.

METHODS

Clinical data and MR images of a total of 236 patients with pathologically confirmed glioblastoma and single brain metastases were retrospectively collected from January 2019 to May 2024 at Provincial Hospital of Shandong First Medical University, and the data were randomly divided into a training set and a test set according to the ratio of 8:2, in which the training set contained 197 cases and the test set contained 39 cases; the images were preprocessed and labeled with the tumor regions. The images were pre-processed and labeled with tumor regions, and different MRI sequences were input individually or in combination to train the deep learning model 3D ResNet-18, and the optimal sequence combinations were obtained by five-fold cross-validation enhancement of the data inputs and training of the deep learning models 3D Vision Transformer (3D Vit), 3D DenseNet, and 3D VGG; the working characteristic curves (ROCs) of subjects were plotted, and the area under the curve (AUC) was calculated. The area under the curve (AUC), accuracy, precision, recall and F1 score were used to evaluate the discriminative performance of the models. In addition, 48 patients with glioblastoma and single brain metastases from January 2020 to December 2022 were collected from the Affiliated Cancer Hospital of Shandong First Medical University as an external test set to compare the discriminative performance, robustness and generalization ability of the four deep learning models.

RESULTS

In the comparison of the discriminative effect of different MRI sequences, the three sequence combinations of T1-CE, T2, and T2-Flair gained discriminative effect, with the accuracy and AUC values of 0.8718 and 0.9305, respectively; after the four deep learning models were inputted into the aforementioned sequence combinations, the accuracy and AUC of the external validation of the 3D ResNet-18 model were 0.8125, respectively, 0.8899, all of which are the highest among all models.

CONCLUSIONS

A combination of multi-sequence MR images and a deep learning model can efficiently identify glioblastoma and solitary brain metastases preoperatively, and the deep learning model 3D ResNet-18 has the highest efficacy in identifying the two types of tumours.

Development of disease diagnosis technology based on coattention cross-fusion of multiomics data

BACKGROUND

Early diagnosis is vital for increasing the rates of curing diseases and patient survival in medicine. With the advancement of biotechnology, the types of bioomics data are increasing. The integration of multiomics data can provide more comprehensive biological information, thereby achieving more accurate diagnoses than single-omics data can. Nevertheless, current multiomics research is often limited to the intelligent diagnosis of a single disease or a few types of omics data and lacks a multiomics disease diagnosis model that can be widely applied to different diseases. Therefore, developing a model that can effectively utilize multiomics data and accurately diagnose diseases has become an important challenge in medical research.

RESULTS

On the basis of vibrational spectroscopy and metabolomics data, this study proposes an innovative coattention cross-fusion model for disease diagnosis on the basis of interactions of multiomics data. The model not only integrates the information of different omics data but also simulates the interactions between these data to achieve accurate diagnosis of diseases. Through comprehensive experiments, our method achieved accuracies of 95.00 %, 94.95 %, and 97.22 % and area under the curve (AUC) values of 95.00 %, 96.77 %, and 99.31 % on the cervical lymph node metastasis of the thyroid, systemic lupus erythematosus, and cancer datasets, respectively, indicating excellent performance in the diagnosis of multiple diseases.

SIGNIFICANCE

The proposed model outperforms existing multiomics models, enhancing medical diagnostic accuracy and offering new approaches for multiomics data use in disease diagnosis. The innovative coattention cross-fusion module enables more effective multiomics data processing and analysis, serving as a potent tool for early and precise disease diagnosis with substantial clinical and research implications.

Value of Bioinformatics Models for Predicting Translational Control of Angiogenesis

Angiogenesis, the formation of new blood vessels, is a fundamental biological process with implications for both physiological functions and pathological conditions. While the transcriptional regulation of angiogenesis, mediated by factors such as HIF-1α (hypoxia-inducible factor 1-alpha) and VEGF (vascular endothelial growth factor), is well-characterized, the translational regulation of this process remains underexplored. Bioinformatics has emerged as an indispensable tool for advancing our understanding of translational regulation, offering predictive models that leverage large data sets to guide research and optimize experimental approaches. However, a significant gap persists between bioinformatics experts and other researchers, limiting the accessibility and utility of these tools in the broader scientific community. To address this divide, user-friendly bioinformatics platforms are being developed to democratize access to predictive analytics and empower researchers across disciplines. Translational control, compared with transcriptional control, offers a more energy-efficient mechanism that facilitates rapid cellular responses to environmental changes. Furthermore, transcriptional regulators themselves are often subject to translational control, emphasizing the interconnected nature of these regulatory layers. Investigating translational regulation requires advanced, accessible bioinformatics tools to analyze RNA structures, interacting micro-RNAs, long noncoding RNAs, and RBPs (RNA-binding proteins). Predictive platforms such as RNA structure, human internal ribosome entry site Atlas, and RBPSuite enable the study of RNA motifs and RNA-protein interactions, shedding light on these critical regulatory mechanisms. This review highlights the transformative role of bioinformatics using widely accessible user-friendly tools with a Web-browser interface to elucidate translational regulation in angiogenesis. The bioinformatics tools discussed extend beyond angiogenesis, with applications in diverse fields, including clinical care. By integrating predictive models and experimental insights, researchers can streamline hypothesis generation, reduce experimental costs, and find novel translational regulators. By bridging the bioinformatics knowledge gap, this review aims to empower researchers worldwide to adopt bioinformatics tools in their work, fostering innovation and accelerating scientific discovery.

A machine learning approach for multimodal data fusion for survival prediction in cancer patients

Technological advancements of the past decade have transformed cancer research, improving patient survival predictions through genotyping and multimodal data analysis. However, there is no comprehensive machine-learning pipeline for comparing methods to enhance these predictions. To address this, a versatile pipeline using The Cancer Genome Atlas (TCGA) data was developed, incorporating various data modalities such as transcripts, proteins, metabolites, and clinical factors. This approach manages challenges like high dimensionality, small sample sizes, and data heterogeneity. By applying different feature extraction and fusion strategies, notably late fusion models, the effectiveness of integrating diverse data types was demonstrated. Late fusion models consistently outperformed single-modality approaches in TCGA lung, breast, and pan-cancer datasets, offering higher accuracy and robustness. This research highlights the potential of comprehensive multimodal data integration in precision oncology to improve survival predictions for cancer patients. The study provides a reusable pipeline for the research community, suggesting future work on larger cohorts.

Cross-modal alignment and contrastive learning for enhanced cancer survival prediction

BACKGROUND AND OBJECTIVE

Integrating multimodal data, such as pathology images and genomics, is crucial for understanding cancer heterogeneity, personalized treatment complexity, and enhancing survival prediction. However, most current prognostic methods are limited to a single domain of histopathology or genomics, inevitably reducing their potential for accurate patient outcome prediction. Despite advancements in the concurrent analysis of pathology and genomic data, existing approaches inadequately address the intricate intermodal relationships.

METHODS

This paper introduces the CPathomic method for multimodal data-based survival prediction. By leveraging whole slide pathology images to guide local pathological features, the method effectively mitigates significant intermodal differences through a cross-modal representational contrastive learning module. Furthermore, it facilitates interactive learning between different modalities through cross-modal and gated attention modules.

RESULTS

The extensive experiments on five public TCGA datasets demonstrate that CPathomic framework effectively bridges modality gaps, consistently outperforming alternative multimodal survival prediction methods.

CONCLUSION

The model we propose, CPathomic, unveils the potential of contrastive learning and cross-modal attention in the representation and fusion of multimodal data, enhancing the performance of patient survival prediction.

MSFT-transformer: a multistage fusion tabular transformer for disease prediction using metagenomic data

More and more recent studies highlight the crucial role of the human microbiome in maintaining health, while modern advancements in metagenomic sequencing technologies have been accumulating data that are associated with human diseases. Although metagenomic data offer rich, multifaceted information, including taxonomic and functional abundance profiles, their full potential remains underutilized, as most approaches rely only on one type of information to discover and understand their related correlations with respect to disease occurrences. To address this limitation, we propose a multistage fusion tabular transformer architecture (MSFT-Transformer), aiming to effectively integrate various types of high-dimensional tabular information extracted from metagenomic data. Its multistage fusion strategy consists of three modules: a fusion-aware feature extraction module in the early stage to improve the extracted information from inputs, an alignment-enhanced fusion module in the mid stage to enforce the retainment of desired information in cross-modal learning, and an integrated feature decision layer in the late stage to incorporate desired cross-modal information. We conduct extensive experiments to evaluate the performance of MSFT-Transformer over state-of-the-art models on five standard datasets. Our results indicate that MSFT-Transformer provides stable performance gains with reduced computational costs. An ablation study illustrates the contributions of all three models compared with a reference multistage fusion transformer without these novel strategies. The result analysis implies the significant potential of the proposed model in future disease prediction with metagenomic data.

Neurosymbolic AI for Reasoning Over Knowledge Graphs: A Survey

Neurosymbolic artificial intelligence (AI) is an increasingly active area of research that combines symbolic reasoning methods with deep learning to leverage their complementary benefits. As knowledge graphs (KGs) are becoming a popular way to represent heterogeneous and multirelational data, methods for reasoning on graph structures have attempted to follow this neurosymbolic paradigm. Traditionally, such approaches have utilized either rule-based inference or generated representative numerical embeddings from which patterns could be extracted. However, several recent studies have attempted to bridge this dichotomy to generate models that facilitate interpretability, maintain competitive performance, and integrate expert knowledge. Therefore, we survey methods that perform neurosymbolic reasoning tasks on KGs and propose a novel taxonomy by which we can classify them. Specifically, we propose three major categories: 1) logically informed embedding approaches; 2) embedding approaches with logical constraints; and 3) rule-learning approaches. Alongside the taxonomy, we provide a tabular overview of the approaches and links to their source code, if available, for more direct comparison. Finally, we discuss the unique characteristics and limitations of these methods and then propose several prospective directions toward which this field of research could evolve.

Multimodal Deep Learning Fusing Clinical and Radiomics Scores for Prediction of Early-Stage Lung Adenocarcinoma Lymph Node Metastasis

RATIONALE AND OBJECTIVES

To develop and validate a multimodal deep learning (DL) model based on computed tomography (CT) images and clinical knowledge to predict lymph node metastasis (LNM) in early lung adenocarcinoma.

MATERIALS AND METHODS

A total of 724 pathologically confirmed early invasive lung adenocarcinoma patients were retrospectively included from two centers. Clinical and CT semantic features of the patients were collected, and 3D radiomics features were extracted from nonenhanced CT images. We proposed a multimodal feature fusion DL network based on the InceptionResNetV2 architecture, which can effectively extract and integrate image and clinical knowledge to predict LNM.

RESULTS

A total of 524 lung adenocarcinoma patients from Center 1 were randomly divided into training (n=418) and internal validation (n=106) sets in a 4:1 ratio, while 200 lung adenocarcinoma patients from Center 2 served as the independent test set. Among the 16 collected clinical and imaging features, 8 were selected: gender, serum carcinoembryonic antigen, cytokeratin 19 fragment antigen 21-1, neuron-specific enolase, tumor size, location, density, and centrality. From the 1595 extracted radiomics features, six key features were identified. The CS-RS-DL fusion model achieved the highest area under the receiver operating characteristic curve in both the internal validation set (0.877) and the independent test set (0.906) compared to other models. The Delong test results for the independent test set indicated that the CS-RS-DL model significantly outperformed the clinical model (0.844), radiomics model (0.850), CS-RS model (0.872), single DL model (0.848), and the CS-DL model (0.875) (all P<0.05). Additionally, the CS-RS-DL model exhibited the highest sensitivity (0.941) and average precision (0.642).

CONCLUSION

The knowledge derived from clinical, radiomics, and DL is complementary in predicting LNM in lung adenocarcinoma. The integration of clinical and radiomics scores through DL can significantly improve the accuracy of lymph node status assessment.

Advancements in Medical Radiology Through Multimodal Machine Learning: A Comprehensive Overview

The majority of data collected and obtained from various sources over a patient's lifetime can be assumed to comprise pertinent information for delivering the best possible treatment. Medical data, such as radiographic and histopathology images, electrocardiograms, and medical records, all guide a physician's diagnostic approach. Nevertheless, most machine learning techniques in the healthcare field emphasize data analysis from a single modality, which is insufficiently reliable. This is especially evident in radiology, which has long been an essential topic of machine learning in healthcare because of its high data density, availability, and interpretation capability. In the future, computer-assisted diagnostic systems must be intelligent to process a variety of data simultaneously, similar to how doctors examine various resources while diagnosing patients. By extracting novel characteristics from diverse medical data sources, advanced identification techniques known as multimodal learning may be applied, enabling algorithms to analyze data from various sources and eliminating the need to train each modality. This approach enhances the flexibility of algorithms by incorporating diverse data. A growing quantity of current research has focused on the exploration of extracting data from multiple sources and constructing precise multimodal machine/deep learning models for medical examinations. A comprehensive analysis and synthesis of recent publications focusing on multimodal machine learning in detecting diseases is provided. Potential future research directions are also identified. This review presents an overview of multimodal machine learning (MMML) in radiology, a field at the cutting edge of integrating artificial intelligence into medical imaging. As radiological practices continue to evolve, the combination of various imaging and non-imaging data modalities is gaining increasing significance. This paper analyzes current methodologies, applications, and trends in MMML while outlining challenges and predicting upcoming research directions. Beginning with an overview of the different data modalities involved in radiology, namely, imaging, text, and structured medical data, this review explains the processes of modality fusion, representation learning, and modality translation, showing how they boost diagnosis efficacy and improve patient care. Additionally, this review discusses key datasets that have been instrumental in advancing MMML research. This review may help clinicians and researchers comprehend the spatial distribution of the field, outline the current level of advancement, and identify areas of research that need to be explored regarding MMML in radiology.

Anatomy-guided slice-description interaction for multimodal brain disease diagnosis based on 3D image and radiological report

Accurate brain disease diagnosis based on radiological images is desired in clinical practice as it can facilitate early intervention and reduce the risk of damage. However, existing unimodal image-based models struggle to process high-dimensional 3D brain imaging data effectively. Multimodal disease diagnosis approaches based on medical images and corresponding radiological reports achieved promising progress with the development of vision-language models. However, most multimodal methods handle 2D images and cannot be directly applied to brain disease diagnosis that uses 3D images. Therefore, in this work we develop a multimodal brain disease diagnosis model that takes 3D brain images and their radiological reports as input. Motivated by the fact that radiologists scroll through image slices and write important descriptions into the report accordingly, we propose a slice-description cross-modality interaction mechanism to realize fine-grained multimodal data interaction. Moreover, since previous medical research has demonstrated potential correlation between anatomical location of anomalies and diagnosis results, we further explore the use of brain anatomical prior knowledge to improve the multimodal interaction. Based on the report description, the prior knowledge filters the image information by suppressing irrelevant regions and enhancing relevant slices. Our method was validated with two brain disease diagnosis tasks. The results indicate that our model outperforms competing unimodal and multimodal methods for brain disease diagnosis. In particular, it has yielded an average accuracy improvement of 15.87% and 7.39% compared with the image-based and multimodal competing methods, respectively.

Integrating Machine Learning and Follow-Up Variables to Improve Early Detection of Hepatocellular Carcinoma in Tyrosinemia Type 1: A Multicenter Study

Hepatocellular carcinoma (HCC) is a major complication of tyrosinemia type 1 (HT-1), an inborn error of metabolism affecting tyrosine catabolism. The risk of HCC is higher in late diagnoses despite treatment. Alpha-fetoprotein (AFP) is widely used to detect liver cancer but has limitations in early-stage HCC detection. This study aimed to implement a machine-learning (ML) approach to identify the most relevant laboratory variables to predict AFP alteration using constrained multidimensional data from Chilean and Italian HT-1 cohorts. A longitudinal retrospective study analyzed 219 records from 35 HT-1 patients, including 8 with HCC and 5 diagnosed through newborn screening. The dataset contained biochemical and demographic variables that were analyzed using the eXtreme Gradient Boosting algorithm, which was trained to predict abnormal AFP levels (>5 ng/mL). Four key variables emerged as significant predictors: alanine transaminase (ALT), alkaline phosphatase, age at diagnosis, and current age. ALT emerged as the most promising indicator of AFP alteration, potentially preceding AFP level changes and improving HCC detection specificity at a cut-off value of 29 UI/L (AUROC = 0.73). Despite limited data from this rare disease, the ML approach successfully analyzed follow-up biomarkers, identifying ALT as an early predictor of AFP elevation and a potential biomarker for HCC progression.

Multicenter Development and Validation of a Multimodal Deep Learning Model to Predict Moderate to Severe AKI

Key Points

We developed and validated a multimodal (structured and unstructured data) model to predict moderate to severe AKI using multicenter data. This multimodal AKI risk score accurately identifies patients who will develop stage 2 AKI over 2 days earlier than serum creatinine alone. The multimodal model performed better than a model based solely on structured data and performed similarly during temporal and site-based validation.

Background

Prior models for the early identification of AKI have used structured data (e.g ., vital signs and laboratory values). We aimed to develop and validate a deep learning model to predict moderate to severe AKI by combining structured data and information from unstructured notes.

Methods

Adults (18 years or older) admitted to the University of Wisconsin (2009–2020) and the University of Chicago Medicine (2016–2022) were eligible for inclusion. Patients were excluded if they had no documented serum creatinine (SCr), ESKD, an admission SCr ≥3.0 mg/dl, developed ≥stage 2 AKI before reaching the wards or intensive care unit, or required dialysis (KRT) within the first 48 hours. Text from unstructured notes was mapped to standardized concept unique identifiers to create predictor variables, and structured data variables were also included. An intermediate fusion deep learning recurrent neural network architecture was used to predict ≥stage 2 AKI within the next 48 hours. This multimodal model was developed in the first 80% of the data and temporally validated in the next 20%.

Results

There were 339,998 admissions in the derivation cohort and 84,581 in the validation cohort, with 12,748 (3%) developing ≥stage 2 AKI. Patients with ≥stage 2 AKI were older, more likely to be male, had higher baseline SCr, and were more commonly in the intensive care unit (P < 0.001 for all). The multimodal model outperformed a model based only on structured data for all outcomes, with an area under the receiver operating characteristic curve (95% confidence interval) of 0.88 (0.88 to 0.88) for predicting ≥stage 2 AKI and 0.93 (0.93 to 0.94) for receiving KRT. The area under the precision-recall-curve for ≥stage 2 AKI was 0.20. The results were similar during external validation.

Conclusions

We developed and validated a multimodal deep learning model using structured and unstructured data that predicts the development of severe AKI across the hospital stay for earlier intervention.

Recent Advances in Artificial Intelligence for Precision Diagnosis and Treatment of Bladder Cancer: A Review

BACKGROUND

Bladder cancer is one of the top ten cancers globally, with its incidence steadily rising in China. Early detection and prognosis risk assessment play a crucial role in guiding subsequent treatment decisions for bladder cancer. However, traditional diagnostic methods such as bladder endoscopy, imaging, or pathology examinations heavily rely on the clinical expertise and experience of clinicians, exhibiting subjectivity and poor reproducibility.

MATERIALS AND METHODS

With the rise of artificial intelligence, novel approaches, particularly those employing deep learning technology, have shown significant advancements in clinical tasks related to bladder cancer, including tumor detection, molecular subtyping identification, tumor staging and grading, prognosis prediction, and recurrence assessment.

RESULTS

Artificial intelligence, with its robust data mining capabilities, enhances diagnostic efficiency and reproducibility when assisting clinicians in decision-making, thereby reducing the risks of misdiagnosis and underdiagnosis. This not only helps alleviate the current challenges of talent shortages and uneven distribution of medical resources but also fosters the development of precision medicine.

CONCLUSIONS

This study provides a comprehensive review of the latest research advances and prospects of artificial intelligence technology in the precise diagnosis and treatment of bladder cancer.

Application of artificial intelligence in the diagnosis of malignant digestive tract tumors: focusing on opportunities and challenges in endoscopy and pathology

BACKGROUND

Malignant digestive tract tumors are highly prevalent and fatal tumor types globally, often diagnosed at advanced stages due to atypical early symptoms, causing patients to miss optimal treatment opportunities. Traditional endoscopic and pathological diagnostic processes are highly dependent on expert experience, facing problems such as high misdiagnosis rates and significant inter-observer variations. With the development of artificial intelligence (AI) technologies such as deep learning, real-time lesion detection with endoscopic assistance and automated pathological image analysis have shown potential in improving diagnostic accuracy and efficiency. However, relevant applications still face challenges including insufficient data standardization, inadequate interpretability, and weak clinical validation.

OBJECTIVE

This study aims to systematically review the current applications of artificial intelligence in diagnosing malignant digestive tract tumors, focusing on the progress and bottlenecks in two key areas: endoscopic examination and pathological diagnosis, and to provide feasible ideas and suggestions for subsequent research and clinical translation.

METHODS

A systematic literature search strategy was adopted to screen relevant studies published between 2017 and 2024 from databases including PubMed, Web of Science, Scopus, and IEEE Xplore, supplemented with searches of early classical literature. Inclusion criteria included studies on malignant digestive tract tumors such as esophageal cancer, gastric cancer, or colorectal cancer, involving the application of artificial intelligence technology in endoscopic diagnosis or pathological analysis. The effects and main limitations of AI diagnosis were summarized through comprehensive analysis of research design, algorithmic methods, and experimental results from relevant literature.

RESULTS

In the field of endoscopy, multiple deep learning models have significantly improved detection rates in real-time polyp detection, early gastric cancer, and esophageal cancer screening, with some commercialized systems successfully entering clinical trials. However, the scale and quality of data across different studies vary widely, and the generalizability of models to multi-center, multi-device environments remains to be verified. In pathological analysis, using convolutional neural networks, multimodal pre-training models, etc., automatic tissue segmentation, tumor grading, and assisted diagnosis can be achieved, showing good scalability in interactive question-answering. Nevertheless, clinical implementation still faces obstacles such as non-uniform data standards, lack of large-scale prospective validation, and insufficient model interpretability and continuous learning mechanisms.

CONCLUSION

Artificial intelligence provides new technological opportunities for endoscopic and pathological diagnosis of malignant digestive tract tumors, achieving positive results in early lesion identification and assisted decision-making. However, to achieve the transition from research to widespread clinical application, data standardization, model reliability, and interpretability still need to be improved through multi-center joint research, and a complete regulatory and ethical system needs to be established. In the future, artificial intelligence will play a more important role in the standardization and precision management of diagnosis and treatment of digestive tract tumors.

Novel pre-spatial data fusion deep learning approach for multimodal volumetric outcome prediction models in radiotherapy

BACKGROUND

Given the recent increased emphasis on multimodal neural networks to solve complex modeling tasks, the problem of outcome prediction for a course of treatment can be framed as fundamentally multimodal in nature. A patient's response to treatment will vary based on their specific anatomy and the proposed treatment plan-these factors are spatial and closely related. However, additional factors may also have importance, such as non-spatial descriptive clinical characteristics, which can be structured as tabular data. It is critical to provide models with as comprehensive of a patient representation as possible, but inputs with differing data structures are incompatible in raw form; traditional models that consider these inputs require feature engineering prior to modeling. In neural networks, feature engineering can be organically integrated into the model itself, under one governing optimization, rather than performed prescriptively beforehand. However, the native incompatibility of different data structures must be addressed. Methods to reconcile structural incompatibilities in multimodal model inputs are called data fusion. We present a novel joint early pre-spatial (JEPS) fusion technique and demonstrate that differences in fusion approach can produce significant model performance differences even when the data is identical.

PURPOSE

To present a novel pre-spatial fusion technique for volumetric neural networks and demonstrate its impact on model performance for pretreatment prediction of overall survival (OS).

METHODS

From a retrospective cohort of 531 head and neck patients treated at our clinic, we prepared an OS dataset of 222 data-complete cases at a 2-year post-treatment time threshold. Each patient's data included CT imaging, dose array, approved structure set, and a tabular summary of the patient's demographics and survey data. To establish single-modality baselines, we fit both a Cox Proportional Hazards model (CPH) and a dense neural network on only the tabular data, then we trained a 3D convolutional neural network (CNN) on only the volume data. Then, we trained five competing architectures for fusion of both modalities: two early fusion models, a late fusion model, a traditional joint fusion model, and the novel JEPS, where clinical data is merged into training upstream of most convolution operations. We used standardized 10-fold cross validation to directly compare the performance of all models on identical train/test splits of patients, using area under the receiver-operator curve (AUC) as the primary performance metric. We used a two-tailed Student t-test to assess the statistical significance (p-value threshold 0.05) of any observed performance differences.

RESULTS

The JEPS design scored the highest, achieving a mean AUC of 0.779 ± 0.080. The late fusion model and clinical-only CPH model scored second and third highest with 0.746 ± 0.066 and 0.720 ± 0.091 mean AUC, respectively. The performance differences between these three models were not statistically significant. All other comparison models scored significantly worse than the top performing JEPS model.

CONCLUSION

For our OS evaluation, our JEPS fusion architecture achieves better integration of inputs and significantly improves predictive performance over most common multimodal approaches. The JEPS fusion technique is easily applied to any volumetric CNN.

Domain-specific information preservation for Alzheimer's disease diagnosis with incomplete multi-modality neuroimages

Although multi-modality neuroimages have advanced the early diagnosis of Alzheimer's Disease (AD), missing modality issue still poses a unique challenge in the clinical practice. Recent studies have tried to impute the missing data so as to utilize all available subjects for training robust multi-modality models. However, these studies may overlook the modality-specific information inherent in multi-modality data, that is, different modalities possess distinct imaging characteristics and focus on different aspects of the disease. In this paper, we propose a domain-specific information preservation (DSIP) framework, consisting of modality imputation stage and status identification stage, for AD diagnosis with incomplete multi-modality neuroimages. In the first stage, a specificity-induced generative adversarial network (SIGAN) is developed to bridge the modality gap and capture modality-specific details for imputing high-quality neuroimages. In the second stage, a specificity-promoted diagnosis network (SPDN) is designed to promote the inter-modality feature interaction and the classifier robustness for identifying disease status accurately. Extensive experiments demonstrate the proposed method significantly outperforms state-of-the-art methods in both modality imputation and status identification tasks.

Multimodal data integration in early-stage breast cancer

The use of biomarkers in breast cancer has significantly improved patient outcomes through targeted therapies, such as hormone therapy anti-Her2 therapy and CDK4/6 or PARP inhibitors. However, existing knowledge does not fully encompass the diverse nature of breast cancer, particularly in triple-negative tumors. The integration of multi-omics and multimodal data has the potential to provide new insights into biological processes, to improve breast cancer patient stratification, enhance prognosis and response prediction, and identify new biomarkers. This review presents a comprehensive overview of the state-of-the-art multimodal (including molecular and image) data integration algorithms developed and with applicability to breast cancer stratification, prognosis, or biomarker identification. We examined the primary challenges and opportunities of these multimodal data integration algorithms, including their advantages, limitations, and critical considerations for future research. We aimed to describe models that are not only academically and preclinically relevant, but also applicable to clinical settings.

Missing-modality enabled multi-modal fusion architecture for medical data

BACKGROUND

Fusion of multi-modal data can improve the performance of deep learning models. However, missing modalities are common in medical data due to patient specificity, which is detrimental to the performance of multi-modal models in applications. Therefore, it is critical to adapt the models to missing modalities.

OBJECTIVE

This study aimed to develop an effective multi-modal fusion architecture for medical data that was robust to missing modalities and further improved the performance for clinical tasks.

METHODS

X-ray chest radiographs for the image modality, radiology reports for the text modality, and structured value data for the tabular data modality were fused in this study. Each modality pair was fused with a Transformer-based bi-modal fusion module, and the three bi-modal fusion modules were then combined into a tri-modal fusion framework. Additionally, multivariate loss functions were introduced into the training process to improve models' robustness to missing modalities during the inference process. Finally, we designed comparison and ablation experiments to validate the effectiveness of the fusion, the robustness to missing modalities, and the enhancements from each key component. Experiments were conducted on MIMIC-IV and MIMIC-CXR datasets with the 14-label disease diagnosis and patient in-hospital mortality prediction task The area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were used to evaluate models' performance.

RESULTS

Our proposed architecture showed superior predictive performance, achieving the average AUROC and AUPRC of 0.916 and 0.551 in the 14-label classification task, 0.816 and 0.392 in the mortality prediction task. while the best average AUROC and AUPRC among the comparison methods were 0.876, 0.492 in the 14-label classification task and 0.806, 0.366 in the mortality prediction task. Both metrics decreased only slightly when tested with modal-incomplete data. Different levels of enhancements were achieved through three key components.

CONCLUSIONS

The proposed multi-modal fusion architecture effectively fused three modalities and showed strong robustness to missing modalities. This architecture holds promise for scaling up to more modalities to enhance the clinical practicality of the model.

An Order-Sensitive Hierarchical Neural Model for Early Lung Cancer Detection Using Dutch Primary Care Notes and Structured Data

Background: Improving prediction models to timely detect lung cancer is paramount. Our aim is to develop and validate prediction models for early detection of lung cancer in primary care, based on free-text consultation notes, that exploit the order and context among words and sentences. Methods: Data of all patients enlisted in 49 general practices between 2002 and 2021 were assessed, and we included those older than 30 years with at least one free-text note. We developed two models using a hierarchical architecture that relies on attention and bidirectional long short-term memory networks. One model used only text, while the other combined text with clinical variables. The models were trained on data excluding the five months leading up to the diagnosis, using target replication and a tuning set, and were tested on a separate dataset for discrimination, PPV, and calibration. Results: A total of 250,021 patients were enlisted, with 1507 having a lung cancer diagnosis. Included in the analysis were 183,012 patients, of which 712 had the diagnosis. From the two models, the combined model showed slightly better performance, achieving an AUROC on the test set of 0.91, an AUPRC of 0.05, and a PPV of 0.034 (0.024, 0.043), and showed good calibration. To early detect one cancer patient, 29 high-risk patients would require additional diagnostic testing. Conclusions: Our models showed excellent discrimination by leveraging the word and sentence structure. Including clinical variables in addition to text slightly improved performance. The number needed to treat holds promise for clinical practice. Investigating external validation and model suitability in clinical practice is warranted.

A Map of the Lipid-Metabolite-Protein Network to Aid Multi-Omics Integration

The integration of multi-omics data offers transformative potential for elucidating complex molecular mechanisms underlying biological processes and diseases. In this study, we developed a lipid-metabolite-protein network that combines a protein-protein interaction network and enzymatic and genetic interactions of proteins with metabolites and lipids to provide a unified framework for multi-omics integration. Using hyperbolic embedding, the network visualizes connections across omics layers, accessible through a user-friendly Shiny R (version 1.10.0) software package. This framework ranks molecules across omics layers based on functional proximity, enabling intuitive exploration. Application in a cardiovascular disease (CVD) case study identified lipids and metabolites associated with CVD-related proteins. The analysis confirmed known associations, like cholesterol esters and sphingomyelin, and highlighted potential novel biomarkers, such as 4-imidazoleacetate and indoleacetaldehyde. Furthermore, we used the network to analyze empagliflozin's temporal effects on lipid metabolism. Functional enrichment analysis of proteins associated with lipid signatures revealed dynamic shifts in biological processes, with early effects impacting phospholipid metabolism and long-term effects affecting sphingolipid biosynthesis. Our framework offers a versatile tool for hypothesis generation, functional analysis, and biomarker discovery. By bridging molecular layers, this approach advances our understanding of disease mechanisms and therapeutic effects, with broad applications in computational biology and precision medicine.

Machine learning prediction of preterm birth in women under 35 using routine biomarkers in a retrospective cohort study

Preterm birth (PTB), defined as delivery before 37 weeks, affects 15 million infants annually, accounting for 11% of live births and over 35% of neonatal deaths. While advanced maternal age (≥ 35 years) is a known risk factor, PTB risk in women under 35 is underexplored. This study aimed to develop a machine learning-based model for PTB prediction in women under 35. A retrospective cohort of 2606 cases (2019-2022) equally split between full-term and preterm births was analyzed. Logistic Regression, LightGBM, Gradient Boosting Decision Tree (GBDT), and XGBoost models were evaluated. External validation was conducted using 803 independent cases (2023). Model performance was assessed using area under the curve (AUC), accuracy, sensitivity, and specificity. SHAP (SHapley Additive exPlanations) values were used to interpret model predictions. The XGBoost model demonstrated superior performance with an AUC of 0.893 (95% CI: 0.860-0.925) on the validation set. In comparison, Logistic Regression, LightGBM, and GBDT achieved AUCs of 0.872, 0.840, and 0.879, respectively. External validation of the XGBoost model yielded an AUC of 0.91 (95% CI: 0.889-0.931). SHAP analysis highlighted seven key predictors: alkaline phosphatase (ALP), alpha-fetoprotein (AFP), hemoglobin (HGB), urea (UREA), lymphocyte count (Lym1), sodium (Na), and red cell distribution width coefficient of variation (RDWCV). The XGBoost model provides accurate PTB risk prediction and key insights for early intervention in women under 35, supporting its potential clinical utility.

The impact of multi-modality fusion and deep learning on adult age estimation based on bone mineral density

INTRODUCTION

Age estimation, especially in adults, presents substantial challenges in different contexts ranging from forensic to clinical applications. Bone mineral density (BMD), with its distinct age-related variations, has emerged as a critical marker in this domain. This study aims to enhance chronological age estimation accuracy using deep learning (DL) incorporating a multi-modality fusion strategy based on BMD.

METHODS

We conducted a retrospective analysis of 4296 CT scans from a Chinese population, covering August 2015 to November 2022, encompassing lumbar, femur, and pubis modalities. Our DL approach, integrating multi-modality fusion, was applied to predict chronological age automatically. The model's performance was evaluated using an internal real-world clinical cohort of 644 scans (December 2022 to May 2023) and an external cadaver validation cohort of 351 scans.

RESULTS

In single-modality assessments, the lumbar modality excelled. However, multi-modality models demonstrated superior performance, evidenced by lower mean absolute errors (MAEs) and higher Pearson's R² values. The optimal multi-modality model exhibited outstanding R² values of 0.89 overall, 0.88 in females, 0.90 in males, with the MAEs of 4.05 overall, 3.69 in females, 4.33 in males in the internal validation cohort. In the external cadaver validation, the model maintained favourable R² values (0.84 overall, 0.89 in females, 0.82 in males) and MAEs (5.01 overall, 4.71 in females, 5.09 in males), highlighting its generalizability across diverse scenarios.

CONCLUSION

The integration of multi-modalities fusion with DL significantly refines the accuracy of adult age estimation based on BMD. The AI-based system that effectively combines multi-modalities BMD data, presenting a robust and innovative tool for accurate AAE, poised to significantly improve both geriatric diagnostics and forensic investigations.

A multi-modal deep learning solution for precise pneumonia diagnosis: the PneumoFusion-Net model

Background

Pneumonia is considered one of the most important causes of morbidity and mortality in the world. Bacterial and viral pneumonia share many similar clinical features, thus making diagnosis a challenging task. Traditional diagnostic method developments mainly rely on radiological imaging and require a certain degree of consulting clinical experience, which can be inefficient and inconsistent. Deep learning for the classification of pneumonia in multiple modalities, especially integrating multiple data, has not been well explored.

Methods

The study introduce the PneumoFusion-Net, a deep learning-based multimodal framework that incorporates CT images, clinical text, numerical lab test results, and radiology reports for improved diagnosis. In the experiments, a dataset of 10,095 pneumonia CT images was used-including associated clinical data-most of which was used for training and validation while keeping part of it for validation on a held-out test set. Five-fold cross-validation was considered in order to evaluate this model, calculating different metrics including accuracy and F1-Score.

Results

PneumoFusion-Net, which achieved 98.96% classification accuracy with a 98% F1-score on the held-out test set, is highly effective in distinguishing bacterial from viral types of pneumonia. This has been highly beneficial for diagnosis, reducing misdiagnosis and further improving homogeneity across various data sets from multiple patients.

Conclusion

PneumoFusion-Net offers an effective and efficient approach to pneumonia classification by integrating diverse data sources, resulting in high diagnostic accuracy. Its potential for clinical integration could significantly reduce the burden of pneumonia diagnosis by providing radiologists and clinicians with a robust, automated diagnostic tool.

Diagnostic value of deep learning of multimodal imaging of thyroid for TI-RADS category 3-5 classification

BACKGROUND

Thyroid nodules classified within the Thyroid Imaging Reporting and Data Systems (TI-RADS) category 3-5 are typically regarded as having varying degrees of malignancy risk, with the risk increasing from TI-RADS 3 to TI-RADS 5. While some of these nodules may undergo fine-needle aspiration (FNA) biopsy to assess their nature, this procedure carries a risk of false negatives and inherent complications. To avoid the need for unnecessary biopsy examination, we explored a method for distinguishing the benign and malignant characteristics of thyroid TI-RADS 3-5 nodules based on deep-learning ultrasound images combined with computed tomography (CT).

METHODS

Thyroid nodules, assessed as American College of Radiology (ACR) TI-RADS category 3-5 through conventional ultrasound, all of which had postoperative pathology results, were examined using both conventional ultrasound and CT before operation. We investigated the effectiveness of deep-learning models based on ultrasound alone, CT alone, and a combination of both imaging modalities using the following metrics: Area Under Curve (AUC), sensitivity, accuracy, and positive predictive value (PPV). Additionally, we compared the diagnostic efficacy of the combined methods with manual readings of ultrasound and CT.

RESULTS

A total of 768 thyroid nodules falling within TI-RADS categories 3-5 were identified across 768 patients. The dataset comprised 499 malignant and 269 benign cases. For the automatic identification of thyroid TI-RADS category 3-5 nodules, deep learning combined with ultrasound and CT demonstrated a significantly higher AUC (0.930; 95% CI: 0.892, 0.969) compared to the application of ultrasound alone AUC (0.901; 95% CI: 0.856, 0.947) or CT alone AUC (0.776; 95% CI: 0.713, 0.840). Additionally, the AUC of combined modalities surpassed that of radiologists'assessments using ultrasound alone AUCmean (0.725；95% CI:0.677, 0.773), CT alone AUCmean (0.617; 95% CI:0.564, 0.669). Deep learning method combined with ultrasound and CT imaging of thyroid can allow more accurate and precise classification of nodules within TI-RADS categories 3-5.

Advance in peptide-based drug development: delivery platforms, therapeutics and vaccines

The successful approval of peptide-based drugs can be attributed to a collaborative effort across multiple disciplines. The integration of novel drug design and synthesis techniques, display library technology, delivery systems, bioengineering advancements, and artificial intelligence have significantly expedited the development of groundbreaking peptide-based drugs, effectively addressing the obstacles associated with their character, such as the rapid clearance and degradation, necessitating subcutaneous injection leading to increasing patient discomfort, and ultimately advancing translational research efforts. Peptides are presently employed in the management and diagnosis of a diverse array of medical conditions, such as diabetes mellitus, weight loss, oncology, and rare diseases, and are additionally garnering interest in facilitating targeted drug delivery platforms and the advancement of peptide-based vaccines. This paper provides an overview of the present market and clinical trial progress of peptide-based therapeutics, delivery platforms, and vaccines. It examines the key areas of research in peptide-based drug development through a literature analysis and emphasizes the structural modification principles of peptide-based drugs, as well as the recent advancements in screening, design, and delivery technologies. The accelerated advancement in the development of novel peptide-based therapeutics, including peptide-drug complexes, new peptide-based vaccines, and innovative peptide-based diagnostic reagents, has the potential to promote the era of precise customization of disease therapeutic schedule.

A novel integrative multimodal classifier to enhance the diagnosis of Parkinson's disease

Parkinson's disease (PD) is a complex, progressive neurodegenerative disorder with high heterogeneity, making early diagnosis difficult. Early detection and intervention are crucial for slowing PD progression. Understanding PD's diverse pathways and mechanisms is key to advancing knowledge. Recent advances in noninvasive imaging and multi-omics technologies have provided valuable insights into PD's underlying causes and biological processes. However, integrating these diverse data sources remains challenging, especially when deriving meaningful low-level features that can serve as diagnostic indicators. This study developed and validated a novel integrative, multimodal predictive model for detecting PD based on features derived from multimodal data, including hematological information, proteomics, RNA sequencing, metabolomics, and dopamine transporter scan imaging, sourced from the Parkinson's Progression Markers Initiative. Several model architectures were investigated and evaluated, including support vector machine, eXtreme Gradient Boosting, fully connected neural networks with concatenation and joint modeling (FCNN_C and FCNN_JM), and a multimodal encoder-based model with multi-head cross-attention (MMT_CA). The MMT_CA model demonstrated superior predictive performance, achieving a balanced classification accuracy of 97.7%, thus highlighting its ability to capture and leverage cross-modality inter-dependencies to aid predictive analytics. Furthermore, feature importance analysis using SHapley Additive exPlanations not only identified crucial diagnostic biomarkers to inform the predictive models in this study but also holds potential for future research aimed at integrated functional analyses of PD from a multi-omics perspective, ultimately revealing targets required for precision medicine approaches to aid treatment of PD aimed at slowing down its progression.

Deep learning-driven survival prediction in pan-cancer studies by integrating multimodal histology-genomic data

Accurate cancer prognosis is essential for personalized clinical management, guiding treatment strategies and predicting patient survival. Conventional methods, which depend on the subjective evaluation of histopathological features, exhibit significant inter-observer variability and limited predictive power. To overcome these limitations, we developed cross-attention transformer-based multimodal fusion network (CATfusion), a deep learning framework that integrates multimodal histology-genomic data for comprehensive cancer survival prediction. By employing self-supervised learning strategy with TabAE for feature extraction and utilizing cross-attention mechanisms to fuse diverse data types, including mRNA-seq, miRNA-seq, copy number variation, DNA methylation variation, mutation data, and histopathological images. By successfully integrating this multi-tiered patient information, CATfusion has become an advanced survival prediction model to utilize the most diverse data types across various cancer types. CATfusion's architecture, which includes a bidirectional multimodal attention mechanism and self-attention block, is adept at synchronizing the learning and integration of representations from various modalities. CATfusion achieves superior predictive performance over traditional and unimodal models, as demonstrated by enhanced C-index and survival area under the curve scores. The model's high accuracy in stratifying patients into distinct risk groups is a boon for personalized medicine, enabling tailored treatment plans. Moreover, CATfusion's interpretability, enabled by attention-based visualization, offers insights into the biological underpinnings of cancer prognosis, underscoring its potential as a transformative tool in oncology.

MMnc: multi-modal interpretable representation for non-coding RNA classification and class annotation

MOTIVATION

As the biological roles and disease implications of non-coding RNAs continue to emerge, the need to thoroughly characterize previously unexplored non-coding RNAs becomes increasingly urgent. These molecules hold potential as biomarkers and therapeutic targets. However, the vast and complex nature of non-coding RNAs data presents a challenge. We introduce MMnc, an interpretable deep-learning approach designed to classify non-coding RNAs into functional groups. MMnc leverages multiple data sources-such as the sequence, secondary structure, and expression-using attention-based multi-modal data integration. This ensures the learning of meaningful representations while accounting for missing sources in some samples.

RESULTS

Our findings demonstrate that MMnc achieves high classification accuracy across diverse non-coding RNA classes. The method's modular architecture allows for the consideration of multiple types of modalities, whereas other tools only consider one or two at most. MMnc is resilient to missing data, ensuring that all available information is effectively utilized. Importantly, the generated attention scores offer interpretable insights into the underlying patterns of the different non-coding RNA classes, potentially driving future non-coding RNA research and applications.

AVAILABILITY AND IMPLEMENTATION

Data and source code can be found at EvryRNA.ibisc.univ-evry.fr/EvryRNA/MMnc.

Integrating artificial intelligence with smartphone-based imaging for cancer detection in vivo

Cancer is a major global health challenge, accounting for nearly one in six deaths worldwide. Early diagnosis significantly improves survival rates and patient outcomes, yet in resource-limited settings, the scarcity of medical resources often leads to late-stage diagnosis. Integrating artificial intelligence (AI) with smartphone-based imaging systems offers a promising solution by providing portable, cost-effective, and widely accessible tools for early cancer detection. This paper introduces advanced smartphone-based imaging systems that utilize various imaging modalities for in vivo detection of different cancer types and highlights the advancements of AI for in vivo cancer detection in smartphone-based imaging. However, these compact smartphone systems face challenges like low imaging quality and restricted computing power. The use of advanced AI algorithms to address the optical and computational limitations of smartphone-based imaging systems provides promising solutions. AI-based cancer detection also faces challenges. Transparency and reliability are critical factors in gaining the trust and acceptance of AI algorithms for clinical application, explainable and uncertainty-aware AI breaks the black box and will shape the future AI development in early cancer detection. The challenges and solutions for improving AI accuracy, transparency, and reliability are general issues in AI applications, the AI technologies, limitations, and potentials discussed in this paper are applicable to a wide range of biomedical imaging diagnostics beyond smartphones or cancer-specific applications. Smartphone-based multimodal imaging systems and deep learning algorithms for multimodal data analysis are also growing trends, as this approach can provide comprehensive information about the tissue being examined. Future opportunities and perspectives of AI-integrated smartphone imaging systems will be to make cutting-edge diagnostic tools more affordable and accessible, ultimately enabling early cancer detection for a broader population.

Robust multi-modal fusion architecture for medical data with knowledge distillation

BACKGROUND

The fusion of multi-modal data has been shown to significantly enhance the performance of deep learning models, particularly on medical data. However, missing modalities are common in medical data due to patient specificity, which poses a substantial challenge to the application of these models.

OBJECTIVE

This study aimed to develop a novel and efficient multi-modal fusion framework for medical datasets that maintains consistent performance, even in the absence of one or more modalities.

METHODS

In this paper, we fused three modalities: chest X-ray radiographs, history of present illness text, and tabular data such as demographics and laboratory tests. A multi-modal fusion module based on pooled bottleneck (PB) attention was proposed in conjunction with knowledge distillation (KD) for enhancing model inference in the case of missing modalities. In addition, we introduced a gradient modulation (GM) method to deal with the unbalanced optimization in multi-modal model training. Finally, we designed comparison and ablation experiments to evaluate the fusion effect, the model robustness to missing modalities, and the contribution of each component (PB, KD, and GM). The evaluation experiments were performed on the MIMIC-IV datasets with the task of predicting in-hospital mortality risk. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC).

RESULTS

The proposed multi-modal fusion framework achieved an AUROC of 0.886 and AUPRC of 0.459, significantly surpassing the performance of baseline models. Even when one or two modalities were missing, our model consistently outperformed the reference models. Ablation of each of the three components resulted in varying degrees of performance degradation, highlighting their distinct contributions to the model's overall effectiveness.

CONCLUSIONS

This innovative multi-modal fusion architecture has demonstrated robustness to missing modalities, and has shown excellent performance in fusing three medical modalities for patient outcome prediction. This study provides a novel idea for addressing the challenge of missing modalities and has the potential be scaled to additional modalities.

A deep learning-based psi CT network effectively predicts early recurrence after hepatectomy in HCC patients

BACKGROUND

Hepatocellular carcinoma (HCC) exhibits a high recurrence rate, and early recurrence significantly jeopardizes patient prognosis, necessitating reliable methods for early recurrence prediction.

METHODS

Utilizing multi-institutional data and integrating deep learning (DL) techniques, we established a neural network based on DenseNet capable of concurrently processing patients' triphasic enhanced CT scans. By incorporating an attention mechanism, the model automatically focuses on regions that significantly impact patient survival. Performance metrics were first evaluated using the concordance index (C-index), calibration curves, and decision curves based on the training and validation cohorts. Finally, class activation map (CAM) techniques were employed to visualize the regions of interest identified by the model. After model construction, five-fold cross-validation was performed to assess overfitting risks and further evaluate model stability.

RESULTS

We retrospectively collected data from 302 cases across five centers, including patients who underwent Partial Hepatectomy between December 2016 and December 2022. During model development, 180 patients from Institution I formed the training cohort, while the remaining patients comprised the validation cohort. The area under the ROC curve (AUC) for two-year outcomes was 0.797 in the validation cohort. Calibration curves, survival curves, and decision curve analysis (DCA) demonstrated the model's robust performance. CAMs revealed that the model primarily focuses on intra-abdominal solid organs, consistent with clinical experience. After model development, datasets were merged for cross-validation. The best model achieved a C-index of 0.774 in the validation cohort, with five-fold cross-validation yielding an average C-index of 0.778. The 95% confidence interval (CI) for the C-index, derived from cross-validation, ranged from 0.762 to 0.793.

CONCLUSION

Our DL-based enhanced CT network shows promise in predicting early recurrence in patients, representing a potential new strategy for early recurrence prediction in HCC.

Deep learning-based multimodal integration of imaging and clinical data for predicting surgical approach in percutaneous transforaminal endoscopic discectomy

BACKGROUND

For cases of multilevel lumbar disc herniation (LDH), selecting the surgical approach for Percutaneous Transforaminal Endoscopic Discectomy (PTED) presents significant challenges and heavily relies on the physician's judgment. This study aims to develop a deep learning (DL)-based multimodal model that provides objective and referenceable support by comprehensively analyzing imaging and clinical data to assist physicians.

METHODS

This retrospective study collected imaging and clinical data from patients with multilevel LDH. Each segmental MR scan was concurrently fed into a multi-input ResNet 50 model to predict the target segment. The target segment scan was then input to a custom model to predict the PTED approach direction. Clinical data, including the patient's lower limb sensory and motor functions, were used as feature variables in a machine learning (ML) model for prediction. Bayesian optimization was employed to determine the optimal weights for the fusion of the two models.

RESULT

The predictive performance of the multimodal model significantly outperformed the DL and ML models. For PTED target segment prediction, the multimodal model achieved an accuracy of 93.8%, while the DL and ML models achieved accuracies of 87.7% and 87.0%, respectively. Regarding the PTED approach direction, the multimodal model had an accuracy of 89.3%, significantly higher than the DL model's 87.8% and the ML model's 87.6%.

CONCLUSION

The multimodal model demonstrated excellent performance in predicting PTED target segments and approach directions. Its predictive performance surpassed that of the individual DL and ML models.

The effect of depth data and upper limb impairment on lightweight monocular RGB human pose estimation models

BACKGROUND AND OBJECTIVES

Markerless vision-based human pose estimation (HPE) is a promising avenue towards scalable data collection in rehabilitation. Deploying this technology will require self-contained systems able to process data efficiently and accurately. The aims of this work are to (1) Determine how depth data affects lightweight monocular red-green-blue (RGB) HPE performance (accuracy and speed), to inform sensor selection and (2) Validate HPE models using data from individuals with physical impairments.

METHODS

Two HPE models were investigated: Dite-HRNet and MobileHumanPose (capable of 2D and 3D HPE, respectively). The models were modified to include depth data as an input using three different fusion techniques: an early fusion method, a simple intermediate fusion method (using concatenation), and a complex intermediate fusion method (using specific fusion blocks, additional convolutional layers, and concatenation). All fusion techniques used RGB-D data, in contrast to the original models which only used RGB data. The models were trained, validated and tested using the CMU Panoptic and Human3.6 M data sets as well as a custom data set. The custom data set includes RGB-D and optical motion capture data of 15 uninjured and 12 post-stroke individuals, while they performed movements involving their upper limbs. HPE model performances were monitored through accuracy and computational efficiency. Evaluation metrics include Mean per Joint Position Error (MPJPE), Floating Point Operations (FLOPs) and frame rates (frames per second).

RESULTS

The early fusion architecture consistently delivered the lowest MPJPE in both 2D and 3D HPE cases while achieving similar FLOPs and frame rates to its RGB counterpart. These results were consistent regardless of the data used for training and testing the HPE models. Comparisons between the uninjured and stroke groups did not reveal a significant effect (all p values > 0.36) of motor impairment on the accuracy of any model.

CONCLUSIONS

Including depth data using an early fusion architecture improves the accuracy-efficiency trade-off of the HPE model. HPE accuracy is not affected by the presence of physical impairments. These results suggest that using depth data with RGB data is beneficial to HPE, and that models trained with data collected from uninjured individuals can generalize to persons with physical impairments.

Multimodal Digital Phenotyping of Behavior in a Neurology Clinic: Development of the Neurobooth Platform and the First Two Years of Data Collection

Quantitative analysis of human behavior is critical for objective characterization of neurological phenotypes, early detection of neurodegenerative diseases, and development of more sensitive measures of disease progression to support clinical trials and translation of new therapies into clinical practice. Sophisticated computational modeling can support these objectives, but requires large, information-rich data sets. This work introduces Neurobooth, a customizable platform for time-synchronized multimodal capture of human behavior. Over a two year period, a Neurobooth implementation integrated into a clinical setting facilitated data collection across multiple behavioral domains from a cohort of 470 individuals (82 controls and 388 with neurologic diseases) who participated in a collective 782 sessions. Visualization of the multimodal time series data demonstrates the presence of rich phenotypic signs across a range of diseases. These data and the open-source platform offer potential for advancing our understanding of neurological diseases and facilitating therapy development, and may be a valuable resource for related fields that study human behavior.

Parallel CNN-Deep Learning Clinical-Imaging Signature for Assessing Pathologic Grade and Prognosis of Soft Tissue Sarcoma Patients

BACKGROUND

Traditional biopsies pose risks and may not accurately reflect soft tissue sarcoma (STS) heterogeneity. MRI provides a noninvasive, comprehensive alternative.

PURPOSE

To assess the diagnostic accuracy of histological grading and prognosis in STS patients when integrating clinical-imaging parameters with deep learning (DL) features from preoperative MR images.

STUDY TYPE

Retrospective/prospective.

POPULATION

354 pathologically confirmed STS patients (226 low-grade, 128 high-grade) from three hospitals and the Cancer Imaging Archive (TCIA), divided into training (n = 185), external test (n = 125), and TCIA cohorts (n = 44). 12 patients (6 low-grade, 6 high-grade) were enrolled into prospective validation cohort.

FIELD STRENGTH/SEQUENCE

1.5 T and 3.0 T/Unenhanced T1-weighted and fat-suppressed-T2-weighted.

ASSESSMENT

DL features were extracted from MR images using a parallel ResNet-18 model to construct DL signature. Clinical-imaging characteristics included age, gender, tumor-node-metastasis stage and MRI semantic features (depth, number, heterogeneity at T1WI/FS-T2WI, necrosis, and peritumoral edema). Logistic regression analysis identified significant risk factors for the clinical model. A DL clinical-imaging signature (DLCS) was constructed by incorporating DL signature with risk factors, evaluated for risk stratification, and assessed for progression-free survival (PFS) in retrospective cohorts, with an average follow-up of 23 ± 22 months.

STATISTICAL TESTS

Logistic regression, Cox regression, Kaplan-Meier curves, log-rank test, area under the receiver operating characteristic curve (AUC)，and decision curve analysis. A P-value <0.05 was considered significant.

RESULTS

The AUC values for DLCS in the external test, TCIA, and prospective test cohorts (0.834, 0.838, 0.819) were superior to clinical model (0.662, 0.685, 0.694). Decision curve analysis showed that the DLCS model provided greater clinical net benefit over the DL and clinical models. Also, the DLCS model was able to risk-stratify patients and assess PFS.

DATA CONCLUSION

The DLCS exhibited strong capabilities in histological grading and prognosis assessment for STS patients, and may have potential to aid in the formulation of personalized treatment plans.

LEVEL OF EVIDENCE: 4

TECHNICAL EFFICACY

Stage 2.

Identifying major depressive disorder based on cerebral blood flow and brain structure: An explainable multimodal learning study

Magnetic resonance imaging (MRI) offers non-invasive assessments of brain structure and function for analyzing brain disorders. With the increasing accumulation of multimodal MRI data in recent years, integrating information from various modalities has become an effective strategy for improving the detection of brain disorders. This study focuses on identifying major depressive disorder (MDD) by using arterial spin labeling (ASL) perfusion MRI in conjunction with structural MRI data. We collected ASL and structural MRI data from 260 participants, including 169 MDD patients and 91 healthy controls. We developed an explainable fusion method to identify MDD, utilizing cerebral blood flow (CBF) data from ASL perfusion MRI and brain tissue volumes from structural MRI. The fusion model, which integrates multimodal data, demonstrated superior predictive performance for MDD. By combining MRI regional volumes with CBF data, we achieved more effective results than using each modality independently. Additionally, we analyzed feature importance and interactions to explain the fusion model. We identified fourteen important features, comprising eight regional volumes and six regional CBF measures, that played a crucial role in the identification of MDD. Furthermore, we found three feature interactions among the important features and seven interactions between structural and functional features, which were particularly prominent in the model. The results of this study suggest that the fusion learning approach, which integrates ASL and structural MRI data, is effective in detecting MDD. Moreover, the study demonstrates that the model explanation method can reveal key features that influence the decisions of models, as well as potential interactions among these key features or between functional and structural features in identifying MDD.

Recognizing drivers' sleep onset by detecting slow eye movement using a parallel multimodal one-dimensional convolutional neural network

Slow eye movements (SEMs) are a reliable physiological marker of drivers' sleep onset, often accompanied by EEG alpha wave attenuation. A parallel multimodal 1D convolutional neural network (PM-1D-CNN) model is proposed to classify SEMs. The model uses two parallel 1D-CNN blocks to extract features from EOG and EEG signals, which are then fused and fed into fully connected layers for classification. Results show that the PM-1D-CNN outperforms the SGL-1D-CNN and Bimodal-LSTM networks in both subject-to-subject and cross-subject evaluations, confirming its effectiveness in detecting sleep onset.

A comprehensive analysis of stroke risk factors and development of a predictive model using machine learning approaches

Stroke is a leading cause of death and disability globally, particularly in China. Identifying risk factors for stroke at an early stage is critical to improving patient outcomes and reducing the overall disease burden. However, the complexity of stroke risk factors requires advanced approaches for accurate prediction. The objective of this study is to identify key risk factors for stroke and develop a predictive model using machine learning techniques to enhance early detection and improve clinical decision-making. Data from the China Health and Retirement Longitudinal Study (2011-2020) were analyzed, classifying participants based on baseline characteristics. We evaluated correlations among 12 chronic diseases and applied machine learning algorithms to identify stroke-associated parameters. A dose-response relationship between these parameters and stroke was assessed using restricted cubic splines with Cox proportional hazards models. A refined predictive model, incorporating age, sex, and key risk factors, was developed. Stroke patients were significantly older (average age 69.03 years) and had a higher proportion of women (53%) compared to non-stroke individuals. Additionally, stroke patients were more likely to reside in rural areas, be unmarried, smoke, and suffer from various diseases. While the 12 chronic diseases were correlated (p < 0.05), the correlation coefficients were generally weak (r < 0.5). Machine learning identified nine parameters significantly associated with stroke risk: TyG-WC, WHtR, TyG-BMI, TyG, TMO, CysC, CREA, SBP, and HDL-C. Of these, TyG-WC, WHtR, TyG-BMI, TyG, CysC, CREA, and SBP exhibited a positive dose-response relationship with stroke risk. In contrast, TMO and HDL-C were associated with reduced stroke risk. In the fully adjusted model, elevated CysC (HR = 2.606, 95% CI 1.869-3.635), CREA (HR = 1.819, 95% CI 1.240-2.668), and SBP (HR = 1.008, 95% CI 1.003-1.012) were significantly associated with increased stroke risk, while higher HDL-C (HR = 0.989, 95% CI 0.984-0.995) and TMO (HR = 0.99995, 95% CI 0.99994-0.99997) were protective. A nomogram model incorporating age, sex, and the identified parameters demonstrated superior predictive accuracy, with a significantly higher Harrell's C-index compared to individual predictors. This study identifies several significant stroke risk factors and presents a predictive model that can enhance early detection of high-risk individuals. Among them, CREA, CysC, SBP, TyG-BMI, TyG, TyG-WC, and WHtR were positively associated with stroke risk, whereas TMO and HDL-C were opposite. This serves as a valuable decision-support resource for clinicians, facilitating more effective prevention and treatment strategies, ultimately improving patient outcomes.

A machine learning approach to leveraging electronic health records for enhanced omics analysis

Omics studies produce a large number of measurements, enabling the development, validation and interpretation of systems-level biological models. Large cohorts are required to power these complex models; yet, the cohort size remains limited due to clinical and budgetary constraints. We introduce clinical and omics multimodal analysis enhanced with transfer learning (COMET), a machine learning framework that incorporates large, observational electronic health record databases and transfer learning to improve the analysis of small datasets from omics studies. By pretraining on electronic health record data and adaptively blending both early and late fusion strategies, COMET overcomes the limitations of existing multimodal machine learning methods. Using two independent datasets, we showed that COMET improved the predictive modelling performance and biological discovery compared with the analysis of omics data with traditional methods. By incorporating electronic health record data into omics analyses, COMET enables more precise patient classifications, beyond the simplistic binary reduction to cases and controls. This framework can be broadly applied to the analysis of multimodal omics studies and reveals more powerful biological insights from limited cohort sizes.

A prediction study on the occurrence risk of heart disease in older hypertensive patients based on machine learning

OBJECTIVE

Constructing a predictive model for the occurrence of heart disease in elderly hypertensive individuals, aiming to provide early risk identification.

METHODS

A total of 934 participants aged 60 and above from the China Health and Retirement Longitudinal Study with a 7-year follow-up (2011-2018) were included. Machine learning methods (logistic regression, XGBoost, DNN) were employed to build a model predicting heart disease risk in hypertensive patients. Model performance was comprehensively assessed using discrimination, calibration, and clinical decision curves.

RESULTS

After a 7-year follow-up of 934 older hypertensive patients, 243 individuals (26.03%) developed heart disease. Older hypertensive patients with baseline comorbid dyslipidemia, chronic pulmonary diseases, arthritis or rheumatic diseases faced a higher risk of future heart disease. Feature selection significantly improved predictive performance compared to the original variable set. The ROC-AUC for logistic regression, XGBoost, and DNN were 0.60 (95% CI: 0.53-0.68), 0.64 (95% CI: 0.57-0.71), and 0.67 (95% CI: 0.60-0.73), respectively, with logistic regression achieving optimal calibration. XGBoost demonstrated the most noticeable clinical benefit as the threshold increased.

CONCLUSION

Machine learning effectively identifies the risk of heart disease in older hypertensive patients based on data from the CHARLS cohort. The results suggest that older hypertensive patients with comorbid dyslipidemia, chronic pulmonary diseases, and arthritis or rheumatic diseases have a higher risk of developing heart disease. This information could facilitate early risk identification for future heart disease in older hypertensive patients.

Open challenges and opportunities in federated foundation models towards biomedical healthcare

This survey explores the transformative impact of foundation models (FMs) in artificial intelligence, focusing on their integration with federated learning (FL) in biomedical research. Foundation models such as ChatGPT, LLaMa, and CLIP, which are trained on vast datasets through methods including unsupervised pretraining, self-supervised learning, instructed fine-tuning, and reinforcement learning from human feedback, represent significant advancements in machine learning. These models, with their ability to generate coherent text and realistic images, are crucial for biomedical applications that require processing diverse data forms such as clinical reports, diagnostic images, and multimodal patient interactions. The incorporation of FL with these sophisticated models presents a promising strategy to harness their analytical power while safeguarding the privacy of sensitive medical data. This approach not only enhances the capabilities of FMs in medical diagnostics and personalized treatment but also addresses critical concerns about data privacy and security in healthcare. This survey reviews the current applications of FMs in federated settings, underscores the challenges, and identifies future research directions including scaling FMs, managing data diversity, and enhancing communication efficiency within FL frameworks. The objective is to encourage further research into the combined potential of FMs and FL, laying the groundwork for healthcare innovations.

Development and validation of a multi-parametric MRI deep-learning model for preoperative lymphovascular invasion evaluation in rectal cancer

Background

Lymphovascular invasion (LVI) is an independent prognostic factor for patients with rectal cancer (RC). Recent studies have shown that deep learning (DL)-based magnetic resonance imaging (MRI) has potential in evaluating the treatment response of RC patients, but the role of MRI-based DL in assessing RC LVI remains unclear. This study sought to develop and validate a DL model to evaluate the LVI status of RC patients preoperatively based on MRI, and to test its performance at an external center.

Methods

The data of 489 patients with surgically confirmed RC were retrospectively collected from two centers. The training set and the internal validation set comprised 320 patients and 80 patients, respectively, from The Second Affiliated Hospital of Harbin Medical University; while the external testing set comprised 89 patients from Xinjiang Production and Construction Corps Tenth Division Beitun Hospital. All the patients underwent MRI examinations before surgery. Two separate image models were constructed based on the three-dimensional (3D) residual network (ResNet)-18 architecture, using only T2-weighted image (T2WI) data and diffusion-weighted image (DWI) data, respectively, to assess LVI, and a combined model was developed that integrated T2WI, DWI, and clinical factors to assess LVI. The performance of the T2WI- and DWI-based models, and the combination model was evaluated using the area under the curve (AUC) and the DeLong test. The clinical utility of these models was assessed by calibration curve analysis and decision curve analysis (DCA).

Results

The T2WI- and DWI-based DL models demonstrated robust capabilities in evaluating LVI in RC in both the internal validation set and the external test set. For the T2WI-based model, the AUC values reached 0.795 and 0.764 in the internal validation set and the external test set, respectively. For the DWI-based model, the AUC values reached 0.822 and 0.825 in the internal validation set and the external test set, respectively. The combined model exhibited superior performance, achieving AUC values of 0.899 and 0.848 in the internal validation set and the external test set, respectively. In the external test set, all three DL models exhibited robust calibration. The DCA also showed that the DWI-based model and the combined model offered a significantly greater overall net benefit in evaluating LVI than the T2WI-based model.

Conclusions

The multi-parametric MRI DL model demonstrated excellent performance in evaluating the LVI status of patients with RC. This model could serve as a complementary method for the non-invasive assessment of LVI in RC.

rU-Net, Multi-Scale Feature Fusion and Transfer Learning: Unlocking the Potential of Cuffless Blood Pressure Monitoring With PPG and ECG

This study introduces an innovative deep-learning model for cuffless blood pressure estimation using PPG and ECG signals, demonstrating state-of-the-art performance on the largest clean dataset, PulseDB. The rU-Net architecture, a fusion of U-Net and ResNet, enhances both generalization and feature extraction accuracy. Accurate multi-scale feature capture is facilitated by short-time Fourier transform (STFT) time-frequency distributions and multi-head attention mechanisms, allowing data-driven feature selection. The inclusion of demographic parameters as supervisory information further elevates performance. On the calibration-based dataset, our model excels, achieving outstanding accuracy (SBP MAE ± std: 4.49 ± 4.86 mmHg, DBP MAE ± std: 2.69 ± 3.10 mmHg), surpassing AAMI standards and earning a BHS Grade A rating. Addressing the challenge of calibration-free data, we propose a fine-tuning-based transfer learning approach. Remarkably, with only 10% data transfer, our model attains exceptional accuracy (SBP MAE ± std: 4.14 ± 5.01 mmHg, DBP MAE ± std: 2.48 ± 2.93 mmHg). This study sets the stage for the development of highly accurate and reliable wearable cuffless blood pressure monitoring devices.

A cross-attention-based deep learning approach for predicting functional stroke outcomes using 4D CTP imaging and clinical metadata

Acute ischemic stroke (AIS) remains a global health challenge, leading to long-term functional disabilities without timely intervention. Spatio-temporal (4D) Computed Tomography Perfusion (CTP) imaging is crucial for diagnosing and treating AIS due to its ability to rapidly assess the ischemic core and penumbra. Although traditionally used to assess acute tissue status in clinical settings, 4D CTP has also been explored in research for predicting stroke tissue outcomes. However, its potential for predicting functional outcomes, especially in combination with clinical metadata, remains unexplored. Thus, this work aims to develop and evaluate a novel multimodal deep learning model for predicting functional outcomes (specifically, 90-day modified Rankin Scale) in AIS patients by combining 4D CTP and clinical metadata. To achieve this, an intermediate fusion strategy with a cross-attention mechanism is introduced to enable a selective focus on the most relevant features and patterns from both modalities. Evaluated on a dataset comprising 70 AIS patients who underwent endovascular mechanical thrombectomy, the proposed model achieves an accuracy (ACC) of 0.77, outperforming conventional late fusion strategies (ACC = 0.73) and unimodal models based on either 4D CTP (ACC = 0.61) or clinical metadata (ACC = 0.71). The results demonstrate the superior capability of the proposed model to leverage complex inter-modal relationships, emphasizing the value of advanced multimodal fusion techniques for predicting functional stroke outcomes.

Current Status and Future Directions of Artificial Intelligence in Post-Traumatic Stress Disorder: A Literature Measurement Analysis

This study aims to explore the current state of research and the applicability of artificial intelligence (AI) at various stages of post-traumatic stress disorder (PTSD), including prevention, diagnosis, treatment, patient self-management, and drug development. We conducted a bibliometric analysis using software tools such as Bibliometrix (version 4.1), VOSviewer (version 1.6.19), and CiteSpace (version 6.3.R1) on the relevant literature from the Web of Science Core Collection (WoSCC). The analysis reveals a significant increase in publications since 2017. Kerry J. Ressler has emerged as the most influential author in the field to date. The United States leads in the number of publications, producing seven times more papers than Canada, the second-ranked country, and demonstrating substantial influence. Harvard University and the Veterans Health Administration are also key institutions in this field. The Journal of Affective Disorders has the highest number of publications and impact in this area. In recent years, keywords related to functional connectivity, risk factors, and algorithm development have gained prominence. The field holds immense research potential, with AI poised to revolutionize PTSD management through early symptom detection, personalized treatment plans, and continuous patient monitoring. However, there are numerous challenges, and fully realizing AI's potential will require overcoming hurdles in algorithm design, data integration, and societal ethics. To promote more extensive and in-depth future research, it is crucial to prioritize the development of standardized protocols for AI implementation, foster interdisciplinary collaboration-especially between AI and neuroscience-and address public concerns about AI's role in healthcare to enhance its acceptance and effectiveness.

AVP-GPT2: A Transformer-Powered Platform for De Novo Generation, Screening, and Explanation of Antiviral Peptides

Human respiratory syncytial virus (RSV) remains a significant global health threat, particularly for vulnerable populations. Despite extensive research, effective antiviral therapies are still limited. To address this urgent need, we present AVP-GPT2, a deep-learning model that significantly outperforms its predecessor, AVP-GPT, in designing and screening antiviral peptides. Trained on a significantly expanded dataset, AVP-GPT2 employs a transformer-based architecture to generate diverse peptide sequences. A multi-modal screening approach, incorporating Star-Transformer and Vision Transformer, enables accurate prediction of antiviral activity and toxicity, leading to the identification of potent and safe candidates. SHAP analysis further enhances interpretability by explaining the underlying mechanisms of peptide activity. Our in vitro experiments confirmed the antiviral efficacy of peptides generated by AVP-GPT2, with some exhibiting EC50 values as low as 0.01 μM and CC50 values > 30 μM. This represents a substantial improvement over AVP-GPT and traditional methods. AVP-GPT2 has the potential to significantly impact antiviral drug discovery by accelerating the identification of novel therapeutic agents. Future research will explore its application to other viral targets and its integration into existing drug development pipelines.