Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

Facial expression deep learning algorithms in the detection of neurological disorders: a systematic review and meta-analysis

BACKGROUND

Neurological disorders, ranging from common conditions like Alzheimer's disease that is a progressive neurodegenerative disorder and remains the most common cause of dementia worldwide to rare disorders such as Angelman syndrome, impose a significant global health burden. Altered facial expressions are a common symptom across these disorders, potentially serving as a diagnostic indicator. Deep learning algorithms, especially convolutional neural networks (CNNs), have shown promise in detecting these facial expression changes, aiding in diagnosing and monitoring neurological conditions.

OBJECTIVES

This systematic review and meta-analysis aimed to evaluate the performance of deep learning algorithms in detecting facial expression changes for diagnosing neurological disorders.

METHODS

Following PRISMA2020 guidelines, we systematically searched PubMed, Scopus, and Web of Science for studies published up to August 2024. Data from 28 studies were extracted, and the quality was assessed using the JBI checklist. A meta-analysis was performed to calculate pooled accuracy estimates. Subgroup analyses were conducted based on neurological disorders, and heterogeneity was evaluated using the I2 statistic.

RESULTS

The meta-analysis included 24 studies from 2019 to 2024, with neurological conditions such as dementia, Bell's palsy, ALS, and Parkinson's disease assessed. The overall pooled accuracy was 89.25% (95% CI 88.75-89.73%). High accuracy was found for dementia (99%) and Bell's palsy (93.7%), while conditions such as ALS and stroke had lower accuracy (73.2%).

CONCLUSIONS

Deep learning models, particularly CNNs, show strong potential in detecting facial expression changes for neurological disorders. However, further work is needed to standardize data sets and improve model robustness for motor-related conditions.

Predicting Progression to Dementia Using Auditory Verbal Learning Test in Community-Dwelling Older Adults Based On Machine Learning

BACKGROUND

Primary healthcare institutions find identifying individuals with dementia particularly challenging. This study aimed to develop machine learning models for identifying predictive features of older adults with normal cognition to develop dementia.

METHODS

We developed four machine learning models: logistic regression, decision tree, random forest, and gradient-boosted trees, predicting dementia of 1,162 older adults with normal cognition at baseline from the Hubei Memory and Aging Cohort Study. All relevant variables collected were included in the models. The Shanghai Aging Study was selected as a replication cohort (n = 1,370) to validate the performance of models including the key features after a wrapper feature selection technique. Both cohorts adopted comparable diagnostic criteria for dementia to most previous cohort studies.

RESULTS

The random forest model exhibited slightly better predictive power using a series of auditory verbal learning test, education, and follow-up time, as measured by overall accuracy (93%) and an area under the curve (AUC) (mean [standard error]: 088 [0.07]). When assessed in the external validation cohort, its performance was deemed acceptable with an AUC of 0.81 (0.15). Conversely, the logistic regression model showed better results in the external validation set, attaining an AUC of 0.88 (0.20).

CONCLUSION

Our machine learning framework offers a viable strategy for predicting dementia using only memory tests in primary healthcare settings. This model can track cognitive changes and provide valuable insights for early intervention.

Fusion of brain imaging genetic data for alzheimer's disease diagnosis and causal factors identification using multi-stream attention mechanisms and graph convolutional networks

Correctly diagnosing Alzheimer's disease (AD) and identifying pathogenic brain regions and genes play a vital role in understanding the AD and developing effective prevention and treatment strategies. Recent works combine imaging and genetic data, and leverage the strengths of both modalities to achieve better classification results. In this work, we propose MCA-GCN, a Multi-stream Cross-Attention and Graph Convolutional Network-based classification method for AD patients. It first constructs a brain region-gene association network based on brain region fMRI time series and gene SNP data. Then it integrates the absolute and relative positions of the brain region time series to obtain a new brain region time series containing temporal information. Then long-range and local association features between brain regions and genes are sequentially aggregated by multi-stream cross-attention and graph convolutional networks. Finally, the learned brain region and gene features are input to the fully connected network to predict AD types. Experimental results on the ADNI dataset show that our model outperforms other methods in AD classification tasks. Moreover, MCA-GCN designed a multi-stage feature scoring process to extract high-risk genes and brain regions related to disease classification.

Weighted Multi-Modal Contrastive Learning Based Hybrid Network for Alzheimer's Disease Diagnosis

Multiple imaging modalities and specific proteins in the cerebrospinal fluid, providing a comprehensive understanding of neurodegenerative disorders, have been widely used for computer-aided diagnosis of Alzheimer's disease (AD). Given the proven effectiveness of contrastive learning in aligning multi-modal representation, in this paper, we investigate effective contrastive learning strategies to learn better cross-modal representations for the integration of multi-modal complementary information. To enhance the overall performance in AD diagnosis, we construct a unified hybrid network that integrates feature learning and classifier learning into an end-to-end framework. Specifically, we propose a weighted multi-modal contrastive learning based on hybrid network (WMCL-HN) method. Firstly, an adaptive weighted strategy is implemented on the multi-modal contrastive learning to dynamically regulate the degree of information exchange across modalities. It assigns higher weights to more important modality pair, thus the most important underlying relationships across modalities can be captured. Secondly, we construct a hybrid network, which employs a curriculum learning strategy that gradually transitions the training from feature learning to classifier learning, ensuring that the learned features are tailored to the diagnostic task. Experimental results on ADNI dataset demonstrate the effectiveness of the proposed WMCL-HN in AD-related diagnosis tasks. The source code is available at https://github.com/pcehnago/WMCL-HN.

Transformer-based approaches for neuroimaging: an in-depth review of their role in classification and regression tasks

In the ever-evolving landscape of deep learning (DL), the transformer model emerges as a formidable neural network architecture, gaining significant traction in neuroimaging-based classification and regression tasks. This paper presents an extensive examination of transformer's application in neuroimaging, surveying recent literature to elucidate its current status and research advancement. Commencing with an exposition on the fundamental principles and structures of the transformer model and its variants, this review navigates through the methodologies and experimental findings pertaining to their utilization in neuroimage classification and regression tasks. We highlight the transformer model's prowess in neuroimaging, showcasing its exceptional performance in classification endeavors while also showcasing its burgeoning potential in regression tasks. Concluding with an assessment of prevailing challenges and future trajectories, this paper proffers insights into prospective research directions. By elucidating the current landscape and envisaging future trends, this review enhances comprehension of transformer's role in neuroimaging tasks, furnishing valuable guidance for further inquiry.

Detection of Alzheimer Disease in Neuroimages Using Vision Transformers: Systematic Review and Meta-Analysis

BACKGROUND

Alzheimer disease (AD) is a progressive condition characterized by cognitive decline and memory loss. Vision transformers (ViTs) are emerging as promising deep learning models in medical imaging, with potential applications in the detection and diagnosis of AD.

OBJECTIVE

This review systematically examines recent studies on the application of ViTs in detecting AD, evaluating the diagnostic accuracy and impact of network architecture on model performance.

METHODS

We conducted a systematic search across major medical databases, including China National Knowledge Infrastructure, CENTRAL (Cochrane Central Register of Controlled Trials), ScienceDirect, PubMed, Web of Science, and Scopus, covering publications from January 1, 2020, to March 1, 2024. A manual search was also performed to include relevant gray literature. The included papers used ViT models for AD detection versus healthy controls based on neuroimaging data, and the included studies used magnetic resonance imaging and positron emission tomography. Pooled diagnostic accuracy estimates, including sensitivity, specificity, likelihood ratios, and diagnostic odds ratios, were derived using random-effects models. Subgroup analyses comparing the diagnostic performance of different ViT network architectures were performed.

RESULTS

The meta-analysis, encompassing 11 studies with 95% CIs and P values, demonstrated pooled diagnostic accuracy: sensitivity 0.925 (95% CI 0.892-0.959; P<.01), specificity 0.957 (95% CI 0.932-0.981; P<.01), positive likelihood ratio 21.84 (95% CI 12.26-38.91; P<.01), and negative likelihood ratio 0.08 (95% CI 0.05-0.14; P<.01). The area under the curve was notably high at 0.924. The findings highlight the potential of ViTs as effective tools for early and accurate AD diagnosis, offering insights for future neuroimaging-based diagnostic approaches.

CONCLUSIONS

This systematic review provides valuable evidence for the utility of ViT models in distinguishing patients with AD from healthy controls, thereby contributing to advancements in neuroimaging-based diagnostic methodologies.

TRIAL REGISTRATION

PROSPERO CRD42024584347; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=584347.

A multi-modal deep learning model for prediction of Ki-67 for meningiomas using pretreatment MR images

This study developed and validated a deep learning network using baseline magnetic resonance imaging (MRI) to predict Ki-67 status in meningioma patients. A total of 1239 patients were retrospectively recruited from three hospitals between January 2010 and December 2023, forming training, internal validation, and two external validation cohorts. A representation learning framework was utilized for modeling, and performance was assessed against existing methods. Furthermore, Kaplan-Meier survival analysis was conducted to investigate whether the model could be used for tumor growth prediction. The model achieved superior results, with areas under the curve (AUCs) of 0.797 for internal testing and 0.808 for generalization, alongside 0.756 and 0.727 for 3- and 5-year tumor growth predictions, respectively. The prediction was significantly associated with the growth of asymptomatic small meningiomas. Overall, the model provides an effective tool for early prediction of Ki-67 and tumor volume growth, aiding in individualized patient management.

Multi-scale multimodal deep learning framework for Alzheimer's disease diagnosis

Multimodal neuroimaging data, including magnetic resonance imaging (MRI) and positron emission tomography (PET), provides complementary information about the brain that can aid in Alzheimer's disease (AD) diagnosis. However, most existing deep learning methods still rely on patch-based extraction from neuroimaging data, which typically yields suboptimal performance due to its isolation from the subsequent network and does not effectively capture the varying scales of structural changes in the cerebrum. Moreover, these methods often simply concatenate multimodal data, ignoring the interactions between them that can highlight discriminative regions and thereby improve the diagnosis of AD. To tackle these issues, we develop a multimodal and multi-scale deep learning model that effectively leverages the interaction between the multimodal and multiscale of the neuroimaging data. First, we employ a convolutional neural network to embed each scale of the multimodal images. Second, we propose multimodal scale fusion mechanisms that utilize both multi-head self-attention and multi-head cross-attention, which capture global relations among the embedded features and weigh each modality's contribution to another, and hence enhancing feature extraction and interaction between each scale of MRI and PET images. Third, we introduce a cross-modality fusion module that includes a multi-head cross-attention to fuse MRI and PET data at different scales and promote global features from the previous attention layers. Finally, all the features from every scale are fused to discriminate between the different stages of AD. We evaluated our proposed method on the ADNI dataset, and the results show that our model achieves better performance than the state-of-the-art methods.

A Modality-Flexible Framework for Alzheimer's Disease Diagnosis Following Clinical Routine

Dementia has high incidence among the elderly, and Alzheimer's disease (AD) is the most common dementia. The procedure of AD diagnosis in clinics usually follows a standard routine consisting of different phases, from acquiring non-imaging tabular data in the screening phase to MR imaging and ultimately to PET imaging. Most of the existing AD diagnosis studies are dedicated to a specific phase using either single or multi-modal data. In this paper, we introduce a modality-flexible classification framework, which is applicable for different AD diagnosis phases following the clinical routine. Specifically, our framework consists of three branches corresponding to three diagnosis phases: 1) a tabular branch using only tabular data for screening phase, 2) an MRI branch using both MRI and tabular data for uncertain cases in screening phase, and 3) ultimately a PET branch for the challenging cases using all the modalities including PET, MRI, and tabular data. To achieve effective fusion of imaging and non-imaging modalities, we introduce an image-tabular transformer block to adaptively scale and shift the image and tabular features according to modality importance determined by the network. The proposed framework is extensively validated on four cohorts containing 6495 subjects. Experiments demonstrate that our framework achieves superior diagnostic performance than the other representative methods across various AD diagnosis tasks, and shows promising performance for all the diagnosis phases, which exhibits great potential for clinical application.

MACFNet: Detection of Alzheimer's disease via multiscale attention and cross-enhancement fusion network

BACKGROUND AND OBJECTIVE

Alzheimer's disease (AD) is a dreaded degenerative disease that results in a profound decline in human cognition and memory. Due to its intricate pathogenesis and the lack of effective therapeutic interventions, early diagnosis plays a paramount role in AD. Recent research based on neuroimaging has shown that the application of deep learning methods by multimodal neural images can effectively detect AD. However, these methods only concatenate and fuse the high-level features extracted from different modalities, ignoring the fusion and interaction of low-level features across modalities. It consequently leads to unsatisfactory classification performance.

METHOD

In this paper, we propose a novel multi-scale attention and cross-enhanced fusion network, MACFNet, which enables the interaction of multi-stage low-level features between inputs to learn shared feature representations. We first construct a novel Cross-Enhanced Fusion Module (CEFM), which fuses low-level features from different modalities through a multi-stage cross-structure. In addition, an Efficient Spatial Channel Attention (ECSA) module is proposed, which is able to focus on important AD-related features in images more efficiently and achieve feature enhancement from different modalities through two-stage residual concatenation. Finally, we also propose a multiscale attention guiding block (MSAG) based on dilated convolution, which can obtain rich receptive fields without increasing model parameters and computation, and effectively improve the efficiency of multiscale feature extraction.

RESULTS

Experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate that our MACFNet has better classification performance than existing multimodal methods, with classification accuracies of 99.59 %, 98.85 %, 99.61 %, and 98.23 % for AD vs. CN, AD vs. MCI, CN vs. MCI and AD vs. CN vs. MCI, respectively, and specificity of 98.92 %, 97.07 %, 99.58 % and 99.04 %, and sensitivity of 99.91 %, 99.89 %, 99.63 % and 97.75 %, respectively.

CONCLUSIONS

The proposed MACFNet is a high-accuracy multimodal AD diagnostic framework. Through the cross mechanism and efficient attention, MACFNet can make full use of the low-level features of different modal medical images and effectively pay attention to the local and global information of the images. This work provides a valuable reference for multi-mode AD diagnosis.

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

FRAMM: Fair ranking with missing modalities for clinical trial site selection

The underrepresentation of gender, racial, and ethnic minorities in clinical trials is a problem undermining the efficacy of treatments on minorities and preventing precise estimates of the effects within these subgroups. We propose FRAMM, a deep reinforcement learning framework for fair trial site selection to help address this problem. We focus on two real-world challenges: the data modalities used to guide selection are often incomplete for many potential trial sites, and the site selection needs to simultaneously optimize for both enrollment and diversity. To address the missing data challenge, FRAMM has a modality encoder with a masked cross-attention mechanism for bypassing missing data. To make efficient trade-offs, FRAMM uses deep reinforcement learning with a reward function designed to simultaneously optimize for both enrollment and fairness. We evaluate FRAMM using real-world historical clinical trials and show that it outperforms the leading baseline in enrollment-only settings while also greatly improving diversity.

Hybrid Fusion of High-Resolution and Ultra-Widefield OCTA Acquisitions for the Automatic Diagnosis of Diabetic Retinopathy

Optical coherence tomography angiography (OCTA) can deliver enhanced diagnosis for diabetic retinopathy (DR). This study evaluated a deep learning (DL) algorithm for automatic DR severity assessment using high-resolution and ultra-widefield (UWF) OCTA. Diabetic patients were examined with 6×6 mm2 high-resolution OCTA and 15×15 mm2 UWF-OCTA using PLEX®Elite 9000. A novel DL algorithm was trained for automatic DR severity inference using both OCTA acquisitions. The algorithm employed a unique hybrid fusion framework, integrating structural and flow information from both acquisitions. It was trained on data from 875 eyes of 444 patients. Tested on 53 patients (97 eyes), the algorithm achieved a good area under the receiver operating characteristic curve (AUC) for detecting DR (0.8868), moderate non-proliferative DR (0.8276), severe non-proliferative DR (0.8376), and proliferative/treated DR (0.9070). These results significantly outperformed detection with the 6×6 mm2 (AUC = 0.8462, 0.7793, 0.7889, and 0.8104, respectively) or 15×15 mm2 (AUC = 0.8251, 0.7745, 0.7967, and 0.8786, respectively) acquisitions alone. Thus, combining high-resolution and UWF-OCTA acquisitions holds the potential for improved early and late-stage DR detection, offering a foundation for enhancing DR management and a clear path for future works involving expanded datasets and integrating additional imaging modalities.