Deep Spatial-Temporal Feature Fusion From Adaptive Dynamic Functional Connectivity for MCI Identification

Bald eagle-optimized transformer networks with temporal-spatial mid-level features for pancreatic tumor classification

The classification and diagnosis of pancreatic tumors present significant challenges due to their inherent complexity and variability. Traditional methods often struggle to capture the dynamic nature of these tumors, highlighting the need for advanced techniques that improve precision and robustness. This study introduces a novel approach that combines temporal-spatial mid-level features (CTSF) with bald eagle search (BES) optimized transformer networks to enhance pancreatic tumor classification. By leveraging temporal-spatial features that encompass both spatial structure and temporal evolution, we employ the BES algorithm to optimize the vision transformer (ViT) and swin transformer (ST) models, significantly enhancing their capacity to process complex datasets. The study underscores the critical role of temporal features in pancreatic tumor classification, enabling the capture of changes over time to improve our understanding of tumor progression and treatment responses. Among the models evaluated-GRU, LSTM, and ViT-the ViTachieved superior performance, with accuracy rates of 94.44%, 89.44%, and 87.22% on the TCIA-Pancreas-CT, Decathlon Pancreas, and NIH-Pancreas-CT datasets, respectively. Spatial features extracted from ResNet50, VGG16, and ST were also essential, with the ST model attaining the highest accuracy of 95.00%, 95.56%, and 93.33% on the same datasets. The integration of temporal and spatial features within the CTSF model resulted in accuracy rates of 96.02%, 97.21%, and 95.06% for the TCIA-Pancreas-CT, Decathlon Pancreas, and NIH-Pancreas-CT datasets, respectively. Furthermore, optimization techniques, particularly hyperparameter tuning, further enhanced performance, with the BES-optimized model achieving the highest accuracy of 98.02%, 98.92%, and 98.89%. The superiority of the CTSF-BES approach was confirmed through the Friedman test and Bonferroni-Dunn test, while execution time analysis demonstrated a favourable balance between performance and efficiency.

Evaluating Cognitive Function and Brain Activity Patterns via Blood Oxygen Level-Dependent Transformer in N-Back Working Memory Tasks

(1) Background: Working memory, which involves temporary storage, information processing, and regulating attention resources, is a fundamental cognitive process and constitutes a significant component of neuroscience research. This study aimed to evaluate brain activation patterns by analyzing functional magnetic resonance imaging (fMRI) time-series data collected during a designed N-back working memory task with varying cognitive demands. (2) Methods: We utilized a novel transformer model, blood oxygen level-dependent transformer (BolT), to extract the activation level features of brain regions in the cognitive process, thereby obtaining the influence weights of regions of interest (ROIs) on the corresponding tasks. (3) Results: Compared with previous studies, our work reached similar conclusions in major brain region performance and provides a more precise analysis for identifying brain activation patterns. For each type of working memory task, we selected the top 5 percent of the most influential ROIs and conducted a comprehensive analysis and discussion. Additionally, we explored the effect of prior knowledge conditions on the performance of different tasks in the same period and the same tasks at different times. (4) Conclusions: The comparison results reflect the brain's adaptive strategies and dependencies in coping with different levels of cognitive demands and the stability optimization of the brain's cognitive processing. This study introduces innovative methodologies for understanding brain function and cognitive processes, highlighting the potential of transformer in cognitive neuroscience. Its findings offer new insights into brain activity patterns associated with working memory, contributing to the broader landscape of neuroscience research.

DEAF-Net: Detail-Enhanced Attention Feature Fusion Network for Retinal Vessel Segmentation

Retinal vessel segmentation is crucial for the diagnosis of ophthalmic and cardiovascular diseases. However, retinal vessels are densely and irregularly distributed, with many capillaries blending into the background, and exhibit low contrast. Moreover, the encoder-decoder-based network for retinal vessel segmentation suffers from irreversible loss of detailed features due to multiple encoding and decoding, leading to incorrect segmentation of the vessels. Meanwhile, the single-dimensional attention mechanisms possess limitations, neglecting the importance of multidimensional features. To solve these issues, in this paper, we propose a detail-enhanced attention feature fusion network (DEAF-Net) for retinal vessel segmentation. First, the detail-enhanced residual block (DERB) module is proposed to strengthen the capacity for detailed representation, ensuring that intricate features are efficiently maintained during the segmentation of delicate vessels. Second, the multidimensional collaborative attention encoder (MCAE) module is proposed to optimize the extraction of multidimensional information. Then, the dynamic decoder (DYD) module is introduced to preserve spatial information during the decoding process and reduce the information loss caused by upsampling operations. Finally, the proposed detail-enhanced feature fusion (DEFF) module composed of DERB, MCAE and DYD modules fuses feature maps from both encoding and decoding and achieves effective aggregation of multi-scale contextual information. The experiments conducted on the datasets of DRIVE, CHASEDB1, and STARE, achieving Sen of 0.8305, 0.8784, and 0.8654, and AUC of 0.9886, 0.9913, and 0.9911 on DRIVE, CHASEDB1, and STARE, respectively, demonstrate the performance of our proposed network, particularly in the segmentation of fine retinal vessels.

Multimodal multiview bilinear graph convolutional network for mild cognitive impairment diagnosis

Mild cognitive impairment (MCI) is a significant predictor of the early progression of Alzheimer's disease (AD) and can serve as an important indicator of disease progression. However, many existing methods focus mainly on the image when processing brain imaging data, ignoring other non-imaging data (e.g., genetic, clinical information, etc.) that may have underlying disease information. In addition, imaging data acquired from different devices may exhibit varying degrees of heterogeneity, potentially resulting in numerous noisy connections during network construction. To address these challenges, this study proposes a Multimodal Multiview Bilinear Graph Convolution (MMBGCN) framework for disease risk prediction. Firstly, grey matter (GM), white matter (WM) and cerebrospinal fluid (CSF) features are extracted from magnetic resonance imaging (MRI), and non-imaging information is combined with the features extracted from MRI to construct a multimodal shared adjacency matrix. The shared adjacency matrix is then used to construct the multiview network so that the effect of potential disease information in the non-imaging information on the model can be considered. Finally, the MRI features extracted by the multiview network are weighted to reduce noise, and then the spatial pattern is restored by bilinear convolution. The features of the recovered spatial patterns are then combined with the genetic information for disease prediction. The proposed method is tested on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset. Extensive experiments demonstrate the superior performance of the proposed framework and its ability to outperform other related algorithms. The average classification accuracy in the binary classification task in this study is 89.6%. The experimental results demonstrate that the method proposed in this study facilitates research on MCI diagnosis using multimodal data.

Prob-sparse self-attention extraction of time-aligned dynamic functional connectivity for ASD diagnosis

Dynamic functional connectivity (DFC) has shown promise in the diagnosis of Autism Spectrum Disorder (ASD). However, extracting highly discriminative information from the complex DFC matrix remains a challenging task. In this paper, we propose an ASD classification framework PSA-FCN which is based on time-aligned DFC and Prob-Sparse Self-Attention to address this problem. Specifically, we introduce Prob-Sparse Self-Attention to selectively extract global features, and use self-attention distillation as a transition at each layer to capture local patterns and reduce dimensionality. Additionally, we construct a time-aligned DFC matrix to mitigate the time sensitivity of DFC and extend the dataset, thereby alleviating model overfitting. Our model is evaluated on fMRI data from the ABIDE NYU site, and the experimental results demonstrate that the model outperforms other methods in the paper with a classification accuracy of 81.8 %. Additionally, our research findings reveal significant variability in the DFC connections of brain regions of ASD patients, including Cuneus (CUN), Lingual gyrus (LING), Superior occipital gyrus (SOG), Posterior cingulate gyrus (PCG), and Precuneus (PCUN), which is consistent with prior research. In summary, our proposed PSA framework shows potential in ASD diagnosis as well as automatic discovery of critical ASD-related biomarkers.

Improving fMRI-Based Autism Severity Identification via Brain Network Distance and Adaptive Label Distribution Learning

Machine learning methodologies have been profoundly researched in the realm of autism spectrum disorder (ASD) diagnosis. Nonetheless, owing to the ambiguity of ASD severity labels and individual differences in ASD severity, current fMRI-based methods for identifying ASD severity still do not achieve satisfactory performance. Besides, the potential association between brain functional networks(BFN) and ASD symptom severity remains under investigation. To address these problems, we propose a low&high-level BFN distance method and an adaptive multi-label distribution(HBFND-AMLD) technique for ASD severity identification. First, a low-level and high-level BFN distance(HBFND) is proposed to construct BFN that reflects differences in ASD severity. This method can measure the distance between the ASD and the health control(HC) on the low-order and high-order BFN respectively, which can distinguish the severity of ASD. After that, a multi-task network is proposed for ASD severity identification which considers the individual differences of ASD severity in communication and society, which considers the individual differences in language and social skills of ASD patients. Finally, a novel adaptive label distribution(ALD) technique is employed to train the ASD severity identification model, effectively preventing network overfitting by restricting label probability distribution. We evaluate the proposed framework on the public ABIDE I dataset. The promising results obtained by our framework outperform the state-of-the-art methods with an increase in identification performance, indicating that it has a potential clinical prospect for practical ASD severity diagnosis.

Multi-Modal Diagnosis of Alzheimer's Disease Using Interpretable Graph Convolutional Networks

The interconnection between brain regions in neurological disease encodes vital information for the advancement of biomarkers and diagnostics. Although graph convolutional networks are widely applied for discovering brain connection patterns that point to disease conditions, the potential of connection patterns that arise from multiple imaging modalities has yet to be fully realized. In this paper, we propose a multi-modal sparse interpretable GCN framework (SGCN) for the detection of Alzheimer's disease (AD) and its prodromal stage, known as mild cognitive impairment (MCI). In our experimentation, SGCN learned the sparse regional importance probability to find signature regions of interest (ROIs), and the connective importance probability to reveal disease-specific brain network connections. We evaluated SGCN on the Alzheimer's Disease Neuroimaging Initiative database with multi-modal brain images and demonstrated that the ROI features learned by SGCN were effective for enhancing AD status identification. The identified abnormalities were significantly correlated with AD-related clinical symptoms. We further interpreted the identified brain dysfunctions at the level of large-scale neural systems and sex-related connectivity abnormalities in AD/MCI. The salient ROIs and the prominent brain connectivity abnormalities interpreted by SGCN are considerably important for developing novel biomarkers. These findings contribute to a better understanding of the network-based disorder via multi-modal diagnosis and offer the potential for precision diagnostics. The source code is available at https://github.com/Houliang-Zhou/SGCN.

Multi-Scale Spatial-Temporal Attention Networks for Functional Connectome Classification

Many neuropsychiatric disorders are considered to be associated with abnormalities in the functional connectivity networks of the brain. The research on the classification of functional connectivity can therefore provide new perspectives for understanding the pathology of disorders and contribute to early diagnosis and treatment. Functional connectivity exhibits a nature of dynamically changing over time, however, the majority of existing methods are unable to collectively reveal the spatial topology and time-varying characteristics. Furthermore, despite the efforts of limited spatial-temporal studies to capture rich information across different spatial scales, they have not delved into the temporal characteristics among different scales. To address above issues, we propose a novel Multi-Scale Spatial-Temporal Attention Networks (MSSTAN) to exploit the multi-scale spatial-temporal information provided by functional connectome for classification. To fully extract spatial features of brain regions, we propose a Topology Enhanced Graph Transformer module to guide the attention calculations in the learning of spatial features by incorporating topology priors. A Multi-Scale Pooling Strategy is introduced to obtain representations of brain connectome at various scales. Considering the temporal dynamic characteristics between dynamic functional connectome, we employ Locality Sensitive Hashing attention to further capture long-term dependencies in time dynamics across multiple scales and reduce the computational complexity of the original attention mechanism. Experiments on three brain fMRI datasets of MDD and ASD demonstrate the superiority of our proposed approach. In addition, benefiting from the attention mechanism in Transformer, our results are interpretable, which can contribute to the discovery of biomarkers. The code is available at https://github.com/LIST-KONG/MSSTAN.

Lightweight skin cancer detection IP hardware implementation using cycle expansion and optimal computation arrays methods

Skin cancer is recognized as one of the most perilous diseases globally. In the field of medical image classification, precise identification of early-stage skin lesions is imperative for accurate diagnosis. However, deploying these algorithms on low-cost devices and attaining high-efficiency operation with minimal energy consumption poses a formidable challenge due to their intricate computational demands. This study proposes a lightweight hardware design based on a convolutional neural network (CNN) for real-time processing of skin disease classifiers on portable devices. Our skin cancer recognition processor utilizes an optimally parallel designed processing engine (PE) for global computation, which greatly reduces hardware resource utilization by multiplexing of computational unit circuits. In addition, a design approach that provides loop unrolling effectively reduces the number of data accesses, thereby reducing computational complexity and logic resource requirements. The hardware circuits in this design perform data inference in convolutional, pooling, and fully connected layers based on 16-bit floating-point numbers. Evaluation of the HAM10000 database dataset shows that the architecture achieves an average classification accuracy of 97.8 %. We are the first to implement an all-hardware FPGA-based skin cancer detection platform that offers a 3.5x speedup in recognition compared to existing skin cancer accelerators at 50 MHz while consuming only 0.48 W of power. The implementation of this hardware architecture meets the major constraints of portable devices, featuring low resource utilization, low power consumption, and cost-effectiveness, while still providing efficient classification and high accuracy results.

Constructing High-Order Functional Connectivity Networks With Temporal Information From fMRI Data

Conducting functional connectivity analysis on functional magnetic resonance imaging (fMRI) data presents a significant and intricate challenge. Contemporary studies typically analyze fMRI data by constructing high-order functional connectivity networks (FCNs) due to their strong interpretability. However, these approaches often overlook temporal information, resulting in suboptimal accuracy. Temporal information plays a vital role in reflecting changes in blood oxygenation level-dependent signals. To address this shortcoming, we have devised a framework for extracting temporal dependencies from fMRI data and inferring high-order functional connectivity among regions of interest (ROIs). Our approach postulates that the current state can be determined by the FCN and the state at the previous time, effectively capturing temporal dependencies. Furthermore, we enhance FCN by incorporating high-order features through hypergraph-based manifold regularization. Our algorithm involves causal modeling of the dynamic brain system, and the obtained directed FC reveals differences in the flow of information under different patterns. We have validated the significance of integrating temporal information into FCN using four real-world fMRI datasets. On average, our framework achieves 12% higher accuracy than non-temporal hypergraph-based and low-order FCNs, all while maintaining a short processing time. Notably, our framework successfully identifies the most discriminative ROIs, aligning with previous research, and thereby facilitating cognitive and behavioral studies.

Temporal Dynamic Synchronous Functional Brain Network for Schizophrenia Classification and Lateralization Analysis

Available evidence suggests that dynamic functional connectivity can capture time-varying abnormalities in brain activity in resting-state cerebral functional magnetic resonance imaging (rs-fMRI) data and has a natural advantage in uncovering mechanisms of abnormal brain activity in schizophrenia (SZ) patients. Hence, an advanced dynamic brain network analysis model called the temporal brain category graph convolutional network (Temporal-BCGCN) was employed. Firstly, a unique dynamic brain network analysis module, DSF-BrainNet, was designed to construct dynamic synchronization features. Subsequently, a revolutionary graph convolution method, TemporalConv, was proposed based on the synchronous temporal properties of features. Finally, the first modular test tool for abnormal hemispherical lateralization in deep learning based on rs-fMRI data, named CategoryPool, was proposed. This study was validated on COBRE and UCLA datasets and achieved 83.62% and 89.71% average accuracies, respectively, outperforming the baseline model and other state-of-the-art methods. The ablation results also demonstrate the advantages of TemporalConv over the traditional edge feature graph convolution approach and the improvement of CategoryPool over the classical graph pooling approach. Interestingly, this study showed that the lower-order perceptual system and higher-order network regions in the left hemisphere are more severely dysfunctional than in the right hemisphere in SZ, reaffirmings the importance of the left medial superior frontal gyrus in SZ. Our code was available at: https://github.com/swfen/Temporal-BCGCN.

Enhancing Autism Spectrum Disorder identification in multi-site MRI imaging: A multi-head cross-attention and multi-context approach for addressing variability in un-harmonized data

Multi-site MRI imaging poses a significant challenge due to the potential variations in images across different scanners at different sites. This variability can introduce ambiguity in further image analysis. Consequently, the image analysis techniques become site-dependent and scanner-dependent, implying that adjustments in the analysis methodologies are necessary for each scanner configuration. Further, implementing real-time modifications becomes intricate, particularly when incorporating a new type of scanner, as it requires adapting the analysis methods accordingly. Taking into account the aforementioned challenge, we have considered its implications for an Autism spectrum disorder (ASD) application. Our objective is to minimize the impact of site and scanner variability in the analysis, aiming to develop a model that remains effective across different scanners and sites. This entails devising a methodology that allows the same model to function seamlessly across multiple scanner configurations and sites. ASD, a behavioral disorder affecting child development, requires early detection. Clinical observation is time-consuming, prompting the use of fMRI with machine/deep learning for expedited diagnosis. Previous methods leverage fMRI's functional connectivity but often rely on less generalized feature extractors and classifiers. Hence, there is significant room for improvement in the generalizability of detection methods across multi-site data, which is acquired from multiple scanners with different settings. In this study, we propose a Cross-Combination Multi-Scale Multi-Context Framework (CCMSMCF) capable of performing neuroimaging-based diagnostic classification of mental disorders for a multi-site dataset. Thus, this framework attains a degree of internal data harmonization, rendering it to some extent site and scanner-agnostic. Our proposed network, CCMSMCF, is constructed by integrating two sub-modules: the Multi-Head Attention Cross-Scale Module (MHACSM) and the Residual Multi-Context Module (RMCN). We also employ multiple loss functions in a novel manner for training the model, which includes Binary Cross Entropy, Dice loss, and Embedding Coupling loss. The model is validated on the Autism Brain Imaging Data Exchange I (ABIDE-I) dataset, which includes data from multiple scanners across different sites, and achieves promising results.

Dynamic functional connections analysis with spectral learning for brain disorder detection

Dynamic functional connections (dFCs), can reveal neural activities, which provides an insightful way of mining the temporal patterns within the human brain and further detecting brain disorders. However, most existing studies focus on the dFCs estimation to identify brain disorders by shallow temporal features and methods, which cannot capture the inherent temporal patterns of dFCs effectively. To address this problem, this study proposes a novel method, named dynamic functional connections analysis with spectral learning (dCSL), to explore inherently temporal patterns of dFCs and further detect the brain disorders. Concretely, dCSL includes two components, dFCs estimation module and dFCs analysis module. In the former, dFCs are estimated via the sliding window technique. In the latter, the spectral kernel mapping is first constructed by combining the Fourier transform with the non-stationary kernel. Subsequently, the spectral kernel mapping is stacked into a deep kernel network to explore higher-order temporal patterns of dFCs through spectral learning. The proposed dCSL, sharing the benefits of deep architecture and non-stationary kernel, can not only calculate the long-range relationship but also explore the higher-order temporal patterns of dFCs. To evaluate the proposed method, a set of brain disorder classification tasks are conducted on several public datasets. As a result, the proposed dCSL achieves 5% accuracy improvement compared with the widely used approaches for analyzing sequence data, 1.3% accuracy improvement compared with the state-of-the-art methods for dFCs. In addition, the discriminative brain regions are explored in the ASD detection task. The findings in this study are consistent with the clinical performance in ASD.

A Novel Method to Identify Mild Cognitive Impairment Using Dynamic Spatio-Temporal Graph Neural Network

Resting-state functional magnetic resonance imaging (rs-fMRI) has been widely used in the identification of mild cognitive impairment (MCI) research, MCI patients are relatively at a higher risk of progression to Alzheimer's disease (AD). However, almost machine learning and deep learning methods are rarely analyzed from the perspective of spatial structure and temporal dimension. In order to make full use of rs-fMRI data, this study constructed a dynamic spatiotemporal graph neural network model, which mainly includes three modules: temporal block, spatial block, and graph pooling block. Our proposed model can extract the BOLD signal of the subject's fMRI data and the spatial structure of functional connections between different brain regions, and improve the decision-making results of the model. In the study of AD, MCI and NC, the classification accuracy reached 83.78% outperforming previously reported, which manifested that our model could effectively learn spatiotemporal, and dynamic spatio-temporal method plays an important role in identifying different groups of subjects. In summary, this paper proposed an end-to-end dynamic spatio-temporal graph neural network model, which uses the information of the temporal dimension and spatial structure in rs-fMRI data, and achieves the improvement of the three classification performance among AD, MCI and NC.

Mild cognitive impairment prediction based on multi-stream convolutional neural networks

BACKGROUND

Mild cognitive impairment (MCI) is the transition stage between the cognitive decline expected in normal aging and more severe cognitive decline such as dementia. The early diagnosis of MCI plays an important role in human healthcare. Current methods of MCI detection include cognitive tests to screen for executive function impairments, possibly followed by neuroimaging tests. However, these methods are expensive and time-consuming. Several studies have demonstrated that MCI and dementia can be detected by machine learning technologies from different modality data. This study proposes a multi-stream convolutional neural network (MCNN) model to predict MCI from face videos.

RESULTS

The total effective data are 48 facial videos from 45 participants, including 35 videos from normal cognitive participants and 13 videos from MCI participants. The videos are divided into several segments. Then, the MCNN captures the latent facial spatial features and facial dynamic features of each segment and classifies the segment as MCI or normal. Finally, the aggregation stage produces the final detection results of the input video. We evaluate 27 MCNN model combinations including three ResNet architectures, three optimizers, and three activation functions. The experimental results showed that the ResNet-50 backbone with Swish activation function and Ranger optimizer produces the best results with an F1-score of 89% at the segment level. However, the ResNet-18 backbone with Swish and Ranger achieves the F1-score of 100% at the participant level.

CONCLUSIONS

This study presents an efficient new method for predicting MCI from facial videos. Studies have shown that MCI can be detected from facial videos, and facial data can be used as a biomarker for MCI. This approach is very promising for developing accurate models for screening MCI through facial data. It demonstrates that automated, non-invasive, and inexpensive MCI screening methods are feasible and do not require highly subjective paper-and-pencil questionnaires. Evaluation of 27 model combinations also found that ResNet-50 with Swish is more stable for different optimizers. Such results provide directions for hyperparameter tuning to further improve MCI predictions.

Explainable spatio-temporal graph evolution learning with applications to dynamic brain network analysis during development

Modeling dynamic interactions among network components is crucial to uncovering the evolution mechanisms of complex networks. Recently, spatio-temporal graph learning methods have achieved noteworthy results in characterizing the dynamic changes of inter-node relations (INRs). However, challenges remain: The spatial neighborhood of an INR is underexploited, and the spatio-temporal dependencies in INRs' dynamic changes are overlooked, ignoring the influence of historical states and local information. In addition, the model's explainability has been understudied. To address these issues, we propose an explainable spatio-temporal graph evolution learning (ESTGEL) model to model the dynamic evolution of INRs. Specifically, an edge attention module is proposed to utilize the spatial neighborhood of an INR at multi-level, i.e., a hierarchy of nested subgraphs derived from decomposing the initial node-relation graph. Subsequently, a dynamic relation learning module is proposed to capture the spatio-temporal dependencies of INRs. The INRs are then used as adjacent information to improve the node representation, resulting in comprehensive delineation of dynamic evolution of the network. Finally, the approach is validated with real data on brain development study. Experimental results on dynamic brain networks analysis reveal that brain functional networks transition from dispersed to more convergent and modular structures throughout development. Significant changes are observed in the dynamic functional connectivity (dFC) associated with functions including emotional control, decision-making, and language processing.

A Multiview Brain Network Transformer Fusing Individualized Information for Autism Spectrum Disorder Diagnosis

Functional connectivity (FC) networks, built from analyses of resting-state magnetic resonance imaging (rs-fMRI), serve as efficacious biomarkers for identifying Autism Spectrum Disorders (ASD) patients. Given the neurobiological heterogeneity across individuals and the unique presentation of ASD symptoms, the fusion of individualized information into diagnosis becomes essential. However, this aspect is overlooked in most methods. Furthermore, the existing methods typically focus on studying direct pairwise connections between brain ROIs, while disregarding interactions between indirectly connected neighbors. To overcome above challenges, we build common FC and individualized FC by tangent pearson embedding (TP) and common orthogonal basis extraction (COBE) respectively, and present a novel multiview brain transformer (MBT) aimed at effectively fusing common and indivinformation of subjects. MBT is mainly constructed by transformer layers with diffusion kernel (DK), fusion quality-inspired weighting module (FQW), similarity loss and orthonormal clustering fusion readout module (OCFRead). DK transformer can incorporate higher-order random walk methods to capture wider interactions among indirectly connected brain regions. FQW promotes adaptive fusion of features between views, and similarity loss and OCFRead are placed on the last layer to accomplish the ultimate integration of information. In our method, TP, DK and FQW modules all help to model wider connectivity in the brain that make up for the shortcomings of traditional methods. We conducted experiments on the public ABIDE dataset based on AAL and CC200 respectively. Our framework has shown promising results, outperforming state-of-the-art methods on both templates. This suggests its potential as a valuable approach for clinical ASD diagnosis.

Spatial-Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data

Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial-temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 ± 4.5%, 72.0 ± 3.8%, and 72.5 ± 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.

STDCformer: Spatial-temporal dual-path cross-attention model for fMRI-based autism spectrum disorder identification

Resting-state functional magnetic resonance imaging (rs-fMRI) is a non-invasive neuroimaging technique widely utilized in the research of Autism Spectrum Disorder (ASD), providing preliminary insights into the potential biological mechanisms underlying ASD. Deep learning techniques have demonstrated significant potential in the analysis of rs-fMRI. However, accurately distinguishing between healthy control group and ASD has been a longstanding challenge. In this regard, this work proposes a model featuring a dual-path cross-attention framework for spatial and temporal patterns, named STDCformer, aiming to enhance the accuracy of ASD identification. STDCformer can preserve both temporal-specific patterns and spatial-specific patterns while explicitly interacting spatiotemporal information in depth. The embedding layer of the STDCformer embeds temporal and spatial patterns in dual paths. For the temporal path, we introduce a perturbation positional encoding to improve the issue of signal misalignment caused by individual differences. For the spatial path, we propose a correlation metric based on Gramian angular field similarity to establish a more specific whole-brain functional network. Subsequently, we interleave the query and key vectors of dual paths to interact spatial and temporal information. We further propose integrating the dual-path attention into a tensor that retains spatiotemporal dimensions and utilizing 2D convolution for feed-forward processing. Our attention layer allows the model to represent spatiotemporal correlations of signals at multiple scales to alleviate issues of information distortion and loss. Our STDCformer demonstrates competitive results compared to state-of-the-art methods on the ABIDE dataset. Additionally, we conducted interpretative analyses of the model to preliminarily discuss the potential physiological mechanisms of ASD. This work once again demonstrates the potential of deep learning technology in identifying ASD and developing neuroimaging biomarkers for ASD.

Deep Self-Reconstruction Fusion Similarity Hashing for the Diagnosis of Alzheimer's Disease on Multi-Modal Data

The pathogenesis of Alzheimer's disease (AD) is extremely intricate, which makes AD patients almost incurable. Recent studies have demonstrated that analyzing multi-modal data can offer a comprehensive perspective on the different stages of AD progression, which is beneficial for early diagnosis of AD. In this paper, we propose a deep self-reconstruction fusion similarity hashing (DS-FSH) method to effectively capture the AD-related biomarkers from the multi-modal data and leverage them to diagnose AD. Given that most existing methods ignore the topological structure of the data, a deep self-reconstruction model based on random walk graph regularization is designed to reconstruct the multi-modal data, thereby learning the nonlinear relationship between samples. Additionally, a fused similarity hash based on anchor graph is proposed to generate discriminative binary hash codes for multi-modal reconstructed data. This allows sample fused similarity to be effectively modeled by a fusion similarity matrix based on anchor graph while modal correlation can be approximated by Hamming distance. Especially, extracted features from the multi-modal data are classified using deep sparse autoencoders classifier. Finally, experiments conduct on the AD Neuroimaging Initiative database show that DS-FSH outperforms comparable methods of AD classification. To conclude, DS-FSH identifies multi-modal features closely associated with AD, which are expected to contribute significantly to understanding of the pathogenesis of AD.

Altered dynamic amplitude of low-frequency fluctuation in individuals at high risk for Alzheimer's disease

The brain's dynamic spontaneous neural activity is significant in supporting cognition; however, how brain dynamics go awry in subjective cognitive decline (SCD) and mild cognitive impairment (MCI) remains unclear. Thus, the current study aimed to investigate the dynamic amplitude of low-frequency fluctuation (dALFF) alterations in patients at high risk for Alzheimer's disease and to explore its correlation with clinical cognitive assessment scales, to identify an early imaging sign for these special populations. A total of 152 participants, including 72 SCD patients, 44 MCI patients and 36 healthy controls (HCs), underwent a resting-state functional magnetic resonance imaging and were assessed with various neuropsychological tests. The dALFF was measured using sliding-window analysis. We employed canonical correlation analysis (CCA) to examine the bi-multivariate correlations between neuropsychological scales and altered dALFF among multiple regions in SCD and MCI patients. Compared to those in the HC group, both the MCI and SCD groups showed higher dALFF values in the right opercular inferior frontal gyrus (voxel P < .001, cluster P < .05, correction). Moreover, the CCA models revealed that behavioural tests relevant to inattention correlated with the dALFF of the right middle frontal gyrus and right opercular inferior frontal gyrus, which are involved in frontoparietal networks (R = .43, P = .024). In conclusion, the brain dynamics of neural activity in frontal areas provide insights into the shared neural basis underlying SCD and MCI.

Integration of temporal & spatial properties of dynamic functional connectivity based on two-directional two-dimensional principal component analysis for disease analysis

Dynamic functional connectivity, derived from resting-state functional magnetic resonance imaging (rs-fMRI), has emerged as a crucial instrument for investigating and supporting the diagnosis of neurological disorders. However, prevalent features of dynamic functional connectivity predominantly capture either temporal or spatial properties, such as mean and global efficiency, neglecting the significant information embedded in the fusion of spatial and temporal attributes. In addition, dynamic functional connectivity suffers from the problem of temporal mismatch, i.e., the functional connectivity of different subjects at the same time point cannot be matched. To address these problems, this article introduces a novel feature extraction framework grounded in two-directional two-dimensional principal component analysis. This framework is designed to extract features that integrate both spatial and temporal properties of dynamic functional connectivity. Additionally, we propose to use Fourier transform to extract temporal-invariance properties contained in dynamic functional connectivity. Experimental findings underscore the superior performance of features extracted by this framework in classification experiments compared to features capturing individual properties.

A Learnable Counter-Condition Analysis Framework for Functional Connectivity-Based Neurological Disorder Diagnosis

To understand the biological characteristics of neurological disorders with functional connectivity (FC), recent studies have widely utilized deep learning-based models to identify the disease and conducted post-hoc analyses via explainable models to discover disease-related biomarkers. Most existing frameworks consist of three stages, namely, feature selection, feature extraction for classification, and analysis, where each stage is implemented separately. However, if the results at each stage lack reliability, it can cause misdiagnosis and incorrect analysis in afterward stages. In this study, we propose a novel unified framework that systemically integrates diagnoses (i.e., feature selection and feature extraction) and explanations. Notably, we devised an adaptive attention network as a feature selection approach to identify individual-specific disease-related connections. We also propose a functional network relational encoder that summarizes the global topological properties of FC by learning the inter-network relations without pre-defined edges between functional networks. Last but not least, our framework provides a novel explanatory power for neuroscientific interpretation, also termed counter-condition analysis. We simulated the FC that reverses the diagnostic information (i.e., counter-condition FC): converting a normal brain to be abnormal and vice versa. We validated the effectiveness of our framework by using two large resting-state functional magnetic resonance imaging (fMRI) datasets, Autism Brain Imaging Data Exchange (ABIDE) and REST-meta-MDD, and demonstrated that our framework outperforms other competing methods for disease identification. Furthermore, we analyzed the disease-related neurological patterns based on counter-condition analysis.

Effective hyper-connectivity network construction and learning: Application to major depressive disorder identification

Functional connectivity (FC) derived from resting-state fMRI (rs-fMRI) is a primary approach for identifying brain diseases, but it is limited to capturing the pairwise correlation between regions-of-interest (ROIs) in the brain. Thus, hyper-connectivity which describes the higher-order relationship among multiple ROIs is receiving increasing attention. However, most hyper-connectivity methods overlook the directionality of connections. The direction of information flow constitutes a pivotal factor in shaping brain activity and cognitive processes. Neglecting this directional aspect can lead to an incomplete understanding of high-order interactions within the brain. To this end, we propose a novel effective hyper-connectivity (EHC) network that integrates direction detection and hyper-connectivity modeling. It characterizes the high-order directional information flow among multiple ROIs, providing a more comprehensive understanding of brain activity. Then, we develop a directed hypergraph convolutional network (DHGCN) to acquire deep representations from EHC network and functional indicators of ROIs. In contrast to conventional hypergraph convolutional networks designed for undirected hypergraphs, DHGCN is specifically tailored to handle directed hypergraph data structures. Moreover, unlike existing methods that primarily focus on fMRI time series, our proposed DHGCN model also incorporates multiple functional indicators, providing a robust framework for feature learning. Finally, deep representations generated via DHGCN, combined with demographic factors, are used for major depressive disorder (MDD) identification. Experimental results demonstrate that the proposed framework outperforms both FC and undirected hyper-connectivity models, as well as surpassing other state-of-the-art methods. The identification of EHC abnormalities through our framework can enhance the analysis of brain function in individuals with MDD.

Dynamic functional network connectivity analysis in schizophrenia based on a spatiotemporal CPD framework

Objective.Dynamic functional network connectivity (dFNC), based on data-driven group independent component (IC) analysis, is an important avenue for investigating underlying patterns of certain brain diseases such as schizophrenia. Canonical polyadic decomposition (CPD) of a higher-way dynamic functional connectivity tensor, can offer an innovative spatiotemporal framework to accurately characterize potential dynamic spatial and temporal fluctuations. Since multi-subject dFNC data from sliding-window analysis are also naturally a higher-order tensor, we propose an innovative sparse and low-rank CPD (SLRCPD) for the three-way dFNC tensor to excavate significant dynamic spatiotemporal aberrant changes in schizophrenia.Approach.The proposed SLRCPD approach imposes two constraints. First, the L1regularization on spatial modules is applied to extract sparse but significant dynamic connectivity and avoid overfitting the model. Second, low-rank constraint is added on time-varying weights to enhance the temporal state clustering quality. Shared dynamic spatial modules, group-specific dynamic spatial modules and time-varying weights can be extracted by SLRCPD. The strength of connections within- and between-IC networks and connection contribution are proposed to inspect the spatial modules. K-means clustering and classification are further conducted to explore temporal group difference.Main results.82 subject resting-state functional magnetic resonance imaging (fMRI) dataset and opening Center for Biomedical Research Excellence (COBRE) schizophrenia dataset both containing schizophrenia patients (SZs) and healthy controls (HCs) were utilized in our work. Three typical dFNC patterns between different brain functional regions were obtained. Compared to the spatial modules of HCs, the aberrant connections among auditory network, somatomotor, visual, cognitive control and cerebellar networks in 82 subject dataset and COBRE dataset were detected. Four temporal states reveal significant differences between SZs and HCs for these two datasets. Additionally, the accuracy values for SZs and HCs classification based on time-varying weights are larger than 0.96.Significance.This study significantly excavates spatio-temporal patterns for schizophrenia disease.

Deep Fusion of Multi-Template Using Spatio-Temporal Weighted Multi-Hypergraph Convolutional Networks for Brain Disease Analysis

Conventional functional connectivity network (FCN) based on resting-state fMRI (rs-fMRI) can only reflect the relationship between pairwise brain regions. Thus, the hyper-connectivity network (HCN) has been widely used to reveal high-order interactions among multiple brain regions. However, existing HCN models are essentially spatial HCN, which reflect the spatial relevance of multiple brain regions, but ignore the temporal correlation among multiple time points. Furthermore, the majority of HCN construction and learning frameworks are limited to using a single template, while the multi-template carries richer information. To address these issues, we first employ multiple templates to parcellate the rs-fMRI into different brain regions. Then, based on the multi-template data, we propose a spatio-temporal weighted HCN (STW-HCN) to capture more comprehensive high-order temporal and spatial properties of brain activity. Next, a novel deep fusion model of multi-template called spatio-temporal weighted multi-hypergraph convolutional network (STW-MHGCN) is proposed to fuse the STW-HCN of multiple templates, which extracts the deep interrelation information between different templates. Finally, we evaluate our method on the ADNI-2 and ABIDE-I datasets for mild cognitive impairment (MCI) and autism spectrum disorder (ASD) analysis. Experimental results demonstrate that the proposed method is superior to the state-of-the-art approaches in MCI and ASD classification, and the abnormal spatio-temporal hyper-edges discovered by our method have significant significance for the brain abnormalities analysis of MCI and ASD.

Enhancing EEG and sEMG Fusion Decoding Using a Multi-Scale Parallel Convolutional Network With Attention Mechanism

Electroencephalography (EEG) and surface electromyography (sEMG) have been widely used in the rehabilitation training of motor function. However, EEG signals have poor user adaptability and low classification accuracy in practical applications, and sEMG signals are susceptible to abnormalities such as muscle fatigue and weakness, resulting in reduced stability. To improve the accuracy and stability of interactive training recognition systems, we propose a novel approach called the Attention Mechanism-based Multi-Scale Parallel Convolutional Network (AM-PCNet) for recognizing and decoding fused EEG and sEMG signals. Firstly, we design an experimental scheme for the synchronous collection of EEG and sEMG signals and propose an ERP-WTC analysis method for channel screening of EEG signals. Then, the AM-PCNet network is designed to extract the time-domain, frequency-domain, and mixed-domain information of the EEG and sEMG fusion spectrogram images, and the attention mechanism is introduced to extract more fine-grained multi-scale feature information of the EEG and sEMG signals. Experiments on datasets obtained in the laboratory have shown that the average accuracy of EEG and sEMG fusion decoding is 96.62%. The accuracy is significantly improved compared with the classification performance of single-mode signals. When the muscle fatigue level reaches 50% and 90%, the accuracy is 92.84% and 85.29%, respectively. This study indicates that using this model to fuse EEG and sEMG signals can improve the accuracy and stability of hand rehabilitation training for patients.

Mapping Multi-Modal Brain Connectome for Brain Disorder Diagnosis via Cross-Modal Mutual Learning

Recently, the study of multi-modal brain connectome has recorded a tremendous increase and facilitated the diagnosis of brain disorders. In this paradigm, functional and structural networks, e.g., functional and structural connectivity derived from fMRI and DTI, are in some manner interacted but are not necessarily linearly related. Accordingly, there remains a great challenge to leverage complementary information for brain connectome analysis. Recently, Graph Convolutional Networks (GNN) have been widely applied to the fusion of multi-modal brain connectome. However, most existing GNN methods fail to couple inter-modal relationships. In this regard, we propose a Cross-modal Graph Neural Network (Cross-GNN) that captures inter-modal dependencies through dynamic graph learning and mutual learning. Specifically, the inter-modal representations are attentively coupled into a compositional space for reasoning inter-modal dependencies. Additionally, we investigate mutual learning in explicit and implicit ways: (1) Cross-modal representations are obtained by cross-embedding explicitly based on the inter-modal correspondence matrix. (2) We propose a cross-modal distillation method to implicitly regularize latent representations with cross-modal semantic contexts. We carry out statistical analysis on the attentively learned correspondence matrices to evaluate inter-modal relationships for associating disease biomarkers. Our extensive experiments on three datasets demonstrate the superiority of our proposed method for disease diagnosis with promising prediction performance and multi-modal connectome biomarker location.

Dynamic functional connectivity analysis with temporal convolutional network for attention deficit/hyperactivity disorder identification

Introduction

Dynamic functional connectivity (dFC), which can capture the abnormality of brain activity over time in resting-state functional magnetic resonance imaging (rs-fMRI) data, has a natural advantage in revealing the abnormal mechanism of brain activity in patients with Attention Deficit/Hyperactivity Disorder (ADHD). Several deep learning methods have been proposed to learn dynamic changes from rs-fMRI for FC analysis, and achieved superior performance than those using static FC. However, most existing methods only consider dependencies of two adjacent timestamps, which is limited when the change is related to the course of many timestamps.

Methods

In this paper, we propose a novel Temporal Dependence neural Network (TDNet) for FC representation learning and temporal-dependence relationship tracking from rs-fMRI time series for automated ADHD identification. Specifically, we first partition rs-fMRI time series into a sequence of consecutive and non-overlapping segments. For each segment, we design an FC generation module to learn more discriminative representations to construct dynamic FCs. Then, we employ the Temporal Convolutional Network (TCN) to efficiently capture long-range temporal patterns with dilated convolutions, followed by three fully connected layers for disease prediction.

Results

As the results, we found that considering the dynamic characteristics of rs-fMRI time series data is beneficial to obtain better diagnostic performance. In addition, dynamic FC networks generated in a data-driven manner are more informative than those constructed by Pearson correlation coefficients.

Discussion

We validate the effectiveness of the proposed approach through extensive experiments on the public ADHD-200 database, and the results demonstrate the superiority of the proposed model over state-of-the-art methods in ADHD identification.

Data-driven multimodal fusion: approaches and applications in psychiatric research

In the era of big data, where vast amounts of information are being generated and collected at an unprecedented rate, there is a pressing demand for innovative data-driven multi-modal fusion methods. These methods aim to integrate diverse neuroimaging perspectives to extract meaningful insights and attain a more comprehensive understanding of complex psychiatric disorders. However, analyzing each modality separately may only reveal partial insights or miss out on important correlations between different types of data. This is where data-driven multi-modal fusion techniques come into play. By combining information from multiple modalities in a synergistic manner, these methods enable us to uncover hidden patterns and relationships that would otherwise remain unnoticed. In this paper, we present an extensive overview of data-driven multimodal fusion approaches with or without prior information, with specific emphasis on canonical correlation analysis and independent component analysis. The applications of such fusion methods are wide-ranging and allow us to incorporate multiple factors such as genetics, environment, cognition, and treatment outcomes across various brain disorders. After summarizing the diverse neuropsychiatric magnetic resonance imaging fusion applications, we further discuss the emerging neuroimaging analyzing trends in big data, such as N-way multimodal fusion, deep learning approaches, and clinical translation. Overall, multimodal fusion emerges as an imperative approach providing valuable insights into the underlying neural basis of mental disorders, which can uncover subtle abnormalities or potential biomarkers that may benefit targeted treatments and personalized medical interventions.

A comprehensive survey of complex brain network representation

Recent years have shown great merits in utilizing neuroimaging data to understand brain structural and functional changes, as well as its relationship to different neurodegenerative diseases and other clinical phenotypes. Brain networks, derived from different neuroimaging modalities, have attracted increasing attention due to their potential to gain system-level insights to characterize brain dynamics and abnormalities in neurological conditions. Traditional methods aim to pre-define multiple topological features of brain networks and relate these features to different clinical measures or demographical variables. With the enormous successes in deep learning techniques, graph learning methods have played significant roles in brain network analysis. In this survey, we first provide a brief overview of neuroimaging-derived brain networks. Then, we focus on presenting a comprehensive overview of both traditional methods and state-of-the-art deep-learning methods for brain network mining. Major models, and objectives of these methods are reviewed within this paper. Finally, we discuss several promising research directions in this field.

Autism spectrum disorder diagnosis based on deep unrolling-based spatial constraint representation

Accurate diagnosis of autism spectrum disorder (ASD) is crucial for effective treatment and prognosis. Functional brain networks (FBNs) constructed from functional magnetic resonance imaging (fMRI) have become a popular tool for ASD diagnosis. However, existing model-driven approaches used to construct FBNs lack the ability to capture potential non-linear relationships between data and labels. Moreover, most existing studies treat the FBNs construction and disease classification as separate steps, leading to large inter-subject variability in the estimated FBNs and reducing the statistical power of subsequent group comparison. To address these limitations, we propose a new approach to FBNs construction called the deep unrolling-based spatial constraint representation (DUSCR) model and integrate it with a convolutional classifier to create an end-to-end framework for ASD recognition. Specifically, the model spatial constraint representation (SCR) is solved using a proximal gradient descent algorithm, and we unroll it into deep networks using the deep unrolling algorithm. Classification is then performed using a convolutional prototype learning model. We evaluated the effectiveness of the proposed method on the ABIDE I dataset and observed a significant improvement in model performance and classification accuracy. The resting state fMRI images are preprocessed into time series data and 3D coordinates of each region of interest. The data are fed into the DUSCR model, a model for building functional brain networks using deep learning instead of traditional models, that we propose, and then the outputs are fed into the convolutional classifier with prototype learning to determine whether the patient has ASD disease.

Global Transformer and Dual Local Attention Network via Deep-Shallow Hierarchical Feature Fusion for Retinal Vessel Segmentation

Clinically, retinal vessel segmentation is a significant step in the diagnosis of fundus diseases. However, recent methods generally neglect the difference of semantic information between deep and shallow features, which fail to capture the global and local characterizations in fundus images simultaneously, resulting in the limited segmentation performance for fine vessels. In this article, a global transformer (GT) and dual local attention (DLA) network via deep-shallow hierarchical feature fusion (GT-DLA-dsHFF) are investigated to solve the above limitations. First, the GT is developed to integrate the global information in the retinal image, which effectively captures the long-distance dependence between pixels, alleviating the discontinuity of blood vessels in the segmentation results. Second, DLA, which is constructed using dilated convolutions with varied dilation rates, unsupervised edge detection, and squeeze-excitation block, is proposed to extract local vessel information, consolidating the edge details in the segmentation result. Finally, a novel deep-shallow hierarchical feature fusion (dsHFF) algorithm is studied to fuse the features in different scales in the deep learning framework, respectively, which can mitigate the attenuation of valid information in the process of feature fusion. We verified the GT-DLA-dsHFF on four typical fundus image datasets. The experimental results demonstrate our GT-DLA-dsHFF achieves superior performance against the current methods and detailed discussions verify the efficacy of the proposed three modules. Segmentation results of diseased images show the robustness of our proposed GT-DLA-dsHFF. Implementation codes will be available on https://github.com/YangLibuaa/GT-DLA-dsHFF.

Learning to Fuse Multiple Brain Functional Networks for Automated Autism Identification

Functional connectivity network (FCN) has become a popular tool to identify potential biomarkers for brain dysfunction, such as autism spectrum disorder (ASD). Due to its importance, researchers have proposed many methods to estimate FCNs from resting-state functional MRI (rs-fMRI) data. However, the existing FCN estimation methods usually only capture a single relationship between brain regions of interest (ROIs), e.g., linear correlation, nonlinear correlation, or higher-order correlation, thus failing to model the complex interaction among ROIs in the brain. Additionally, such traditional methods estimate FCNs in an unsupervised way, and the estimation process is independent of the downstream tasks, which makes it difficult to guarantee the optimal performance for ASD identification. To address these issues, in this paper, we propose a multi-FCN fusion framework for rs-fMRI-based ASD classification. Specifically, for each subject, we first estimate multiple FCNs using different methods to encode rich interactions among ROIs from different perspectives. Then, we use the label information (ASD vs. healthy control (HC)) to learn a set of fusion weights for measuring the importance/discrimination of those estimated FCNs. Finally, we apply the adaptively weighted fused FCN on the ABIDE dataset to identify subjects with ASD from HCs. The proposed FCN fusion framework is straightforward to implement and can significantly improve diagnostic accuracy compared to traditional and state-of-the-art methods.

BolT: Fused window transformers for fMRI time series analysis

Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.

Personalized Functional Connectivity Based Spatio-Temporal Aggregated Attention Network for MCI Identification

Functional connectivity (FC) networks deri- ved from resting-state magnetic resonance image (rs-fMRI) are effective biomarkers for identifying mild cognitive impairment (MCI) patients. However, most FC identification methods simply extract features from group-averaged brain templates, and neglect inter-subject functional variations. Furthermore, the existing methods generally concentrate on spatial correlation among brain regions, resulting in the inefficient capture of the fMRI temporal features. To address these limitations, we propose a novel personalized functional connectivity based dual-branch graph neural network with spatio-temporal aggregated attention (PFC-DBGNN-STAA) for MCI identification. Specifically, a personalized functional connectivity (PFC) template is firstly constructed to align 213 functional regions across samples and generate discriminative individualized FC features. Secondly, a dual-branch graph neural network (DBGNN) is conducted by aggregating features from the individual- and group-level templates with the cross-template FC, which is beneficial to improve the feature discrimination by considering dependency between templates. Finally, a spatio-temporal aggregated attention (STAA) module is investigated to capture the spatial and dynamic relationships between functional regions, which solves the limitation of insufficient temporal information utilization. We evaluate our proposed method on 442 samples from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, and achieve the accuracies of 90.1%, 90.3%, 83.3% for normal control (NC) vs. early MCI (EMCI), EMCI vs. late MCI (LMCI), and NC vs. EMCI vs. LMCI classification tasks, respectively, indicating that our method boosts MCI identification performance and outperforms state-of-the-art methods.

Dynamic multi-site graph convolutional network for autism spectrum disorder identification

Multi-site learning has attracted increasing interests in autism spectrum disorder (ASD) identification tasks by its efficacy on capturing data heterogeneity of neuroimaging taken from different medical sites. However, existing multi-site graph convolutional network (MSGCN) often ignores the correlations between different sites, and may obtain suboptimal identification results. Moreover, current feature extraction methods characterizing temporal variations of functional magnetic resonance imaging (fMRI) signals require the time series to be of the same length and cannot be directly applied to multi-site fMRI datasets. To address these problems, we propose a dual graph based dynamic multi-site graph convolutional network (DG-DMSGCN) for multi-site ASD identification. First, a sliding-window dual-graph convolutional network (SW-DGCN) is introduced for feature extraction, simultaneously capturing temporal and spatial features of fMRI data with different series lengths. Then we aggregate the features extracted from multiple medical sites through a novel dynamic multi-site graph convolutional network (DMSGCN), which effectively considers the correlations between different sites and is beneficial to improve identification performance. We evaluate the proposed DG-DMSGCN on public ABIDE I dataset containing data from 17 medical sites. The promising results obtained by our framework outperforms the state-of-the-art methods with increase in identification accuracy, indicating that it has a potential clinical prospect for practical ASD diagnosis. Our codes are available on https://github.com/Junling-Du/DG-DMSGCN.

MCFSA-Net: A multi-scale channel fusion and spatial activation network for retinal vessel segmentation

As the only vascular tissue that can be directly viewed in vivo, retinal vessels are medically important in assisting the diagnosis of ocular and cardiovascular diseases. They generally appear as different morphologies and uneven thickness in fundus images. Therefore, the single-scale segmentation method may fail to capture abundant morphological features, suffering from the deterioration in vessel segmentation, especially for tiny vessels. To alleviate this issue, we propose a multi-scale channel fusion and spatial activation network (MCFSA-Net) for retinal vessel segmentation with emphasis on tiny ones. Specifically, the Hybrid Convolution-DropBlock (HC-Drop) is first used to extract deep features of vessels and construct multi-scale feature maps by progressive down-sampling. Then, the Channel Cooperative Attention Fusion (CCAF) module is designed to handle different morphological vessels in a multi-scale manner. Finally, the Global Spatial Activation (GSA) module is introduced to aggregate global feature information for improving the attention on tiny vessels in the spatial domain and realizing effective segmentation for them. Experiments are carried out on three datasets including DRIVE, CHASE_DB1, and STARE. Our retinal vessel segmentation method achieves Accuracy of 96.95%, 97.57%, and 97.83%, and F1 score of 82.67%, 81.82%, and 82.95% in the above datasets, respectively. Qualitative and quantitative analysis show that the proposed method outperforms current advanced vessel segmentation methods, especially for tiny vessels.

Multicenter and Multichannel Pooling GCN for Early AD Diagnosis Based on Dual-Modality Fused Brain Network

For significant memory concern (SMC) and mild cognitive impairment (MCI), their classification performance is limited by confounding features, diverse imaging protocols, and limited sample size. To address the above limitations, we introduce a dual-modality fused brain connectivity network combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), and propose three mechanisms in the current graph convolutional network (GCN) to improve classifier performance. First, we introduce a DTI-strength penalty term for constructing functional connectivity networks. Stronger structural connectivity and bigger structural strength diversity between groups provide a higher opportunity for retaining connectivity information. Second, a multi-center attention graph with each node representing a subject is proposed to consider the influence of data source, gender, acquisition equipment, and disease status of those training samples in GCN. The attention mechanism captures their different impacts on edge weights. Third, we propose a multi-channel mechanism to improve filter performance, assigning different filters to features based on feature statistics. Applying those nodes with low-quality features to perform convolution would also deteriorate filter performance. Therefore, we further propose a pooling mechanism, which introduces the disease status information of those training samples to evaluate the quality of nodes. Finally, we obtain the final classification results by inputting the multi-center attention graph into the multi-channel pooling GCN. The proposed method is tested on three datasets (i.e., an ADNI 2 dataset, an ADNI 3 dataset, and an in-house dataset). Experimental results indicate that the proposed method is effective and superior to other related algorithms, with a mean classification accuracy of 93.05% in our binary classification tasks. Our code is available at: https://github.com/Xuegang-S.

A ViT-AMC Network With Adaptive Model Fusion and Multiobjective Optimization for Interpretable Laryngeal Tumor Grading From Histopathological Images

The tumor grading of laryngeal cancer pathological images needs to be accurate and interpretable. The deep learning model based on the attention mechanism-integrated convolution (AMC) block has good inductive bias capability but poor interpretability, whereas the deep learning model based on the vision transformer (ViT) block has good interpretability but weak inductive bias ability. Therefore, we propose an end-to-end ViT-AMC network (ViT-AMCNet) with adaptive model fusion and multiobjective optimization that integrates and fuses the ViT and AMC blocks. However, existing model fusion methods often have negative fusion: 1). There is no guarantee that the ViT and AMC blocks will simultaneously have good feature representation capability. 2). The difference in feature representations learning between the ViT and AMC blocks is not obvious, so there is much redundant information in the two feature representations. Accordingly, we first prove the feasibility of fusing the ViT and AMC blocks based on Hoeffding's inequality. Then, we propose a multiobjective optimization method to solve the problem that ViT and AMC blocks cannot simultaneously have good feature representation. Finally, an adaptive model fusion method integrating the metrics block and the fusion block is proposed to increase the differences between feature representations and improve the deredundancy capability. Our methods improve the fusion ability of ViT-AMCNet, and experimental results demonstrate that ViT-AMCNet significantly outperforms state-of-the-art methods. Importantly, the visualized interpretive maps are closer to the region of interest of concern by pathologists, and the generalization ability is also excellent. Our code is publicly available at https://github.com/Baron-Huang/ViT-AMCNet.

A spatio-temporal decomposition framework for dynamic functional connectivity in the human brain

Much recent attention has been directed toward investigating the spatial and temporal organization of brain dynamics, but the rules which constrain the variation of spatio-temporal organization in functional connectivity under different brain states remain unclear. Here, we developed a novel computational approach based on tensor decomposition and regularization to represent dynamic functional connectivity as a linear combination of dynamic modules and time-varying weights. In this approach, dynamic modules represent co-activating functional connectivity patterns, and time-varying weights represent the temporal expression of dynamic modules. We applied this dynamic decomposition model (DDM) on a resting-state fMRI dataset and found that whole-brain dynamic functional connectivity can be decomposed as a linear combination of eight dynamic modules which we summarize as 'high order modules' and 'primary-high order modules', according to their spatial attributes and correspondence with existing intrinsic functional brain networks. By clustering the time-varying weights, we identified five brain states including three major states and two minor states. We found that state transitions mainly occurred between the three major states, and that temporal variation of dynamic modules may contribute to brain state transitions. We then conceptualized the variability of weights as the flexibility of the corresponding dynamic modules and found that different dynamic modules exhibit different amounts of flexibility and contribute to different cognitive measures. Finally, we applied DDM to a schizophrenia resting-state fMRI dataset and found that atypical flexibility of dynamic modules correlates with impaired cognitive flexibility in schizophrenia. Overall, this work provides a quantitative framework that characterizes temporal variation in the topology of dynamic functional connectivity.

Deep Spatio-Temporal Attention-based Recurrent Network from Dynamic Adaptive Functional Connectivity for MCI Identification

Most existing methods of constructing dynamic functional connectivity (dFC) network obtain the connectivity strength via the sliding window correlation (SWC) method, which estimates the connectivity strength at each time segment, rather than at each time point, and thus is difficult to produce accurate dFC network due to the influence of the window type and window width. Furthermore, the deep learning methods may not capture the discriminative spatio-temporal information that is closely related to disease, thus impacting the performance of (mild cognitive impairment) MCI identification. In this paper, a novel spatio-temporal attention-based bidirectional gated recurrent unit (STA-BiGRU) network is proposed to extract inherent spatio-temporal information from a dynamic adaptive functional connectivity (dAFC) network for MCI diagnosis. Specifically, we adopt a group lasso-based Kalman filter algorithm to obtain the dAFC network with more accurate connectivity strength at each time step. Then a spatial attention module with self-attention and a temporal attention module with multiple temporal attention vectors are incorporated into the BiGRU network to extract more discriminative disease-related spatio-temporal information. Finally, the spatio-temporal regularizations are employed to better guide the attention learning of STA-BiGRU network to enhance the robustness of the deep network. Experimental results show that the proposed framework achieves mean accuracies of 90.2%, 90.0%, and 81.5%, respectively, for three MCI classification tasks. This study provides a more effective deep spatio-temporal attention-based recurrent network and obtains good performance and interpretability of deep learning for psychiatry diagnosis research.

Dual Encoder-Based Dynamic-Channel Graph Convolutional Network With Edge Enhancement for Retinal Vessel Segmentation

Retinal vessel segmentation with deep learning technology is a crucial auxiliary method for clinicians to diagnose fundus diseases. However, the deep learning approaches inevitably lose the edge information, which contains spatial features of vessels while performing down-sampling, leading to the limited segmentation performance of fine blood vessels. Furthermore, the existing methods ignore the dynamic topological correlations among feature maps in the deep learning framework, resulting in the inefficient capture of the channel characterization. To address these limitations, we propose a novel dual encoder-based dynamic-channel graph convolutional network with edge enhancement (DE-DCGCN-EE) for retinal vessel segmentation. Specifically, we first design an edge detection-based dual encoder to preserve the edge of vessels in down-sampling. Secondly, we investigate a dynamic-channel graph convolutional network to map the image channels to the topological space and synthesize the features of each channel on the topological map, which solves the limitation of insufficient channel information utilization. Finally, we study an edge enhancement block, aiming to fuse the edge and spatial features in the dual encoder, which is beneficial to improve the accuracy of fine blood vessel segmentation. Competitive experimental results on five retinal image datasets validate the efficacy of the proposed DE-DCGCN-EE, which achieves more remarkable segmentation results against the other state-of-the-art methods, indicating its potential clinical application.

Parkinson's Disease Classification and Clinical Score Regression via United Embedding and Sparse Learning From Longitudinal Data

Parkinson's disease (PD) is known as an irreversible neurodegenerative disease that mainly affects the patient's motor system. Early classification and regression of PD are essential to slow down this degenerative process from its onset. In this article, a novel adaptive unsupervised feature selection approach is proposed by exploiting manifold learning from longitudinal multimodal data. Classification and clinical score prediction are performed jointly to facilitate early PD diagnosis. Specifically, the proposed approach performs united embedding and sparse regression, which can determine the similarity matrices and discriminative features adaptively. Meanwhile, we constrain the similarity matrix among subjects and exploit the l_2,p norm to conduct sparse adaptive control for obtaining the intrinsic information of the multimodal data structure. An effective iterative optimization algorithm is proposed to solve this problem. We perform abundant experiments on the Parkinson's Progression Markers Initiative (PPMI) data set to verify the validity of the proposed approach. The results show that our approach boosts the performance on the classification and clinical score regression of longitudinal data and surpasses the state-of-the-art approaches.

A Multi-Modal and Multi-Atlas Integrated Framework for Identification of Mild Cognitive Impairment

BACKGROUND

Multi-modal neuroimaging with appropriate atlas is vital for effectively differentiating mild cognitive impairment (MCI) from healthy controls (HC).

METHODS

The resting-state functional magnetic resonance imaging (rs-fMRI) and structural MRI (sMRI) of 69 MCI patients and 61 HC subjects were collected. Then, the gray matter volumes obtained from the sMRI and Hurst exponent (HE) values calculated from rs-fMRI data in the Automated Anatomical Labeling (AAL-90), Brainnetome (BN-246), Harvard-Oxford (HOA-112) and AAL3-170 atlases were extracted, respectively. Next, these characteristics were selected with a minimal redundancy maximal relevance algorithm and a sequential feature collection method in single or multi-modalities, and only the optimal features were retained after this procedure. Lastly, the retained characteristics were served as the input features for the support vector machine (SVM)-based method to classify MCI patients, and the performance was estimated with a leave-one-out cross-validation (LOOCV).

RESULTS

Our proposed method obtained the best 92.00% accuracy, 94.92% specificity and 89.39% sensitivity with the sMRI in AAL-90 and the fMRI in HOA-112 atlas, which was much better than using the single-modal or single-atlas features.

CONCLUSION

The results demonstrated that the multi-modal and multi-atlas integrated method could effectively recognize MCI patients, which could be extended into various neurological and neuropsychiatric diseases.

Diagnosis of Amnesic Mild Cognitive Impairment Using MGS-WBC and VGBN-LM Algorithms

Computer-aided diagnosis (CAD) has undergone rapid development with the advent of advanced neuroimaging and machine learning methods. Nevertheless, how to extract discriminative features from the limited and high-dimensional data is not ideal, especially for amnesic mild cognitive impairment (aMCI) data based on resting-state functional magnetic resonance imaging (rs-fMRI). Furthermore, a robust and reliable system for aMCI detection is conducive to timely detecting and screening subjects at a high risk of Alzheimer's disease (AD). In this scenario, we first develop the mask generation strategy based on within-class and between-class criterion (MGS-WBC), which primarily aims at reducing data redundancy and excavating multiscale features of the brain. Concurrently, vector generation for brain networks based on Laplacian matrix (VGBN-LM) is presented to obtain the global features of the functional network. Finally, all multiscale features are fused to further improve the diagnostic performance of aMCI. Typical classifiers for small data learning, such as naive Bayesian (NB), linear discriminant analysis (LDA), logistic regression (LR), and support vector machines (SVMs), are adopted to evaluate the diagnostic performance of aMCI. This study helps to reveal discriminative neuroimaging features, and outperforms the state-of-the-art methods, providing new insights for the intelligent construction of CAD system of aMCI.

Feature aggregation graph convolutional network based on imaging genetic data for diagnosis and pathogeny identification of Alzheimer's disease

The roles of brain regions activities and gene expressions in the development of Alzheimer's disease (AD) remain unclear. Existing imaging genetic studies usually has the problem of inefficiency and inadequate fusion of data. This study proposes a novel deep learning method to efficiently capture the development pattern of AD. First, we model the interaction between brain regions and genes as node-to-node feature aggregation in a brain region-gene network. Second, we propose a feature aggregation graph convolutional network (FAGCN) to transmit and update the node feature. Compared with the trivial graph convolutional procedure, we replace the input from the adjacency matrix with a weight matrix based on correlation analysis and consider common neighbor similarity to discover broader associations of nodes. Finally, we use a full-gradient saliency graph mechanism to score and extract the pathogenetic brain regions and risk genes. According to the results, FAGCN achieved the best performance among both traditional and cutting-edge methods and extracted AD-related brain regions and genes, providing theoretical and methodological support for the research of related diseases.

Improved Multiple Vector Representations of Images and Robust Dictionary Learning

Each sparse representation classifier has different classification accuracy for different samples. It is difficult to achieve good performance with a single feature classification model. In order to balance the large-scale information and global features of images, a robust dictionary learning method based on image multi-vector representation is proposed in this paper. First, this proposed method generates a reasonable virtual image for the original image and obtains the multi-vector representation of all images. Second, the same dictionary learning algorithm is used for each vector representation to obtain multiple sets of image features. The proposed multi-vector representation can provide a good global understanding of the whole image contour and increase the content of dictionary learning. Last, the weighted fusion algorithm is used to classify the test samples. The introduction of influencing factors and the automatic adjustment of the weights of each classifier in the final decision results have a significant indigenous effect on better extracting image features. The study conducted experiments on the proposed algorithm on a number of widely used image databases. A large number of experimental results show that it effectively improves the accuracy of image classification. At the same time, to fully dig and exploit possible representation diversity might be a better way to lead to potential various appearances and high classification accuracy concerning the image.

A dynamic graph convolutional neural network framework reveals new insights into connectome dysfunctions in ADHD

The pathological mechanism of attention deficit hyperactivity disorder (ADHD) is incompletely specified, which leads to difficulty in precise diagnosis. Functional magnetic resonance imaging (fMRI) has emerged as a common neuroimaging technique for studying the brain functional connectome. Most existing methods that have either ignored or simply utilized graph structure, do not fully leverage the potentially important topological information which may be useful in characterizing brain disorders. There is a crucial need for designing novel and efficient approaches which can capture such information. To this end, we propose a new dynamic graph convolutional network (dGCN), which is trained with sparse brain regional connections from dynamically calculated graph features. We also develop a novel convolutional readout layer to improve graph representation. Our extensive experimental analysis demonstrates significantly improved performance of dGCN for ADHD diagnosis compared with existing machine learning and deep learning methods. Visualizations of the salient regions of interest (ROIs) and connectivity based on informative features learned by our model show that the identified functional abnormalities mainly involve brain regions in temporal pole, gyrus rectus, and cerebellar gyri from temporal lobe, frontal lobe, and cerebellum, respectively. A positive correlation was further observed between the identified connectomic abnormalities and ADHD symptom severity. The proposed dGCN model shows great promise in providing a functional network-based precision diagnosis of ADHD and is also broadly applicable to brain connectome-based study of mental disorders.

Multi-Scale Graph Representation Learning for Autism Identification With Functional MRI

Resting-state functional MRI (rs-fMRI) has been widely used for the early diagnosis of autism spectrum disorder (ASD). With rs-fMRI, the functional connectivity networks (FCNs) are usually constructed for representing each subject, with each element representing the pairwise relationship between brain region-of-interests (ROIs). Previous studies often first extract handcrafted network features (such as node degree and clustering coefficient) from FCNs and then construct a prediction model for ASD diagnosis, which largely requires expert knowledge. Graph convolutional networks (GCNs) have recently been employed to jointly perform FCNs feature extraction and ASD identification in a data-driven manner. However, existing studies tend to focus on the single-scale topology of FCNs by using one single atlas for ROI partition, thus ignoring potential complementary topology information of FCNs at different spatial scales. In this paper, we develop a multi-scale graph representation learning (MGRL) framework for rs-fMRI based ASD diagnosis. The MGRL consists of three major components: (1) multi-scale FCNs construction using multiple brain atlases for ROI partition, (2) FCNs representation learning via multi-scale GCNs, and (3) multi-scale feature fusion and classification for ASD diagnosis. The proposed MGRL is evaluated on 184 subjects from the public Autism Brain Imaging Data Exchange (ABIDE) database with rs-fMRI scans. Experimental results suggest the efficacy of our MGRL in FCN feature extraction and ASD identification, compared with several state-of-the-art methods.