Structured and Sparse Canonical Correlation Analysis as a Brain-Wide Multi-Modal Data Fusion Approach

Structure-adaptive canonical correlation analysis for microbiome multi-omics data

Sparse canonical correlation analysis (sCCA) has been a useful approach for integrating different high-dimensional datasets by finding a subset of correlated features that explain the most correlation in the data. In the context of microbiome studies, investigators are always interested in knowing how the microbiome interacts with the host at different molecular levels such as genome, methylol, transcriptome, metabolome and proteome. sCCA provides a simple approach for exploiting the correlation structure among multiple omics data and finding a set of correlated omics features, which could contribute to understanding the host-microbiome interaction. However, existing sCCA methods do not address compositionality, and its application to microbiome data is thus not optimal. This paper proposes a new sCCA framework for integrating microbiome data with other high-dimensional omics data, accounting for the compositional nature of microbiome sequencing data. It also allows integrating prior structure information such as the grouping structure among bacterial taxa by imposing a "soft" constraint on the coefficients through varying penalization strength. As a result, the method provides significant improvement when the structure is informative while maintaining robustness against a misspecified structure. Through extensive simulation studies and real data analysis, we demonstrate the superiority of the proposed framework over the state-of-the-art approaches.

A Biometric Identification for Multi-Modal Biomedical Signals in Geriatric Care

With the acceleration of global population aging, the elderly have an increasing demand for home care and nursing institutions, and the significance of health prevention and management of the elderly has become increasingly prominent. In this context, we propose a biometric recognition method for multi-modal biomedical signals. This article focuses on three key signals that can be picked up by wearable devices: ECG, PPG, and breath (RESP). The RESP signal is introduced into the existing two-mode signal identification for multi-mode identification. Firstly, the features of the signal in the time-frequency domain are extracted. To represent deep features in a low-dimensional feature space and expedite authentication tasks, PCA and LDA are employed for dimensionality reduction. MCCA is used for feature fusion, and SVM is used for identification. The accuracy and performance of the system were evaluated using both public data sets and self-collected data sets, with an accuracy of more than 99.5%. The experimental data fully show that this method significantly improves the accuracy of identity recognition. In the future, combined with the signal monitoring function of wearable devices, it can quickly identify individual elderly people with abnormal conditions, provide safer and more efficient medical services for the elderly, and relieve the pressure on medical resources.

Alzheimer's disease diagnosis from multi-modal data via feature inductive learning and dual multilevel graph neural network

Multi-modal data can provide complementary information of Alzheimer's disease (AD) and its development from different perspectives. Such information is closely related to the diagnosis, prevention, and treatment of AD, and hence it is necessary and critical to study AD through multi-modal data. Existing learning methods, however, usually ignore the influence of feature heterogeneity and directly fuse features in the last stages. Furthermore, most of these methods only focus on local fusion features or global fusion features, neglecting the complementariness of features at different levels and thus not sufficiently leveraging information embedded in multi-modal data. To overcome these shortcomings, we propose a novel framework for AD diagnosis that fuses gene, imaging, protein, and clinical data. Our framework learns feature representations under the same feature space for different modalities through a feature induction learning (FIL) module, thereby alleviating the impact of feature heterogeneity. Furthermore, in our framework, local and global salient multi-modal feature interaction information at different levels is extracted through a novel dual multilevel graph neural network (DMGNN). We extensively validate the proposed method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and experimental results demonstrate our method consistently outperforms other state-of-the-art multi-modal fusion methods. The code is publicly available on the GitHub website. (https://github.com/xiankantingqianxue/MIA-code.git).

A novel interactive deep cascade spectral graph convolutional network with multi-relational graphs for disease prediction

Graph neural networks (GNNs) have recently grown in popularity for disease prediction. Existing GNN-based methods primarily build the graph topological structure around a single modality and combine it with other modalities to acquire feature representations of acquisitions. The complicated relationship in each modality, however, may not be well highlighted due to its specificity. Further, relatively shallow networks restrict adequate extraction of high-level features, affecting disease prediction performance. Accordingly, this paper develops a new interactive deep cascade spectral graph convolutional network with multi-relational graphs (IDCGN) for disease prediction tasks. Its crucial points lie in constructing multiple relational graphs and dual cascade spectral graph convolution branches with interaction (DCSGBI). Specifically, the former designs a pairwise imaging-based edge generator and a pairwise non-imaging-based edge generator from different modalities by devising two learnable networks, which adaptively capture graph structures and provide various views of the same acquisition to aid in disease diagnosis. Again, DCSGBI is established to enrich high-level semantic information and low-level details of disease data. It devises a cascade spectral graph convolution operator for each branch and incorporates the interaction strategy between different branches into the network, successfully forming a deep model and capturing complementary information from diverse branches. In this manner, more favorable and sufficient features are learned for a reliable diagnosis. Experiments on several disease datasets reveal that IDCGN exceeds state-of-the-art models and achieves promising results.

Modelling-based joint embedding of histology and genomics using canonical correlation analysis for breast cancer survival prediction

Traditional approaches to predicting breast cancer patients' survival outcomes were based on clinical subgroups, the PAM50 genes, or the histological tissue's evaluation. With the growth of multi-modality datasets capturing diverse information (such as genomics, histology, radiology and clinical data) about the same cancer, information can be integrated using advanced tools and have improved survival prediction. These methods implicitly exploit the key observation that different modalities originate from the same cancer source and jointly provide a complete picture of the cancer. In this work, we investigate the benefits of explicitly modelling multi-modality data as originating from the same cancer under a probabilistic framework. Specifically, we consider histology and genomics as two modalities originating from the same breast cancer under a probabilistic graphical model (PGM). We construct maximum likelihood estimates of the PGM parameters based on canonical correlation analysis (CCA) and then infer the underlying properties of the cancer patient, such as survival. Equivalently, we construct CCA-based joint embeddings of the two modalities and input them to a learnable predictor. Real-world properties of sparsity and graph-structures are captured in the penalized variants of CCA (pCCA) and are better suited for cancer applications. For generating richer multi-dimensional embeddings with pCCA, we introduce two novel embedding schemes that encourage orthogonality to generate more informative embeddings. The efficacy of our proposed prediction pipeline is first demonstrated via low prediction errors of the hidden variable and the generation of informative embeddings on simulated data. When applied to breast cancer histology and RNA-sequencing expression data from The Cancer Genome Atlas (TCGA), our model can provide survival predictions with average concordance-indices of up to 68.32% along with interpretability. We also illustrate how the pCCA embeddings can be used for survival analysis through Kaplan-Meier curves.

Homogeneous-Multiset-CCA-Based Brain Covariation and Contravariance Connectivity Network Modeling

Brain connectivity networks based on functional magnetic resonance imaging (fMRI) have expanded our understanding of brain functions in both healthy and diseased states. However, most current studies construct connectivity networks using averaged regional time courses with the strong assumption that the activities of voxels contained in each brain region are similar, ignoring their possible variations. Additionally, pairwise correlation analysis is often adopted with more attention to positive relationships, while joint interactions at the network level as well as anti-correlations are less investigated. In this paper, to provide a new strategy for regional activity representation and brain connectivity modeling, a novel homogeneous multiset canonical correlation analysis (HMCCA) model is proposed, which enforces sign constraints on the weights of voxels to guarantee homogeneity within each brain region. It is capable of obtaining regional representative signals and constructing covariation and contravariance networks simultaneously, at both group and subject levels. Validations on two sessions of fMRI data verified its reproducibility and reliability when dealing with brain connectivity networks. Further experiments on subjects with and without Parkinson's disease (PD) revealed significant alterations in brain connectivity patterns, which were further associated with clinical scores and demonstrated superior prediction ability, indicating its potential in clinical practice.

Multiview Regularized Discriminant Canonical Correlation Analysis: Sequential Extraction of Relevant Features From Multiblock Data

One of the important issues associated with real-life high-dimensional data analysis is how to extract significant and relevant features from multiview data. The multiset canonical correlation analysis (MCCA) is a well-known statistical method for multiview data integration. It finds a linear subspace that maximizes the correlations among different views. However, the existing methods to find the multiset canonical variables are computationally very expensive, which restricts the application of the MCCA in real-life big data analysis. The covariance matrix of each high-dimensional view may also suffer from the singularity problem due to the limited number of samples. Moreover, the MCCA-based existing feature extraction algorithms are, in general, unsupervised in nature. In this regard, a new supervised feature extraction algorithm is proposed, which integrates multimodal multidimensional data sets by solving maximal correlation problem of the MCCA. A new block matrix representation is introduced to reduce the computational complexity for computing the canonical variables of the MCCA. The analytical formulation enables efficient computation of the multiset canonical variables under supervised ridge regression optimization technique. It deals with the "curse of dimensionality" problem associated with high-dimensional data and facilitates the sequential generation of relevant features with significantly lower computational cost. The effectiveness of the proposed multiblock data integration algorithm, along with a comparison with other existing methods, is demonstrated on several benchmark and real-life cancer data.

Dysconnection and cognition in schizophrenia: A spectral dynamic causal modeling study

Schizophrenia (SZ) is a severe mental disorder characterized by failure of functional integration (aka dysconnection) across the brain. Recent functional connectivity (FC) studies have adopted functional parcellations to define subnetworks of large-scale networks, and to characterize the (dys)connection between them, in normal and clinical populations. While FC examines statistical dependencies between observations, model-based effective connectivity (EC) can disclose the causal influences that underwrite the observed dependencies. In this study, we investigated resting state EC within seven large-scale networks, in 66 SZ and 74 healthy subjects from a public dataset. The results showed that a remarkable 33% of the effective connections (among subnetworks) of the cognitive control network had been pathologically modulated in SZ. Further dysconnection was identified within the visual, default mode and sensorimotor networks of SZ subjects, with 24%, 20%, and 11% aberrant couplings. Overall, the proportion of discriminative connections was remarkably larger in EC (24%) than FC (1%) analysis. Subsequently, to study the neural correlates of impaired cognition in SZ, we conducted a canonical correlation analysis between the EC parameters and the cognitive scores of the patients. As such, the self-inhibitions of supplementary motor area and paracentral lobule (in the sensorimotor network) and the excitatory connection from parahippocampal gyrus to inferior temporal gyrus (in the cognitive control network) were significantly correlated with the social cognition, reasoning/problem solving and working memory capabilities of the patients. Future research can investigate the potential of whole-brain EC as a biomarker for diagnosis of brain disorders and for neuroimaging-based cognitive assessment.

A joint multi-modal learning method for early-stage knee osteoarthritis disease classification

Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Furthermore, a visual analysis of the important features in the multimodal data verified the relations among the modalities when classifying the grade of knee OA disease.

A multimodal fusion method for Alzheimer's disease based on DCT convolutional sparse representation

Introduction

The medical information contained in magnetic resonance imaging (MRI) and positron emission tomography (PET) has driven the development of intelligent diagnosis of Alzheimer's disease (AD) and multimodal medical imaging. To solve the problems of severe energy loss, low contrast of fused images and spatial inconsistency in the traditional multimodal medical image fusion methods based on sparse representation. A multimodal fusion algorithm for Alzheimer' s disease based on the discrete cosine transform (DCT) convolutional sparse representation is proposed.

Methods

The algorithm first performs a multi-scale DCT decomposition of the source medical images and uses the sub-images of different scales as training images, respectively. Different sparse coefficients are obtained by optimally solving the sub-dictionaries at different scales using alternating directional multiplication method (ADMM). Secondly, the coefficients of high-frequency and low-frequency subimages are inverse DCTed using an improved L1 parametric rule combined with improved spatial frequency novel sum-modified SF (NMSF) to obtain the final fused images.

Results and discussion

Through extensive experimental results, we show that our proposed method has good performance in contrast enhancement, texture and contour information retention.

A Review of Fusion Methods for Omics and Imaging Data

The development of omics data and biomedical images has greatly advanced the progress of precision medicine in diagnosis, treatment, and prognosis. The fusion of omics and imaging data, i.e., omics-imaging fusion, offers a new strategy for understanding complex diseases. However, due to a variety of issues such as the limited number of samples, high dimensionality of features, and heterogeneity of different data types, efficiently learning complementary or associated discriminative fusion information from omics and imaging data remains a challenge. Recently, numerous machine learning methods have been proposed to alleviate these problems. In this review, from the perspective of fusion levels and fusion methods, we first provide an overview of preprocessing and feature extraction methods for omics and imaging data, and comprehensively analyze and summarize the basic forms and variations of commonly used and newly emerging fusion methods, along with their advantages, disadvantages and the applicable scope. We then describe public datasets and compare experimental results of various fusion methods on the ADNI and TCGA datasets. Finally, we discuss future prospects and highlight remaining challenges in the field.

Multi-Modal Graph Learning for Disease Prediction

Benefiting from the powerful expressive capability of graphs, graph-based approaches have been popularly applied to handle multi-modal medical data and achieved impressive performance in various biomedical applications. For disease prediction tasks, most existing graph-based methods tend to define the graph manually based on specified modality (e.g., demographic information), and then integrated other modalities to obtain the patient representation by Graph Representation Learning (GRL). However, constructing an appropriate graph in advance is not a simple matter for these methods. Meanwhile, the complex correlation between modalities is ignored. These factors inevitably yield the inadequacy of providing sufficient information about the patient's condition for a reliable diagnosis. To this end, we propose an end-to-end Multi-modal Graph Learning framework (MMGL) for disease prediction with multi-modality. To effectively exploit the rich information across multi-modality associated with the disease, modality-aware representation learning is proposed to aggregate the features of each modality by leveraging the correlation and complementarity between the modalities. Furthermore, instead of defining the graph manually, the latent graph structure is captured through an effective way of adaptive graph learning. It could be jointly optimized with the prediction model, thus revealing the intrinsic connections among samples. Our model is also applicable to the scenario of inductive learning for those unseen data. An extensive group of experiments on two disease prediction tasks demonstrates that the proposed MMGL achieves more favorable performance. The code of MMGL is available at https://github.com/SsGood/MMGL.

Structured sparse multiset canonical correlation analysis of simultaneous fNIRS and EEG provides new insights into the human action-observation network

The action observation network (AON) is a network of brain regions involved in the execution and observation of a given action. The AON has been investigated in humans using mostly electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), but shared neural correlates of action observation and action execution are still unclear due to lack of ecologically valid neuroimaging measures. In this study, we used concurrent EEG and functional Near Infrared Spectroscopy (fNIRS) to examine the AON during a live-action observation and execution paradigm. We developed structured sparse multiset canonical correlation analysis (ssmCCA) to perform EEG-fNIRS data fusion. MCCA is a generalization of CCA to more than two sets of variables and is commonly used in medical multimodal data fusion. However, mCCA suffers from multi-collinearity, high dimensionality, unimodal feature selection, and loss of spatial information in interpreting the results. A limited number of participants (small sample size) is another problem in mCCA, which leads to overfitted models. Here, we adopted graph-guided (structured) fused least absolute shrinkage and selection operator (LASSO) penalty to mCCA to conduct feature selection, incorporating structural information amongst the variables (i.e., brain regions). Benefitting from concurrent recordings of brain hemodynamic and electrophysiological responses, the proposed ssmCCA finds linear transforms of each modality such that the correlation between their projections is maximized. Our analysis of 21 right-handed participants indicated that the left inferior parietal region was active during both action execution and action observation. Our findings provide new insights into the neural correlates of AON which are more fine-tuned than the results from each individual EEG or fNIRS analysis and validate the use of ssmCCA to fuse EEG and fNIRS datasets.

Multidataset Independent Subspace Analysis With Application to Multimodal Fusion

Unsupervised latent variable models-blind source separation (BSS) especially-enjoy a strong reputation for their interpretability. But they seldom combine the rich diversity of information available in multiple datasets, even though multidatasets yield insightful joint solutions otherwise unavailable in isolation. We present a direct, principled approach to multidataset combination that takes advantage of multidimensional subspace structures. In turn, we extend BSS models to capture the underlying modes of shared and unique variability across and within datasets. Our approach leverages joint information from heterogeneous datasets in a flexible and synergistic fashion. We call this method multidataset independent subspace analysis (MISA). Methodological innovations exploiting the Kotz distribution for subspace modeling, in conjunction with a novel combinatorial optimization for evasion of local minima, enable MISA to produce a robust generalization of independent component analysis (ICA), independent vector analysis (IVA), and independent subspace analysis (ISA) in a single unified model. We highlight the utility of MISA for multimodal information fusion, including sample-poor regimes ( N = 600 ) and low signal-to-noise ratio, promoting novel applications in both unimodal and multimodal brain imaging data.

A technical review of canonical correlation analysis for neuroscience applications

Collecting comprehensive data sets of the same subject has become a standard in neuroscience research and uncovering multivariate relationships among collected data sets have gained significant attentions in recent years. Canonical correlation analysis (CCA) is one of the powerful multivariate tools to jointly investigate relationships among multiple data sets, which can uncover disease or environmental effects in various modalities simultaneously and characterize changes during development, aging, and disease progressions comprehensively. In the past 10 years, despite an increasing number of studies have utilized CCA in multivariate analysis, simple conventional CCA dominates these applications. Multiple CCA-variant techniques have been proposed to improve the model performance; however, the complicated multivariate formulations and not well-known capabilities have delayed their wide applications. Therefore, in this study, a comprehensive review of CCA and its variant techniques is provided. Detailed technical formulation with analytical and numerical solutions, current applications in neuroscience research, and advantages and limitations of each CCA-related technique are discussed. Finally, a general guideline in how to select the most appropriate CCA-related technique based on the properties of available data sets and particularly targeted neuroscience questions is provided.

Latent Representation Learning for Alzheimer's Disease Diagnosis With Incomplete Multi-Modality Neuroimaging and Genetic Data

The fusion of complementary information contained in multi-modality data [e.g., magnetic resonance imaging (MRI), positron emission tomography (PET), and genetic data] has advanced the progress of automated Alzheimer's disease (AD) diagnosis. However, multi-modality based AD diagnostic models are often hindered by the missing data, i.e., not all the subjects have complete multi-modality data. One simple solution used by many previous studies is to discard samples with missing modalities. However, this significantly reduces the number of training samples, thus leading to a sub-optimal classification model. Furthermore, when building the classification model, most existing methods simply concatenate features from different modalities into a single feature vector without considering their underlying associations. As features from different modalities are often closely related (e.g., MRI and PET features are extracted from the same brain region), utilizing their inter-modality associations may improve the robustness of the diagnostic model. To this end, we propose a novel latent representation learning method for multi-modality based AD diagnosis. Specifically, we use all the available samples (including samples with incomplete modality data) to learn a latent representation space. Within this space, we not only use samples with complete multi-modality data to learn a common latent representation, but also use samples with incomplete multi-modality data to learn independent modality-specific latent representations. We then project the latent representations to the label space for AD diagnosis. We perform experiments using 737 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, and the experimental results verify the effectiveness of our proposed method.

Performing Sparse Regularization and Dimension Reduction Simultaneously in Multimodal Data Fusion

Collecting multiple modalities of neuroimaging data on the same subject is increasingly becoming the norm in clinical practice and research. Fusing multiple modalities to find related patterns is a challenge in neuroimaging analysis. Canonical correlation analysis (CCA) is commonly used as a symmetric data fusion technique to find related patterns among multiple modalities. In CCA-based data fusion, principal component analysis (PCA) is frequently applied as a preprocessing step to reduce data dimension followed by CCA on dimension-reduced data. PCA, however, does not differentiate between informative voxels from non-informative voxels in the dimension reduction step. Sparse PCA (sPCA) extends traditional PCA by adding sparse regularization that assigns zero weights to non-informative voxels. In this study, sPCA is incorporated into CCA-based fusion analysis and applied on neuroimaging data. A cross-validation method is developed and validated to optimize the parameters in sPCA. Different simulations are carried out to evaluate the improvement by introducing sparsity constraint to PCA. Four fusion methods including sPCA+CCA, PCA+CCA, parallel ICA and sparse CCA were applied on structural and functional magnetic resonance imaging data of mild cognitive impairment subjects and normal controls. Our results indicate that sPCA significantly can reduce the impact of non-informative voxels and lead to improved statistical power in uncovering disease-related patterns by a fusion analysis.

A Projection CCA Method for Effective fMRI Data Analysis

OBJECTIVE

Canonical correlation analysis (CCA) is a data-driven method that has been successfully used in functional magnetic resonance imaging (fMRI) data analysis. Standard CCA extracts meaningful information from a pair of data sets by seeking pairs of linear combinations from two sets of variables with maximum pairwise correlation. So far, however, this method has been used without incorporating prior information available for fMRI data. In this paper, we address this issue by proposing a new CCA method named pCCA (for projection CCA).

METHODS

The proposed method is obtained by projection onto a set of basis vectors that better characterize temporal information in the fMRI data set. A methodology is presented to describe the basis selection process using discrete cosine transform (DCT) basis functions. Employing DCT guides the estimated canonical variates, yielding a more computationally efficient CCA procedure.

RESULTS

The performance gain of the proposed pCCA algorithm over standard and regularized CCA is illustrated on both simulated and real fMRI datasets from resting state, block paradigm task-related and event-related experiments. The results have shown that the proposed pCCA successfully extracts latent components from the task as well as resting-state datasets with increased specificity of the activated voxels.

CONCLUSION

In addition to offering a new CCA approach, when applied on fMRI data, the proposed algorithm adapts to variations of brain activity patterns and reveals the functionally connected brain regions.

SIGNIFICANCE

The proposed method can be seen as a regularized CCA method where regularization is introduced via basis expansion, which has the advantage of enforcing smoothness on canonical components.

Neonatal brain resting-state functional connectivity imaging modalities

Infancy is the most critical period in human brain development. Studies demonstrate that subtle brain abnormalities during this state of life may greatly affect the developmental processes of the newborn infants. One of the rapidly developing methods for early characterization of abnormal brain development is functional connectivity of the brain at rest. While the majority of resting-state studies have been conducted using magnetic resonance imaging (MRI), there is clear evidence that resting-state functional connectivity (rs-FC) can also be evaluated using other imaging modalities. The aim of this review is to compare the advantages and limitations of different modalities used for the mapping of infants' brain functional connectivity at rest. In addition, we introduce photoacoustic tomography, a novel functional neuroimaging modality, as a complementary modality for functional mapping of infants' brain.