A selective review of multimodal fusion methods in schizophrenia

Multivariate brain-behaviour associations in psychiatric disorders

Mapping brain-behaviour associations is paramount to understand and treat psychiatric disorders. Standard approaches involve investigating the association between one brain and one behavioural variable (univariate) or multiple variables against one brain/behaviour feature ('single' multivariate). Recently, large multimodal datasets have propelled a new wave of studies that leverage on 'doubly' multivariate approaches capable of parsing the multifaceted nature of both brain and behaviour simultaneously. Within this movement, canonical correlation analysis (CCA) and partial least squares (PLS) emerge as the most popular techniques. Both seek to capture shared information between brain and behaviour in the form of latent variables. We provide an overview of these methods, review the literature in psychiatric disorders, and discuss the main challenges from a predictive modelling perspective. We identified 39 studies across four diagnostic groups: attention deficit and hyperactive disorder (ADHD, k = 4, N = 569), autism spectrum disorders (ASD, k = 6, N = 1731), major depressive disorder (MDD, k = 5, N = 938), psychosis spectrum disorders (PSD, k = 13, N = 1150) and one transdiagnostic group (TD, k = 11, N = 5731). Most studies (67%) used CCA and focused on the association between either brain morphology, resting-state functional connectivity or fractional anisotropy against symptoms and/or cognition. There were three main findings. First, most diagnoses shared a link between clinical/cognitive symptoms and two brain measures, namely frontal morphology/brain activity and white matter association fibres (tracts between cortical areas in the same hemisphere). Second, typically less investigated behavioural variables in multivariate models such as physical health (e.g., BMI, drug use) and clinical history (e.g., childhood trauma) were identified as important features. Finally, most studies were at risk of bias due to low sample size/feature ratio and/or in-sample testing only. We highlight the importance of carefully mitigating these sources of bias with an exemplar application of CCA.

Characterizing Typhoon-related Posttraumatic Stress Disorder Based on Multimodal Fusion of Structural, Diffusion, and Functional Magnetic Resonance Imaging

Diagnosing posttraumatic stress disorder (PTSD) using only single-modality images is controversial. We aimed to use multimodal magnetic resonance imaging (MRI) combining structural, diffusion, and functional MRI to possibly provide a more comprehensive viewpoint on the decisive characteristics of PTSD patients. Typhoon-exposed individuals with (n = 26) and without PTSD (n = 32) and healthy volunteers (n = 30) were enrolled. Five MRI features from three modalities, including two resting-state functional MRI (rs-fMRI) features (amplitude of low-frequency fluctuation, ALFF; and regional homogeneity, ReHo), one structural MRI feature (gray matter density, GM), and two diffusion tensor imaging (DTI) features (fractional anisotropy, FA; and mean diffusivity, MD) were investigated simultaneously with a multimodal canonical correlation analysis + joint independent component analysis model to identify abnormalities in the PTSD brain. We identified statistical differences between PTSD patients and healthy controls in terms of 1 rs-fMRI (ALFF, ReHo) alterations in the superior frontal gyrus, precuneus, inferior parietal lobule (IPL), anterior cingulate cortex (ACC), and posterior cingulate cortex (PCC), 2 DTI (FA, MD) changes in the pons, genu, and splenium of the corpus callosum, and 3 Structural MRI abnormalities in the precuneus, IPL, ACC, and PCC. A novel ReHo component was found to distinguish PTSD and trauma-exposed controls, including the precuneus, IPL, middle frontal gyrus, middle occipital gyrus, and cerebellum. This study reveals that PTSD individuals exhibit intertwined functional and structural anomalies within the default mode network. Some alterations within this network may serve as a potential marker to distinguish between PTSD patients and trauma-exposed controls.

The network characteristics in schizophrenia with prominent negative symptoms: a multimodal fusion study

Previous studies on putative neural mechanisms of negative symptoms in schizophrenia mainly used single modal imaging data, and seldom utilized schizophrenia patients with prominent negative symptoms (PNS).This study adopted the multimodal fusion method and recruited a homogeneous sample with PNS. We aimed to identify negative symptoms-related structural and functional neural correlates of schizophrenia. Structural magnetic resonance imaging (sMRI) and resting-state functional MRI (rs-fMRI) were performed in 31 schizophrenia patients with PNS and 33 demographically matched healthy controls.Compared to healthy controls, schizophrenia patients with PNS exhibited significantly altered functional activations in the default mode network (DMN) and had structural gray matter volume (GMV) alterations in the cerebello-thalamo-cortical network. Correlational analyses showed that negative symptoms severity was significantly correlated with the cerebello-thalamo-cortical structural network, but not with the DMN network in schizophrenia patients with PNS.Our findings highlight the important role of the cerebello-thalamo-cortical structural network underpinning the neuropathology of negative symptoms in schizophrenia. Future research should recruit a large sample and schizophrenia patients without PNS, and apply adjustments for multiple comparison, to verify our preliminary findings.

Voxel-wise Fusion of Resting fMRI Networks and Gray Matter Volume for Alzheimer's Disease Classification using Deep Multimodal Learning

Alzheimer's disease (AD) is a prevalent neurodegenerative disorder requiring accurate and early diagnosis for effective treatment. Resting-state functional magnetic resonance imaging (rs-fMRI) and gray matter volume analysis from structural MRI have emerged as valuable tools for investigating AD-related brain alterations. However, the potential benefits of integrating these modalities using deep learning techniques remain unexplored. In this study, we propose a novel framework that fuses composite images of multiple rs-fMRI networks (called voxelwise intensity projection) and gray matter segmentation images through a deep learning approach for improved AD classification. We demonstrate the superiority of fMRI networks over commonly used metrics such as amplitude of low-frequency fluctuations (ALFF) and fractional ALFF in capturing spatial maps critical for AD classification. We use a multi-channel convolutional neural network incorporating the AlexNet dropout architecture to effectively model spatial and temporal dependencies in the integrated data. Extensive experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset of AD patients and cognitively normal (CN) validate the efficacy of our approach, showcasing improved classification performance of 94.12% test accuracy and an area under the curve (AUC) score of 97.79 compared to existing methods. Our results show that the fusion results generally outperformed the unimodal results. The saliency visualizations also show significant differences in the hippocampus, amygdala, putamen, caudate nucleus, and regions of basal ganglia which are in line with the previous neurobiological literature. Our research offers a novel method to enhance our grasp of AD pathology. By integrating data from various functional networks with structural MRI insights, we significantly improve diagnostic accuracy. This accuracy is further boosted by the effective visualization of this combined information. This lays the groundwork for further studies focused on providing a more accurate and personalized approach to AD diagnosis. The proposed framework and insights gained from fMRI networks provide a promising avenue for future research in deep multimodal fusion and neuroimaging analysis.

Network Differential in Gaussian Graphical Models from Multimodal Neuroimaging Data. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023;2023:1-6. [PMID: 38083176 DOI: 10.1109/embc40787.2023.10340856] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Multimodal brain network analysis has the potential to provide insights into the mechanisms of brain disorders. Most previous studies have analyzed either unimodal brain graphs or focused on local/global graphic metrics with little consideration of details of disrupted paths in the patient group. As we show, the combination of multimodal brain graphs and disrupted path-based analysis can be highly illuminating to recognize path-based disease biomarkers. In this study, we first propose a way to estimate multimodal brain graphs using static functional network connectivity (sFNC) and gray matter features using a Gaussian graphical model of schizophrenia versus controls. Next, applying the graph theory approach we identify disconnectors or connectors in the patient group graph that create additional paths or cause absent paths compared to the control graph. Results showed several edges in the schizophrenia group graph that trigger missing or additional paths. Identified edges associated with these disrupted paths were identified both within and between dFNC and gray matter which highlights the importance of considering multimodal studies and moving beyond pairwise edges to provide a more comprehensive understanding of brain disorders.Clinical Relevance- We identified a path-based biomarker in schizophrenia, by imitating the structure of paths in a multimodal (sMIR+fMRI) brain graph of the control group. Identified cross-modal edges associated with disrupted paths were related to the middle temporal gyrus and cerebellar regions.

Joint structural-functional magnetic resonance imaging features are associated with diagnosis and real-world functioning in patients with schizophrenia

OBJECTIVE

Earlier evidence suggested that structural-functional covariation in schizophrenia patients (SCZ) is associated with cognition, a predictor of functioning. Moreover, studies suggested that functional brain abnormalities of schizophrenia may be related with structural network features. However, only few studies have investigated the relationship between structural-functional covariation and both diagnosis and functioning in SCZ. We hypothesized that structural-functional covariation networks associated with diagnosis are related to real-world functioning in SCZ.

METHODS

We performed joint Independent Component Analysis on T1 images and resting-state fMRI-based Degree Centrality (DC) maps from 89 SCZ and 285 controls. Structural-functional covariation networks in which we found a main effect of diagnosis underwent correlation analysis to investigate their relationship with functioning. Covariation networks showing a significant association with both diagnosis and functioning underwent univariate analysis to better characterize group-level differences at the spatial level.

RESULTS

A structural-functional covariation network characterized by frontal, temporal, parietal and thalamic structural estimates significantly covaried with temporo-parietal resting-state DC. Compared with controls, SCZ had reduced structural-functional covariation within this network (p_FDR = 0.005). The same measure correlated positively with both social and occupational functioning (both p_FDR = 0.042). Univariate analyses revealed grey matter deviations in SCZ compared with controls within this structural-functional network in hippocampus, cerebellum, thalamus, orbito-frontal cortex, and insula. No group differences were found in DC.

CONCLUSIONS

Findings support the existence of a phenotypical association between group-level differences and inter-individual heterogeneity of functional deficits in SCZ. Given that only the joint structural/functional analysis revealed this association, structural-functional covariation may be a potentially relevant schizophrenia phenotype.

Cognitive Implications of Correlated Structural Network Changes in Schizophrenia

Background

Schizophrenia is a brain disorder characterized by diffuse, diverse, and wide-spread changes in gray matter volume (GM) and white matter structure (fractional anisotropy, FA), as well as cognitive impairments that greatly impact an individual’s quality of life. While the relationship of each of these image modalities and their links to schizophrenia status and cognitive impairment has been investigated separately, a multimodal fusion via parallel independent component analysis (pICA) affords the opportunity to explore the relationships between the changes in GM and FA, and the implications these network changes have on cognitive performance.

Methods

Images from 73 subjects with schizophrenia (SZ) and 82 healthy controls (HC) were drawn from an existing dataset. We investigated 12 components from each feature (FA and GM). Loading coefficients from the images were used to identify pairs of features that were significantly correlated and showed significant group differences between HC and SZ. MANCOVA analysis uncovered the relationships the identified spatial maps had with age, gender, and a global cognitive performance score.

Results

Three component pairs showed significant group differences (HC > SZ) in both gray and white matter measurements. Two of the component pairs identified networks of gray matter that drove significant relationships with cognition (HC > SZ) after accounting for age and gender. The gray and white matter structural networks identified in these three component pairs pull broadly from many regions, including the right and left thalamus, lateral occipital cortex, multiple regions of the middle temporal gyrus, precuneus cortex, postcentral gyrus, cingulate gyrus/cingulum, lingual gyrus, and brain stem.

Conclusion

The results of this multimodal analysis adds to our understanding of how the relationship between GM, FA, and cognition differs between HC and SZ by highlighting the correlated intermodal covariance of these structural networks and their differential relationships with cognitive performance. Previous unimodal research has found similar areas of GM and FA differences between these groups, and the cognitive deficits associated with SZ have been well documented. This study allowed us to evaluate the intercorrelated covariance of these structural networks and how these networks are involved the differences in cognitive performance between HC and SZ.

Multivariate Analysis and Modelling of multiple Brain endOphenotypes: Let's MAMBO!

Imaging genetic studies aim to test how genetic information influences brain structure and function by combining neuroimaging-based brain features and genetic data from the same individual. Most studies focus on individual correlation and association tests between genetic variants and a single measurement of the brain. Despite the great success of univariate approaches, given the capacity of neuroimaging methods to provide a multiplicity of cerebral phenotypes, the development and application of multivariate methods become crucial. In this article, we review novel methods and strategies focused on the analysis of multiple phenotypes and genetic data. We also discuss relevant aspects of multi-trait modelling in the context of neuroimaging data.

Neural Signatures of Memory Encoding in Schizophrenia Are Modulated by Antipsychotic Treatment

There is no pharmacological treatment to remediate cognitive impairment in schizophrenia (SZ). It is imperative to characterize underlying pathologies of memory processing in order to effectively develop new treatments. In this longitudinal study, we combined functional magnetic resonance imaging during a memory encoding task with proton MR spectroscopy to measure hippocampal glutamate + glutamine (Glx). Seventeen SZ were scanned while unmedicated and after 6 weeks of treatment with risperidone and compared to a group of matched healthy controls (HC) scanned 6 weeks apart. Unmedicated patients showed reduced blood oxygen level dependent (BOLD) response in several regions, including the hippocampus, and greater BOLD response in regions of the default mode network (DMN) during correct memory encoding. Post hoc contrasts from significant group by time interactions indicated reduced hippocampal BOLD response at baseline with subsequent increase following treatment. Hippocampal Glx was not different between groups at baseline, but at week 6, hippocampal Glx was significantly lower in SZ compared to HC. Finally, in unmedicated SZ, higher hippocampal Glx predicted less deactivation of the BOLD response in regions of the DMN. Using 2 brain imaging modalities allowed us to concurrently investigate different mechanisms involved in memory encoding dysfunction in SZ. Hippocampal pathology during memory encoding stems from decreased hippocampal recruitment and faulty deactivation of the DMN, and hippocampal recruitment during encoding can be modulated by antipsychotic treatment. High Glx in unmedicated patients predicted less deactivation of the DMN; these results suggest a mechanism by which faulty DMN deactivation, a hallmark of pathological findings in SZ, is achieved.

Hallucination-Specific structure-function associations in schizophrenia

Combining structural (sMRI) and functional magnetic resonance imaging (fMRI) data in schizophrenia patients with and without auditory hallucinations (9 SZ_AVH, 12 SZ_nAVH), 18 patients with bipolar disorder, and 22 healthy controls, we examined whether cortical thinning was associated with abnormal activity in functional brain networks associated with auditory hallucinations. Language-task fMRI data were combined with mean cortical thickness values from 148 brain regions in a constrained principal component analysis (CPCA) to identify brain structure-function associations predictable from group differences. Two components emerged from the multimodal analysis. The "AVH component" highlighted an association of frontotemporal and cingulate thinning with altered brain activity characteristic of hallucinations among patients with AVH. In contrast, the "Bipolar component" distinguished bipolar patients from healthy controls and linked increased activity in the language network with cortical thinning in the left occipital-temporal lobe. Our findings add to a body of evidence of the biological underpinnings of hallucinations and illustrate a method for multimodal data analysis of structure-function associations in psychiatric illness.

Structural Covariance Reveals Alterations in Control and Salience Network Integrity in Chronic Schizophrenia

Schizophrenia (SCZ) is recognized as a disorder of distributed brain dysconnectivity. While progress has been made delineating large-scale functional networks in SCZ, little is known about alterations in grey matter integrity of these networks. We used a multivariate approach to identify the structural covariance of the salience, default, motor, visual, fronto-parietal control, and dorsal attention networks. We derived individual scores reflecting covariance in each structural image for a given network. Seed-based multivariate analyses were conducted on structural images in a discovery (n = 90) and replication (n = 74) sample of SCZ patients and healthy controls. We first validated patterns across all networks, consistent with well-established functional connectivity reports. Next, across two SCZ samples, we found reliable and robust reductions in structural integrity of the fronto-parietal control and salience networks, but not default, dorsal attention, motor and sensory networks. Well-powered exploratory analyses failed to identify relationships with symptoms. These findings provide evidence of selective structural decline in associative networks in SCZ. Such decline may be linked with recently identified functional disturbances in associative networks, providing more sensitive multi-modal network-level probes in SCZ. Absence of symptom effects suggests that identified disturbances may underlie a trait-type marker in SCZ.

Movement disorder and sensorimotor abnormalities in schizophrenia and other psychoses - European consensus on assessment and perspectives

Over the last three decades, movement disorder as well as sensorimotor and psychomotor functioning in schizophrenia (SZ) and other psychoses has gained greater scientific and clinical relevance as an intrinsic component of the disease process of psychotic illness; this extends to early psychosis prediction, early detection of motor side effects of antipsychotic medication, clinical outcome monitoring, treatment of psychomotor syndromes (e.g. catatonia), and identification of new targets for non-invasive brain stimulation. In 2017, a systematic cooperation between working groups interested in movement disorder and sensorimotor/psychomotor functioning in psychoses was initiated across European universities. As a first step, the members of this group would like to introduce and define the theoretical aspects of the sensorimotor domain in SZ and other psychoses. This consensus paper is based on a synthesis of scientific evidence, good clinical practice and expert opinions that were discussed during recent conferences hosted by national and international psychiatric associations. While reviewing and discussing the recent theoretical and experimental work on neural mechanisms and clinical implications of sensorimotor behavior, we here seek to define the key principles and elements of research on movement disorder and sensorimotor/psychomotor functioning in psychotic illness. Finally, the members of this European group anticipate that this consensus paper will stimulate further multimodal and prospective studies on hypo- and hyperkinetic movement disorders and sensorimotor/psychomotor functioning in SZ and other psychotic disorders.

Meta-Modal Information Flow: A Method for Capturing Multimodal Modular Disconnectivity in Schizophrenia

OBJECTIVE

Multimodal measurements of the same phenomena provide complementary information and highlight different perspectives, albeit each with their own limitations. A focus on a single modality may lead to incorrect inferences, which is especially important when a studied phenomenon is a disease. In this paper, we introduce a method that takes advantage of multimodal data in addressing the hypotheses of disconnectivity and dysfunction within schizophrenia (SZ).

METHODS

We start with estimating and visualizing links within and among extracted multimodal data features using a Gaussian graphical model (GGM). We then propose a modularity-based method that can be applied to the GGM to identify links that are associated with mental illness across a multimodal data set. Through simulation and real data, we show our approach reveals important information about disease-related network disruptions that are missed with a focus on a single modality. We use functional MRI (fMRI), diffusion MRI (dMRI), and structural MRI (sMRI) to compute the fractional amplitude of low frequency fluctuations (fALFF), fractional anisotropy (FA), and gray matter (GM) concentration maps. These three modalities are analyzed using our modularity method.

RESULTS

Our results show missing links that are only captured by the cross-modal information that may play an important role in disconnectivity between the components.

CONCLUSION

We identified multimodal (fALFF, FA and GM) disconnectivity in the default mode network area in patients with SZ, which would not have been detectable in a single modality.

SIGNIFICANCE

The proposed approach provides an important new tool for capturing information that is distributed among multiple imaging modalities.

Multimodal Data Fusion of Deep Learning and Dynamic Functional Connectivity Features to Predict Alzheimer's Disease Progression

Early prediction of diseased brain conditions is critical for curing illness and preventing irreversible neuronal dysfunction and loss. Generically regarding the different neuroimaging modalities as filtered, complementary insights of brain's anatomical and functional organization, multimodal data fusion could be hypothesized to enhance the predictive power as compared to a unimodal prediction of disease progression. More recently, deep learning (DL) based methods on structural MRI (sMRI) data have outperformed classical machine learning approaches in several neuroimaging applications including diagnostic classification and prediction. Similarly, functional MRI (fMRI) features estimated using a dynamic (i.e. time-varying) functional connectivity (FC) approach have been found to be more discriminative and predictive of the clinical diagnosis than those based on the static FC approach. Motivated by this, we introduce a novel multimodal data fusion framework featuring deep residual learning of non-linear sMRI features and dynamic FC (dFC) based extraction of fMRI features to predict the subset of individuals with mild cognitive impairments who would progress to Alzheimer's disease within a time-period of three years from the baseline scanning sessions. Our cross-validated results from the developed multimodal (sMRI-fMRI) data fusion framework demonstrate a significant improvement in performance over the unimodal prediction analyses with the fMRI (p = 7.03 x 10^-7) and sMRI (p = 6.72 x 10^-4) modalities. As such, the findings in this work highlight the benefits of combining multiple neuroimaging data modalities via data fusion, corroborate the predictive value of the tested DL and dFC features and argue in favor of exploration of similar approaches to learn neuroanatomical and functional alterations in the neuroimaging data.

Unraveling Diagnostic Biomarkers of Schizophrenia Through Structure-Revealing Fusion of Multi-Modal Neuroimaging Data

Fusing complementary information from different modalities can lead to the discovery of more accurate diagnostic biomarkers for psychiatric disorders. However, biomarker discovery through data fusion is challenging since it requires extracting interpretable and reproducible patterns from data sets, consisting of shared/unshared patterns and of different orders. For example, multi-channel electroencephalography (EEG) signals from multiple subjects can be represented as a third-order tensor with modes: subject, time, and channel, while functional magnetic resonance imaging (fMRI) data may be in the form of subject by voxel matrices. Traditional data fusion methods rearrange higher-order tensors, such as EEG, as matrices to use matrix factorization-based approaches. In contrast, fusion methods based on coupled matrix and tensor factorizations (CMTF) exploit the potential multi-way structure of higher-order tensors. The CMTF approach has been shown to capture underlying patterns more accurately without imposing strong constraints on the latent neural patterns, i.e., biomarkers. In this paper, EEG, fMRI, and structural MRI (sMRI) data collected during an auditory oddball task (AOD) from a group of subjects consisting of patients with schizophrenia and healthy controls, are arranged as matrices and higher-order tensors coupled along the subject mode, and jointly analyzed using structure-revealing CMTF methods [also known as advanced CMTF (ACMTF)] focusing on unique identification of underlying patterns in the presence of shared/unshared patterns. We demonstrate that joint analysis of the EEG tensor and fMRI matrix using ACMTF reveals significant and biologically meaningful components in terms of differentiating between patients with schizophrenia and healthy controls while also providing spatial patterns with high resolution and improving the clustering performance compared to the analysis of only the EEG tensor. We also show that these patterns are reproducible, and study reproducibility for different model parameters. In comparison to the joint independent component analysis (jICA) data fusion approach, ACMTF provides easier interpretation of EEG data by revealing a single summary map of the topography for each component. Furthermore, fusion of sMRI data with EEG and fMRI through an ACMTF model provides structural patterns; however, we also show that when fusing data sets from multiple modalities, hence of very different nature, preprocessing plays a crucial role.

Linked 4-Way Multimodal Brain Differences in Schizophrenia in a Large Chinese Han Population

Multimodal fusion has been regarded as a promising tool to discover covarying patterns of multiple imaging types impaired in brain diseases, such as schizophrenia (SZ). In this article, we aim to investigate the covarying abnormalities underlying SZ in a large Chinese Han population (307 SZs, 298 healthy controls [HCs]). Four types of magnetic resonance imaging (MRI) features, including regional homogeneity (ReHo) from resting-state functional MRI, gray matter volume (GM) from structural MRI, fractional anisotropy (FA) from diffusion MRI, and functional network connectivity (FNC) resulted from group independent component analysis, were jointly analyzed by a data-driven multivariate fusion method. Results suggest that a widely distributed network disruption appears in SZ patients, with synchronous changes in both functional and structural regions, especially the basal ganglia network, salience network (SAN), and the frontoparietal network. Such a multimodal coalteration was also replicated in another independent Chinese sample (40 SZs, 66 HCs). Our results on auditory verbal hallucination (AVH) also provide evidence for the hypothesis that prefrontal hypoactivation and temporal hyperactivation in SZ may lead to failure of executive control and inhibition, which is relevant to AVH. In addition, impaired working memory performance was found associated with GM reduction and FA decrease in SZ in prefrontal and superior temporal area, in both discovery and replication datasets. In summary, by leveraging multiple imaging and clinical information into one framework to observe brain in multiple views, we can integrate multiple inferences about SZ from large-scale population and offer unique perspectives regarding the missing links between the brain function and structure that may not be achieved by separate unimodal analyses.

Multimodal Neuroimaging: Basic Concepts and Classification of Neuropsychiatric Diseases

Neuroimaging techniques are widely used in neuroscience to visualize neural activity, to improve our understanding of brain mechanisms, and to identify biomarkers-especially for psychiatric diseases; however, each neuroimaging technique has several limitations. These limitations led to the development of multimodal neuroimaging (MN), which combines data obtained from multiple neuroimaging techniques, such as electroencephalography, functional magnetic resonance imaging, and yields more detailed information about brain dynamics. There are several types of MN, including visual inspection, data integration, and data fusion. This literature review aimed to provide a brief summary and basic information about MN techniques (data fusion approaches in particular) and classification approaches. Data fusion approaches are generally categorized as asymmetric and symmetric. The present review focused exclusively on studies based on symmetric data fusion methods (data-driven methods), such as independent component analysis and principal component analysis. Machine learning techniques have recently been introduced for use in identifying diseases and biomarkers of disease. The machine learning technique most widely used by neuroscientists is classification-especially support vector machine classification. Several studies differentiated patients with psychiatric diseases and healthy controls with using combined datasets. The common conclusion among these studies is that the prediction of diseases increases when combining data via MN techniques; however, there remain a few challenges associated with MN, such as sample size. Perhaps in the future N-way fusion can be used to combine multiple neuroimaging techniques or nonimaging predictors (eg, cognitive ability) to overcome the limitations of MN.

Different levels of facial expression recognition in patients with first-episode schizophrenia: A functional MRI study

Background

The impairment of facial expression recognition has become a biomarker for early identification of first-episode schizophrenia, and this kind of research is increasing.

Aims

To explore the differences in brain area activation using different degrees of disgusted facial expression recognition in antipsychotic-naïve patients with first-episode schizophrenia and healthy controls.

Methods

In this study, facial expression recognition tests were performed on 30 first-episode, antipsychotic-naïve patients with schizophrenia (treatment group) and 30 healthy subjects (control group) with matched age, educational attainment and gender. Functional MRI was used for comparing the differences of the brain areas of activation between the two groups.

Results

The average response time difference between the patient group and the control group in the ‘high degree of disgust’ facial expression recognition task was statistically significant (1.359 (0.408)/2.193 (0.625), F=26.65, p<0.001), and the correct recognition rate of the treatment group was lower than that of the control group (41.05 (22.25)/59.84 (13.91, F=19.81, p<0.001). Compared with the control group, the left thalamus, right lingual gyrus and right middle temporal gyrus were negatively activated in the patients with first-episode schizophrenia in the ‘high degree of disgust’ emotion recognition, and there was a significant activation in the left and right middle temporal gyrus and the right caudate nucleus. However, there was no significant activation difference in the ‘low degree of disgust’ recognition.

Conclusions

In patients with first-episode schizophrenia, the areas of facial recognition impairment are significantly different in different degrees of disgust facial expression recognition.

Constrained temporal parallel decomposition for EEG-fMRI fusion

OBJECTIVE

Multimodal neuroimaging has become a common practice in neuroscience research. Simultaneous EEG-fMRI is a popular multimodal recording approach due to the complementary spatiotemporal relationship between the two modalities. Several data fusion techniques have been proposed in the literature for EEG-fMRI fusion, including joint-ICA and parallel-ICA frameworks. Previous EEG-fMRI fusion approaches have used sensor-level EEG features. Recently, we introduced source-space ICA for EEG-MEG source reconstruction and component identification, which was shown to be a superior alternative to sensor-space ICA.

APPROACH

Here, we extend source-space ICA to the fusion of EEG-fMRI data. Additionally, we incorporate the use of a paradigm signal (constrained) and a lag-based signal decomposition approach to accommodate recent findings demonstrating the potentially variable lag structure between electrophysiological and BOLD signals. We evaluated this method on simulated concurrent EEG-fMRI during a boxcar task design, as well as real concurrent EEG-fMRI data from three participants performing an N-Back working memory task. The block diagram of the algorithm and corresponding source codes are provided.

MAIN RESULTS

Based on the results of the real working memory task, for all three subjects, one frontal theta component, and one right posterior alpha component had the highest contribution coefficients (~0.5) to the paradigm-related fused component. There were also two more alpha band components with contribution coefficients of 0.3. The highest contributing fMRI component (~0.8) was one known in the literature to be related to the attention network. The second fMRI component was related to the well-known default mode network, with a contribution coefficient of 0.3.

SIGNIFICANCE

The proposed EEG-fMRI fusion approach, is capable of estimating the brain maps of the EEG and fMRI for the fused components and account for the variable lag structure between electrophysiological and BOLD signals.

Motor dysfunction as research domain across bipolar, obsessive-compulsive and neurodevelopmental disorders

Although genuine motor abnormalities (GMA) are frequently found in schizophrenia, they are also considered as an intrinsic feature of bipolar, obsessive-compulsive, and neurodevelopmental disorders with early onset such as autism, ADHD, and Tourette syndrome. Such transnosological observations strongly suggest a common neural pathophysiology. This systematic review highlights the evidence on GMA and their neuroanatomical substrates in bipolar, obsessive-compulsive, and neurodevelopmental disorders. The data lends support for a common pattern contributing to GMA expression in these diseases that seems to be related to cerebello-thalamo-cortical, fronto-parietal, and cortico-subcortical motor circuit dysfunction. The identified studies provide first evidence for a motor network dysfunction as a correlate of early neurodevelopmental deviance prior to clinical symptom expression. There are also first hints for a developmental risk factor model of these mental disorders. An in-depth analysis of motor networks and related patho-(physiological) mechanisms will not only help promoting Research Domain Criteria (RDoC) Motor System construct, but also facilitate the development of novel psychopharmacological models, as well as the identification of neurobiologically plausible target sites for non-invasive brain stimulation.

Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns "alignments" (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

Application of Graph Theory to Assess Static and Dynamic Brain Connectivity: Approaches for Building Brain Graphs

Human brain connectivity is complex. Graph theory based analysis has become a powerful and popular approach for analyzing brain imaging data, largely because of its potential to quantitatively illuminate the networks, the static architecture in structure and function, the organization of dynamic behavior over time, and disease related brain changes. The first step in creating brain graphs is to define the nodes and edges connecting them. We review a number of approaches for defining brain nodes including fixed versus data-driven nodes. Expanding the narrow view of most studies which focus on static and/or single modality brain connectivity, we also survey advanced approaches and their performances in building dynamic and multi-modal brain graphs. We show results from both simulated and real data from healthy controls and patients with mental illnesse. We outline the advantages and challenges of these various techniques. By summarizing and inspecting recent studies which analyzed brain imaging data based on graph theory, this article provides a guide for developing new powerful tools to explore complex brain networks.

Relations between structural and EEG-based graph metrics in healthy controls and schizophrenia patients

Our aim was to assess structural and functional networks in schizophrenia patients; and the possible prediction of the latter based on the former. The possible dependence of functional network properties on structural alterations has not been analyzed in schizophrenia. We applied averaged path-length (PL), clustering coefficient, and density (D) measurements to data from diffusion magnetic resonance and electroencephalography in 39 schizophrenia patients and 79 controls. Functional data were collected for the global and theta frequency bands during an odd-ball task, prior to stimulus delivery and at the corresponding processing window. Connectivity matrices were constructed from tractography and registered cortical segmentations (structural) and phase-locking values (functional). Both groups showed a significant electroencephalographic task-related modulation (change between prestimulus and response windows) in the global and theta bands. Patients showed larger structural PL and prestimulus density in the global and theta bands, and lower PL task-related modulation in the theta band. Structural network values predicted prestimulus global band values in controls and global band task-related modulation in patients. Abnormal functional values found in patients (prestimulus density in the global and theta bands and task-related modulation in the theta band) were not predicted by structural data in this group. Structural and functional network abnormalities respectively predicted cognitive performance and positive symptoms in patients. Taken together, the alterations in the structural and functional theta networks in the patients and the lack of significant relations between these alterations, suggest that these types of network abnormalities exist in different groups of schizophrenia patients.

Four-way multimodal fusion of 7 T imaging data using an mCCA+jICA model in first-episode schizophrenia

Acquisition of multimodal brain imaging data for the same subject has become more common leading to a growing interest in determining the intermodal relationships between imaging modalities to further elucidate the pathophysiology of schizophrenia. Multimodal data have previously been individually analyzed and subsequently integrated; however, these analysis techniques lack the ability to examine true modality inter-relationships. The utilization of a multiset canonical correlation and joint independent component analysis (mCCA + jICA) model for data fusion allows shared or distinct abnormalities between modalities to be examined. In this study, first-episode schizophrenia patients (n_SZ =19) and matched controls (n_HC =21) completed a resting-state functional magnetic resonance imaging (fMRI) scan at 7 T. Grey matter (GM), white matter (WM), cerebrospinal fluid (CSF), and amplitude of low frequency fluctuation (ALFF) maps were used as features in a mCCA + jICA model. Results of the mCCA + jICA model indicated three joint group-discriminating components (GM-CSF, WM-ALFF, GM-ALFF) and two modality-unique group-discriminating components (GM, WM). The joint component findings are highlighted by GM basal ganglia, somatosensory, parietal lobe, and thalamus abnormalities associated with ventricular CSF volume; WM occipital and frontal lobe abnormalities associated with temporal lobe function; and GM frontal, temporal, parietal, and occipital lobe abnormalities associated with caudate function. These results support and extend major findings throughout the literature using independent single modality analyses. The multimodal fusion of 7 T data in this study provides a more comprehensive illustration of the relationships between underlying neuronal abnormalities associated with schizophrenia than examination of imaging data independently.

Multimodal Fusion With Reference: Searching for Joint Neuromarkers of Working Memory Deficits in Schizophrenia

By exploiting cross-information among multiple imaging data, multimodal fusion has often been used to better understand brain diseases. However, most current fusion approaches are blind, without adopting any prior information. There is increasing interest to uncover the neurocognitive mapping of specific clinical measurements on enriched brain imaging data; hence, a supervised, goal-directed model that employs prior information as a reference to guide multimodal data fusion is much needed and becomes a natural option. Here, we proposed a fusion with reference model called "multi-site canonical correlation analysis with reference + joint-independent component analysis" (MCCAR+jICA), which can precisely identify co-varying multimodal imaging patterns closely related to the reference, such as cognitive scores. In a three-way fusion simulation, the proposed method was compared with its alternatives on multiple facets; MCCAR+jICA outperforms others with higher estimation precision and high accuracy on identifying a target component with the right correspondence. In human imaging data, working memory performance was utilized as a reference to investigate the co-varying working memory-associated brain patterns among three modalities and how they are impaired in schizophrenia. Two independent cohorts (294 and 83 subjects respectively) were used. Similar brain maps were identified between the two cohorts along with substantial overlaps in the central executive network in fMRI, salience network in sMRI, and major white matter tracts in dMRI. These regions have been linked with working memory deficits in schizophrenia in multiple reports and MCCAR+jICA further verified them in a repeatable, joint manner, demonstrating the ability of the proposed method to identify potential neuromarkers for mental disorders.

Schizophrenia Shows Disrupted Links between Brain Volume and Dynamic Functional Connectivity

Studies featuring multimodal neuroimaging data fusion for understanding brain function and structure, or disease characterization, leverage the partial information available in each of the modalities to reveal data variations not exhibited through the independent analyses. Similar to other complex syndromes, the characteristic brain abnormalities in schizophrenia may be better understood with the help of the additional information conveyed by leveraging an advanced modeling method involving multiple modalities. In this study, we propose a novel framework to fuse feature spaces corresponding to functional magnetic resonance imaging (functional) and gray matter (structural) data from 151 schizophrenia patients and 163 healthy controls. In particular, the features for the functional and structural modalities include dynamic (i.e., time-varying) functional network connectivity (dFNC) maps and the intensities of the gray matter (GM) maps, respectively. The dFNC maps are estimated from group independent component analysis (ICA) network time-courses by first computing windowed functional correlations using a sliding window approach, and then estimating subject specific states from this windowed data using temporal ICA followed by spatio-temporal regression. For each subject, the functional data features are horizontally concatenated with the corresponding GM features to form a combined feature space that is subsequently decomposed through a symmetric multimodal fusion approach involving a combination of multiset canonical correlation analysis (mCCA) and joint ICA (jICA). Our novel combined analyses successfully linked changes in the two modalities and revealed significantly disrupted links between GM volumes and time-varying functional connectivity in schizophrenia. Consistent with prior research, we found significant group differences in GM comprising regions in the superior parietal lobule, precuneus, postcentral gyrus, medial/superior frontal gyrus, superior/middle temporal gyrus, insula and fusiform gyrus, and several significant aberrations in the inter-regional functional connectivity strength as well. Importantly, structural and dFNC measures have independently shown changes associated with schizophrenia, and in this work we begin the process of evaluating the links between the two, which could shed light on the illness beyond what we can learn from a single imaging modality. In future work, we plan to evaluate replication of the inferred structure-function relationships in independent partitions of larger multi-modal schizophrenia datasets.

Blind Source Separation for Unimodal and Multimodal Brain Networks: A Unifying Framework for Subspace Modeling

In the past decade, numerous advances in the study of the human brain were fostered by successful applications of blind source separation (BSS) methods to a wide range of imaging modalities. The main focus has been on extracting "networks" represented as the underlying latent sources. While the broad success in learning latent representations from multiple datasets has promoted the wide presence of BSS in modern neuroscience, it also introduced a wide variety of objective functions, underlying graphical structures, and parameter constraints for each method. Such diversity, combined with a host of datatype-specific know-how, can cause a sense of disorder and confusion, hampering a practitioner's judgment and impeding further development. We organize the diverse landscape of BSS models by exposing its key features and combining them to establish a novel unifying view of the area. In the process, we unveil important connections among models according to their properties and subspace structures. Consequently, a high-level descriptive structure is exposed, ultimately helping practitioners select the right model for their applications. Equipped with that knowledge, we review the current state of BSS applications to neuroimaging. The gained insight into model connections elicits a broader sense of generalization, highlighting several directions for model development. In light of that, we discuss emerging multi-dataset multidimensional (MDM) models and summarize their benefits for the study of the healthy brain and disease-related changes.

Brain mechanisms of Change in Addictions Treatment: Models, Methods, and Emerging Findings

Increased understanding of "how" and "for whom" treatment works at the level of the brain has potential to transform addictions treatment through the development of innovative neuroscience-informed interventions. The 2015 Science of Change meeting bridged the fields of neuroscience and psychotherapy research to identify brain mechanisms of behavior change that are "common" across therapies, and "specific" to distinct behavioral interventions. Conceptual models of brain mechanisms underlying effects of Cognitive Behavioral Therapy, mindfulness interventions, and Motivational Interviewing were discussed. Presentations covered methods for integrating neuroimaging into psychotherapy research, and novel analytic approaches. Effects of heavy substance use on the brain, and recovery of brain functioning with sustained abstinence, which may be facilitated by cognitive training, were reviewed. Neuroimaging provides powerful tools for determining brain mechanisms underlying psychotherapy and medication effects, predicting and monitoring outcomes, developing novel interventions that target specific brain circuits, and identifying for whom an intervention will be effective.

Classifying Schizophrenia Using Multimodal Multivariate Pattern Recognition Analysis: Evaluating the Impact of Individual Clinical Profiles on the Neurodiagnostic Performance

Previous studies have shown that structural brain changes are among the best-studied candidate markers for schizophrenia (SZ) along with functional connectivity (FC) alterations of resting-state (RS) patterns. This study aimed to investigate effects of clinical and sociodemographic variables on the classification by applying multivariate pattern analysis (MVPA) to both gray matter (GM) volume and FC measures in patients with SZ and healthy controls (HC). RS and structural magnetic resonance imaging data (sMRI) from 74 HC and 71 SZ patients were obtained from a Mind Research Network COBRE dataset available via COINS (http://coins.mrn.org/dx). We used a MVPA framework using support-vector machines embedded in a repeated, nested cross-validation to generate a multi-modal diagnostic system and evaluate its generalizability. The dependence of neurodiagnostic performance on clinical and sociodemographic variables was evaluated. The RS classifier showed a slightly higher accuracy (70.5%) compared to the structural classifier (69.7%). The combination of sMRI and RS outperformed single MRI modalities classification by reaching 75% accuracy. The RS based moderator analysis revealed that the neurodiagnostic performance was driven by older SZ patients with an earlier illness onset and more pronounced negative symptoms. In contrast, there was no linear relationship between the clinical variables and neuroanatomically derived group membership measures. This study achieved higher accuracy distinguishing HC from SZ patients by fusing 2 imaging modalities. In addition the results of RS based moderator analysis showed that age of patients, as well as their age at the illness onset were the most important clinical features.

Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness

It is becoming increasingly clear that combining multi-modal brain imaging data is able to provide more information for individual subjects by exploiting the rich multimodal information that exists. However, the number of studies that do true multimodal fusion (i.e. capitalizing on joint information among modalities) is still remarkably small given the known benefits. In part, this is because multi-modal studies require broader expertise in collecting, analyzing, and interpreting the results than do unimodal studies. In this paper, we start by introducing the basic reasons why multimodal data fusion is important and what it can do, and importantly how it can help us avoid wrong conclusions and help compensate for imperfect brain imaging studies. We also discuss the challenges that need to be confronted for such approaches to be more widely applied by the community. We then provide a review of the diverse studies that have used multimodal data fusion (primarily focused on psychosis) as well as provide an introduction to some of the existing analytic approaches. Finally, we discuss some up-and-coming approaches to multi-modal fusion including deep learning and multimodal classification which show considerable promise. Our conclusion is that multimodal data fusion is rapidly growing, but it is still underutilized. The complexity of the human brain coupled with the incomplete measurement provided by existing imaging technology makes multimodal fusion essential in order to mitigate against misdirection and hopefully provide a key to finding the missing link(s) in complex mental illness.

Motor dysfunction within the schizophrenia-spectrum: A dimensional step towards an underappreciated domain

At the beginning of the 20th century, genuine motor abnormalities (GMA) were considered to be intricately linked to schizophrenia. Subsequently, however, GMA have been increasingly regarded as unspecific transdiagnostic phenomena or related to side effects of antipsychotic treatment. Despite possible medication confounds, within the schizophrenia spectrum GMA have been categorized into three broad categories, i.e. neurological soft signs, abnormal involuntary movements and catatonia. Schizophrenia patients show a substantial overlap across a broad range of distinct motor signs and symptoms suggesting a prominent involvement of the motor system in disease pathophysiology. There have been several attempts to increase reliability and validity in diagnosing schizophrenia based on behavior and neurobiology, yet relatively little attention has been paid to the motor domain in the past. Nevertheless, accumulating neuroscientific evidence suggests the possibility of a motor endophenotype in schizophrenia, and that GMA could represent a specific dimension within the schizophrenia-spectrum. Here, we review current neuroimaging research on GMA in schizophrenia with an emphasis on distinct and common mechanisms of brain dysfunction. Based on a dimensional approach we show that multimodal neuroimaging combined with fine-grained clinical examination can result in a comprehensive characterization of structural and functional brain changes that are presumed to underlie core GMA in schizophrenia. We discuss the possibility of a distinct motor domain, together with its implications for future research. Investigating GMA by means of multimodal neuroimaging can essentially contribute at identifying novel and biologically reliable phenotypes in psychiatry.

Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia

Functional connectivity (FC) patterns obtained from resting-state functional magnetic resonance imaging data are commonly employed to study neuropsychiatric conditions by using pattern classifiers such as the support vector machine (SVM). Meanwhile, a deep neural network (DNN) with multiple hidden layers has shown its ability to systematically extract lower-to-higher level information of image and speech data from lower-to-higher hidden layers, markedly enhancing classification accuracy. The objective of this study was to adopt the DNN for whole-brain resting-state FC pattern classification of schizophrenia (SZ) patients vs. healthy controls (HCs) and identification of aberrant FC patterns associated with SZ. We hypothesized that the lower-to-higher level features learned via the DNN would significantly enhance the classification accuracy, and proposed an adaptive learning algorithm to explicitly control the weight sparsity in each hidden layer via L1-norm regularization. Furthermore, the weights were initialized via stacked autoencoder based pre-training to further improve the classification performance. Classification accuracy was systematically evaluated as a function of (1) the number of hidden layers/nodes, (2) the use of L1-norm regularization, (3) the use of the pre-training, (4) the use of framewise displacement (FD) removal, and (5) the use of anatomical/functional parcellation. Using FC patterns from anatomically parcellated regions without FD removal, an error rate of 14.2% was achieved by employing three hidden layers and 50 hidden nodes with both L1-norm regularization and pre-training, which was substantially lower than the error rate from the SVM (22.3%). Moreover, the trained DNN weights (i.e., the learned features) were found to represent the hierarchical organization of aberrant FC patterns in SZ compared with HC. Specifically, pairs of nodes extracted from the lower hidden layer represented sparse FC patterns implicated in SZ, which was quantified by using kurtosis/modularity measures and features from the higher hidden layer showed holistic/global FC patterns differentiating SZ from HC. Our proposed schemes and reported findings attained by using the DNN classifier and whole-brain FC data suggest that such approaches show improved ability to learn hidden patterns in brain imaging data, which may be useful for developing diagnostic tools for SZ and other neuropsychiatric disorders and identifying associated aberrant FC patterns.

Sparse models for correlative and integrative analysis of imaging and genetic data

The development of advanced medical imaging technologies and high-throughput genomic measurements has enhanced our ability to understand their interplay as well as their relationship with human behavior by integrating these two types of datasets. However, the high dimensionality and heterogeneity of these datasets presents a challenge to conventional statistical methods; there is a high demand for the development of both correlative and integrative analysis approaches. Here, we review our recent work on developing sparse representation based approaches to address this challenge. We show how sparse models are applied to the correlation and integration of imaging and genetic data for biomarker identification. We present examples on how these approaches are used for the detection of risk genes and classification of complex diseases such as schizophrenia. Finally, we discuss future directions on the integration of multiple imaging and genomic datasets including their interactions such as epistasis.

PUNCH: Population Characterization of Heterogeneity

Neuropsychiatric disorders are notoriously heterogeneous in their presentation, which precludes straightforward and objective description of the differences between affected and typical populations that therefore makes finding reliable biomarkers a challenge. This difficulty underlines the need for reliable methods to capture sample characteristics of heterogeneity using a single continuous measure, incorporating the multitude of scores used to describe different aspects of functioning. This study addresses this challenge by proposing a general method of identifying and quantifying the heterogeneity of any clinical population using a severity measure called the PUNCH (Population Characterization of Heterogeneity). PUNCH is a decision level fusion technique to incorporate decisions of various phenotypic scores, while providing interpretable weights for scores. We provide applications of our framework to simulated datasets and to a large sample of youth with Autism Spectrum Disorder (ASD). Next we stratify PUNCH scores in our ASD sample and show how severity moderates findings of group differences in diffusion weighted brain imaging data; more severely affected subgroups of ASD show expanded differences compared to age and gender matched healthy controls. Results demonstrate the ability of our measure in quantifying the underlying heterogeneity of the clinical samples, and suggest its utility in providing researchers with reliable severity assessments incorporating population heterogeneity.

The multimodal connectivity of the hippocampal complex in auditory and visual hallucinations

Hallucinations constitute one of the most representative and disabling symptoms of schizophrenia. Several Magnetic Resonance Imaging (MRI) findings support the hypothesis that distinct patterns of connectivity, particularly within networks involving the hippocampal complex (HC), could be associated with different hallucinatory modalities. The aim of this study was to investigate HC connectivity as a function of the hallucinatory modality, that is, auditory or visual. Two carefully selected subgroups of schizophrenia patients with only auditory hallucinations (AH) or with audio-visual hallucinations (A+VH) were compared using the following three complementary multimodal MRI methods: resting state functional MRI, diffusion MRI and structural MRI were used to analyze seed-based Functional Connectivity (sb-FC), Tract-Based Spatial Statistics (TBSS) and shape analysis, respectively. Sb-FC was significantly higher between the HC, the medial prefrontal cortex (mPFC) and the caudate nuclei in A+VH patients compared with the AH group. Conversely, AH patients exhibited a higher sb-FC between the HC and the thalamus in comparison with the A+VH group. In the A+VH group, TBSS showed specific higher white matter connectivity in the pathways connecting the HC with visual areas, such as the forceps major and the inferior-fronto-occipital fasciculus than in the AH group. Finally, shape analysis showed localized hippocampal hypertrophy in the A+VH group. Functional results support the fronto-limbic dysconnectivity hypothesis of schizophrenia, while specific structural findings indicate that plastic changes are associated with hallucinations. Together, these results suggest that there are distinct connectivity patterns in patients with schizophrenia that depend on the sensory-modality, with specific involvement of the HC in visual hallucinations.

Discriminative analysis of non-linear brain connectivity in schizophrenia: an fMRI Study

Background: Dysfunctional integration of distributed brain networks is believed to be the cause of schizophrenia, and resting-state functional connectivity analyses of schizophrenia have attracted considerable attention in recent years. Unfortunately, existing functional connectivity analyses of schizophrenia have been mostly limited to linear associations.

Objective: The objective of the present study is to evaluate the discriminative power of non-linear functional connectivity and identify its changes in schizophrenia.

Method: A novel measure utilizing the extended maximal information coefficient was introduced to construct non-linear functional connectivity. In conjunction with multivariate pattern analysis, the new functional connectivity successfully discriminated schizophrenic patients from healthy controls with relative higher accuracy rate than the linear measure.

Result: We found that the strength of the identified non-linear functional connections involved in the classification increased in patients with schizophrenia, which was opposed to its linear counterpart. Further functional network analysis revealed that the changes of the non-linear and linear connectivity have similar but not completely the same spatial distribution in human brain.

Conclusion: The classification results suggest that the non-linear functional connectivity provided useful discriminative power in diagnosis of schizophrenia, and the inverse but similar spatial distributed changes between the non-linear and linear measure may indicate the underlying compensatory mechanism and the complex neuronal synchronization underlying the symptom of schizophrenia.

Function-structure associations of the brain: evidence from multimodal connectivity and covariance studies

Despite significant advances in multimodal imaging techniques and analysis approaches, unimodal studies are still the predominant way to investigate brain changes or group differences, including structural magnetic resonance imaging (sMRI), functional MRI (fMRI), diffusion tensor imaging (DTI) and electroencephalography (EEG). Multimodal brain studies can be used to understand the complex interplay of anatomical, functional and physiological brain alterations or development, and to better comprehend the biological significance of multiple imaging measures. To examine the function-structure associations of the brain in a more comprehensive and integrated manner, we reviewed a number of multimodal studies that combined two or more functional (fMRI and/or EEG) and structural (sMRI and/or DTI) modalities. In this review paper, we specifically focused on multimodal neuroimaging studies on cognition, aging, disease and behavior. We also compared multiple analysis approaches, including univariate and multivariate methods. The possible strengths and limitations of each method are highlighted, which can guide readers when selecting a method based on a given research question. In particular, we believe that multimodal fusion approaches will shed further light on the neuronal mechanisms underlying the major structural and functional pathophysiological features of both the healthy brain (e.g. development) or the diseased brain (e.g. mental illness) and, in the latter case, may provide a more sensitive measure than unimodal imaging for disease classification, e.g. multimodal biomarkers, which potentially can be used to support clinical diagnosis based on neuroimaging techniques.

Combination of Resting State fMRI, DTI, and sMRI Data to Discriminate Schizophrenia by N-way MCCA + jICA

Multimodal brain imaging data have shown increasing utility in answering both scientifically interesting and clinically relevant questions. Each brain imaging technique provides a different view of brain function or structure, while multimodal fusion capitalizes on the strength of each and may uncover hidden relationships that can merge findings from separate neuroimaging studies. However, most current approaches have focused on pair-wise fusion and there is still relatively little work on N-way data fusion and examination of the relationships among multiple data types. We recently developed an approach called “mCCA + jICA” as a novel multi-way fusion method which is able to investigate the disease risk factors that are either shared or distinct across multiple modalities as well as the full correspondence across modalities. In this paper, we applied this model to combine resting state fMRI (amplitude of low-frequency fluctuation, ALFF), gray matter (GM) density, and DTI (fractional anisotropy, FA) data, in order to elucidate the abnormalities underlying schizophrenia patients (SZs, n = 35) relative to healthy controls (HCs, n = 28). Both modality-common and modality-unique abnormal regions were identified in SZs, which were then used for successful classification for seven modality-combinations, showing the potential for a broad applicability of the mCCA + jICA model and its results. In addition, a pair of GM-DTI components showed significant correlation with the positive symptom subscale of Positive and Negative Syndrome Scale (PANSS), suggesting that GM density changes in default model network along with white-matter disruption in anterior thalamic radiation are associated with increased positive PANSS. Findings suggest the DTI anisotropy changes in frontal lobe may relate to the corresponding functional/structural changes in prefrontal cortex and superior temporal gyrus that are thought to play a role in the clinical expression of SZ.

Three-way (N-way) fusion of brain imaging data based on mCCA+jICA and its application to discriminating schizophrenia

Multimodal fusion is an effective approach to better understand brain diseases. However, most such instances have been limited to pair-wise fusion; because there are often more than two imaging modalities available per subject, there is a need for approaches that can combine multiple datasets optimally. In this paper, we extended our previous two-way fusion model called "multimodal CCA+joint ICA", to three or N-way fusion, that enables robust identification of correspondence among N data types and allows one to investigate the important question of whether certain disease risk factors are shared or distinct across multiple modalities. We compared "mCCA+jICA" with its alternatives in a 3-way fusion simulation and verified its advantages in both decomposition accuracy and modal linkage detection. We also applied it to real functional Magnetic Resonance Imaging (fMRI)-Diffusion Tensor Imaging (DTI) and structural MRI fusion to elucidate the abnormal architecture underlying schizophrenia (n=97) relative to healthy controls (n=116). Both modality-common and modality-unique abnormal regions were identified in schizophrenia. Specifically, the visual cortex in fMRI, the anterior thalamic radiation (ATR) and forceps minor in DTI, and the parietal lobule, cuneus and thalamus in sMRI were linked and discriminated between patients and controls. One fMRI component with regions of activity in motor cortex and superior temporal gyrus individually discriminated schizophrenia from controls. Finally, three components showed significant correlation with duration of illness (DOI), suggesting that lower gray matter volumes in parietal, frontal, and temporal lobes and cerebellum are associated with increased DOI, along with white matter disruption in ATR and cortico-spinal tracts. Findings suggest that the identified fractional anisotropy changes may relate to the corresponding functional/structural changes in the brain that are thought to play a role in the clinical expression of schizophrenia. The proposed "mCCA+jICA" method showed promise for elucidating the joint or coupled neuronal abnormalities underlying mental illnesses and improves our understanding of the disease process.

Gene, brains, and environment-genetic neuroimaging of depression

Depression, conceptualized as major depressive disorder (MDD), is a complex psychiatric disorder with multiple behavioral changes and alterations in various brain regions. Biochemically, serotonin and others substances like GABA, glutamate, norepinephrin, adrenaline/noradrenaline play an essential role in the pathogenesis of MDD. The paper reviews recent human neuroimaging findings on how the genes underlying these biochemical substances modulate neural activity, behavior, and ultimately clinical symptoms. Current data provide solid evidence that genes related to serotonin impact emotion-related neural activity in the amygdala and the anterior cingulate cortex. By contrast, evidence is not as strong for genes related to biochemical substances other than serotonin and other regions of the brain. The review concludes with discussing future genetic, neural, and clinical challenges that point out the central role of gene × environment and brain × environment interactions as genetic and neural predispositions of depression.