Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Interpersonal neural synchrony and mental disorders: unlocking potential pathways for clinical interventions

Introduction

Interpersonal synchronization involves the alignment of behavioral, affective, physiological, and brain states during social interactions. It facilitates empathy, emotion regulation, and prosocial commitment. Mental disorders characterized by social interaction dysfunction, such as Autism Spectrum Disorder (ASD), Reactive Attachment Disorder (RAD), and Social Anxiety Disorder (SAD), often exhibit atypical synchronization with others across multiple levels. With the introduction of the "second-person" neuroscience perspective, our understanding of interpersonal neural synchronization (INS) has improved, however, so far, it has hardly impacted the development of novel therapeutic interventions.

Methods

To evaluate the potential of INS-based treatments for mental disorders, we performed two systematic literature searches identifying studies that directly target INS through neurofeedback (12 publications; 9 independent studies) or brain stimulation techniques (7 studies), following PRISMA guidelines. In addition, we narratively review indirect INS manipulations through behavioral, biofeedback, or hormonal interventions. We discuss the potential of such treatments for ASD, RAD, and SAD and using a systematic database search assess the acceptability of neurofeedback (4 studies) and neurostimulation (4 studies) in patients with social dysfunction.

Results

Although behavioral approaches, such as engaging in eye contact or cooperative actions, have been shown to be associated with increased INS, little is known about potential long-term consequences of such interventions. Few proof-of-concept studies have utilized brain stimulation techniques, like transcranial direct current stimulation or INS-based neurofeedback, showing feasibility and preliminary evidence that such interventions can boost behavioral synchrony and social connectedness. Yet, optimal brain stimulation protocols and neurofeedback parameters are still undefined. For ASD, RAD, or SAD, so far no randomized controlled trial has proven the efficacy of direct INS-based intervention techniques, although in general brain stimulation and neurofeedback methods seem to be well accepted in these patient groups.

Discussion

Significant work remains to translate INS-based manipulations into effective treatments for social interaction disorders. Future research should focus on mechanistic insights into INS, technological advancements, and rigorous design standards. Furthermore, it will be key to compare interventions directly targeting INS to those targeting other modalities of synchrony as well as to define optimal target dyads and target synchrony states in clinical interventions.

Data leakage inflates prediction performance in connectome-based machine learning models

Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Testing the sensitivity of diagnosis-derived patterns in functional brain networks to symptom burden in a Norwegian youth sample

Aberrant brain network development represents a putative aetiological component in mental disorders, which typically emerge during childhood and adolescence. Previous studies have identified resting-state functional connectivity (RSFC) patterns reflecting psychopathology, but the generalisability to other samples and politico-cultural contexts has not been established. We investigated whether a previously identified cross-diagnostic case-control and autism spectrum disorder (ASD)-specific pattern of RSFC (discovery sample; aged 5-21 from New York City, USA; n = 1666) could be validated in a Norwegian convenience-based youth sample (validation sample; aged 9-25 from Oslo, Norway; n = 531). As a test of generalisability, we investigated if these diagnosis-derived RSFC patterns were sensitive to levels of symptom burden in both samples, based on an independent measure of symptom burden. Both the cross-diagnostic and ASD-specific RSFC pattern were validated across samples. Connectivity patterns were significantly associated with thematically appropriate symptom dimensions in the discovery sample. In the validation sample, the ASD-specific RSFC pattern showed a weak, inverse relationship with symptoms of conduct problems, hyperactivity and prosociality, while the cross-diagnostic pattern was not significantly linked to symptoms. Diagnosis-derived connectivity patterns in a developmental clinical US sample were validated in a convenience sample of Norwegian youth, however, they were not associated with mental health symptoms.

The effects of data leakage on connectome-based machine learning models

Predictive modeling has now become a central technique in neuroimaging to identify complex brain-behavior relationships and test their generalizability to unseen data. However, data leakage, which unintentionally breaches the separation between data used to train and test the model, undermines the validity of predictive models. Previous literature suggests that leakage is generally pervasive in machine learning, but few studies have empirically evaluated the effects of leakage in neuroimaging data. Although leakage is always an incorrect practice, understanding the effects of leakage on neuroimaging predictive models provides insight into the extent to which leakage may affect the literature. Here, we investigated the effects of leakage on machine learning models in two common neuroimaging modalities, functional and structural connectomes. Using over 400 different pipelines spanning four large datasets and three phenotypes, we evaluated five forms of leakage fitting into three broad categories: feature selection, covariate correction, and lack of independence between subjects. As expected, leakage via feature selection and repeated subjects drastically inflated prediction performance. Notably, other forms of leakage had only minor effects (e.g., leaky site correction) or even decreased prediction performance (e.g., leaky covariate regression). In some cases, leakage affected not only prediction performance, but also model coefficients, and thus neurobiological interpretations. Finally, we found that predictive models using small datasets were more sensitive to leakage. Overall, our results illustrate the variable effects of leakage on prediction pipelines and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Morphologic alterations of the fear circuitry: the role of sex hormones and oral contraceptives

Background

Endogenous sex hormones and oral contraceptives (OCs) have been shown to influence key regions implicated in fear processing. While OC use has been found to impact brain morphology, methodological challenges remain to be addressed, such as avoiding selection bias between OC users and non-users, as well as examining potential lasting effects of OC intake.

Objective

We investigated the current and lasting effects of OC use, as well as the interplay between the current hormonal milieu and history of hormonal contraception use on structural correlates of the fear circuitry. We also examined the role of endogenous and exogenous sex hormones within this network.

Methods

We recruited healthy adults aged 23-35 who identified as women currently using (n = 62) or having used (n = 37) solely combined OCs, women who never used any hormonal contraceptives (n = 40), or men (n = 41). Salivary endogenous sex hormones and current users' salivary ethinyl estradiol (EE) were assessed using liquid chromatography - tandem mass spectrometry. Using structural magnetic resonance imaging, we extracted surface-based gray matter volumes (GMVs) and cortical thickness (CT) for regions of interest of the fear circuitry. Exploratory whole-brain analyses were conducted with surface-based and voxel-based morphometry methods.

Results

Compared to men, all three groups of women exhibited a larger GMV of the dorsal anterior cingulate cortex, while only current users showed a thinner ventromedial prefrontal cortex. Irrespective of the menstrual cycle phase, never users exhibited a thicker right anterior insular cortex than past users. While associations with endogenous sex hormones remain unclear, we showed that EE dosage in current users had a greater influence on brain anatomy compared to salivary EE levels and progestin androgenicity, with lower doses being associated with smaller cortical GMVs.

Discussion

Our results highlight a sex difference for the dorsal anterior cingulate cortex GMV (a fear-promoting region), as well as a reduced CT of the ventromedial prefrontal cortex (a fear-inhibiting region) specific to current OC use. Precisely, this finding was driven by lower EE doses. These findings may represent structural vulnerabilities to anxiety and stress-related disorders. We showed little evidence of durable anatomical effects, suggesting that OC intake can (reversibly) affect fear-related brain morphology.

Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines

BACKGROUND AND OBJECTIVE

Mental workload (MWL) is a relevant construct involved in all cognitively demanding activities, and its assessment is an important goal in many research fields. This paper aims at evaluating the reproducibility and sensitivity of MWL assessment from EEG signals considering the effects of different electrode configurations and pre-processing pipelines (PPPs).

METHODS

Thirteen young healthy adults were enrolled and were asked to perform 45 min of Simon's task to elicit a cognitive demand. EEG data were collected using a 32-channel system with different electrode configurations (fronto-parietal; Fz and Pz; Cz) and analyzed using different PPPs, from the simplest bandpass filtering to the combination of filtering, Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA). The reproducibility of MWL indexes estimation and the sensitivity of their changes were assessed using Intraclass Correlation Coefficient and statistical analysis.

RESULTS

MWL assessed with different PPPs showed reliability ranging from good to very good in most of the electrode configurations (average consistency > 0.87 and average absolute agreement > 0.92). Larger fronto-parietal electrode configurations, albeit being more affected by the choice of PPPs, provide better sensitivity in the detection of MWL changes if compared to a single-electrode configuration (18 vs. 10 statistically significant differences detected, respectively).

CONCLUSIONS

The most complex PPPs have been proven to ensure good reliability (>0.90) and sensitivity in all experimental conditions. In conclusion, we propose to use at least a two-electrode configuration (Fz and Pz) and complex PPPs including at least the ICA algorithm (even better including ASR) to mitigate artifacts and obtain reliable and sensitive MWL assessment during cognitive tasks.