Reproducibility and replicability in neuroimaging data analysis

Time-Varying Spatial Propagation of Brain Networks in fMRI Data

Spontaneous neural activity coherently relays information across the brain. Several efforts have been made to understand how spontaneous neural activity evolves at the macro-scale level as measured by resting-state functional magnetic resonance imaging (rsfMRI). Previous studies observe the global patterns and flow of information in rsfMRI using methods such as sliding window or temporal lags. However, to our knowledge, no studies have examined spatial propagation patterns evolving with time across multiple overlapping 4D networks. Here, we propose a novel approach to study how dynamic states of the brain networks spatially propagate and evaluate whether these propagating states contain information relevant to mental illness. We implement a lagged windowed correlation approach to capture voxel-wise network-specific spatial propagation patterns in dynamic states. Results show systematic spatial state changes over time, which we confirmed are replicable across multiple scan sessions using human connectome project data. We observe networks varying in propagation speed; for example, the default mode network (DMN) propagates slowly and remains positively correlated with blood oxygenation level-dependent (BOLD) signal for 6-8 s, whereas the visual network propagates much quicker. We also show that summaries of network-specific propagative patterns are linked to schizophrenia. More specifically, we find significant group differences in multiple dynamic parameters between patients with schizophrenia and controls within four large-scale networks: default mode, temporal lobe, subcortical, and visual network. Individuals with schizophrenia spend more time in certain propagating states. In summary, this study introduces a promising general approach to exploring the spatial propagation in dynamic states of brain networks and their associated complexity and reveals novel insights into the neurobiology of schizophrenia.

Higher amplitudes of visual networks are associated with trait- but not state-depression

Despite depression being a leading cause of global disability, neuroimaging studies have struggled to identify replicable neural correlates of depression or explain limited variance. This challenge may, in part, stem from the intertwined state (current symptoms; variable) and trait (general propensity; stable) experiences of depression.Here, we sought to disentangle state from trait experiences of depression by leveraging a longitudinal cohort and stratifying individuals into four groups: those in remission ('trait depression group'), those with large longitudinal severity changes in depression symptomatology ('state depression group'), and their respective matched control groups (total analytic n = 1030). We hypothesized that spatial network organization would be linked to trait depression due to its temporal stability, whereas functional connectivity between networks would be more sensitive to state-dependent depression symptoms due to its capacity to fluctuate.We identified 15 large-scale probabilistic functional networks from resting-state fMRI data and performed group comparisons on the amplitude, connectivity, and spatial overlap between these networks, using matched control participants as reference. Our findings revealed higher amplitude in visual networks for the trait depression group at the time of remission, in contrast to controls. This observation may suggest altered visual processing in individuals predisposed to developing depression over time. No significant group differences were observed in any other network measures for the trait-control comparison, nor in any measures for the state-control comparison. These results underscore the overlooked contribution of visual networks to the psychopathology of depression and provide evidence for distinct neural correlates between state and trait experiences of depression.

Imaging-genomic spatial-modality attentive fusion for studying neuropsychiatric disorders

Multimodal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of subjects' underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domain specific, leading to data being represented in dissimilar subspaces, and (ii) fused data are often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multimodal artificial intelligence model with a novel fusion module inspired by a bottleneck attention module. We use deep neural networks to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multimodal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (p < .0001), outperforming state-of-the-art unimodal and multimodal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network's decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multimodal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to SZ. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.

Risk of intraoperative hemorrhage during cesarean scar ectopic pregnancy surgery: development and validation of an interpretable machine learning prediction model

Background

Current models for predicting intraoperative hemorrhage in cesarean scar ectopic pregnancy (CSEP) are constrained by known risk factors and conventional statistical methods. Our objective is to develop an interpretable prediction model using machine learning (ML) techniques to assess the risk of intraoperative hemorrhage during CSEP in women, followed by external validation and clinical application.

Methods

This multicenter retrospective study utilized electronic medical record (EMR) data from four tertiary medical institutions. The model was developed using data from 1680 patients with CSEP diagnosed and treated at Qilu Hospital of Shandong University, Chongqing Health Center for Women and Children, and Dezhou Maternal and Child Health Care Hospital between January 1, 2008, and December 31, 2023. External validation data were obtained from Liao Cheng Dong Chang Fu District Maternal and Child Health Care Hospital between January 1, 2021, and December 31, 2023. Random forest (RF), Lasso, Boruta, and Extreme Gradient Boosting (XGBoost) were employed to identify the most influential variables in the model development data set; the best variables were selected based on reaching the λmin value. Model development involved eight machine learning methods with ten-fold cross-validation. Accuracy and decision curve analysis (DCA) were used to assess model performance for selection of the optimal model. Internal validation of the model utilized area under the receiver operating characteristic curve (AUC), sensitivity, specificity, Matthews correlation coefficient, and F1 score. These same indicators were also applied to evaluate external validation performance of the model. Finally, visualization techniques were used to present the optimal model which was then deployed for clinical application via network applications.

Findings

Setting λmin at the value of 0.003, the optimal variable combination containing 9 variables was selected for model development. The optimal prediction model (Bayes) had an accuracy of 0.879 (95% CI: 0.857-0.901) an AUC of 0.882 (95% CI: 0.860-0.904), a DCA curve maximum threshold probability of 0.41, and a maximum return of 7.86%. The internal validation accuracy was 0.869 (95% CI: 0.847-0.891), an AUC of 0.822 (95% CI: 0.801-0.843), a sensitivity of 0.938, a specificity of 0.422, a Matthews correlation coefficient of 0.392, and an F1 score of 0.925. In the external validation, the accuracy was 0.936 (95% CI: 0.913-0.959), an AUC of 0.853 (95% CI: 0.832-0.874), a sensitivity of 0.954, a specificity of 0.5, a Matthews correlation coefficient of 0.365, and an F1 score of 0.966. This indicates that the prediction model performed well in both internal and external validation.

Interpretation

The developed prediction model, deployed in the network application, is capable of forecasting the risk of intraoperative hemorrhage during CSEP. This tool can facilitate targeted preoperative assessment and clinical decision-making for clinicians. Prospective data should be utilized in future studies to further validate the extended applicability of the model.

Funding

Natural Science Foundation of Shandong Province; Qilu Hospital of Shandong University.

Inter-Modality Source Coupling: A Fully-Automated Whole-Brain Data-Driven Structure-Function Fingerprint Shows Replicable Links to Reading in a Large-Scale (N∼8K) Analysis

OBJECTIVE

Both structural and functional brain changes have been individually associated with developing cognitive processes such as reading. However, there is limited research about the combined influence of resting-state functional and structural magnetic resonance imaging (rs-fMRI and sMRI) features in reading development, which could provide insights into the interplay between brain structure and function in shaping cognitive growth. We propose a method called inter-modality source coupling (IMSC) to study the coupling between the rs-fMRI and sMRI and its relationship to reading ability in school-age children.

METHODS

This approach is applied to baseline data from four thousand participants (9-11 years) and replicated in a second group. Our analysis focused on the relationship of IMSC to overall reading score.

RESULTS

Our findings indicate that higher reading ability was linked with increased function-structure coupling among higher-level cortical regions, particularly those links between the inferior parietal lobule and inferior frontal areas, and conversely, lower reading ability was associated with enhanced function-structure coupling among the fusiform and lingual gyrus. Our study found evidence of spatial correspondence between the data indicating an interplay between brain structure and function in our participants.

CONCLUSION

Our approach revealed a linked pattern of whole brain structure to the corresponding functional connectivity pattern that correlated with reading ability. This novel IMSC analysis method provides a new approach to study the multimodal relationship between brain function and structure.

SIGNIFICANCE

These findings have interesting implications for understanding the multimodal complexity underlying the development of the neural basis for reading ability in school-aged children.

The language network ages well: Preserved selectivity, lateralization, and within-network functional synchronization in older brains

Healthy aging is associated with structural and functional brain changes. However, cognitive abilities differ from one another in how they change with age: whereas executive functions, like working memory, show age-related decline, aspects of linguistic processing remain relatively preserved (Hartshorne et al., 2015). This heterogeneity of the cognitive-behavioral landscape in aging predicts differences among brain networks in whether and how they should change with age. To evaluate this prediction, we used individual-subject fMRI analyses ('precision fMRI') to examine the language-selective network (Fedorenko et al., 2024) and the Multiple Demand (MD) network, which supports executive functions (Duncan et al., 2020), in older adults (n=77) relative to young controls (n=470). In line with past claims, relative to young adults, the MD network of older adults shows weaker and less spatially extensive activations during an executive function task and reduced within-network functional synchronization. However, in stark contrast to the MD network, we find remarkable preservation of the language network in older adults. Their language network responds to language as strongly and selectively as in younger adults, and is similarly lateralized and internally synchronized. In other words, the language network of older adults looks indistinguishable from that of younger adults. Our findings align with behavioral preservation of language skills in aging and suggest that some networks remain young-like, at least on standard measures of function and connectivity.

A survey of brain functional network extraction methods using fMRI data

Functional network (FN) analyses play a pivotal role in uncovering insights into brain function and understanding the pathophysiology of various brain disorders. This paper focuses on classical and advanced methods for deriving brain FNs from functional magnetic resonance imaging (fMRI) data. We systematically review their foundational principles, advantages, shortcomings, and interrelations, encompassing both static and dynamic FN extraction approaches. In the context of static FN extraction, we present hypothesis-driven methods such as region of interest (ROI)-based approaches as well as data-driven methods including matrix decomposition, clustering, and deep learning. For dynamic FN extraction, both window-based and windowless methods are surveyed with respect to the estimation of time-varying FN and the subsequent computation of FN states. We also discuss the scope of application of the various methods and avenues for future improvements.

Inter-modality source coupling: a fully automated whole-brain data-driven structure-function fingerprint shows replicable links to reading in large-scale (N~8K) analysis

We propose a method called inter-modality source coupling (IMSC) to study the coupling between the function and structure of the brain and its relationship to reading ability in school-age children. We performed an analysis focused on the relationship of IMSC to overall reading score. Our findings indicate that key brain regions are linked to differences in function-structure coupling in the context of reading ability and language processing. Higher reading ability was linked with increased function-structure coupling among higher-level cortical regions and lower reading ability was associated with enhanced function-structure coupling among occipital networks and early language processes areas.

Large-Scale Cortical Network Analysis and Classification of MI-BCI Tasks Based on Bayesian Nonnegative Matrix Factorization

Motor imagery (MI) is a high-level cognitive process that has been widely applied to clinical rehabilitation and brain-computer interfaces (BCIs). However, the decoding of MI tasks still faces challenges, and the neural mechanisms underlying its application are unclear, which seriously hinders the development of MI-based clinical applications and BCIs. Here, we combined EEG source reconstruction and Bayesian nonnegative matrix factorization (NMF) methods to construct large-scale cortical networks of left-hand and right-hand MI tasks. Compared to right-hand MI, the results showed that the significantly increased functional network connectivities (FNCs) mainly located among the visual network (VN), sensorimotor network (SMN), right temporal network, right central executive network, and right parietal network in the left-hand MI at the β (13-30Hz) and all (8-30Hz) frequency bands. For the network properties analysis, we found that the clustering coefficient, global efficiency, and local efficiency were significantly increased and characteristic path length was significantly decreased in left-hand MI compared to right-hand MI at the β and all frequency bands. These network pattern differences indicated that the left-hand MI may need more modulation of multiple large-scale networks (i.e., VN and SMN) mainly located in the right hemisphere. Finally, based on the spatial pattern network of FNC and network properties, we propose a classification model. The proposed model achieves a top classification accuracy of 78.2% in cross-subject two-class MI-BCI tasks. Overall, our findings provide new insights into the neural mechanisms of MI and a potential network biomarker to identify MI-BCI tasks.

Multimodal active subspace analysis for computing assessment oriented subspaces from neuroimaging data

BACKGROUND

For successful biomarker discovery, it is essential to develop computational frameworks that summarize high-dimensional neuroimaging data in terms of involved sub-systems of the brain, while also revealing underlying heterogeneous functional and structural changes covarying with specific cognitive and biological traits. However, unsupervised decompositions do not inculcate clinical assessment information, while supervised approaches extract only individual feature importance, thereby impeding qualitative interpretation at the level of subspaces.

NEW METHOD

We present a novel framework to extract robust multimodal brain subspaces associated with changes in a given cognitive or biological trait. Our approach involves active subspace learning on the gradients of a trained machine learning model followed by clustering to extract and summarize the most salient and consistent subspaces associated with the target variable.

RESULTS

Through a rigorous cross-validation procedure on an Alzheimer's disease (AD) dataset, our framework successfully extracts multimodal subspaces specific to a given clinical assessment (e.g., memory and other cognitive skills), and also retains predictive performance in standard machine learning algorithms. We also show that the salient active subspace directions occur consistently across randomly sub-sampled repetitions of the analysis.

COMPARISON WITH EXISTING METHOD(S)

Compared to existing unsupervised decompositions based on principle component analysis, the subspace components in our framework retain higher predictive information.

CONCLUSIONS

As an important step towards biomarker discovery, our framework not only uncovers AD-related brain regions in the associated brain subspaces, but also enables automated identification of multiple underlying structural and functional sub-systems of the brain that collectively characterize changes in memory and proficiency in cognitive skills related to brain disorders like AD.

Intra-V1 functional networks and classification of observed stimuli

Introduction

Previous studies suggest that co-fluctuations in neural activity within V1 (measured with fMRI) carry information about observed stimuli, potentially reflecting various cognitive mechanisms. This study explores the neural sources shaping this information by using different fMRI preprocessing methods. The common response to stimuli shared by all individuals can be emphasized by using inter-subject correlations or de-emphasized by deconvolving the fMRI with hemodynamic response functions (HRFs) before calculating the correlations. The latter approach shifts the balance towards participant-idiosyncratic activity.

Methods

Here, we used multivariate pattern analysis of intra-V1 correlation matrices to predict the Level or Shape of observed Navon letters employing the types of correlations described above. We assessed accuracy in inter-subject prediction of specific conjunctions of properties, and attempted intra-subject cross-classification of stimulus properties (i.e., prediction of one feature despite changes in the other). Weight maps from successful classifiers were projected onto the visual field. A control experiment investigated eye-movement patterns during stimuli presentation.

Results

All inter-subject classifiers accurately predicted the Level and Shape of specific observed stimuli. However, successful intra-subject cross-classification was achieved only for stimulus Level, but not Shape, regardless of preprocessing scheme. Weight maps for successful Level classification differed between inter-subject correlations and deconvolved correlations. The latter revealed asymmetries in visual field link strength that corresponded to known perceptual asymmetries. Post-hoc measurement of eyeball fMRI signals did not find differences in gaze between stimulus conditions, and a control experiment (with derived simulations) also suggested that eye movements do not explain the stimulus-related changes in V1 topology.

Discussion

Our findings indicate that both inter-subject common responses and participant-specific activity contribute to the information in intra-V1 co-fluctuations, albeit through distinct sub-networks. Deconvolution, that enhances subject-specific activity, highlighted interhemispheric links for Global stimuli. Further exploration of intra-V1 networks promises insights into the neural basis of attention and perceptual organization.

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Over the past two decades, advances in computational power and data availability combined with increased accessibility to pre-trained models have led to an exponential rise in machine learning (ML) publications. While ML may have the potential to transform healthcare, this sharp increase in ML research output without focus on methodological rigor and standard reporting guidelines has fueled a reproducibility crisis. In addition, the rapidly growing complexity of these models compromises their interpretability, which currently impedes their successful and widespread clinical adoption. In medicine, where failure of such models may have severe implications for patients' health, the high requirements for accuracy, robustness, and interpretability confront ML researchers with a unique set of challenges. In this review, we discuss the semantics of reproducibility and interpretability, as well as related issues and challenges, and outline possible solutions to counteracting the "black box". To foster reproducibility, standard reporting guidelines need to be further developed and data or code sharing encouraged. Editors and reviewers may equally play a critical role by establishing high methodological standards and thus preventing the dissemination of low-quality ML publications. To foster interpretable learning, the use of simpler models more suitable for medical data can inform the clinician how results are generated based on input data. Model-agnostic explanation tools, sensitivity analysis, and hidden layer representations constitute further promising approaches to increase interpretability. Balancing model performance and interpretability are important to ensure clinical applicability. We have now reached a critical moment for ML in medicine, where addressing these issues and implementing appropriate solutions will be vital for the future evolution of the field.

Highlight results, don't hide them: Enhance interpretation, reduce biases and improve reproducibility

Most neuroimaging studies display results that represent only a tiny fraction of the collected data. While it is conventional to present "only the significant results" to the reader, here we suggest that this practice has several negative consequences for both reproducibility and understanding. This practice hides away most of the results of the dataset and leads to problems of selection bias and irreproducibility, both of which have been recognized as major issues in neuroimaging studies recently. Opaque, all-or-nothing thresholding, even if well-intentioned, places undue influence on arbitrary filter values, hinders clear communication of scientific results, wastes data, is antithetical to good scientific practice, and leads to conceptual inconsistencies. It is also inconsistent with the properties of the acquired data and the underlying biology being studied. Instead of presenting only a few statistically significant locations and hiding away the remaining results, studies should "highlight" the former while also showing as much as possible of the rest. This is distinct from but complementary to utilizing data sharing repositories: the initial presentation of results has an enormous impact on the interpretation of a study. We present practical examples and extensions of this approach for voxelwise, regionwise and cross-study analyses using publicly available data that was analyzed previously by 70 teams (NARPS; Botvinik-Nezer, et al., 2020), showing that it is possible to balance the goals of displaying a full set of results with providing the reader reasonably concise and "digestible" findings. In particular, the highlighting approach sheds useful light on the kind of variability present among the NARPS teams' results, which is primarily a varied strength of agreement rather than disagreement. Using a meta-analysis built on the informative "highlighting" approach shows this relative agreement, while one using the standard "hiding" approach does not. We describe how this simple but powerful change in practice-focusing on highlighting results, rather than hiding all but the strongest ones-can help address many large concerns within the field, or at least to provide more complete information about them. We include a list of practical suggestions for results reporting to improve reproducibility, cross-study comparisons and meta-analyses.

Symmetric data-driven fusion of diffusion tensor MRI: Age differences in white matter

In the past 20 years, white matter (WM) microstructure has been studied predominantly using diffusion tensor imaging (DTI). Decreases in fractional anisotropy (FA) and increases in mean (MD) and radial diffusivity (RD) have been consistently reported in healthy aging and neurodegenerative diseases. To date, DTI parameters have been studied individually (e.g., only FA) and separately (i.e., without using the joint information across them). This approach gives limited insights into WM pathology, increases the number of multiple comparisons, and yields inconsistent correlations with cognition. To take full advantage of the information in a DTI dataset, we present the first application of symmetric fusion to study healthy aging WM. This data-driven approach allows simultaneous examination of age differences in all four DTI parameters. We used multiset canonical correlation analysis with joint independent component analysis (mCCA + jICA) in cognitively healthy adults (age 20-33, n = 51 and age 60-79, n = 170). Four-way mCCA + jICA yielded one high-stability modality-shared component with co-variant patterns of age differences in RD and AD in the corpus callosum, internal capsule, and prefrontal WM. The mixing coefficients (or loading parameters) showed correlations with processing speed and fluid abilities that were not detected by unimodal analyses. In sum, mCCA + jICA allows data-driven identification of cognitively relevant multimodal components within the WM. The presented method should be further extended to clinical samples and other MR techniques (e.g., myelin water imaging) to test the potential of mCCA+jICA to discriminate between different WM disease etiologies and improve the diagnostic classification of WM diseases.

IABC: A Toolbox for Intelligent Analysis of Brain Connectivity

Brain functional networks and connectivity have played an important role in exploring brain function for understanding the brain and disclosing the mechanisms of brain disorders. Independent component analysis (ICA) is one of the most widely applied data-driven methods to extract brain functional networks/connectivity. However, it is hard to guarantee the reliability of networks/connectivity due to the randomness of component order and the difficulty in selecting an optimal component number in ICA. To facilitate the analysis of brain functional networks and connectivity using ICA, we developed a MATLAB toolbox called Intelligent Analysis of Brain Connectivity (IABC). IABC incorporates our previously proposed group information guided independent component analysis (GIG-ICA), NeuroMark, and splitting-merging assisted reliable ICA (SMART ICA) methods, which can estimate reliable individual-subject neuroimaging measures for further analysis. After user inputs functional magnetic resonance imaging (fMRI) data of multiple subjects that are regularly organized (e.g., in Brain Imaging Data Structure (BIDS)) and clicks a few buttons to set parameters, IABC automatically outputs brain functional networks, their related time courses, and functional network connectivity of each subject. All these neuroimaging measures are promising for providing clues in understanding brain function and differentiating brain disorders.