A hitchhiker's guide to working with large, open-source neuroimaging datasets

Predicting high-level visual areas in the absence of task fMRI

The ventral visual stream is organized into units, or functional regions of interest (fROIs), specialized for processing high-level visual categories. Task-based fMRI scans ("localizers") are typically used to identify each individual's nuanced set of fROIs. The unique landscape of an individual's functional activation may rely in large part on their specialized connectivity patterns; recent studies corroborate this by showing that connectivity can predict individual differences in neural responses. We focus on the ventral visual stream and ask: how well can an individual's resting state functional connectivity localize their fROIs for face, body, scene, and object perception? And are the neural processors for any particular visual category better predicted by connectivity than others, suggesting a tighter mechanistic relationship between connectivity and function? We found, among 18 fROIs predicted from connectivity for each subject, all but one were selective for their preferred visual category. Defining an individual's fROIs based on their connectivity patterns yielded regions that were more selective than regions identified from previous studies or atlases in nearly all cases. Overall, we found that in the absence of a domain-specific localizer task, a 10-min resting state scan can be reliably used for defining these fROIs.

Improving Predictability, Test-Retest Reliability and Generalisability of Brain-Wide Associations for Cognitive Abilities via Multimodal Stacking

Brain-wide association studies (BWASs) have attempted to relate cognitive abilities with brain phenotypes, but have been challenged by issues such as predictability, test-retest reliability, and cross-cohort generalisability. To tackle these challenges, we proposed "stacking" that combines brain magnetic resonance imaging of different modalities, from task-fMRI contrasts and functional connectivity during tasks and rest to structural measures, into one prediction model. We benchmarked the benefits of stacking, using the Human Connectome Projects: Young Adults and Aging and the Dunedin Multidisciplinary Health and Development Study. For predictability, stacked models led to out-of-sample r ∼.5-.6 when predicting cognitive abilities at the time of scanning and 36 years earlier. For test-retest reliability, stacked models reached an excellent level of reliability (ICC>.75), even when we stacked only task-fMRI contrasts together. For generalisability, a stacked model with non-task MRI built from one dataset significantly predicted cognitive abilities in other datasets. Altogether, stacking is a viable approach to undertake the three challenges of BWAS for cognitive abilities.

Data leakage inflates prediction performance in connectome-based machine learning models

Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.

Cross Atlas Remapping via Optimal Transport (CAROT): Creating connectomes for different atlases when raw data is not available

Open-source, publicly available neuroimaging datasets - whether from large-scale data collection efforts or pooled from multiple smaller studies - offer unprecedented sample sizes and promote generalization efforts. Releasing data can democratize science, increase the replicability of findings, and lead to discoveries. Partly due to patient privacy, computational, and data storage concerns, researchers typically release preprocessed data with the voxelwise time series parcellated into a map of predefined regions, known as an atlas. However, releasing preprocessed data also limits the choices available to the end-user. This is especially true for connectomics, as connectomes created from different atlases are not directly comparable. Since there exist several atlases with no gold standards, it is unrealistic to have processed, open-source data available from all atlases. Together, these limitations directly inhibit the potential benefits of open-source neuroimaging data. To address these limitations, we introduce Cross Atlas Remapping via Optimal Transport (CAROT) to find a mapping between two atlases. This approach allows data processed from one atlas to be directly transformed into a connectome based on another atlas without the need for raw data access. To validate CAROT, we compare reconstructed connectomes against their original counterparts (i.e., connectomes generated directly from an atlas), demonstrate the utility of transformed connectomes in downstream analyses, and show how a connectome-based predictive model can generalize to publicly available data that was processed with different atlases. Overall, CAROT can reconstruct connectomes from an extensive set of atlases - without needing the raw data - allowing already processed connectomes to be easily reused in a wide range of analyses while eliminating redundant processing efforts. We share this tool as both source code and as a stand-alone web application (http://carotproject.com/).

Positron emission tomography and magnetic resonance imaging methods and datasets within the Dominantly Inherited Alzheimer Network (DIAN)

The Dominantly Inherited Alzheimer Network (DIAN) is an international collaboration studying autosomal dominant Alzheimer disease (ADAD). ADAD arises from mutations occurring in three genes. Offspring from ADAD families have a 50% chance of inheriting their familial mutation, so non-carrier siblings can be recruited for comparisons in case-control studies. The age of onset in ADAD is highly predictable within families, allowing researchers to estimate an individual's point in the disease trajectory. These characteristics allow candidate AD biomarker measurements to be reliably mapped during the preclinical phase. Although ADAD represents a small proportion of AD cases, understanding neuroimaging-based changes that occur during the preclinical period may provide insight into early disease stages of 'sporadic' AD also. Additionally, this study provides rich data for research in healthy aging through inclusion of the non-carrier controls. Here we introduce the neuroimaging dataset collected and describe how this resource can be used by a range of researchers.

Pre-pandemic brain structure and function and adolescent psychopathology during the COVID-19 pandemic

The COVID-19 pandemic has presented unprecedented challenges for youths and families, dramatically increasing exposure to stressors and stress-related psychopathology. Increasing work has leveraged pre-pandemic neuroimaging data to predict adolescent psychopathology and stress responses during the pandemic, with a particular focus on internalizing symptoms. We review this recent literature on pre-pandemic brain structure and function and adolescent internalizing psychopathology during the pandemic. At present, existing studies have not consistently identified specific alterations in brain structure and function that predict anxiety or depressive symptoms during the pandemic. In contrast, exposure to stress and adversity before and during the pandemic as well as access to peer and family support have emerged as consistent and reliable predictors of youth mental health during the pandemic.

Connectome-based machine learning models are vulnerable to subtle data manipulations

Neuroimaging-based predictive models continue to improve in performance, yet a widely overlooked aspect of these models is "trustworthiness," or robustness to data manipulations. High trustworthiness is imperative for researchers to have confidence in their findings and interpretations. In this work, we used functional connectomes to explore how minor data manipulations influence machine learning predictions. These manipulations included a method to falsely enhance prediction performance and adversarial noise attacks designed to degrade performance. Although these data manipulations drastically changed model performance, the original and manipulated data were extremely similar (r = 0.99) and did not affect other downstream analysis. Essentially, connectome data could be inconspicuously modified to achieve any desired prediction performance. Overall, our enhancement attacks and evaluation of existing adversarial noise attacks in connectome-based models highlight the need for counter-measures that improve the trustworthiness to preserve the integrity of academic research and any potential translational applications.

Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization

Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.

Multisite test-retest reliability and compatibility of brain metrics derived from FreeSurfer versions 7.1, 6.0, and 5.3

Automatic neuroimaging processing tools provide convenient and systematic methods for extracting features from brain magnetic resonance imaging scans. One tool, FreeSurfer, provides an easy-to-use pipeline to extract cortical and subcortical morphometric measures. There have been over 25 stable releases of FreeSurfer, with different versions used across published works. The reliability and compatibility of regional morphometric metrics derived from the most recent version releases have yet to be empirically assessed. Here, we used test-retest data from three public data sets to determine within-version reliability and between-version compatibility across 42 regional outputs from FreeSurfer versions 7.1, 6.0, and 5.3. Cortical thickness from v7.1 was less compatible with that of older versions, particularly along the cingulate gyrus, where the lowest version compatibility was observed (intraclass correlation coefficient 0.37-0.61). Surface area of the temporal pole, frontal pole, and medial orbitofrontal cortex, also showed low to moderate version compatibility. We confirm low compatibility between v6.0 and v5.3 of pallidum and putamen volumes, while those from v7.1 were compatible with v6.0. Replication in an independent sample showed largely similar results for measures of surface area and subcortical volumes, but had lower overall regional thickness reliability and compatibility. Batch effect correction may adjust for some inter-version effects when most sites are run with one version, but results vary when more sites are run with different versions. Age associations in a quality controlled independent sample (N = 106) revealed version differences in results of downstream statistical analysis. We provide a reference to highlight the regional metrics that may yield recent version-related inconsistencies in published findings. An interactive viewer is provided at http://data.brainescience.org/Freesurfer_Reliability/.

Interpreting mental state decoding with deep learning models

In mental state decoding, researchers aim to identify the set of mental states (e.g., experiencing happiness or fear) that can be reliably identified from the activity patterns of a brain region (or network). Deep learning (DL) models are highly promising for mental state decoding because of their unmatched ability to learn versatile representations of complex data. However, their widespread application in mental state decoding is hindered by their lack of interpretability, difficulties in applying them to small datasets, and in ensuring their reproducibility and robustness. We recommend approaching these challenges by leveraging recent advances in explainable artificial intelligence (XAI) and transfer learning, and also provide recommendations on how to improve the reproducibility and robustness of DL models in mental state decoding.

Functional Connectome-Based Predictive Modeling in Autism

Autism is a heterogeneous neurodevelopmental condition, and functional magnetic resonance imaging-based studies have helped advance our understanding of its effects on brain network activity. We review how predictive modeling, using measures of functional connectivity and symptoms, has helped reveal key insights into this condition. We discuss how different prediction frameworks can further our understanding of the brain-based features that underlie complex autism symptomatology and consider how predictive models may be used in clinical settings. Throughout, we highlight aspects of study interpretation, such as data decay and sampling biases, that require consideration within the context of this condition. We close by suggesting exciting future directions for predictive modeling in autism.

Functional Connectivity of the Chemosenses: A Review

Functional connectivity approaches have long been used in cognitive neuroscience to establish pathways of communication between and among brain regions. However, the use of these analyses to better understand how the brain processes chemosensory information remains nascent. In this review, we conduct a literature search of all functional connectivity papers of olfaction, gustation, and chemesthesis, with 103 articles discovered in total. These publications largely use approaches of seed-based functional connectivity and psychophysiological interactions, as well as effective connectivity approaches such as Granger Causality, Dynamic Causal Modeling, and Structural Equation Modeling. Regardless of modality, studies largely focus on elucidating neural correlates of stimulus qualities such as identity, pleasantness, and intensity, with task-based paradigms most frequently implemented. We call for further "model free" or data-driven approaches in predictive modeling to craft brain-behavior relationships that are free from a priori hypotheses and not solely based on potentially irreproducible literature. Moreover, we note a relative dearth of resting-state literature, which could be used to better understand chemosensory networks with less influence from motion artifacts induced via gustatory or olfactory paradigms. Finally, we note a lack of genomics data, which could clarify individual and heritable differences in chemosensory perception.

Reporting details of neuroimaging studies on individual traits prediction: A literature survey

Using machine-learning tools to predict individual phenotypes from neuroimaging data is one of the most promising and hence dynamic fields in systems neuroscience. Here, we perform a literature survey of the rapidly work on phenotype prediction in healthy subjects or general population to sketch out the current state and ongoing developments in terms of data, analysis methods and reporting. Excluding papers on age-prediction and clinical applications, which form a distinct literature, we identified a total 108 papers published since 2007. In these, memory, fluid intelligence and attention were most common phenotypes to be predicted, which resonates with the observation that roughly a quarter of the papers used data from the Human Connectome Project, even though another half recruited their own cohort. Sample size (in terms of training and external test sets) and prediction accuracy (from internal and external validation respectively) did not show significant temporal trends. Prediction accuracy was negatively correlated with sample size of the training set, but not the external test set. While known to be optimistic, leave-one-out cross-validation (LOO CV) was the prevalent strategy for model validation (n = 48). Meanwhile, 27 studies used external validation with external test set. Both numbers showed no significant temporal trends. The most popular learning algorithm was connectome-based predictive modeling introduced by the Yale team. Other common learning algorithms were linear regression, relevance vector regression (RVR), support vector regression (SVR), least absolute shrinkage and selection operator (LASSO), and elastic net. Meanwhile, the amount of data from self-recruiting studies (but not studies using open, shared dataset) was positively correlated with internal validation prediction accuracy. At the same time, self-recruiting studies also reported a significantly higher internal validation prediction accuracy than those using open, shared datasets. Data type and participant age did not significantly influence prediction accuracy. Confound control also did not influence prediction accuracy after adjusted for other factors. To conclude, most of the current literature is probably quite optimistic with internal validation using LOO CV. More efforts should be made to encourage the use of external validation with external test sets to further improve generalizability of the models.

A protocol for working with open-source neuroimaging datasets

Large, publicly available neuroimaging datasets are becoming increasingly common, but their use presents challenges because of insufficient knowledge of the tool options for data processing and proper data organization. Here, we describe a protocol to lessen these barriers. We describe the steps for the search and download of the open-source dataset. We detail the steps for proper data management and practical guidelines for data analysis. Finally, we give instructions for data and result sharing on public repositories and preprint services. For complete details on the use and execution of this profile, please refer to Horien et al. (2021).

FAIRly big: A framework for computationally reproducible processing of large-scale data

Large-scale datasets present unique opportunities to perform scientific investigations with unprecedented breadth. However, they also pose considerable challenges for the findability, accessibility, interoperability, and reusability (FAIR) of research outcomes due to infrastructure limitations, data usage constraints, or software license restrictions. Here we introduce a DataLad-based, domain-agnostic framework suitable for reproducible data processing in compliance with open science mandates. The framework attempts to minimize platform idiosyncrasies and performance-related complexities. It affords the capture of machine-actionable computational provenance records that can be used to retrace and verify the origins of research outcomes, as well as be re-executed independent of the original computing infrastructure. We demonstrate the framework's performance using two showcases: one highlighting data sharing and transparency (using the studyforrest.org dataset) and another highlighting scalability (using the largest public brain imaging dataset available: the UK Biobank dataset).

The role of artificial intelligence in paediatric neuroradiology

Imaging plays a fundamental role in the managing childhood neurologic, neurosurgical and neuro-oncological disease. Employing multi-parametric MRI techniques, such as spectroscopy and diffusion- and perfusion-weighted imaging, to the radiophenotyping of neuroradiologic conditions is becoming increasingly prevalent, particularly with radiogenomic analyses correlating imaging characteristics with molecular biomarkers of disease. However, integration into routine clinical practice remains elusive. With modern multi-parametric MRI now providing additional data beyond anatomy, informing on histology, biology and physiology, such metric-rich information can present as information overload to the treating radiologist and, as such, information relevant to an individual case can become lost. Artificial intelligence techniques are capable of modelling the vast radiologic, biological and clinical datasets that accompany childhood neurologic disease, such that this information can become incorporated in upfront prognostic modelling systems, with artificial intelligence techniques providing a plausible approach to this solution. This review examines machine learning approaches than can be used to underpin such artificial intelligence applications, with exemplars for each machine learning approach from the world literature. Then, within the specific use case of paediatric neuro-oncology, we examine the potential future contribution for such artificial intelligence machine learning techniques to offer solutions for patient care in the form of decision support systems, potentially enabling personalised medicine within this domain of paediatric radiologic practice.

Dear reviewers: Responses to common reviewer critiques about infant neuroimaging studies

The field of adult neuroimaging relies on well-established principles in research design, imaging sequences, processing pipelines, as well as safety and data collection protocols. The field of infant magnetic resonance imaging, by comparison, is a young field with tremendous scientific potential but continuously evolving standards. The present article aims to initiate a constructive dialog between researchers who grapple with the challenges and inherent limitations of a nascent field and reviewers who evaluate their work. We address 20 questions that researchers commonly receive from research ethics boards, grant, and manuscript reviewers related to infant neuroimaging data collection, safety protocols, study planning, imaging sequences, decisions related to software and hardware, and data processing and sharing, while acknowledging both the accomplishments of the field and areas of much needed future advancements. This article reflects the cumulative knowledge of experts in the FIT’NG community and can act as a resource for both researchers and reviewers alike seeking a deeper understanding of the standards and tradeoffs involved in infant neuroimaging.

•

The field of infant MRI is young with evolving standards.

•

We address 20 questions that researchers commonly receive reviewers.

•

These come from research ethics boards, grant, and manuscript reviewers.

•

This article reflects the cumulative knowledge of experts in the FIT’NG community.

Modularity maximization as a flexible and generic framework for brain network exploratory analysis

The modular structure of brain networks supports specialized information processing, complex dynamics, and cost-efficient spatial embedding. Inter-individual variation in modular structure has been linked to differences in performance, disease, and development. There exist many data-driven methods for detecting and comparing modular structure, the most popular of which is modularity maximization. Although modularity maximization is a general framework that can be modified and reparamaterized to address domain-specific research questions, its application to neuroscientific datasets has, thus far, been narrow. Here, we highlight several strategies in which the "out-of-the-box" version of modularity maximization can be extended to address questions specific to neuroscience. First, we present approaches for detecting "space-independent" modules and for applying modularity maximization to signed matrices. Next, we show that the modularity maximization frame is well-suited for detecting task- and condition-specific modules. Finally, we highlight the role of multi-layer models in detecting and tracking modules across time, tasks, subjects, and modalities. In summary, modularity maximization is a flexible and general framework that can be adapted to detect modular structure resulting from a wide range of hypotheses. This article highlights multiple frontiers for future research and applications.

How prenatal exposures shape the infant brain: Insights from infant neuroimaging studies

Brain development during the prenatal period is rapid and unparalleled by any other time during development. Biological systems undergoing rapid development are at higher risk for disorganizing influences. Therefore, certain prenatal exposures impact brain development, increasing risk for negative neurodevelopmental outcome. While prenatal exposures have been associated with cognitive and behavioral outcomes later in life, the underlying macroscopic brain pathways remain unclear. Here, we review magnetic resonance imaging (MRI) studies investigating the association between prenatal exposures and infant brain development focusing on prenatal exposures via maternal physical health factors, maternal mental health factors, and maternal drug and medication use. Further, we discuss the need for studies to consider multiple prenatal exposures in parallel and suggest future directions for this body of research.

Large, open datasets for human connectomics research: Considerations for reproducible and responsible data use

Large, open datasets have emerged as important resources in the field of human connectomics. In this review, the evolution of data sharing involving magnetic resonance imaging is described. A summary of the challenges and progress in conducting reproducible data analyses is provided, including description of recent progress made in the development of community guidelines and recommendations, software and data management tools, and initiatives to enhance training and education. Finally, this review concludes with a discussion of ethical conduct relevant to analyses of large, open datasets and a researcher's responsibility to prevent further stigmatization of historically marginalized racial and ethnic groups. Moving forward, future work should include an enhanced emphasis on the social determinants of health, which may further contextualize findings among diverse population-based samples. Leveraging the progress to date and guided by interdisciplinary collaborations, the future of connectomics promises to be an impressive era of innovative research, yielding a more inclusive understanding of brain structure and function.

Virtual Connectomic Datasets in Alzheimer's Disease and Aging Using Whole-Brain Network Dynamics Modelling

Large neuroimaging datasets, including information about structural connectivity (SC) and functional connectivity (FC), play an increasingly important role in clinical research, where they guide the design of algorithms for automated stratification, diagnosis or prediction. A major obstacle is, however, the problem of missing features [e.g., lack of concurrent DTI SC and resting-state functional magnetic resonance imaging (rsfMRI) FC measurements for many of the subjects]. We propose here to address the missing connectivity features problem by introducing strategies based on computational whole-brain network modeling. Using two datasets, the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset and a healthy aging dataset, for proof-of-concept, we demonstrate the feasibility of virtual data completion (i.e., inferring “virtual FC” from empirical SC or “virtual SC” from empirical FC), by using self-consistent simulations of linear and nonlinear brain network models. Furthermore, by performing machine learning classification (to separate age classes or control from patient subjects), we show that algorithms trained on virtual connectomes achieve discrimination performance comparable to when trained on actual empirical data; similarly, algorithms trained on virtual connectomes can be used to successfully classify novel empirical connectomes. Completion algorithms can be combined and reiterated to generate realistic surrogate connectivity matrices in arbitrarily large number, opening the way to the generation of virtual connectomic datasets with network connectivity information comparable to the one of the original data.

Decision Models and Technology Can Help Psychiatry Develop Biomarkers

Why is psychiatry unable to define clinically useful biomarkers? We explore this question from the vantage of data and decision science and consider biomarkers as a form of phenotypic data that resolves a well-defined clinical decision. We introduce a framework that systematizes different forms of phenotypic data and further introduce the concept of decision model to describe the strategies a clinician uses to seek out, combine, and act on clinical data. Though many medical specialties rely on quantitative clinical data and operationalized decision models, we observe that, in psychiatry, clinical data are gathered and used in idiosyncratic decision models that exist solely in the clinician's mind and therefore are outside empirical evaluation. This, we argue, is a fundamental reason why psychiatry is unable to define clinically useful biomarkers: because psychiatry does not currently quantify clinical data, decision models cannot be operationalized and, in the absence of an operationalized decision model, it is impossible to define how a biomarker might be of use. Here, psychiatry might benefit from digital technologies that have recently emerged specifically to quantify clinically relevant facets of human behavior. We propose that digital tools might help psychiatry in two ways: first, by quantifying data already present in the standard clinical interaction and by allowing decision models to be operationalized and evaluated; second, by testing whether new forms of data might have value within an operationalized decision model. We reference successes from other medical specialties to illustrate how quantitative data and operationalized decision models improve patient care.