Brain-phenotype predictions can survive across diverse real-world data

Recent work suggests that machine learning models predicting psychiatric treatment outcomes based on clinical data may fail when applied to unharmonized samples. Neuroimaging predictive models offer the opportunity to incorporate neurobiological information, which may be more robust to dataset shifts. Yet, among the minority of neuroimaging studies that undertake any form of external validation, there is a notable lack of attention to generalization across dataset-specific idiosyncrasies. Research settings, by design, remove the between-site variations that real-world and, eventually, clinical applications demand. Here, we rigorously test the ability of a range of predictive models to generalize across three diverse, unharmonized samples: the Philadelphia Neurodevelopmental Cohort (n=1291), the Healthy Brain Network (n=1110), and the Human Connectome Project in Development (n=428). These datasets have high inter-dataset heterogeneity, encompassing substantial variations in age distribution, sex, racial and ethnic minority representation, recruitment geography, clinical symptom burdens, fMRI tasks, sequences, and behavioral measures. We demonstrate that reproducible and generalizable brain-behavior associations can be realized across diverse dataset features with sample sizes in the hundreds. Results indicate the potential of functional connectivity-based predictive models to be robust despite substantial inter-dataset variability. Notably, for the HCPD and HBN datasets, the best predictions were not from training and testing in the same dataset (i.e., cross-validation) but across datasets. This result suggests that training on diverse data may improve prediction in specific cases. Overall, this work provides a critical foundation for future work evaluating the generalizability of neuroimaging predictive models in real-world scenarios and clinical settings.

Connectome-based prediction of craving in gambling disorder and cocaine use disorder

INTRODUCTION

Craving, involving intense and urgent desires to engage in specific behaviours, is a feature of addictions. Multiple studies implicate regions of salience/limbic networks and basal ganglia, fronto-parietal, medial frontal regions in craving in addictions. However, prior studies have not identified common neural networks that reliably predict craving across substance and behavioural addictions.

METHODS

Functional magnetic resonance imaging during an audiovisual cue-reactivity task and connectome-based predictive modelling (CPM), a data-driven method for generating brain-behavioural models, were used to study individuals with cocaine-use disorder and gambling disorder. Functions of nodes and networks relevant to craving were identified and interpreted based on meta-analytic data.

RESULTS

Craving was predicted by neural connectivity across disorders. The highest degree nodes were mostly located in the prefrontal cortex. Overall, the prediction model included complex networks including motor/sensory, fronto-parietal, and default-mode networks. The decoding revealed high functional associations with components of memory, valence ratings, physiological responses, and finger movement/motor imagery.

CONCLUSIONS

Craving could be predicted across substance and behavioural addictions. The model may reflect general neural mechanisms of craving despite specificities of individual disorders. Prefrontal regions associated with working memory and autobiographical memory seem important in predicting craving. For further validation, the model should be tested in diverse samples and contexts.

Altered Brain Dynamics Across Bipolar Disorder and Schizophrenia During Rest and Task Switching Revealed by Overlapping Brain States

BACKGROUND

Individuals with bipolar disorder (BD) and schizophrenia (SCZ) show aberrant brain dynamics (i.e., altered recruitment or traversal through different brain states over time). Existing investigations of brain dynamics typically assume that one dominant brain state characterizes each time point. However, as multiple brain states likely are engaged at any given moment, this approach can obscure alterations in less prominent but critical brain states. Here, we examined brain dynamics in BD and SCZ by implementing a novel framework that simultaneously assessed the engagement of multiple brain states.

METHODS

Four recurring brain states were identified by applying nonlinear manifold learning and k-means clustering to the Human Connectome Project task-based functional magnetic resonance imaging data. We then assessed moment-to-moment state engagement in 2 independent samples of healthy control participants and patients with BD or SCZ using resting-state (N = 336) or task-based (N = 217) functional magnetic resonance imaging data. Relative state engagement and state engagement variability were extracted and compared across groups using multivariate analysis of covariance, controlling for site, medication, age, and sex.

RESULTS

Our framework identified dynamic alterations in BD and SCZ, while a state discretization approach revealed no significant group differences. Participants with BD or SCZ showed reduced state engagement variability, but not relative state engagement, across multiple brain states during resting-state and task-based functional magnetic resonance imaging. We found decreased state engagement variability in older participants and preliminary evidence suggesting an association with avolition.

CONCLUSIONS

Assessing multiple brain states simultaneously can reflect the complexity of aberrant brain dynamics in BD and SCZ, providing a more comprehensive understanding of the neural mechanisms underpinning these conditions.

Gradient-based enhancement attacks in biomedical machine learning

The prevalence of machine learning in biomedical research is rapidly growing, yet the trustworthiness of such research is often overlooked. While some previous works have investigated the ability of adversarial attacks to degrade model performance in medical imaging, the ability to falsely improve performance via recently-developed "enhancement attacks" may be a greater threat to biomedical machine learning. In the spirit of developing attacks to better understand trustworthiness, we developed two techniques to drastically enhance prediction performance of classifiers with minimal changes to features: 1) general enhancement of prediction performance, and 2) enhancement of a particular method over another. Our enhancement framework falsely improved classifiers' accuracy from 50% to almost 100% while maintaining high feature similarities between original and enhanced data (Pearson's r's > 0.99). Similarly, the method-specific enhancement framework was effective in falsely improving the performance of one method over another. For example, a simple neural network outperformed logistic regression by 17% on our enhanced dataset, although no performance differences were present in the original dataset. Crucially, the original and enhanced data were still similar (r = 0.99). Our results demonstrate the feasibility of minor data manipulations to achieve any desired prediction performance, which presents an interesting ethical challenge for the future of biomedical machine learning. These findings emphasize the need for more robust data provenance tracking and other precautionary measures to ensure the integrity of biomedical machine learning research. Code is available at https://github.com/mattrosenblatt7/enhancement_EPIMI.

Cross Atlas Remapping via Optimal Transport (CAROT): Creating connectomes for different atlases when raw data is not available

Open-source, publicly available neuroimaging datasets - whether from large-scale data collection efforts or pooled from multiple smaller studies - offer unprecedented sample sizes and promote generalization efforts. Releasing data can democratize science, increase the replicability of findings, and lead to discoveries. Partly due to patient privacy, computational, and data storage concerns, researchers typically release preprocessed data with the voxelwise time series parcellated into a map of predefined regions, known as an atlas. However, releasing preprocessed data also limits the choices available to the end-user. This is especially true for connectomics, as connectomes created from different atlases are not directly comparable. Since there exist several atlases with no gold standards, it is unrealistic to have processed, open-source data available from all atlases. Together, these limitations directly inhibit the potential benefits of open-source neuroimaging data. To address these limitations, we introduce Cross Atlas Remapping via Optimal Transport (CAROT) to find a mapping between two atlases. This approach allows data processed from one atlas to be directly transformed into a connectome based on another atlas without the need for raw data access. To validate CAROT, we compare reconstructed connectomes against their original counterparts (i.e., connectomes generated directly from an atlas), demonstrate the utility of transformed connectomes in downstream analyses, and show how a connectome-based predictive model can generalize to publicly available data that was processed with different atlases. Overall, CAROT can reconstruct connectomes from an extensive set of atlases - without needing the raw data - allowing already processed connectomes to be easily reused in a wide range of analyses while eliminating redundant processing efforts. We share this tool as both source code and as a stand-alone web application (http://carotproject.com/).

Machine Learning and Prediction in Fetal, Infant, and Toddler Neuroimaging: A Review and Primer

Predictive models in neuroimaging are increasingly designed with the intent to improve risk stratification and support interventional efforts in psychiatry. Many of these models have been developed in samples of children school-aged or older. Nevertheless, despite growing evidence that altered brain maturation during the fetal, infant, and toddler (FIT) period modulates risk for poor mental health outcomes in childhood, these models are rarely implemented in FIT samples. Applications of predictive modeling in children of these ages provide an opportunity to develop powerful tools for improved characterization of the neural mechanisms underlying development. To facilitate the broader use of predictive models in FIT neuroimaging, we present a brief primer and systematic review on the methods used in current predictive modeling FIT studies. Reflecting on current practices in more than 100 studies conducted over the past decade, we provide an overview of topics, modalities, and methods commonly used in the field and under-researched areas. We then outline ethical and future considerations for neuroimaging researchers interested in predicting health outcomes in early life, including researchers who may be relatively new to either advanced machine learning methods or using FIT data. Altogether, the last decade of FIT research in machine learning has provided a foundation for accelerating the prediction of early-life trajectories across the full spectrum of illness and health.

Stacking multiple optimal transport policies to map functional connectomes

Connectomics is a popular approach for understanding the brain with neuroimaging data. However, a connectome generated from one atlas is different in size, topology, and scale compared to a connectome generated from another. Consequently, connectomes generated from different atlases cannot be used in the same analysis. This limitation hinders efforts toward increasing sample size and demonstrating generalizability across datasets. Recently, we proposed Cross Atlas Remapping via Optimal Transport (CAROT) to find a spatial mapping between a pair of atlases based on a set of training data. The mapping transforms timeseries fMRI data parcellated with an atlas to form a connectome based on a different one. Crucially, CAROT does not need raw fMRI data and thus does not require re-processing, which can otherwise be time-consuming and expensive. The current CAROT implementation leverages information from several source atlases to create robust mappings for a target atlas. In this work, we extend CAROT to combine existing mappings between a source and target atlas for an arbitrary number of mappings. This extension (labeled Stacking CAROT) allows mappings between a pair of atlases to be created once and re-used with other pre-trained mappings to create new mappings as needed. Reconstructed connectomes from Stacking CAROT perform as well as those from CAROT in downstream analyses. Importantly, Stacking CAROT significantly reduces training time and storage requirements compared to CAROT. Overall, Stacking CAROT improves previous versions of CAROT.

Predicting the future of neuroimaging predictive models in mental health

Predictive modeling using neuroimaging data has the potential to improve our understanding of the neurobiology underlying psychiatric disorders and putatively information interventions. Accordingly, there is a plethora of literature reviewing published studies, the mathematics underlying machine learning, and the best practices for using these approaches. As our knowledge of mental health and machine learning continue to evolve, we instead aim to look forward and "predict" topics that we believe will be important in current and future studies. Some of the most discussed topics in machine learning, such as bias and fairness, the handling of dirty data, and interpretable models, may be less familiar to the broader community using neuroimaging-based predictive modeling in psychiatry. In a similar vein, transdiagnostic research and targeting brain-based features for psychiatric intervention are modern topics in psychiatry that predictive models are well-suited to tackle. In this work, we target an audience who is a researcher familiar with the fundamental procedures of machine learning and who wishes to increase their knowledge of ongoing topics in the field. We aim to accelerate the utility and applications of neuroimaging-based predictive models for psychiatric research by highlighting and considering these topics. Furthermore, though not a focus, these ideas generalize to neuroimaging-based predictive modeling in other clinical neurosciences and predictive modeling with different data types (e.g., digital health data).

A hitchhiker's guide to working with large, open-source neuroimaging datasets

Large datasets that enable researchers to perform investigations with unprecedented rigor are growing increasingly common in neuroimaging. Due to the simultaneous increasing popularity of open science, these state-of-the-art datasets are more accessible than ever to researchers around the world. While analysis of these samples has pushed the field forward, they pose a new set of challenges that might cause difficulties for novice users. Here we offer practical tips for working with large datasets from the end-user's perspective. We cover all aspects of the data lifecycle: from what to consider when downloading and storing the data to tips on how to become acquainted with a dataset one did not collect and what to share when communicating results. This manuscript serves as a practical guide one can use when working with large neuroimaging datasets, thus dissolving barriers to scientific discovery.

Transdiagnostic, Connectome-Based Prediction of Memory Constructs Across Psychiatric Disorders

Memory deficits are observed in a range of psychiatric disorders, but it is unclear whether memory deficits arise from a shared brain correlate across disorders or from various dysfunctions unique to each disorder. Connectome-based predictive modeling is a computational method that captures individual differences in functional connectomes associated with behavioral phenotypes such as memory. We used publicly available task-based functional MRI data from patients with schizophrenia (n = 33), bipolar disorder (n = 34), attention deficit hyper-activity disorder (n = 32), and healthy controls (n = 73) to model the macroscale brain networks associated with working, short- and long-term memory. First, we use 10-fold and leave-group-out analyses to demonstrate that the same macroscale brain networks subserve memory across diagnostic groups and that individual differences in memory performance are related to individual differences within networks distributed throughout the brain, including the subcortex, default mode network, limbic network, and cerebellum. Next, we show that diagnostic groups are associated with significant differences in whole-brain functional connectivity that are distinct from the predictive models of memory. Finally, we show that models trained on the transdiagnostic sample generalize to novel, healthy participants (n = 515) from the Human Connectome Project. These results suggest that despite significant differences in whole-brain patterns of functional connectivity between diagnostic groups, the core macroscale brain networks that subserve memory are shared.