From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Beyond Increasing Sample Sizes: Optimizing Effect Sizes in Neuroimaging Research on Individual Differences

Linking neurobiology to relatively stable individual differences in cognition, emotion, motivation, and behavior can require large sample sizes to yield replicable results. Given the nature of between-person research, sample sizes at least in the hundreds are likely to be necessary in most neuroimaging studies of individual differences, regardless of whether they are investigating the whole brain or more focal hypotheses. However, the appropriate sample size depends on the expected effect size. Therefore, we propose four strategies to increase effect sizes in neuroimaging research, which may help to enable the detection of replicable between-person effects in samples in the hundreds rather than the thousands: (1) theoretical matching between neuroimaging tasks and behavioral constructs of interest; (2) increasing the reliability of both neural and psychological measurement; (3) individualization of measures for each participant; and (4) using multivariate approaches with cross-validation instead of univariate approaches. We discuss challenges associated with these methods and highlight strategies for improvements that will help the field to move toward a more robust and accessible neuroscience of individual differences.

Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability : Reliability of computational model parameters

Computational modeling of behavior is increasingly being adopted as a standard methodology in psychology, cognitive neuroscience, and computational psychiatry. This approach involves estimating parameters in a computational (or cognitive) model that represents the computational processes of the underlying behavior. In this approach, the reliability of the parameter estimates is an important issue. The use of hierarchical (Bayesian) approaches, which place a prior on each model parameter of the individual participants, is thought to improve the reliability of the parameters. However, the characteristics of reliability in parameter estimates, especially when individual-level priors are assumed, as in hierarchical models, have not yet been fully discussed. Furthermore, the suitability of different reliability measures for assessing parameter reliability is not thoroughly understood. In this study, we conduct a systematic examination of these issues through theoretical analysis and numerical simulations, focusing specifically on reinforcement learning models. We note that the heterogeneity in the estimation precision of individual parameters, particularly with priors, can skew reliability measures toward individuals with higher precision. We further note that there are two factors that reduce reliability, namely estimation error and intersession variation in the true parameters, and we discuss how to evaluate these factors separately. Based on the considerations of this study, we present several recommendations and cautions for assessing the reliability of the model parameters.

The value of error-correcting responses for cognitive assessment in games

Traditional conflict-based cognitive assessment tools are highly behaviorally restrictive, which prevents them from capturing the dynamic nature of human cognition, such as the tendency to make error-correcting responses. The cognitive game Tunnel Runner measures interference control, response inhibition, and response-rule switching in a less restrictive manner than traditional cognitive assessment tools by giving players movement control after an initial response and encouraging error-correcting responses. Nevertheless, error-correcting responses remain unused due to a limited understanding of what they measure and how to use them. To facilitate the use of error-correcting responses to measure and understand human cognition, we developed theoretically-grounded measures of error-correcting responses in Tunnel Runner and assessed whether they reflected the same cognitive functions measured via initial responses. Furthermore, we evaluated the measurement potential of error-correcting responses. We found that initial and error-correcting responses similarly reflected players' response inhibition and interference control, but not their response-rule switching. Furthermore, combining the two response types increased the reliability of interference control and response inhibition measurements. Lastly, error-correcting responses showed the potential to measure response inhibition on their own. Our results pave the way toward understanding and using post-decision change of mind data for cognitive measurement and other research and application contexts.

Test-retest reliability of the play-or-pass version of the Iowa Gambling Task

The Iowa Gambling Task (IGT) is used to assess decision-making in clinical populations. The original IGT does not disambiguate reward and punishment learning; however, an adaptation of the task, the "play-or-pass" IGT, was developed to better distinguish between reward and punishment learning. We evaluated the test-retest reliability of measures of reward and punishment learning from the play-or-pass IGT and examined associations with self-reported measures of reward/punishment sensitivity and internalizing symptoms. Participants completed the task across two sessions, and we calculated mean-level differences and rank-order stability of behavioral measures across the two sessions using traditional scoring, involving session-wide choice proportions, and computational modeling, involving estimates of different aspects of trial-level learning. Measures using both approaches were reliable; however, computational modeling provided more insights regarding between-session changes in performance, and how performance related to self-reported measures of reward/punishment sensitivity and internalizing symptoms. Our results show promise in using the play-or-pass IGT to assess decision-making; however, further work is still necessary to validate the play-or-pass IGT.

Evidence for a competitive relationship between executive functions and statistical learning

The ability of the brain to extract patterns from the environment and predict future events, known as statistical learning, has been proposed to interact in a competitive manner with prefrontal lobe-related networks and their characteristic cognitive or executive functions. However, it remains unclear whether these cognitive functions also possess a competitive relationship with implicit statistical learning across individuals and at the level of latent executive function components. In order to address this currently unknown aspect, we investigated, in two independent experiments (NStudy1 = 186, NStudy2 = 157), the relationship between implicit statistical learning, measured by the Alternating Serial Reaction Time task, and executive functions, measured by multiple neuropsychological tests. In both studies, a modest, but consistent negative correlation between implicit statistical learning and most executive function measures was observed. Factor analysis further revealed that a factor representing verbal fluency and complex working memory seemed to drive these negative correlations. Thus, the antagonistic relationship between implicit statistical learning and executive functions might specifically be mediated by the updating component of executive functions or/and long-term memory access.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Process models of verbal memory in cancer survivors: Bayesian process modeling approach to variation in test scores

INTRODUCTION

Verbal memory is a complex and fundamental aspect of human cognition. However, traditional sum-score analyses of verbal learning tests oversimplify underlying verbal memory processes. We propose using process models to subdivide memory into multiple processes, which helps in localizing the most affected processes in impaired verbal memory. Additionally, the model can be used to address score and process variability. This study aims to investigate the effects of cancer and its treatment on verbal memory, as well as provide a demonstration of how process models can be used to investigate the uncertainty in neuropsychological test scores.

METHOD

We present an investigation of memory process scores in non-CNS cancer survivors (n = 184) and no-cancer controls (n = 204). The participants completed the Amsterdam Cognition Scan (ACS), in which classical neuropsychological tests are digitally recreated for online at-home administration. We analyzed data from the ACS equivalent of a Verbal Learning Test using both traditional sum scores and a Bayesian process model.

RESULTS

Analysis of the sum score indicated that patients scored lower than controls on immediate recall but found no difference for delayed recall. The process model analysis indicated a small difference between patients and controls in immediate retrieval from both the partially learned and learned states, with no differences in learning or delayed retrieval processes. Individual-level analysis shows considerable uncertainty for sum scores. Sum scores were more certain than single trials. Retrieval parameters also showed less uncertainty than learning parameters.

CONCLUSION

The Bayesian process model increased the informativity of Verbal Learning test data, by showing uncertainty of the traditional sum score measurements as well as how the underlying processes differed between populations. Additionally, the model grants insight into underlying memory processes for individuals and how these processes vary within and between them.