Value-free random exploration is linked to impulsivity

Individual differences in sequential decision-making

1People differ widely in how they make decisions in uncertain environments. While many studies leverage this variability to measure differences in specific cognitive processes and parameters, the key dimension(s) of individual variability in uncertain decision-making tasks has not been identified. Here, we analyzed behavioral data from 1001 participants performing a restless three-armed bandit task, where reward probabilities fluctuated unpredictably over time. Using a novel analytical approach that controlled for the stochasticity in this tasks, we identified a dominant nonlinear axis of individual variability. We found that this primary axis of variability was strongly and selectively correlated with the probability of exploration, as inferred by latent state modeling. This suggests that the major factor shaping individual differences in bandit task performance is the tendency to explore (versus exploit), rather than personality characteristics, reinforcement learning model parameters, or low-level strategies. Certain demographic characteristics also predicted variance along this principle axis: participants at the exploratory end tended to be younger than participants at the exploitative end, and self-identified men were overrepresented at both extremes. Together, these findings offer a principled framework for understanding individual differences in task behavior while highlighting the cognitive and demographic factors that shape individual differences in decision-making under uncertainty.

Day-to-day fluctuations in motivation drive effort-based decision-making

Internal states like motivation fluctuate substantially over time. However, studies of the neurocomputational mechanims of motivated behavior have failed to capture this. Here, we examined how naturalistic ups and downs in state motivation influence the subjective value of reward and effort. In a microlongitudinal design (N = 155, state timepoints = 3,344, decision-making tasks = 845), we captured fluctuations in state and effort-based decision-making using smartphone-based momentary assessments as people went about their daily lives. We found that both state and trait have independent and multiplicative effects on decision-making. State-behavior coupling was particularly pronounced in individuals with higher trait apathy, meaning that their choices were even more state dependent. Using computational modeling, we demonstrate that state motivation prospectively boosted reward sensitivity, making people more willing to exert effort in future. Our results show that day-to-day fluctuations in state and cognition are tightly linked and critical for understanding fundamental human behaviors and mental ill-health.

Emotion dysregulation and impulsivity as overlapping symptoms in adult Attention-Deficit/Hyperactivity Disorder and Borderline Personality Disorder: severity profiles and associations with childhood traumatization and personality functioning

BACKGROUND

Increased levels of emotion dysregulation and impulsive behavior are overlapping symptoms in adult Attention-Deficit/Hyperactivity Disorder (aADHD) and Borderline Personality Disorder (BPD), both symptom domains reflecting on inhibitory control, although from different angles. Our aims were to describe their differences in the above conditions, investigate their associations with childhood traumatization, and to explore the potential mediation of emotion dysregulation and impulsivity between childhood traumas and personality functioning.

METHODS

Young adults between 18 and 36 years diagnosed with aADHD (n = 100) and BPD (n = 63) were investigated with structured clinical interviews, while age-matched healthy controls (n = 100) were screened for psychiatric disorders. Patients with aADHD-BPD comorbidity were excluded from further analyses. The Difficulties in Emotion Regulation Scale, the Barratt Impulsiveness Scale, the Level of Personality Functioning Scale, and the Childhood Trauma Questionnaire-Short Form were administered to investigate trait measures and childhood traumatization, respectively. Behavioral impulsivity and delay aversion were assessed using selected tests of the Cambridge Neuropsychological Test Automated Battery, and a computerized decision-making paradigm based on the Rogers decision-making task, respectively.

RESULTS

Significantly higher levels of emotion dysregulation and impulsivity were present both in the aADHD and BPD groups, however with different profiles. Waiting and stopping impulsivity was selectively higher among aADHD patients compared to healthy controls. The BPD group reported higher levels of emotion dysregulation in all domains, and demonstrated increased delay aversion among uncertain conditions in decision-making. Higher levels of childhood trauma were associated with emotion dysregulation, trait impulsivity, and delay aversion across groups. Emotion regulatory capacity played a significant mediating role between childhood traumatization and the level of personality functioning.

CONCLUSIONS

Inhibitory control profiles of the aADHD and BPD groups were divergent. Childhood traumatization was associated with lower levels of personality functioning in adulthood, independently of diagnosis, an effect mediated more by emotion dysregulation, rather than impulsivity. These findings have various clinical implications for the treatment of aADHD and BPD, including psychoeducation, pharmacological interventions, and psychotherapy targeting specific symptom domains.

Latent mechanisms of language disorganization relate to specific dimensions of psychopathology

Comprehensible communication is critical for social functioning and well-being. In psychopathology, incoherent discourse is assumed to reflect disorganized thinking, which is classically linked to psychotic disorders. However, people do not express everything that comes to mind, rendering inferences from discourse to the underlying structure of thought challenging. Indeed, a range of psychopathologies are linked to self-reported disorganized thinking in the absence of language output incoherence. Here we combine natural language processing and computational modeling of free association to detail the relationship between disorganized thinking and language (in)coherence in a large sample of participants varying across different dimensions of psychopathology. Our approach allowed us to differentiate between disorganized thinking, disinhibited thought expression and deliberate creativity. We find evidence for both under-regulated and over-regulated disorganized thinking, which relate to two specific dimensions of psychopathology: self-reported eccentricity and suspiciousness. Broadly, these results underscore the theoretical progress afforded by analyzing latent dimensions underlying behavior and psychopathology.

Individual differences in information demand have a low dimensional structure predicted by some curiosity traits

To understand human learning and progress, it is crucial to understand curiosity. But how consistent is curiosity's conception and assessment across scientific research disciplines? We present the results of a large collaborative project assessing the correspondence between curiosity measures in personality psychology and cognitive science. A total of 820 participants completed 15 personality trait measures and 9 cognitive tasks that tested multiple aspects of information demand. We show that shared variance across the cognitive tasks was captured by a dimension reflecting directed (uncertainty-driven) versus random (stochasticity-driven) exploration and individual differences along this axis were significantly and consistently predicted by personality traits. However, the personality metrics that best predicted information demand were not the central curiosity traits of openness to experience, deprivation sensitivity, and joyous exploration, but instead included more peripheral curiosity traits (need for cognition, thrill seeking, and stress tolerance) and measures not traditionally associated with curiosity (extraversion and behavioral inhibition). The results suggest that the umbrella term "curiosity" reflects a constellation of cognitive and emotional processes, only some of which are shared between personality measures and cognitive tasks. The results reflect the distinct methods that are used in these fields, indicating a need for caution in comparing results across fields and for future interdisciplinary collaborations to strengthen our emerging understanding of curiosity.

A causal role of the right dorsolateral prefrontal cortex in random exploration

Decision to explore new options with uncertain outcomes or exploit familiar options with known outcomes is a fundamental challenge that the brain faces in almost all real-life decisions. Previous studies have shown that humans use two main explorative strategies to negotiate this explore-exploit tradeoff. Exploring for the sake of information is called directed exploration, and exploration driven by behavioral variability is known as random exploration. While previous neuroimaging studies have shown different neural correlates for these explorative strategies, including right frontopolar cortex (FPC), right dorsolateral prefrontal cortex (DLPFC), and dorsal anterior cingulate cortex (dACC), there is still a lack of causal evidence for most of these brain regions. Here, we focused on the right DLPFC, which was previously supported to be involved in exploration. Using the continuous theta burst stimulation (cTBS) and Horizon task on twenty-five healthy right-handed adult participants, we showed that inhibiting rDLPFC did not change directed exploration but selectively reduced random exploration, by increasing reward sensitivity over the average reward of each option. This suggests a causal role for rDLPFC in random exploration, and further supports dissociable neural implementations for these two explorative strategies.

The effects of threat on complex decision-making: evidence from a virtual environment

Individuals living and working in dangerous settings (e.g., first responders and military personnel) make complex decisions amidst serious threats. However, controlled studies on decision-making under threat are limited given obvious ethical concerns. Here, we embed a complex decision-making task within a threatening, immersive virtual environment. Based on the Iowa Gambling Task (IGT), a paradigm widely used to study complex decision-making, the task requires participants to make a series of choices to escape a collapsing building. In Study 1 we demonstrate that, as with the traditional IGT, participants learn to make advantageous decisions over time and that their behavioural data can be described by reinforcement-learning based computational models. In Study 2 we created threatening and neutral versions of the environment. In the threat condition, participants performed worse, taking longer to improve from baseline and scoring lower through the final trials. Computational modelling further revealed that participants in the threat condition were more responsive to short term rewards and less likely to perseverate on a given choice. These findings suggest that when threat is integral to decision-making, individuals make more erratic choices and focus on short term gains. They furthermore demonstrate the utility of virtual environments for making threat integral to cognitive tasks.

Multiple and subject-specific roles of uncertainty in reward-guided decision-making

Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices. First, stable and psychologically meaningful individual differences in uncertainty preferences actually range from seeking to avoidance, which can manifest as null group-level effects. Second, uncertainty modulates the use of basic decision heuristics that imperfectly exploit immediate rewards: a repetition bias and win-stay-lose-shift heuristic. These heuristics interact with uncertainty, favoring heuristic choices under higher uncertainty. These results, highlighting the rich and varied structure of reward-based choice, are a step to understanding its functional basis and dysfunction in psychopathology.

Dynamics of spontaneous thoughts: Exploration, attentional profile and the segmentation of the stream of thoughts

For a long time, clinical knowledge and first-person reports have pointed to individual differences in the dynamics of spontaneous thoughts, in particular in the extreme case of psychiatric conditions (e.g. racing thoughts in Attention Deficit / Hyperactivity Disorder, ADHD; rumination in depression). We used a novel procedure to investigate this individual variability by combining verbal fluency tasks and introspective reports of thought content. Our goal was twofold. First, we tested the hypothesis that a greater segmentation of the stream of thoughts would be associated with trait inattention, in line with subjective reports of ADHD patients. Second, we tested whether the segmentation of the stream of thoughts increased with an increased tendency for exploratory behavior, following recent theoretical claims on the mechanisms underpinning the generation of spontaneous thoughts. Our results support both hypotheses, shedding light on the factors contributing to the individual variability in the dynamics of the stream of thought.

The explore/exploit trade-off: An ecologically valid and translational framework that can advance mechanistic understanding of eating disorders

The explore/exploit trade-off is a decision-making process that is conserved across species and balances exploring unfamiliar choices of unknown value with choosing familiar options of known value to maximize reward. This framework is rooted in behavioral ecology and has traditionally been used to study maladaptive versus adaptive non-human animal foraging behavior. Researchers have begun to recognize the potential utility of understanding human decision-making and psychopathology through the explore/exploit trade-off. In this article, we propose that explore/exploit trade-off holds promise for advancing our mechanistic understanding of decision-making processes that confer vulnerability for and maintain eating pathology due to its neurodevelopmental bases, conservation across species, and ability to be mathematically modeled. We present a model for how suboptimal explore/exploit decision-making can promote disordered eating and present recommendations for future research applying this framework to eating pathology. Taken together, the explore/exploit trade-off provides a translational framework for expanding etiologic and maintenance models of eating pathology, given developmental changes in explore/exploit decision-making that coincide in time with the emergence of eating pathology and evidence of biased explore/exploit decision-making in psychopathology. Additionally, understanding explore/exploit decision-making in eating disorders may improve knowledge of their underlying pathophysiology, informing targeted clinical interventions such as neuromodulation and pharmacotherapy. PUBLIC SIGNIFICANCE STATEMENT: The explore/exploit trade-off is a cross-species decision-making process whereby organisms choose between a known option with a known reward or sampling unfamiliar options. We hypothesize that imbalanced explore/exploit decision-making can promote disordered eating and present preliminary data. We propose that explore/exploit trade-off has significant potential to advance understanding of the neurocognitive and neurodevelopmental mechanisms of eating pathology, which could ultimately guide revisions of etiologic models and inform novel interventions.

Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder

Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.

Developmental changes in exploration resemble stochastic optimization

Human development is often described as a 'cooling off' process, analogous to stochastic optimization algorithms that implement a gradual reduction in randomness over time. Yet there is ambiguity in how to interpret this analogy, due to a lack of concrete empirical comparisons. Using data from n = 281 participants ages 5 to 55, we show that cooling off does not only apply to the single dimension of randomness. Rather, human development resembles an optimization process of multiple learning parameters, for example, reward generalization, uncertainty-directed exploration and random temperature. Rapid changes in parameters occur during childhood, but these changes plateau and converge to efficient values in adulthood. We show that while the developmental trajectory of human parameters is strikingly similar to several stochastic optimization algorithms, there are important differences in convergence. None of the optimization algorithms tested were able to discover reliably better regions of the strategy space than adult participants on this task.

Understanding patch foraging strategies across development

Patch foraging is a near-ubiquitous behaviour across the animal kingdom and characterises many decision-making domains encountered by humans. We review how a disposition to explore in adolescence may reflect the evolutionary conditions under which hunter-gatherers foraged for resources. We propose that neurocomputational mechanisms responsible for reward processing, learning, and cognitive control facilitate the transition from exploratory strategies in adolescence to exploitative strategies in adulthood - where individuals capitalise on known resources. This developmental transition may be disrupted by psychopathology, as there is emerging evidence of biases in explore/exploit choices in mental health problems. Explore/exploit choices may be an informative marker for mental health across development and future research should consider this feature of decision-making as a target for clinical intervention.

Diminished prospective mental representations of reward mediate reward learning strategies among youth with internalizing symptoms

BACKGROUND

Adolescent internalizing symptoms and trauma exposure have been linked with altered reward learning processes and decreased ventral striatal responses to rewarding cues. Recent computational work on decision-making highlights an important role for prospective representations of the imagined outcomes of different choices. This study tested whether internalizing symptoms and trauma exposure among youth impact the generation of prospective reward representations during decision-making and potentially mediate altered behavioral strategies during reward learning.

METHODS

Sixty-one adolescent females with varying exposure to interpersonal violence exposure (n = 31 with histories of physical or sexual assault) and severity of internalizing symptoms completed a social reward learning task during fMRI. Multivariate pattern analyses (MVPA) were used to decode neural reward representations at the time of choice.

RESULTS

MVPA demonstrated that rewarding outcomes could accurately be decoded within several large-scale distributed networks (e.g. frontoparietal and striatum networks), that these reward representations were reactivated prospectively at the time of choice in proportion to the expected probability of receiving reward, and that youth with behavioral strategies that favored exploiting high reward options demonstrated greater prospective generation of reward representations. Youth internalizing symptoms, but not trauma exposure characteristics, were negatively associated with both the behavioral strategy of exploiting high reward options as well as the prospective generation of reward representations in the striatum.

CONCLUSIONS

These data suggest diminished prospective mental simulation of reward as a mechanism of altered reward learning strategies among youth with internalizing symptoms.

Acetylcholine and noradrenaline enhance foraging optimality in humans

Foraging theory prescribes when optimal foragers should leave the current option for more rewarding alternatives. Actual foragers often exploit options longer than prescribed by the theory, but it is unclear how this foraging suboptimality arises. We investigated whether the upregulation of cholinergic, noradrenergic, and dopaminergic systems increases foraging optimality. In a double-blind, between-subject design, participants (N = 160) received placebo, the nicotinic acetylcholine receptor agonist nicotine, a noradrenaline reuptake inhibitor reboxetine, or a preferential dopamine reuptake inhibitor methylphenidate, and played the role of a farmer who collected milk from patches with different yield. Across all groups, participants on average overharvested. While methylphenidate had no effects on this bias, nicotine, and to some extent also reboxetine, significantly reduced deviation from foraging optimality, which resulted in better performance compared to placebo. Concurring with amplified goal-directedness and excluding heuristic explanations, nicotine independently also improved trial initiation and time perception. Our findings elucidate the neurochemical basis of behavioral flexibility and decision optimality and open unique perspectives on psychiatric disorders affecting these functions.

A low dimensional manifold of human exploratory behavior reveals opposing roles for apathy and anxiety

Exploration-exploitation decision-making is a feature of daily life that is altered in a number of neuropsychiatric conditions. Humans display a range of exploration and exploitation behaviors, which can be affected by apathy and anxiety. It remains unknown how factors underlying decision-making generate the spectrum of observed exploration-exploitation behavior and how they relate to states of anxiety and apathy. Here, we report a latent structure underlying sequential exploration and exploitation decisions that explains variation in anxiety and apathy. 1001 participants in a gender-balanced sample completed a three-armed restless bandit task along with psychiatric symptom surveys. Using dimensionality reduction methods, we found that decision sequences reduced to a low-dimensional manifold. The axes of this manifold explained individual differences in the balance between states of exploration and exploitation and the stability of those states, as determined by a statistical mechanics model of decision-making. Position along the balance axis was correlated with opposing symptoms of behavioral apathy and anxiety, while position along the stability axis correlated with the level of emotional apathy. This result resolves a paradox over how these symptoms can be correlated in samples but have opposite effects on behavior. Furthermore, this work provides a basis for using behavioral manifolds to reveal relationships between behavioral dynamics and affective states, with important implications for behavioral measurement approaches to neuropsychiatric conditions.

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Most psychiatric disorders do not occur in isolation, and most psychiatric symptom dimensions are not uniquely expressed within a single diagnostic category. Current treatments fail to work for around 25% to 40% of individuals, perhaps due at least in part to an overreliance on diagnostic categories in treatment development and allocation. In this review, we describe ongoing efforts in the field to surmount these challenges and precisely characterize psychiatric symptom dimensions using large-scale studies of unselected samples via remote, online, and "citizen science" efforts that take a dimensional, mechanistic approach. We discuss the importance that efforts to identify meaningful psychiatric dimensions be coupled with careful computational modeling to formally specify, test, and potentially falsify candidate mechanisms that underlie transdiagnostic symptom dimensions. We refer to this approach, i.e., where symptom dimensions are identified and validated against computationally well-defined neurocognitive processes, as computational factor modeling. We describe in detail some recent applications of this method to understand transdiagnostic cognitive processes that include model-based planning, metacognition, appetitive processing, and uncertainty estimation. In this context, we highlight how computational factor modeling has been used to identify specific associations between cognition and symptom dimensions and reveal previously obscured relationships, how findings generalize to smaller in-person clinical and nonclinical samples, and how the method is being adapted and optimized beyond its original instantiation. Crucially, we discuss next steps for this area of research, highlighting the value of more direct investigations of treatment response that bridge the gap between basic research and the clinic.