Disrupted state transition learning as a computational marker of compulsivity

The Experience-Experience Gap: Distributional Learning Is Associated with a Divergence of Preferences from Estimations

Recent landmark studies show that the brain is equipped to learn not just average expected outcomes, but entire distributions of expected outcomes. Yet the role of such distributional learning in shaping human decision-making remains to be determined. To study this question, we designed two tasks where participants experienced different outcome distributions, provided their estimates of each, and reported their preferences among them. In one task, which facilitated distributional learning, participants' preferences significantly diverged from their own estimates, consistent with predictions of Prospect Theory. Conversely, in a task that hindered distributional learning, the divergence of preferences from estimates was eliminated. Computational modelling showed how distributional learning may be responsible for disassociating preferences from estimations by enabling the application of a utility function to different potential outcomes. Our findings offer a new understanding of when and how preferences deviate from normative decision-making, a fundamental question in the study of human rationality.

Schemas, reinforcement learning and the medial prefrontal cortex

Schemas are rich and complex knowledge structures about the typical unfolding of events in a context; for example, a schema of a dinner at a restaurant. In this Perspective, we suggest that reinforcement learning (RL), a computational theory of learning the structure of the world and relevant goal-oriented behaviour, underlies schema learning. We synthesize literature about schemas and RL to offer that three RL principles might govern the learning of schemas: learning via prediction errors, constructing hierarchical knowledge using hierarchical RL, and dimensionality reduction through learning a simplified and abstract representation of the world. We then suggest that the orbitomedial prefrontal cortex is involved in both schemas and RL due to its involvement in dimensionality reduction and in guiding memory reactivation through interactions with posterior brain regions. Last, we hypothesize that the amount of dimensionality reduction might underlie gradients of involvement along the ventral-dorsal and posterior-anterior axes of the orbitomedial prefrontal cortex. More specific and detailed representations might engage the ventral and posterior parts, whereas abstraction might shift representations towards the dorsal and anterior parts of the medial prefrontal cortex.

Impact of provoked stress on model-free and model-based reinforcement learning in individuals with alcohol use disorder

Background

From both clinical and theoretical perspectives, understanding the functionality of evaluative reinforcement learning mechanisms (Model-Free, MF, and Model-Based, MB) under provoked stress, particularly in Alcohol Use Disorder (AUD), is crucial yet underexplored. This study aims to evaluate whether individuals with AUD who do not seek treatment show a greater tendency towards retrospective behaviors (MF) rather than prospective and deliberative simulations (MB) compared to controls. Additionally, it examines the impact of induced social stress on these decision-making processes.

Methods

A cohort comprising 117 participants, including 55 individuals with AUD and 62 controls, was examined. Acute social stress was induced through the socially evaluated cold pressor task (SECPT), followed by engagement in a Two-Step Markov task to assess MB and MF learning tendencies. We measured hypothalamic-pituitary-adrenal axis stress response using salivary cortisol levels.

Results

Both groups showed similar baseline cortisol levels and responses to the SECPT. Our findings indicate that participants with AUD exhibit a reduced reliance on MB strategies compared to those without AUD. Furthermore, stress decreases reliance on MB strategies in healthy participants, but this effect is not observed in those with AUD.

Conclusion

An atypical pattern of stress modulation impacting the balance between MB and MF reinforcement learning was identified in individuals with AUD who are not seeking treatment. Potential explanations for these findings and their clinical implications are explored.

Survival in a world of complex dangers

How did our nomadic ancestors continually adapt to the seemingly limitless and unpredictable number of dangers in the natural world? We argue that human defensive behaviors are dynamically constructed to facilitate survival in capricious and itinerant environments. We first hypothesize that internal and external states result in state constructions that combine to form a meta-representation. When a threat is detected, it triggers the action construction. Action constructions are formed through two contiguous survival strategies: generalization strategies, which are used when encountering new threats and ecologies. Generalization strategies are associated with cognitive representations that have high dimensionality and which furnish flexible psychological constructs, including relations between threats, and imagination, and which converge through the construction of defensive states. We posit that generalization strategies drive 'explorative' behaviors including information seeking, where the goal is to increase knowledge that can be used to mitigate current and future threats. Conversely, specialization strategies entail lower dimensional representations, which underpin specialized, sometimes reflexive, or habitual survival behaviors that are 'exploitative'. Together, these strategies capture a central adaptive feature of human survival systems: self-preservation in response to a myriad of threats.

Proactive control for conflict resolution is intact in subclinical obsessive-compulsive individuals

Background

Obsessive-compulsive (OC) traits (i.e., tendency to implement stereotyped behaviors to avoid negative consequences) are transversally observed in psychiatric disorders largely differing in terms of clinical manifestations and etiopathogenesis. Interestingly, OC traits were also extensively found in the prodromal phases of the full-blown psychopathology and in healthy relatives of affected individuals. Moreover, OC traits were found to be associated—and possibly underlain by—cognitive control impairments. Nonetheless, the role of such interplay in the onset of OC disorders is yet to be understood. We hypothesized that OC traits are associated with abnormalities in proactively implement cognitive control for solving conflict.

Methods

We administered healthy individuals (n = 104) with the perifoveal spatial Stroop task to measure their ability of solving conflict in a proactive fashion, and with Obsessive-Compulsive Inventory (OCI) to stratify population according to the severity of OC traits.

Results

Analysis of response times by means of Linear Mixed-effect models revealed that proactive control performance was not associated with and the severity of OC traits. Furthermore, an equivalence test (Two One-Sided Test) revealed that the association between OCI scores and task performance was equivalent to zero.

Conclusion

These results suggest that the interplay between OC traits and proactive control abnormalities might not contribute to the development of OC-related disorders. Therefore, the role of other cognitive endophenotypes should be scrutinized for exploiting alternative prevention and intervention strategies.

Beliefs, compulsive behavior and reduced confidence in control

OCD has been conceptualized as a disorder arising from dysfunctional beliefs, such as overestimating threats or pathological doubts. Yet, how these beliefs lead to compulsions and obsessions remains unclear. Here, we develop a computational model to examine the specific beliefs that trigger and sustain compulsive behavior in a simple symptom-provoking scenario. Our results demonstrate that a single belief disturbance-a lack of confidence in the effectiveness of one's preventive (harm-avoiding) actions-can trigger and maintain compulsions and is directly linked to compulsion severity. This distrust can further explain a number of seemingly unrelated phenomena in OCD, including the role of not-just-right feelings, the link to intolerance to uncertainty, perfectionism, and overestimation of threat, and deficits in reversal and state learning. Our simulations shed new light on which underlying beliefs drive compulsive behavior and highlight the important role of perceived ability to exert control for OCD.

Using smartphones to optimise and scale-up the assessment of model-based planning

Model-based planning is thought to protect against over-reliance on habits. It is reduced in individuals high in compulsivity, but effect sizes are small and may depend on subtle features of the tasks used to assess it. We developed a diamond-shooting smartphone game that measures model-based planning in an at-home setting, and varied the game's structure within and across participants to assess how it affects measurement reliability and validity with respect to previously established correlates of model-based planning, with a focus on compulsivity. Increasing the number of trials used to estimate model-based planning did remarkably little to affect the association with compulsivity, because the greatest signal was in earlier trials. Associations with compulsivity were higher when transition ratios were less deterministic and depending on the reward drift utilised. These findings suggest that model-based planning can be measured at home via an app, can be estimated in relatively few trials using certain design features, and can be optimised for sensitivity to compulsive symptoms in the general population.

Interactive cognitive maps support flexible behavior under threat

In social environments, survival can depend upon inferring and adapting to other agents' goal-directed behavior. However, it remains unclear how humans achieve this, despite the fact that many decisions must account for complex, dynamic agents acting according to their own goals. Here, we use a predator-prey task (total n = 510) to demonstrate that humans exploit an interactive cognitive map of the social environment to infer other agents' preferences and simulate their future behavior, providing for flexible, generalizable responses. A model-based inverse reinforcement learning model explained participants' inferences about threatening agents' preferences, with participants using this inferred knowledge to enact generalizable, model-based behavioral responses. Using tree-search planning models, we then found that behavior was best explained by a planning algorithm that incorporated simulations of the threat's goal-directed behavior. Our results indicate that humans use a cognitive map to determine other agents' preferences, facilitating generalized predictions of their behavior and effective responses.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Identifying Transdiagnostic Mechanisms in Mental Health Using Computational Factor Modeling

Most psychiatric disorders do not occur in isolation, and most psychiatric symptom dimensions are not uniquely expressed within a single diagnostic category. Current treatments fail to work for around 25% to 40% of individuals, perhaps due at least in part to an overreliance on diagnostic categories in treatment development and allocation. In this review, we describe ongoing efforts in the field to surmount these challenges and precisely characterize psychiatric symptom dimensions using large-scale studies of unselected samples via remote, online, and "citizen science" efforts that take a dimensional, mechanistic approach. We discuss the importance that efforts to identify meaningful psychiatric dimensions be coupled with careful computational modeling to formally specify, test, and potentially falsify candidate mechanisms that underlie transdiagnostic symptom dimensions. We refer to this approach, i.e., where symptom dimensions are identified and validated against computationally well-defined neurocognitive processes, as computational factor modeling. We describe in detail some recent applications of this method to understand transdiagnostic cognitive processes that include model-based planning, metacognition, appetitive processing, and uncertainty estimation. In this context, we highlight how computational factor modeling has been used to identify specific associations between cognition and symptom dimensions and reveal previously obscured relationships, how findings generalize to smaller in-person clinical and nonclinical samples, and how the method is being adapted and optimized beyond its original instantiation. Crucially, we discuss next steps for this area of research, highlighting the value of more direct investigations of treatment response that bridge the gap between basic research and the clinic.

Rethinking model-based and model-free influences on mental effort and striatal prediction errors

A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

Computational perspectives on human fear and anxiety

Fear and anxiety are adaptive emotions that serve important defensive functions, yet in excess, they can be debilitating and lead to poor mental health. Computational modelling of behaviour provides a mechanistic framework for understanding the cognitive and neurobiological bases of fear and anxiety, and has seen increasing interest in the field. In this brief review, we discuss recent developments in the computational modelling of human fear and anxiety. Firstly, we describe various reinforcement learning strategies that humans employ when learning to predict or avoid threat, and how these relate to symptoms of fear and anxiety. Secondly, we discuss initial efforts to explore, through a computational lens, approach-avoidance conflict paradigms that are popular in animal research to measure fear- and anxiety-relevant behaviours. Finally, we discuss negative biases in decision-making in the face of uncertainty in anxiety.

The role of the lateral orbitofrontal cortex in creating cognitive maps

We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.

Humans perseverate on punishment avoidance goals in multigoal reinforcement learning

Managing multiple goals is essential to adaptation, yet we are only beginning to understand computations by which we navigate the resource demands entailed in so doing. Here, we sought to elucidate how humans balance reward seeking and punishment avoidance goals, and relate this to variation in its expression within anxious individuals. To do so, we developed a novel multigoal pursuit task that includes trial-specific instructed goals to either pursue reward (without risk of punishment) or avoid punishment (without the opportunity for reward). We constructed a computational model of multigoal pursuit to quantify the degree to which participants could disengage from the pursuit goals when instructed to, as well as devote less model-based resources toward goals that were less abundant. In general, participants (n = 192) were less flexible in avoiding punishment than in pursuing reward. Thus, when instructed to pursue reward, participants often persisted in avoiding features that had previously been associated with punishment, even though at decision time these features were unambiguously benign. In a similar vein, participants showed no significant downregulation of avoidance when punishment avoidance goals were less abundant in the task. Importantly, we show preliminary evidence that individuals with chronic worry may have difficulty disengaging from punishment avoidance when instructed to seek reward. Taken together, the findings demonstrate that people avoid punishment less flexibly than they pursue reward. Future studies should test in larger samples whether a difficulty to disengage from punishment avoidance contributes to chronic worry.