Model-Based and Model-Free Control Predicts Alcohol Consumption Developmental Trajectory in Young Adults: A 3-Year Prospective Study

Negative affect-driven impulsivity as hierarchical model-based overgeneralization

'If your mouth is burned by milk, you blow before you eat yogurt' ('Sütten ağzı yanan yoğurdu üfleyerek yer'). This Turkish proverb advises caution based on past experiences when similar situations are encountered. However, although we may infer similarities across experiences, each situation is a complex combination of many features, and generalizing across situations based on perceived similarities may not achieve desired outcomes when obtaining them depends on more subtle or overlooked features. Here, we examine how models of generalization can uncover the model-based (MB) processes underlying reactive and rigid behaviors traditionally considered model-free (MF). Our novel conceptualization suggests that emotionally driven impulsive behaviors stem from a propensity to overgeneralize based on surface-level similarities, hindering the incorporation of other informative, discriminant cues.

The role of psychological distress in understanding the relationship between habitual decision-making and addictive behaviors

Existing models are currently inadequate in explaining the relationship between habitual decision-making and different expressions of addictive behaviors. The current study investigates the role of psychological distress as a key factor in disrupting decision-making processes in the context of substance and behavioral addictions. A large community sample (N = 668) completed a gamified two-stage task to investigate the link between model-free (habitual) task behavior and a wide range of addictive behaviors. Addictive behaviors included substance use (alcohol, nicotine, and illicit drug use) and behavioral addictions (problematic use of the internet, addictive eating, shopping, and gambling). The relationship between habitual task behavior and addictive engagement was investigated using structural equation modelling with a bifactor latent variable structure, which was modeled and tested; one for substance use and one for behavioral addictions. For participants with higher levels of psychological distress, greater habitual task behavior was a significant predictor of behavioral addiction risk (β = -0.188, SE = 2.061, p = .016), specifically problematic use of the internet (β = -0.148, SE = 0.045, p = .018) and eating behaviors (β = -0.191, SE = 0.016, p < .001). These findings support our proposition that psychological distress disrupts cognitive control, leading to a greater reliance on habitual decision-making and non-substance addictive behaviors. This highlights the potential importance of habit-based interventions in combination with stress-reduction and mental health-promoting techniques to improve outcomes and minimize harm, especially in the context of behavioral addictions.

Impact of provoked stress on model-free and model-based reinforcement learning in individuals with alcohol use disorder

Background

From both clinical and theoretical perspectives, understanding the functionality of evaluative reinforcement learning mechanisms (Model-Free, MF, and Model-Based, MB) under provoked stress, particularly in Alcohol Use Disorder (AUD), is crucial yet underexplored. This study aims to evaluate whether individuals with AUD who do not seek treatment show a greater tendency towards retrospective behaviors (MF) rather than prospective and deliberative simulations (MB) compared to controls. Additionally, it examines the impact of induced social stress on these decision-making processes.

Methods

A cohort comprising 117 participants, including 55 individuals with AUD and 62 controls, was examined. Acute social stress was induced through the socially evaluated cold pressor task (SECPT), followed by engagement in a Two-Step Markov task to assess MB and MF learning tendencies. We measured hypothalamic-pituitary-adrenal axis stress response using salivary cortisol levels.

Results

Both groups showed similar baseline cortisol levels and responses to the SECPT. Our findings indicate that participants with AUD exhibit a reduced reliance on MB strategies compared to those without AUD. Furthermore, stress decreases reliance on MB strategies in healthy participants, but this effect is not observed in those with AUD.

Conclusion

An atypical pattern of stress modulation impacting the balance between MB and MF reinforcement learning was identified in individuals with AUD who are not seeking treatment. Potential explanations for these findings and their clinical implications are explored.

Disorders of compulsivity: Deficits in arbitrating learning strategies

While previous research has shown that compulsivity is related to an imbalance between goal-directed and habitual learning systems, very little is known about whether this effect is due to the impairment of a single system or the impairment of the arbitration mechanism that determines which system controls behaviour at any given moment; the current study aims to address this disagreement. Nineteen alcohol use disorder, 30 obsessive-compulsive disorder (OCD) and 20 major depressive disorder patients and corresponding sex- and age-matched controls performed two-choice, three-stage Markov decision-making paradigm. Model-based and mode-free reinforcement learning models were used to independently fitted their behavioural data. Alcohol use disorder and OCD patients showed less model-based strategy choice than healthy controls in task conditions where the model-based strategy was optimal. Only OCD patients showed higher behavioural control system switching in task conditions where model-free use was optimal. Major depressive disorder patients did not differ from the matched control in both. These findings suggest that dysfunction in arbitration control between dual systems may be the basis for diverse disorders involving compulsivity.

Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism

Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.

The ReCoDe addiction research consortium: Losing and regaining control over drug intake-Findings and future perspectives

Substance use disorders (SUDs) are seen as a continuum ranging from goal-directed and hedonic drug use to loss of control over drug intake with aversive consequences for mental and physical health and social functioning. The main goals of our interdisciplinary German collaborative research centre on Losing and Regaining Control over Drug Intake (ReCoDe) are (i) to study triggers (drug cues, stressors, drug priming) and modifying factors (age, gender, physical activity, cognitive functions, childhood adversity, social factors, such as loneliness and social contact/interaction) that longitudinally modulate the trajectories of losing and regaining control over drug consumption under real-life conditions. (ii) To study underlying behavioural, cognitive and neurobiological mechanisms of disease trajectories and drug-related behaviours and (iii) to provide non-invasive mechanism-based interventions. These goals are achieved by: (A) using innovative mHealth (mobile health) tools to longitudinally monitor the effects of triggers and modifying factors on drug consumption patterns in real life in a cohort of 900 patients with alcohol use disorder. This approach will be complemented by animal models of addiction with 24/7 automated behavioural monitoring across an entire disease trajectory; i.e. from a naïve state to a drug-taking state to an addiction or resilience-like state. (B) The identification and, if applicable, computational modelling of key molecular, neurobiological and psychological mechanisms (e.g., reduced cognitive flexibility) mediating the effects of such triggers and modifying factors on disease trajectories. (C) Developing and testing non-invasive interventions (e.g., Just-In-Time-Adaptive-Interventions (JITAIs), various non-invasive brain stimulations (NIBS), individualized physical activity) that specifically target the underlying mechanisms for regaining control over drug intake. Here, we will report on the most important results of the first funding period and outline our future research strategy.

Arbitration between model-free and model-based control is not affected by transient changes in tonic serotonin levels

BACKGROUND

Serotonin has been suggested to modulate decision-making by influencing the arbitration between model-based and model-free control. Disruptions in these control mechanisms are involved in mental disorders such as drug dependence or obsessive-compulsive disorder. While previous reports indicate that lower brain serotonin levels reduce model-based control, it remains unknown whether increases in serotonergic availability might thus increase model-based control. Moreover, the mediating neural mechanisms have not been studied yet.

AIM

The first aim of this study was to investigate whether increased/decreased tonic serotonin levels affect the arbitration between model-free and model-based control. Second, we aimed to identify the underlying neural processes.

METHODS

We employed a sequential two-stage Markov decision-task and measured brain responses during functional magnetic resonance imaging in 98 participants in a randomized, double-blind cross-over within-subject design. To investigate the influence of serotonin on the balance between model-free and model-based control, we used a tryptophan intervention with three intervention levels (loading, balanced, depletion). We hypothesized that model-based behaviour would increase with higher serotonin levels.

RESULTS

We found evidence that neither model-free nor model-based control were affected by changes in tonic serotonin levels. Furthermore, our tryptophan intervention did not elicit relevant changes in Blood-Oxygenation-Level Dependent activity.

Model-based and model-free mechanisms in methamphetamine use disorder

People with methamphetamine use disorder (MUD) struggle to shift their behaviour from methamphetamine-orientated habits to goal-oriented choices. The model-based/model-free framework is well suited to understand this difficulty by unpacking the computational mechanisms that support experienced-based (model-free) and goal-directed (model-based) choices. We aimed to examine whether 1) participants with MUD differed from controls on behavioural proxies and/or computational mechanisms of model-based/model-free choices; 2) model-based/model-free decision-making correlated with MUD symptoms; and 3) model-based/model-free deficits improved over six weeks in the group with MUD. Participants with MUD and controls with similar age, IQ and socioeconomic status completed the Two-Step Task at treatment commencement (MUD n = 30, Controls n = 31) and six weeks later (MUD n = 23, Controls n = 26). We examined behavioural proxies of model-based/model-free decisions using mixed logistic regression, and their underlying mechanisms using computational modelling. At a behavioural level, participants with MUD were more likely to switch their choices following rewarded actions, although this pattern improved at follow up. At a computational level, groups were similar in their use of model-based mechanisms, but participants with MUD were less likely to apply model-free mechanisms and less likely to repeat rewarded actions. We did not find evidence that individual differences in model-based or model-free parameters were associated with greater severity of methamphetamine dependence, nor did we find that group differences in computational parameters changed between baseline and follow-up assessment. Decision-making challenges in people with MUD are likely related to difficulties in pursuing choices previously associated with positive outcomes.

Elevated Amygdala Responses During De Novo Pavlovian Conditioning in Alcohol Use Disorder Are Associated With Pavlovian-to-Instrumental Transfer and Relapse Latency

Background

Contemporary learning theories of drug addiction ascribe a key role to Pavlovian learning mechanisms in the development, maintenance, and relapse of addiction. In fact, cue-reactivity research has demonstrated the power of alcohol-associated cues to activate the brain's reward system, which has been linked to craving and subsequent relapse. However, whether de novo Pavlovian conditioning is altered in alcohol use disorder (AUD) has rarely been investigated.

Methods

To characterize de novo Pavlovian conditioning in AUD, 62 detoxified patients with AUD and 63 matched healthy control participants completed a Pavlovian learning task as part of a Pavlovian-to-instrumental transfer paradigm during a functional magnetic resonance imaging session. Patients were followed up for 12 months to assess drinking behavior and relapse status.

Results

While patients and healthy controls did not differ in their ability to explicitly acquire the contingencies between conditioned and unconditioned stimuli, patients with AUD displayed significantly stronger amygdala responses toward Pavlovian cues, an effect primarily driven by stronger blood oxygen level-dependent differentiation during learning from reward compared with punishment. Moreover, in patients compared with controls, differential amygdala responses during conditioning were positively related to the ability of Pavlovian stimuli to influence ongoing instrumental choice behavior measured during a subsequent Pavlovian-to-instrumental transfer test. Finally, patients who relapsed within the 12-month follow-up period showed an inverse association between amygdala activity during conditioning and relapse latency.

Conclusions

We provide evidence of altered neural correlates of de novo Pavlovian conditioning in patients with AUD, especially for appetitive stimuli. Thus, heightened processing of Pavlovian cues might constitute a behaviorally relevant mechanism in alcohol addiction.

Neurocognitive predictors of addiction-related outcomes: A systematic review of longitudinal studies

It is well-established that addiction is typically associated with a distinct pattern of neurocognitive functioning with a consensus that it is typified by impaired top-down executive control and aberrant risk-reward processing. Despite a consensus that neurocognition plays an important role in characterizing and maintaining addictive disorders, there is a lack of systematic, bottom-up synthesis of quantitative evidence showing that neurocognition predicts addictive behaviors, and which neurocognitive constructs have the best predictive validity. This systematic review aimed to assess whether cognitive control and risk-reward processes as defined by the Research Domain Criteria (RDoC) predict the development and maintenance of addictive behaviors specifically, consumption, severity, and relapse. The findings from this review expose the substantial lack of evidence for neurocognition predicting addiction outcomes. However, there is evidence that suggests reward-related neurocognitive processes may be important for the detection of early risk for addiction, as well as a potentially viable target for designing novel, more effective interventions.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Rethinking model-based and model-free influences on mental effort and striatal prediction errors

A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.

Susceptibility to interference between Pavlovian and instrumental control predisposes risky alcohol use developmental trajectory from ages 18 to 24

Pavlovian cues can influence ongoing instrumental behaviour via Pavlovian-to-instrumental transfer (PIT) processes. While appetitive Pavlovian cues tend to promote instrumental approach, they are detrimental when avoidance behaviour is required, and vice versa for aversive cues. We recently reported that susceptibility to interference between Pavlovian and instrumental control assessed via a PIT task was associated with risky alcohol use at age 18. We now investigated whether such susceptibility also predicts drinking trajectories until age 24, based on AUDIT (Alcohol Use Disorders Identification Test) consumption and binge drinking (gramme alcohol/drinking occasion) scores. The interference PIT effect, assessed at ages 18 and 21 during fMRI, was characterized by increased error rates (ER) and enhanced neural responses in the ventral striatum (VS), the lateral and dorsomedial prefrontal cortices (dmPFC) during conflict, that is, when an instrumental approach was required in the presence of an aversive Pavlovian cue or vice versa. We found that a stronger VS response during conflict at age 18 was associated with a higher starting point of both drinking trajectories but predicted a decrease in binge drinking. At age 21, high ER and enhanced neural responses in the dmPFC were associated with increasing AUDIT-C scores over the next 3 years until age 24. Overall, susceptibility to interference between Pavlovian and instrumental control might be viewed as a predisposing mechanism towards hazardous alcohol use during young adulthood, and the identified high-risk group may profit from targeted interventions.

Translational models of addiction phenotypes to advance addiction pharmacotherapy

Alcohol and substance use disorders are heterogeneous conditions with limited effective treatment options. While there have been prior attempts to classify addiction subtypes, they have not been translated into clinical practice. In an effort to better understand heterogeneity in psychiatric disorders, the National Institute for Mental Health Research Domain Criteria (RDoC) has challenged scientists to think beyond diagnostic symptoms and to consider the underlying features of psychopathology from a neuroscience-based framework. The field of addiction has grappled with this approach by considering several key constructs with the potential to capture RDoC domains. This critical review will focus on the efforts to apply translational models of addiction phenomenology in human clinical samples, including their relative strengths and weaknesses. Opportunities for forward and reverse translation are also discussed. Deep behavioral phenotyping using neuroscience-informed batteries shows promise for a better understanding of the clinical neuroscience of addiction and advancing precision medicine for alcohol and substance use disorders.

The effect of body image dissatisfaction on goal-directed decision making in a population marked by negative appearance beliefs and disordered eating

Eating disorders are associated with one of the highest mortality rates among all mental disorders, yet there is very little research about them within the newly emerging and promising field of computational psychiatry. As such, we focus on investigating a previously unexplored, yet core aspect of eating disorders-body image dissatisfaction. We continue a freshly opened debate about model-based learning and its trade-off against model-free learning-a proxy for goal-directed and habitual behaviour. We perform a behavioural study that utilises a two-step decision-making task and a reinforcement learning model to understand the effect of body image dissatisfaction on model-based learning in a population characterised by high scores of disordered eating and negative appearance beliefs, as recruited using Prolific. We find a significantly reduced model-based contribution in the body image dissatisfaction task condition in the population of interest as compared to a healthy control. This finding suggests general deficits in deliberate control in this population, leading to habitual, compulsive-like behaviours (body checking) dominating the experience. Importantly, the results may inform treatment approaches, which could focus on enhancing the reliance on goal-directed decision making to help cope with unwanted behaviours.

Prelimbic cortex neural encoding dynamically tracks expected outcome value

Animals must modify their behavior based on updated expected outcomes in a changing environment. Prelimbic cortex (PrL) neural encoding during learning predicts, and is necessary for, appropriately altering behavior based on a new expected outcome value following devaluation. We aimed to determine how PrL neural activity encodes reward predictive cues after the expected outcome value of those cues is decreased following conditioned taste aversion. In one post-devaluation session, rats were tested under extinction to determine their ability to alter their behavior to the expected outcome values (i.e., extinction test). In a second post-devaluation session, rats were tested with the newly devalued outcome delivered so that the rats experienced the updated outcome value within the session (i.e., re-exposure test). We found that PrL neural encoding of the cue associated with the devalued reward predicted the ability of rats to suppress behavior in the extinction test session, but not in the re-exposure test session. While all rats were able to successfully devalue the outcome during conditioned taste aversion, a subset of rats continued to consume the devalued outcome in the re-exposure test session. We found differential patterns of PrL neural encoding in the population of rats that did not avoid the devalued outcome during the re-exposure test compared to the rats that successfully avoided the devalued outcome. Our findings suggest that PrL neural encoding dynamically tracks expected outcome values, and differential neural encoding in the PrL to reward predictive cues following expected outcome value changes may contribute to distinct behavioral phenotypes.

Psychopathic tendency in violent offenders is associated with reduced aversive Pavlovian inhibition of behavior and associated striatal BOLD signal

Background

Violent offenders with psychopathic tendencies are characterized by instrumental, i.e., planned, callous, and unemotional (aggressive) behavior and have been shown to exhibit abnormal aversive processing. However, the consequences of abnormal aversive processing for instrumental action and associated neural mechanisms are unclear.

Materials and methods

Here we address this issue by using event-related functional magnetic resonance imaging (fMRI) in 15 violent offenders with high psychopathic tendencies and 18 matched controls during the performance of an aversive Pavlovian-to-instrumental transfer paradigm. This paradigm allowed us to assess the degree to which aversive Pavlovian cues affect instrumental action and associated neural signaling.

Results

Psychopathic tendency scores were associated with an attenuation of aversive Pavlovian inhibition of instrumental action. Moreover, exploratory analyses revealed an anomalous positive association between aversive inhibition of action and aversive inhibition of BOLD signal in the caudate nucleus of violent offenders with psychopathic tendencies. In addition, psychopathic tendency also correlated positively with amygdala reactivity during aversive versus neutral cues in Pavlovian training.

Conclusion

These findings strengthen the hypothesis that psychopathic tendencies in violent offenders are related to abnormal impact of aversive processing on instrumental behavior. The neural effects raise the possibility that this reflects deficient transfer of aversive Pavlovian inhibitory biases onto neural systems that implement instrumental action, including the caudate nucleus.

The modulation of acute stress on model-free and model-based reinforcement learning in gambling disorder

BACKGROUND AND AIMS

Experiencing acute stress is common in behavioral addictions such as gambling disorder. Additionally, like most substance-induced addictions, aberrant decision-making wherein a reactive habit-induced response (conceptualized as a Model-free [MF] in reinforcement learning) suppresses a flexible goal-directed response (conceptualized as a Model-based [MB]) is also common in gambling disorder. In the current study we investigated the influence of acute stress on the balance between habitual response and the goal-directed system.

METHODS

A sample of N = 116 problem gamblers (PG) and healthy controls (HC) performed an acute stress task - the Socially Evaluated Cold pressure task (SECPT) - or a control task. Self-reported stress and salivary cortisol were collected as measures of acute stress. Following the SECPT, participants performed the Two-Step Markov Task to account for the relative contribution of MB and MF strategies. Additionally, verbal working memory and IQ measures were collected to account for their mediating effects on the orchestration between MB/MF and the impact of stress.

RESULTS

Both groups had comparable baseline and stress-induced cortisol response to the SECPT. Non-stressed PG displayed lower MB learning than HC. MANOVA and regression analyses showed a deleterious effect of stress-induced cortisol response on the orchestration between MB and MF learning in HC but not in PG. These effects remained when controlling for working memory and IQ.

DISCUSSION AND CONCLUSIONS

We found an abnormal pattern of modulation of stress on the orchestration between MB and MF learning among PG. Several interpretations and future research directions are discussed.

Amygdala response predicts clinical symptom reduction in patients with borderline personality disorder: A pilot fMRI study

Borderline personality disorder (BPD) is a prevalent, devastating, and heterogeneous psychiatric disorder. Treatment success is highly variable within this patient group. A cognitive neuroscientific approach to BPD might contribute to precision psychiatry by identifying neurocognitive factors that predict who will benefit from a specific treatment. Here, we build on observations that BPD is accompanied by the enhanced impact of the aversive effect on behavior and abnormal neural signaling in the amygdala. We assessed whether BPD is accompanied by abnormal aversive regulation of instrumental behavior and associated neural signaling, in a manner that is predictive of symptom reduction after therapy. We tested a clinical sample of 15 female patients with BPD, awaiting dialectical behavior therapy (DBT), and 16 matched healthy controls using fMRI and an aversive Pavlovian-to-instrumental transfer (PIT) task that assesses how instrumental behaviors are influenced by aversive Pavlovian stimuli. Patients were assessed 1 year after the start of DBT to quantify changes in BPD symptom severity. At baseline, behavioral aversive PIT and associated neural signaling did not differ between groups. However, the BOLD signal in the amygdala measured during aversive PIT was associated with symptom reduction at 1-year follow-up: higher PIT-related aversive amygdala signaling before treatment was associated with reduced clinical improvement at follow-up. Thus, within the evaluated group of BPD patients, the BOLD signal in the amygdala before treatment was related to clinical symptom reduction 1 year after the start of treatment. The results suggest that less PIT-related responsiveness of the amygdala increases the chances of treatment success. We note that the relatively small sample size is a limitation of this study and that replication is warranted.

Reinforcement learning detuned in addiction: integrative and translational approaches

Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.