Model-Free Temporal-Difference Learning and Dopamine in Alcohol Dependence: Examining Concepts From Theory and Animals in Human Imaging

The utility of a latent-cause framework for understanding addiction phenomena

Computational models of addiction often rely on a model-free reinforcement learning (RL) formulation, owing to the close associations between model-free RL, habitual behavior and the dopaminergic system. However, such formulations typically do not capture key recurrent features of addiction phenomena such as craving and relapse. Moreover, they cannot account for goal-directed aspects of addiction that necessitate contrasting, model-based formulations. Here we synthesize a growing body of evidence and propose that a latent-cause framework can help unify our understanding of several recurrent phenomena in addiction, by viewing them as the inferred return of previous, persistent "latent causes". We demonstrate that applying this framework to Pavlovian and instrumental settings can help account for defining features of craving and relapse such as outcome-specificity, generalization, and cyclical dynamics. Finally, we argue that this framework can bridge model-free and model-based formulations, and account for individual variability in phenomenology by accommodating the memories, beliefs, and goals of those living with addiction, motivating a centering of the individual, subjective experience of addiction and recovery.

Computational analysis of probabilistic reversal learning deficits in male subjects with alcohol use disorder

BACKGROUND

Alcohol use disorder is characterized by perseverative alcohol use despite negative consequences. This hallmark feature of addiction potentially relates to impairments in behavioral flexibility, which can be measured by probabilistic reversal learning (PRL) paradigms. We here aimed to examine the cognitive mechanisms underlying impaired PRL task performance in patients with alcohol use disorder (AUDP) using computational models of reinforcement learning.

METHODS

Twenty-eight early abstinent AUDP and 27 healthy controls (HC) performed an extensive PRL paradigm. We compared conventional behavioral variables of choices (perseveration; correct responses) between groups. Moreover, we fitted Bayesian computational models to the task data to compare differences in latent cognitive variables including reward and punishment learning and choice consistency between groups.

RESULTS

AUDP and HC did not significantly differ with regard to direct perseveration rates after reversals. However, AUDP made overall less correct responses and specifically showed decreased win-stay behavior compared to HC. Interestingly, AUDP showed premature switching after no or little negative feedback but elevated proneness to stay when accumulation of negative feedback would make switching a more optimal option. Computational modeling revealed that AUDP compared to HC showed enhanced learning from punishment, a tendency to learn less from positive feedback and lower choice consistency.

CONCLUSION

Our data do not support the assumption that AUDP are characterized by increased perseveration behavior. Instead our findings provide evidence that enhanced negative reinforcement and decreased non-drug-related reward learning as well as diminished choice consistency underlie dysfunctional choice behavior in AUDP.

Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic "computational fingerprint", will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Brain responses to anticipating and receiving beer: Comparing light, at-risk, and dependent alcohol users

Impaired brain processing of alcohol‐related rewards has been suggested to play a central role in alcohol use disorder. Yet, evidence remains inconsistent and mainly originates from studies in which participants passively observe alcohol cues or taste alcohol. Here, we designed a protocol in which beer consumption was predicted by incentive cues and contingent on instrumental action closer to real life situations. We predicted that anticipating and receiving beer (compared with water) would elicit activity in the brain reward network and that this activity would correlate with drinking level across participants.

The sample consisted of 150 beer‐drinking males, aged 18 to 25 years. Three groups were defined based on alcohol use disorders identification test (AUDIT) scores: light drinkers (n = 39), at‐risk drinkers (n = 64), and dependent drinkers (n = 47). fMRI measures were obtained while participants engaged in the beer incentive delay task involving beer‐ and water‐predicting cues followed by real sips of beer or water.

During anticipation, outcome notification and delivery of beer compared with water, higher activity was found in a reward‐related brain network including the dorsal medial prefrontal cortex, orbitofrontal cortex, and amygdala. Yet, no activity was observed in the striatum, and no differences were found between the groups.

Our results reveal that anticipating, obtaining, and tasting beer activates parts of the brain reward network, but that these brain responses do not differentiate between different drinking levels.

Learning to lose control: A process-based account of behavioral addiction

Learning psycho(bio)logy has developed a solid corpus of evidence and theory regarding behavior control modes. The present article briefly reviews that literature and its influence on recent models in which the transition from goal-directed to compulsive behavior is identified as the main process underlying substance use disorders. This literature is also relevant to non-substance addictive disorders, and serves as basis to propose a restricted definition of behavioral addiction relying on the presence of behavior-specific compulsivity. Complementarily, we consider whether some activities can become disordered while remaining mostly goal-driven. Based on reinforcement learning models, relative outcome utility computation is proposed as an alternative mechanism through which dysfunctional behaviors (even not qualifying as addictive) can override adaptive ones, causing functional impairment. Beyond issues of conceptual delimitation, recommendations are made regarding the importance of identifying individual etiological pathways to dysregulated behavior, the necessity of accurately profiling at-risk individuals, and the potential hazards of symptom-based diagnosis. In our view, the validity of these recommendations does not depend on the position one takes in the nosological debate.

Neuroimaging reward, craving, learning, and cognitive control in substance use disorders: review and implications for treatment

Substance use disorder is a leading causes of preventable disease and mortality. Drugs of abuse cause molecular and cellular changes in specific brain regions and these neuroplastic changes are thought to play a role in the transition to uncontrolled drug use. Neuroimaging has identified neural substrates associated with problematic substance use and may offer clues to reduce its burden on the patient and society. Here, we provide a narrative review of neuroimaging studies that have examined the structures and circuits associated with reward, cues and craving, learning, and cognitive control in substance use disorders. Most studies use advanced MRI or positron emission tomography (PET). Many studies have focused on the dopamine neurons of the ventral tegmental area, and the regions where these neurons terminate, such as the striatum and prefrontal cortex. Decreases in dopamine receptors and transmission have been found in chronic users of drugs, alcohol, and nicotine. Recent studies also show evidence of differences in structure and function in substance users relative to controls in brain regions involved in salience evaluation, such as the insula and anterior cingulate cortex. Balancing between reward-related bottom-up and cognitive-control-related top-down processes is discussed in the context of neuromodulation as a potential treatment. Finally, some of the challenges for understanding substance use disorder using neuroimaging methods are discussed.

No association of goal-directed and habitual control with alcohol consumption in young adults

Alcohol dependence is a mental disorder that has been associated with an imbalance in behavioral control favoring model-free habitual over model-based goal-directed strategies. It is as yet unknown, however, whether such an imbalance reflects a predisposing vulnerability or results as a consequence of repeated and/or excessive alcohol exposure. We, therefore, examined the association of alcohol consumption with model-based goal-directed and model-free habitual control in 188 18-year-old social drinkers in a two-step sequential decision-making task while undergoing functional magnetic resonance imaging before prolonged alcohol misuse could have led to severe neurobiological adaptations. Behaviorally, participants showed a mixture of model-free and model-based decision-making as observed previously. Measures of impulsivity were positively related to alcohol consumption. In contrast, neither model-free nor model-based decision weights nor the trade-off between them were associated with alcohol consumption. There were also no significant associations between alcohol consumption and neural correlates of model-free or model-based decision quantities in either ventral striatum or ventromedial prefrontal cortex. Exploratory whole-brain functional magnetic resonance imaging analyses with a lenient threshold revealed early onset of drinking to be associated with an enhanced representation of model-free reward prediction errors in the posterior putamen. These results suggest that an imbalance between model-based goal-directed and model-free habitual control might rather not be a trait marker of alcohol intake per se.