The role of learning-related dopamine signals in addiction vulnerability

Implementations of sign- and goal-tracking behavior in humans: A scoping review

Animal research has identified two major phenotypes in the tendency to attribute incentive salience to a reward-associated cue. Individuals called "sign-trackers" (STs) preferentially approach the cue, assigning both predictive and incentive values to it. In contrast, individuals called "goal-trackers" (GTs) preferentially approach the location of the upcoming reward, assigning only a predictive value to the cue. The ST/GT model has been shown to be relevant to understanding addiction vulnerability and other pathological behaviors in animals. Therefore, recent studies tried to implement this animal model in the human population. This scoping review aimed to identify and map evidence of human sign- and goal-tracking. Studies that explicitly measured human sign- and goal-tracking or related phenomena (e.g., attentional bias induced by reward-related cues), using paradigms in line with the animal model, were eligible for this review. We searched for published, unpublished, and gray literature (PhD theses, posters, conference papers) through the following databases: MEDLINE, Scopus, PsycINFO, Embase, OSF, and Google Scholar. The JBI scoping review methodology was adopted. Screening and extraction were carried out by three reviewers, in pairs. A total of 48 studies were identified. These studies used various experimental paradigms and used the term "sign-tracking" inconsistently, sometimes implicitly or not at all. We conclude that the literature on human sign-tracking is very heterogeneous on many levels. Overall, evidence supports the existence of sign- and goal-tracking behaviors in humans, although further validated research is crucially needed.

Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction

BACKGROUND

Drugs like opioids are potent reinforcers thought to co-opt value-based decisions by overshadowing other rewarding outcomes, but how this happens at a neurocomputational level remains elusive. Range adaptation is a canonical process of fine-tuning representations of value based on reward context. Here, we tested whether recent opioid exposure impacts range adaptation in opioid use disorder, potentially explaining why shifting decision making away from drug taking during this vulnerable period is so difficult.

METHODS

Participants who had recently (<90 days) used opioids (n = 34) or who had abstained from opioid use for ≥ 90 days (n = 20) and comparison control participants (n = 44) completed a reinforcement learning task designed to induce robust contextual modulation of value. Two models were used to assess the latent process that participants engaged while making their decisions: 1) a Range model that dynamically tracks context and 2) a standard Absolute model that assumes stationary, objective encoding of value.

RESULTS

Control participants and ≥90-days-abstinent participants with opioid use disorder exhibited choice patterns consistent with range-adapted valuation. In contrast, participants with recent opioid use were more prone to learn and encode value on an absolute scale. Computational modeling confirmed the behavior of most control participants and ≥90-days-abstinent participants with opioid use disorder (75%), but a minority in the recent use group (38%), was better fit by the Range model than the Absolute model. Furthermore, the degree to which participants relied on range adaptation correlated with duration of continuous abstinence and subjective craving/withdrawal.

CONCLUSIONS

Reduced context adaptation to available rewards could explain difficulty deciding about smaller (typically nondrug) rewards in the aftermath of drug exposure.

More widespread and rigid neuronal representation of reward expectation underlies impulsive choices

Impulsive choices prioritize smaller, more immediate rewards over larger, delayed, or potentially uncertain rewards. Impulsive choices are a critical aspect of substance use disorders and maladaptive decision-making across the lifespan. Here, we sought to understand the neuronal underpinnings of expected reward and risk estimation on a trial-by-trial basis during impulsive choices. To do so, we acquired electrical recordings from the human brain while participants carried out a risky decision-making task designed to measure choice impulsivity. Behaviorally, we found a reward-accuracy tradeoff, whereby more impulsive choosers were more accurate at the task, opting for a more immediate reward while compromising overall task performance. We then examined how neuronal populations across frontal, temporal, and limbic brain regions parametrically encoded reinforcement learning model variables, namely reward and risk expectation and surprise, across trials. We found more widespread representations of reward value expectation and prediction error in more impulsive choosers, whereas less impulsive choosers preferentially represented risk expectation. A regional analysis of reward and risk encoding highlighted the anterior cingulate cortex for value expectation, the anterior insula for risk expectation and surprise, and distinct regional encoding between impulsivity groups. Beyond describing trial-by-trial population neuronal representations of reward and risk variables, these results suggest impaired inhibitory control and model-free learning underpinnings of impulsive choice. These findings shed light on neural processes underlying reinforced learning and decision-making in uncertain environments and how these processes may function in psychiatric disorders.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Excitatory and Inhibitory Signaling in the Nucleus Accumbens Encode Different Aspects of a Pavlovian Cue in Sign Tracking and Goal Tracking Rats

When a Pavlovian cue is presented separately from its associated reward, some animals will acquire a sign tracking (ST) response - approach and/or interaction with the cue - while others will acquire a goal tracking response - approach to the site of reward. We have previously shown that cue-evoked excitations in the nucleus accumbens (NAc) encode the vigor of both behaviors; in contrast, reward-related responses diverge over the course of training, possibly reflecting neurochemical differences between sign tracker and goal tracker individuals. However, a substantial subset of neurons in the NAc exhibit inhibitory, rather than excitatory, cue-evoked responses, and the evolution of their signaling during Pavlovian conditioning remains unknown. Using single-neuron recordings in behaving rats, we show that NAc neurons with cue-evoked inhibitions have distinct coding properties from neurons with cue-evoked excitations. Cue-evoked inhibitions become more numerous over the course of training and, like excitations, may encode the vigor of sign tracking and goal tracking behavior. However, the responses of cue-inhibited neurons do not evolve differently between sign tracker and goal tracker individuals. Moreover, cue-evoked inhibitions, unlike excitations, are insensitive to extinction of the cue-reward relationship. Finally, we show that cue-evoked excitations are greatly diminished by reward devaluation, while inhibitory cue responses are virtually unaffected. Overall, these findings converge with existing evidence that cue-excited neurons in NAc, but not cue-inhibited neurons, are profoundly sensitive to the same behavior variations that are often associated with changes in dopamine release.

Food reinforcement architecture: A framework for impulsive and compulsive overeating and food abuse

Few reward-based theories address key drivers of susceptibility to food cues and consumption beyond fullness. Decision-making and habit formation are governed by reinforcement-based learning processes that, when overstimulated, can drive unregulated hedonically motivated overeating. Here, a model food reinforcement architecture is proposed that uses fundamental concepts in reinforcement and decision-making to identify maladaptive eating habits that can lead to obesity. This model is unique in that it identifies metabolic drivers of reward and incorporates neuroscience, computational decision-making, and psychology to map overeating and obesity. Food reinforcement architecture identifies two paths to overeating: a propensity for hedonic targeting of food cues contributing to impulsive overeating and lack of satiation that contributes to compulsive overeating. A combination of those paths will result in a conscious and subconscious drive to overeat independent of negative consequences, leading to food abuse and/or obesity. Use of this model to identify aberrant reinforcement learning processes and decision-making systems that can serve as markers of overeating risk may provide an opportunity for early intervention in obesity.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outcome from those that are not sensitive to such changes. To address this question, we combined a Pavlovian learning task with outcome devaluation, eye-tracking and functional magnetic resonance imaging. We used computational modeling to identify brain regions involved in learning stimulus-reward associations and stimulus-stimulus associations, by testing for brain areas correlating with reward-prediction errors and state-prediction errors, respectively. We found that, contrary to theoretical predictions about reward prediction errors being exclusively model-free, voxels correlating with reward prediction errors in the ventral striatum and subgenual anterior cingulate cortex were sensitive to devaluation. On the other hand, brain areas correlating with state prediction errors were found to be devaluation insensitive. In a supplementary analysis, we distinguished brain regions encoding predictions about outcome taste identity from those involved in encoding predictions about its expected spatial location. A subset of regions involved in taste identity predictions were devaluation sensitive while those involved in encoding predictions about spatial location were devaluation insensitive. These findings provide insights into the role of multiple associative mechanisms in the brain in mediating Pavlovian conditioned behavior - illustrating how distinct neural pathways can in parallel produce both devaluation sensitive and devaluation insensitive behaviors.

Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Analysis of individual differences in neurofeedback training illuminates successful self-regulation of the dopaminergic midbrain

The dopaminergic midbrain is associated with reinforcement learning, motivation and decision-making - functions often disturbed in neuropsychiatric disorders. Previous research has shown that dopaminergic midbrain activity can be endogenously modulated via neurofeedback. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examine whether the activation of particular brain regions associates with successful regulation transfer when feedback is no longer available. Moreover, to elucidate mechanisms underlying effective self-regulation, we study the relation of successful transfer with learning (temporal difference coding) outside the midbrain during neurofeedback training and with individual reward sensitivity in a monetary incentive delay (MID) task. Fifty-nine participants underwent neurofeedback training either in standard (Study 1 N = 15, Study 2 N = 28) or control feedback group (Study 1, N = 16). We find that successful self-regulation is associated with prefrontal reward sensitivity in the MID task (N = 25), with a decreasing relation between prefrontal activity and midbrain learning signals during neurofeedback training and with increased activity within cognitive control areas during transfer. The association between midbrain self-regulation and prefrontal temporal difference and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings provide insights in the control of midbrain activity and may facilitate individually tailoring neurofeedback training.

A mathematical model of reward-mediated learning in drug addiction

Substances of abuse are known to activate and disrupt neuronal circuits in the brain reward system. We propose a simple and easily interpretable dynamical systems model to describe the neurobiology of drug addiction that incorporates the psychiatric concepts of reward prediction error, drug-induced incentive salience, and opponent process theory. Drug-induced dopamine releases activate a biphasic reward response with pleasurable, positive "a-processes" (euphoria, rush) followed by unpleasant, negative "b-processes" (cravings, withdrawal). Neuroadaptive processes triggered by successive intakes enhance the negative component of the reward response, which the user compensates for by increasing drug dose and/or intake frequency. This positive feedback between physiological changes and drug self-administration leads to habituation, tolerance, and, eventually, to full addiction. Our model gives rise to qualitatively different pathways to addiction that can represent a diverse set of user profiles (genetics, age) and drug potencies. We find that users who have, or neuroadaptively develop, a strong b-process response to drug consumption are most at risk for addiction. Finally, we include possible mechanisms to mitigate withdrawal symptoms, such as through the use of methadone or other auxiliary drugs used in detoxification.

Association of the OPRM1 A118G polymorphism and Pavlovian-to-instrumental transfer: Clinical relevance for alcohol dependence

BACKGROUND

Pavlovian-to-instrumental transfer (PIT) quantifies the extent to which a stimulus that has been associated with reward or punishment alters operant behaviour. In alcohol dependence (AD), the PIT effect serves as a paradigmatic model of cue-induced relapse. Preclinical studies have suggested a critical role of the opioid system in modulating Pavlovian-instrumental interactions. The A118G polymorphism of the OPRM1 gene affects opioid receptor availability and function. Furthermore, this polymorphism interacts with cue-induced approach behaviour and is a potential biomarker for pharmacological treatment response in AD. In this study, we tested whether the OPRM1 polymorphism is associated with the PIT effect and relapse in AD.

METHODS

Using a PIT task, we examined three independent samples: young healthy subjects (N = 161), detoxified alcohol-dependent patients (N = 186) and age-matched healthy controls (N = 105). We used data from a larger study designed to assess the role of learning mechanisms in the development and maintenance of AD. Subjects were genotyped for the A118G (rs1799971) polymorphism of the OPRM1 gene. Relapse was assessed after three months.

RESULTS

In all three samples, participants with the minor OPRM1 G-Allele (G+ carriers) showed increased expression of the PIT effect in the absence of learning differences. Relapse was not associated with the OPRM1 polymorphism. Instead, G+ carriers displaying increased PIT effects were particularly prone to relapse.

CONCLUSION

These results support a role for the opioid system in incentive salience motivation. Furthermore, they inform a mechanistic model of aberrant salience processing and are in line with the pharmacological potential of opioid receptor targets in the treatment of AD.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Advances in the computational understanding of mental illness

Computational psychiatry is a rapidly growing field attempting to translate advances in computational neuroscience and machine learning into improved outcomes for patients suffering from mental illness. It encompasses both data-driven and theory-driven efforts. Here, recent advances in theory-driven work are reviewed. We argue that the brain is a computational organ. As such, an understanding of the illnesses arising from it will require a computational framework. The review divides work up into three theoretical approaches that have deep mathematical connections: dynamical systems, Bayesian inference and reinforcement learning. We discuss both general and specific challenges for the field, and suggest ways forward.

Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic "computational fingerprint", will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Drug-Induced Conditioned Place Preference and Its Practical Use in Substance Use Disorder Research

The conditioned place preference (CPP) paradigm is a well-established model utilized to study the role of context associations in reward-related behaviors, including both natural rewards and drugs of abuse. In this review article, we discuss the basic history, various uses, and considerations that are tied to this technique. There are many potential takeaway implications of this model, including negative affective states, conditioned drug effects, memory, and motivation, which are all considered here. We also discuss the neurobiology of CPP including relevant brain regions, molecular signaling cascades, and neuromodulatory systems. We further examine some of our prior findings and how they integrate CPP with self-administration paradigms. Overall, by describing the fundamentals of CPP, findings from the past few decades, and implications of using CPP as a research paradigm, we have endeavored to support the case that the CPP method is specifically advantageous for studying the role of a form of Pavlovian learning that associates drug use with the surrounding environment.

Ketamine Blocks Morphine-Induced Conditioned Place Preference and Anxiety-Like Behaviors in Mice

Patients suffering from opioid use disorder often relapse during periods of abstinence, which is posited to be caused by negative affective states that drive motivated behaviors. Here, we explored whether conditioning mice with morphine in a conditioned place preference (CPP) training paradigm evoked anxiety-like behavior during morphine abstinence. To do this, mice were conditioned with morphine (10 mg/kg, i.p.) for 5 days. Twenty-four hours following conditioning, anxiety levels were tested by measuring time in the open arms of the elevated plus-maze. The next day, mice were placed in the three-compartment chamber to measure morphine-induced CPP. Our results show that following morphine conditioning, mice spent significantly less time in the open arm of the elevated plus-maze and expressed robust morphine CPP on CPP test day. Furthermore, we found that an acute treatment with (R,S)-ketamine (10 mg/kg, i.p.), a medication demonstrating promise for preventing anxiety-related phenotypes, 30 min before testing on post-conditioning day 1, increased time spent in the open arm of the elevated plus-maze in saline- and morphine-conditioned mice. Additionally, we found that the second injection of ketamine 30 min before CPP tests on post-conditioning day 2 prevented morphine-induced CPP, which lasted for up to 28 days post-conditioning. Furthermore, we found that conditioning mice with 10% (w/v) sucrose using an oral self-administration procedure did not evoke anxiety-like behavior, but elicited robust CPP, which was attenuated by ketamine treatment 30 min before CPP tests. Overall, our results suggest that the ketamine-induced block of morphine CPP may not be attributed solely to alleviating negative affective states, but potentially through impaired memory of morphine-context associations.

An Embodied Neurocomputational Framework for Organically Integrating Biopsychosocial Processes: An Application to the Role of Social Support in Health and Disease

OBJECTIVE

Two distinct perspectives-typically referred to as the biopsychosocial and biomedical models-currently guide clinical practice. Although the role of psychosocial factors in contributing to physical and mental health outcomes is widely recognized, the biomedical model remains dominant. This is due in part to (a) the largely nonmechanistic focus of biopsychosocial research and (b) the lack of specificity it currently offers in guiding clinicians to focus on social, psychological, and/or biological factors in individual cases. In this article, our objective is to provide an evidence-based and theoretically sophisticated mechanistic model capable of organically integrating biopsychosocial processes.

METHODS

To construct this model, we provide a narrative review of recent advances in embodied cognition and predictive processing within computational neuroscience, which offer mechanisms for understanding individual differences in social perceptions, visceral responses, health-related behaviors, and their interactions. We also review current evidence for bidirectional influences between social support and health as a detailed illustration of the novel conceptual resources offered by our model.

RESULTS

When integrated, these advances highlight multiple mechanistic causal pathways between psychosocial and biological variables.

CONCLUSIONS

By highlighting these pathways, the resulting model has important implications motivating a more psychologically sophisticated, person-specific approach to future research and clinical application in the biopsychosocial domain. It also highlights the potential for quantitative computational modeling and the design of novel interventions. Finally, it should aid in guiding future research in a manner capable of addressing the current criticisms/limitations of the biopsychosocial model and may therefore represent an important step in bridging the gap between it and the biomedical perspective.

Sign-tracking behavior is sensitive to outcome devaluation in a devaluation context-dependent manner: implications for analyzing habitual behavior

Motivationally attractive cues can draw in behavior in a phenomenon termed incentive salience. Incentive cue attraction is an important model for animal models of drug seeking and relapse. One question of interest is the extent to which the pursuit of motivationally attractive cues is related to the value of the paired outcome or can become unrelated and habitual. We studied this question using a sign-tracking (ST) paradigm in rats, in which a lever stimulus preceding food reward comes to elicit conditioned lever-interaction behavior. We asked whether reinforcer devaluation by means of conditioned taste aversion, a classic test of habitual behavior, can modify ST to incentive cues, and whether this depends upon the manner in which reinforcer devaluation takes place. In contrast to several recent reports, we conclude that ST is indeed sensitive to reinforcer devaluation. However, this effect depends critically upon the congruence between the context in which taste aversion is learned and the context in which it is tested. When the taste aversion successfully transfers to the testing context, outcome value strongly influences ST behavior, both when the outcome is withheld (in extinction) and when animals can learn from outcome feedback (reacquisition). When taste aversion does not transfer to the testing context, ST remains high. In total, the extent to which ST persists after outcome devaluation is closely related to the extent to which that outcome is truly devalued in the task context. We believe this effect of context on devaluation can reconcile contradictory findings about the flexibility/inflexibility of ST. We discuss this literature and relate our findings to the study of habits generally.

Increased Goal Tracking in Adolescent Rats Is Goal-Directed and Not Habit-Like

When a cue is paired with reward in a different location, some animals will approach the site of reward during the cue, a behavior called goal tracking, while other animals will approach and interact with the cue itself: a behavior called sign tracking. Sign tracking is thought to reflect a tendency to transfer incentive salience from the reward to the cue. Adolescence is a time of heightened sensitivity to rewards, including environmental cues that have been associated with rewards, which may account for increased impulsivity and vulnerability to drug abuse. Surprisingly, however, studies have shown that adolescents are actually less likely to interact with the cue (i.e., sign track) than adult animals. We reasoned that adolescents might show decreased sign tracking, accompanied by increased apparent goal tracking, because they tend to attribute incentive salience to a more reward-proximal "cue": the food magazine. On the other hand, adolescence is also a time of enhanced exploratory behavior, novelty-seeking, and behavioral flexibility. Therefore, adolescents might truly express more goal-directed reward-seeking and less inflexible habit-like approach to a reward-associated cue. Using a reward devaluation procedure to distinguish between these two hypotheses, we found that adolescents indeed exhibit more goal tracking, and less sign tracking, than a comparable group of adults. Moreover, adolescents' goal tracking behavior is highly sensitive to reward devaluation and therefore goal-directed and not habit-like.

Model-Free and Model-Based Influences in Addiction-Related Behaviors

BACKGROUND

Disruptions in the decision-making processes that guide action selection are a core feature of many psychiatric disorders, including addiction. Decision making is influenced by the goal-directed and habitual systems that can be computationally characterized using model-based and model-free reinforcement learning algorithms, respectively. Recent evidence suggests an imbalance in the influence of these reinforcement learning systems on behavior in individuals with substance dependence, but it is unknown whether these disruptions are a manifestation of chronic drug use and/or are a preexisting risk factor for addiction.

METHODS

We trained adult male rats on a multistage decision-making task to quantify model-free and model-based processes before and after self-administration of methamphetamine or saline.

RESULTS

Individual differences in model-free, but not model-based, learning prior to any drug use predicted subsequent methamphetamine self-administration; rats with lower model-free behavior took more methamphetamine than rats with higher model-free behavior. This relationship was selective to model-free updating following a rewarded, but not unrewarded, choice. Both model-free and model-based learning were reduced in rats following methamphetamine self-administration, which was due to a decrement in the ability of rats to use unrewarded outcomes appropriately. Moreover, the magnitude of drug-induced disruptions in model-free learning was not correlated with disruptions in model-based behavior, indicating that drug self-administration independently altered both reinforcement learning strategies.

CONCLUSIONS

These findings provide direct evidence that model-free and model-based learning mechanisms are involved in select aspects of addiction vulnerability and pathology, and they provide a unique behavioral platform for conducting systems-level analyses of decision making in preclinical models of mental illness.

Neural correlates of instrumental responding in the context of alcohol-related cues index disorder severity and relapse risk

The influence of Pavlovian conditioned stimuli on ongoing behavior may contribute to explaining how alcohol cues stimulate drug seeking and intake. Using a Pavlovian-instrumental transfer task, we investigated the effects of alcohol-related cues on approach behavior (i.e., instrumental response behavior) and its neural correlates, and related both to the relapse after detoxification in alcohol-dependent patients. Thirty-one recently detoxified alcohol-dependent patients and 24 healthy controls underwent instrumental training, where approach or non-approach towards initially neutral stimuli was reinforced by monetary incentives. Approach behavior was tested during extinction with either alcohol-related or neutral stimuli (as Pavlovian cues) presented in the background during functional magnetic resonance imaging (fMRI). Patients were subsequently followed up for 6 months. We observed that alcohol-related background stimuli inhibited the approach behavior in detoxified alcohol-dependent patients (t = - 3.86, p < .001), but not in healthy controls (t = - 0.92, p = .36). This behavioral inhibition was associated with neural activation in the nucleus accumbens (NAcc) (t₍₃₀₎ = 2.06, p < .05). Interestingly, both the effects were only present in subsequent abstainers, but not relapsers and in those with mild but not severe dependence. Our data show that alcohol-related cues can acquire inhibitory behavioral features typical of aversive stimuli despite being accompanied by a stronger NAcc activation, suggesting salience attribution. The fact that these findings are restricted to abstinence and milder illness suggests that they may be potential resilience factors.Clinical trial: LeAD study, http://www.lead-studie.de , NCT01679145.

A microfluidic platform for continuous monitoring of dopamine homeostasis in dopaminergic cells

Homeostasis of dopamine, a classical neurotransmitter, is a key indicator of neuronal health. Dysfunction in the regulation of dopamine is implicated in a long list of neurological disorders, including addiction, depression, and neurodegeneration. The existing methods used to evaluate dopamine homeostasis in vitro are inconvenient and do not allow for continuous non-destructive measurement. In response to this challenge, we introduce an integrated microfluidic system that combines dopaminergic cell culture and differentiation with electroanalytical measurements of extracellular dopamine in real-time at any point during an assay. We used the system to examine the behavior of differentiated SH-SY5Y cells upon exposure to four dopamine transporter ant/agonists (cocaine, ketamine, epigallocatechin gallate, and amphetamine) and study their pharmacokinetics. The IC₅₀ values of cocaine, ketamine, and epigallocatechin gallate were determined to be (average ± standard deviation) 3.7 ± 1.1 µM, 51.4 ± 17.9 µM, and 2.6 ± 0.8 µM, respectively. Furthermore, we used the new system to study amphetamine-mediated dopamine release to probe the related phenomena of dopamine transporter-mediated reverse-transport and dopamine release from vesicles. We propose that this platform, which is the first platform to simultaneously evaluate uptake and release, could be useful to screen for drugs and other agents that target dopaminergic neurons and the function of the dopamine transporter. More broadly, this platform should be adaptable for any application that could benefit from high-temporal resolution electroanalysis combined with multi-day cell culture using small numbers of cells.

Sign Tracking and Goal Tracking Are Characterized by Distinct Patterns of Nucleus Accumbens Activity

During Pavlovian conditioning, if a cue (e.g., lever extension) predicts reward delivery in a different location (e.g., a food magazine), some individuals will come to approach and interact with the cue, a behavior known as sign tracking (ST), and others will approach the site of reward, a behavior known as goal tracking (GT). In rats, the acquisition of ST versus GT behavior is associated with distinct profiles of dopamine release in the nucleus accumbens (NAc), but it is unknown whether it is associated with different patterns of accumbens neural activity. Therefore, we recorded from individual neurons in the NAc core during the acquisition, maintenance, and extinction of ST and GT behavior. Even though NAc dopamine is specifically important for the acquisition and expression of ST, we found that cue-evoked excitatory responses encode the vigor of both ST and GT behavior. In contrast, among sign trackers only, there was a prominent decrease in reward-related activity over the course of training, which may reflect the decreasing reward prediction error encoded by phasic dopamine. Finally, both behavior and cue-evoked activity were relatively resistant to extinction in sign trackers, as compared with goal trackers, although a subset of neurons in both groups retained their cue-evoked responses. Overall, the results point to the convergence of multiple forms of reward learning in the NAc.

Differential involvement of dopamine receptor subtypes in the acquisition of Pavlovian sign-tracking and goal-tracking responses

RATIONALE

Previous work has identified that different forms of Pavlovian conditioned approach, sign-tracking and goal-tracking, are governed by distinct neurochemical mechanisms when compared in animals predisposed to learning one form vs. the other.

OBJECTIVES

The present study aimed to investigate whether these are also neurochemically distinct processes in a population of animals capable of developing either response when this is manipulated via the use of distinct conditioned stimuli (CS).

METHODS

Rats were trained on one of two Pavlovian conditioning procedures in which the CS was either a lever, which elicits sign-tracking, or an auditory click, which elicits goal-tracking. The differential involvement of dopamine D1- and D2-receptors (D1R; D2R) in the acquisition of approach types was investigated via systemic administration of antagonists selective to one or both receptor subtypes during Pavlovian training.

RESULTS

Results indicate that dopaminergic signalling is important for the acquisition of both sign-tracking and goal-tracking responses. However, whilst development of sign-tracking to a lever depends on activity at both D1R and D2R, development of goal-tracking in response to a click was shown to depend only on activity at D1R.

CONCLUSIONS

We suggest that the importance of D1R activity in both sign- and goal-tracking acquisition reflects a general role in learning Pavlovian associations, which aligns with data implicating dopamine in prediction error processes. In contrast, the selective involvement of D2R activity in sign-tracking acquisition may reflect its importance in motivational processes such as incentive salience attribution.

Extinction to amphetamine-associated context in female rats is dependent upon conditioned orienting

RATIONALE

Females are considered more susceptible to the reinforcing effects of drugs and subsequently at increased risk for drug abuse and relapse after treatment. Estrogen is known to facilitate drug effects in females. However, other factors which contribute to the incidence of drug abuse are important to identify in order to recognize early risk factors and develop effective prevention and treatment schemes. Cue-directed behavior (aka sign tracking) has been implicated as a behavioral phenotype which identifies populations susceptible to drug abuse, partly due to its association with impulsivity and heightened dopamine function.

OBJECTIVES AND METHODS

In this study, we investigate the viability of conditioned orienting (a form of cue-directed behavior) as a potential phenotype which predicts drug proclivity in female rats. In addition, we examine any influence endogenous female hormones across the estrous cycle may have on conditioned orienting and drug proclivity.

RESULTS AND CONCLUSIONS

Utilizing an amphetamine-conditioned place preference task, results suggest that the orienting phenotype is an effective predictor of drug proclivity in females. Rats exhibiting enhanced orienting behavior show more robust preference for an amphetamine-associated context and are more resistant to extinction of this preference than nonorienting counterparts. Furthermore, both conditioned orienting behavior and conditioned place preference are minimally influenced by the estrous cycle.

Sign-tracking is an expectancy-mediated behavior that relies on prediction error mechanisms

When discrete localizable stimuli are used during appetitive Pavlovian conditioning, "sign-tracking" and "goal-tracking" responses emerge. Sign-tracking is observed when conditioned responding is directed toward the CS, whereas goal-tracking manifests as responding directed to the site of expected reward delivery. These behaviors seem to rely on distinct, though overlapping neural circuitries, and, possibly, distinct psychological processes as well, and are thought to be related to addiction vulnerability. One currently popular view is that sign-tracking reflects an incentive motivational process, whereas goal-tracking reflects the influence of more top-down cognitive processes. To test these ideas, we used illness-induced outcome-devaluation and Kamin blocking procedures to determine whether these behaviors rely on similar or distinct underlying associative mechanisms. In Experiments 1 and 2 we showed that outcome-devaluation reduced sign-tracking responses, demonstrating that sign-tracking is controlled by reward expectancies. We also observed that post-CS goal-tracking in these animals is also devaluation sensitive. To test whether these two types of behaviors rely on similar or different prediction error mechanisms, we next tested whether Kamin blocking effects could be observed across these two classes of behaviors. In Experiment 3 we asked if sign-tracking to a lever CS could block the development of goal-tracking to a tone CS; whereas in Experiment 4, we examined whether goal-tracking to a tone CS could block sign-tracking to a lever CS. In both experiments blocking effects were observed suggesting that both sign- and goal-tracking emerge via a common prediction error mechanism. Collectively, the studies reported here suggest that the psychological mechanisms mediating sign- and goal-tracking are more similar than is commonly acknowledged.

Drunk decisions: Alcohol shifts choice from habitual towards goal-directed control in adolescent intermediate-risk drinkers

BACKGROUND

Studies in humans and animals suggest a shift from goal-directed to habitual decision-making in addiction. We therefore tested whether acute alcohol administration reduces goal-directed and promotes habitual decision-making, and whether these effects are moderated by self-reported drinking problems.

METHODS

Fifty-three socially drinking males completed the two-step task in a randomised crossover design while receiving an intravenous infusion of ethanol (blood alcohol level=80 mg%), or placebo. To minimise potential bias by long-standing heavy drinking and subsequent neuropsychological impairment, we tested 18- to 19-year-old adolescents.

RESULTS

Alcohol administration consistently reduced habitual, model-free decisions, while its effects on goal-directed, model-based behaviour varied as a function of drinking problems measured with the Alcohol Use Disorders Identification Test. While adolescents with low risk for drinking problems (scoring <8) exhibited an alcohol-induced numerical reduction in goal-directed choices, intermediate-risk drinkers showed a shift away from habitual towards goal-directed decision-making, such that alcohol possibly even improved their performance.

CONCLUSIONS

We assume that alcohol disrupted basic cognitive functions underlying habitual and goal-directed decisions in low-risk drinkers, thereby enhancing hasty choices. Further, we speculate that intermediate-risk drinkers benefited from alcohol as a negative reinforcer that reduced unpleasant emotional states, possibly displaying a novel risk factor for drinking in adolescence.

Disconnection of basolateral amygdala and insular cortex disrupts conditioned approach in Pavlovian lever autoshaping

Previously established individual differences in appetitive approach and devaluation sensitivity observed in goal- and sign-trackers may be attributed to differences in the acquisition, modification, or use of associative information in basolateral amygdala (BLA) pathways. Here, we sought to determine the extent to which communication of associative information between BLA and anterior portions of insular cortex (IC) supports ongoing Pavlovian conditioned approach behaviors in sign- and goal-tracking rats, in the absence of manipulations to outcome value. We hypothesized that the BLA mediates goal-, but not sign- tracking approach through interactions with the IC, a brain region involved in supporting flexible behavior. We first trained rats in Pavlovian lever autoshaping to determine their sign- or goal-tracking tendency. During alternating test sessions, we gave unilateral intracranial injections of vehicle or a cocktail of gamma-aminobutyric acid (GABA) receptor agonists, baclofen and muscimol, unilaterally into the BLA and contralaterally or ipsilaterally into the IC prior to reinforced lever autoshaping sessions. Consistent with our hypothesis we found that contralateral inactivation of BLA and IC increased the latency to approach the food cup and decreased the number of food cup contacts in goal-trackers. While contralateral inactivation of BLA and IC did not affect the total number of lever contacts in sign-trackers, this manipulation increased the latency to approach the lever. Ipsilateral inactivation of BLA and IC did not impact approach behaviors in Pavlovian lever autoshaping. These findings, contrary to our hypothesis, suggest that communication between BLA and IC maintains a representation of initially learned appetitive associations that commonly support the initiation of Pavlovian conditioned approach behavior regardless of whether it is directed at the cue or the location of reward delivery.

No association of goal-directed and habitual control with alcohol consumption in young adults

Alcohol dependence is a mental disorder that has been associated with an imbalance in behavioral control favoring model-free habitual over model-based goal-directed strategies. It is as yet unknown, however, whether such an imbalance reflects a predisposing vulnerability or results as a consequence of repeated and/or excessive alcohol exposure. We, therefore, examined the association of alcohol consumption with model-based goal-directed and model-free habitual control in 188 18-year-old social drinkers in a two-step sequential decision-making task while undergoing functional magnetic resonance imaging before prolonged alcohol misuse could have led to severe neurobiological adaptations. Behaviorally, participants showed a mixture of model-free and model-based decision-making as observed previously. Measures of impulsivity were positively related to alcohol consumption. In contrast, neither model-free nor model-based decision weights nor the trade-off between them were associated with alcohol consumption. There were also no significant associations between alcohol consumption and neural correlates of model-free or model-based decision quantities in either ventral striatum or ventromedial prefrontal cortex. Exploratory whole-brain functional magnetic resonance imaging analyses with a lenient threshold revealed early onset of drinking to be associated with an enhanced representation of model-free reward prediction errors in the posterior putamen. These results suggest that an imbalance between model-based goal-directed and model-free habitual control might rather not be a trait marker of alcohol intake per se.

Alcohol devaluation has dissociable effects on distinct components of alcohol behaviour

RATIONALE

Substance-related behaviour is often viewed as an appetitive behaviour, motivated by the reinforcing effects of the drug. However, there are various indices of substance motivation (e.g. attentional bias, behavioural economic demand, craving) and it is unclear how these are related or whether they play an important role in all types of substance-related behaviour.

OBJECTIVES

(1) To determine the effect of alcohol devaluation on several indices of alcohol motivation and goal-directed and cue-elicited alcohol behaviour. (2) To investigate which components of motivation mediate any effect of devaluation on behaviour.

METHODS

Sixty-two social drinkers gave baseline measures of alcohol craving, behavioural economic demand and choice for alcohol vs. soft drink. Participants tasted alcohol which was either unadulterated (control) or adulterated with a bitter solution (devaluation) before craving and demand were measured again. Alcohol choice was assessed in several phases: extinction (evaluating goal-directed behaviour), in the presence of drink cues (Pavlovian-to-instrumental transfer (PIT, cue-elicited behaviour)), and reacquisition. Attentional bias (AB) was measured by tracking eye movements towards the drink cues during novel PIT trials where both cues were presented. Finally, consumption was evaluated in a taste test.

RESULTS

Alcohol devaluation reduced alcohol-related demand, AB, alcohol choice in all phases, and consumption. Alcohol cues presented during PIT increased alcohol choice above baseline irrespective of devaluation. AB and demand for alcohol fully mediated the effect of devaluation on alcohol choice during extinction, AB fully mediated the effect on cue-elicited (specific PIT) alcohol choice and alcohol consumption.

CONCLUSIONS

Alcohol behaviour in social drinkers is largely sensitive to devaluation, i.e. governed by current motivational value of the drug (suggesting goal-directed behaviour). However, a dissociable form of stimulus control can also drive alcohol-seeking independently of drug value (specific PIT). Mediation analyses suggests that AB may play a paradoxical role in both forms of alcohol seeking and consumption.

The Roles of Phasic and Tonic Dopamine in Tic Learning and Expression

Tourette syndrome (TS) prominently involves dopaminergic disturbances, but the precise nature of those disturbances has remained elusive. A substantial body of empirical work and recent computational models have characterized the specific roles of phasic and tonic dopamine (DA) in action learning and selection, respectively. Using insights from this work and models, we suggest that TS involves increases in both phasic and tonic DA, which produce increased propensities for tic learning and expression, respectively. We review the evidence from reinforcement-learning and habit-learning studies in TS, which supports the idea that TS involves increased phasic DA responses; we also review the evidence that tics engage the habit-learning circuitry. On the basis of these findings, we suggest that tics are exaggerated, maladaptive, and persistent motor habits reinforced by aberrant, increased phasic DA responses. Increased tonic DA amplifies the tendency to execute learned tics and also provides a fertile ground of motor hyperactivity for tic learning. We review evidence suggesting that antipsychotics may counter both the increased propensity for tic expression, by increasing excitability in the indirect pathway, and the increased propensity for tic learning, by shifting plasticity in the indirect pathway toward long-term potentiation (and possibly also through more complex mechanisms). Finally, we review evidence suggesting that low doses of DA agonists that effectively treat TS decrease both phasic and tonic DA, thereby also reducing the propensity for both tic learning and tic expression, respectively.

Computational approaches to schizophrenia: A perspective on negative symptoms

Schizophrenia is a heterogeneous spectrum disorder often associated with detrimental negative symptoms. In recent years, computational approaches to psychiatry have attracted growing attention. Negative symptoms have shown some overlap with general cognitive impairments and were also linked to impaired motivational processing in brain circuits implementing reward prediction. In this review, we outline how computational approaches may help to provide a better understanding of negative symptoms in terms of the potentially underlying behavioural and biological mechanisms. First, we describe the idea that negative symptoms could arise from a failure to represent reward expectations to enable flexible behavioural adaptation. It has been proposed that these impairments arise from a failure to use prediction errors to update expectations. Important previous studies focused on processing of so-called model-free prediction errors where learning is determined by past rewards only. However, learning and decision-making arise from multiple cognitive mechanisms functioning simultaneously, and dissecting them via well-designed tasks in conjunction with computational modelling is a promising avenue. Second, we move on to a proof-of-concept example on how generative models of functional imaging data from a cognitive task enable the identification of subgroups of patients mapping on different levels of negative symptoms. Combining the latter approach with behavioural studies regarding learning and decision-making may allow the identification of key behavioural and biological parameters distinctive for different dimensions of negative symptoms versus a general cognitive impairment. We conclude with an outlook on how this computational framework could, at some point, enrich future clinical studies.

Associative learning contributes to the increased water intake observed after daily injections of angiotensin II

Daily injections of angiotensin II (AngII) cause a progressive increase of water intake that resembles a classically ascribed non-associative sensitization. Consistent with the presumption that the observed increase in intake was sensitization, we hypothesized that it resulted from a pharmacological interaction between AngII and its receptor. To test this hypothesis, and remove the influence of drinking itself, we implemented a delay in water access after injection of AngII (icv) on four consecutive 'induction days,' and then measured intake on the next day ('test day') when rats were allowed to drink immediately after AngII. The delay in water access effectively reduced water intake on the four induction days, and rats with longer delays in access drank less on the test day than did rats allowed to drink immediately after AngII on the induction days. Additional experiments ruled out a role for a conditioned drinking response to the injection alone, and demonstrated a lack of conditioned appetition after pairing injections of AngII with water given by intragastric catheter. Taken together, these findings suggest that the increased drinking observed after daily injections of AngII is a conditioned appetition after repeated pairings of AngII and water intake. We further conclude that repeated stimulation of the AngII receptor alone is not sufficient to drive appetition.

Neurobiological Basis of Individual Variation in Stimulus-Reward Learning

Cues in the environment can guide behavior in adaptive ways, leading one towards valuable resources such as food, water, or a potential mate. However, cues in the environment may also serve as powerful motivators that lead to maladaptive patterns of behavior, such as addiction. Importantly, and central to this article, there is considerable individual variation in the extent to which reward cues gain motivational control over behavior. Here we describe an animal model that captures this individual variation, allowing us to better understand the psychological and neurobiological processes that contribute to cue-evoked behaviors. When a discrete cue is paired with a food reward in a Pavlovian manner it acquires greater control over motivated behavior in some rats ("sign-trackers, STs) than in others ("goal-trackers", GTs). We review studies that have exploited this animal model to parse the neurobiological mechanisms involved in learning associations between stimuli vs. those involved in attributing incentive salience to those same stimuli. The latter seems to be dependent on dopamine and subcortical circuits, whereas the former may engage more cortical "top-down" mechanisms.

Subanesthetic ketamine decreases the incentive-motivational value of reward-related cues

The attribution of incentive-motivational value to reward-related cues contributes to cue-induced craving and relapse in addicted patients. Recently, it was demonstrated that subanesthetic ketamine increases motivation to quit and decreases cue-induced craving in cocaine-dependent individuals. Although the underlying mechanism of this effect is currently unknown, one possibility is that subanesthetic ketamine decreases the incentive-motivational value of reward-related cues. In the present study, we used a Pavlovian conditioned approach procedure to identify sign-trackers, rats that attribute incentive-motivational value to reward-related cues, and goal-trackers, rats that assign only predictive value to reward-related cues. This model is of interest because sign-trackers are more vulnerable to cue-induced reinstatement of drug-seeking behavior and will persist in this drug-seeking behavior despite adverse consequences. We tested the effect of subanesthetic ketamine on the expression of Pavlovian conditioned approach behavior and the conditioned reinforcing properties of a reward-related cue in sign- and goal-trackers. We found that subanesthetic ketamine decreased sign-tracking and increased goal-tracking behavior in sign-trackers, though it had no effect on conditioned reinforcement. These results suggest that subanesthetic ketamine may be a promising pharmacotherapy for addiction that acts by decreasing the incentive-motivational value of reward-related cues.

Neurobiology of opioid dependence in creating addiction vulnerability

Opioid drugs are potent modulators of many physiological and psychological processes. When given acutely, they can elicit the signature responses of euphoria and analgesia that societies have coveted for centuries. Repeated, or chronic, use of opioids induces adaptive or allostatic changes that modify neuronal circuitry and create an altered normality — the “drug-dependent” state. This state, at least that exhibited by those maintained continuously on long-acting opioid drugs such as methadone or buprenorphine, is generally indistinguishable from the drug-naïve state for most overt behaviors. The consequences of the allostatic changes (cellular, circuit, and system adaptations) that accompany the drug-dependent state are revealed during drug withdrawal. Drug cessation triggers a temporally orchestrated allostatic re-establishment of neuronal systems, which is manifested as opposing physiological and psychological effects to those exhibited by acute drug intoxication. Some withdrawal symptoms, such as physical symptoms (sweating, shaking, and diarrhea) resolve within days, whilst others, such as dysphoria, insomnia, and anxiety, can linger for months, and some adaptations, such as learned associations, may be established for life. We will briefly discuss the cellular mechanisms and neural circuitry that contribute to the opioid drug-dependent state, inferring an emerging role for neuroinflammation. We will argue that opioid addictive behaviors result from a learned relationship between opioids and relief from an existing or withdrawal-induced anxiogenic and/or dysphoric state. Furthermore, a future stressful life event can recall the memory that opioid drugs alleviate negative affect (despair, sadness, and anxiety) and thereby precipitate craving, resulting in relapse. A learned association of relief of aversive states would fuel drug craving in vulnerable people living in an increasingly stressful society. We suggest that this route to addiction is contributive to the current opioid epidemic in the USA.

Computational psychiatry as a bridge from neuroscience to clinical applications

Translating advances in neuroscience into benefits for patients with mental illness presents enormous challenges because it involves both the most complex organ, the brain, and its interaction with a similarly complex environment. Dealing with such complexities demands powerful techniques. Computational psychiatry combines multiple levels and types of computation with multiple types of data in an effort to improve understanding, prediction and treatment of mental illness. Computational psychiatry, broadly defined, encompasses two complementary approaches: data driven and theory driven. Data-driven approaches apply machine-learning methods to high-dimensional data to improve classification of disease, predict treatment outcomes or improve treatment selection. These approaches are generally agnostic as to the underlying mechanisms. Theory-driven approaches, in contrast, use models that instantiate prior knowledge of, or explicit hypotheses about, such mechanisms, possibly at multiple levels of analysis and abstraction. We review recent advances in both approaches, with an emphasis on clinical applications, and highlight the utility of combining them.

Toward isolating the role of dopamine in the acquisition of incentive salience attribution

Stimulus-reward learning has been heavily linked to the reward-prediction error learning hypothesis and dopaminergic function. However, some evidence suggests dopaminergic function may not strictly underlie reward-prediction error learning, but may be specific to incentive salience attribution. Utilizing a Pavlovian conditioned approach procedure consisting of two stimuli that were equally reward-predictive (both undergoing reward-prediction error learning) but functionally distinct in regard to incentive salience (levers that elicited sign-tracking and tones that elicited goal-tracking), we tested the differential role of D1 and D2 dopamine receptors and nucleus accumbens dopamine in the acquisition of sign- and goal-tracking behavior and their associated conditioned reinforcing value within individuals. Overall, the results revealed that both D1 and D2 inhibition disrupted performance of sign- and goal-tracking. However, D1 inhibition specifically prevented the acquisition of sign-tracking to a lever, instead promoting goal-tracking and decreasing its conditioned reinforcing value, while neither D1 nor D2 signaling was required for goal-tracking in response to a tone. Likewise, nucleus accumbens dopaminergic lesions disrupted acquisition of sign-tracking to a lever, while leaving goal-tracking in response to a tone unaffected. Collectively, these results are the first evidence of an intraindividual dissociation of dopaminergic function in incentive salience attribution from reward-prediction error learning, indicating that incentive salience, reward-prediction error, and their associated dopaminergic signaling exist within individuals and are stimulus-specific. Thus, individual differences in incentive salience attribution may be reflective of a differential balance in dopaminergic function that may bias toward the attribution of incentive salience, relative to reward-prediction error learning only.

An omission procedure reorganizes the microstructure of sign-tracking while preserving incentive salience

Appetitive sign-tracking, in which reward-paired cues elicit approach that can result in cue interaction, demonstrates how cues acquire motivational value. For example, rats will approach and subsequently interact with a lever insertion cue that signals food delivery upon its retraction. However, lever deflections are rapidly reduced once rats are trained on an omission schedule in which lever interactions cancel food delivery. Here we evaluated the change in sign-tracking response topography in rats exposed to such an omission procedure. Lever deflections dropped precipitously when they canceled reward. However, rats that were on an omission schedule continued to approach, sniff, and contact the lever without pressing it, and did so at comparable rates to rats that were not under an omission schedule. Thus, sign-tracking was maintained, albeit in a different manner, following omission. Such findings show that the motivational attraction to reward cues can be expressed with remarkable persistence and flexibility.

Don't Think, Just Feel the Music: Individuals with Strong Pavlovian-to-Instrumental Transfer Effects Rely Less on Model-based Reinforcement Learning

Behavioral choice can be characterized along two axes. One axis distinguishes reflexive, model-free systems that slowly accumulate values through experience and a model-based system that uses knowledge to reason prospectively. The second axis distinguishes Pavlovian valuation of stimuli from instrumental valuation of actions or stimulus-action pairs. This results in four values and many possible interactions between them, with important consequences for accounts of individual variation. We here explored whether individual variation along one axis was related to individual variation along the other. Specifically, we asked whether individuals' balance between model-based and model-free learning was related to their tendency to show Pavlovian interferences with instrumental decisions. In two independent samples with a total of 243 participants, Pavlovian-instrumental transfer effects were negatively correlated with the strength of model-based reasoning in a two-step task. This suggests a potential common underlying substrate predisposing individuals to both have strong Pavlovian interference and be less model-based and provides a framework within which to interpret the observation of both effects in addiction.

The effects of methylphenidate on cerebral responses to conflict anticipation and unsigned prediction error in a stop-signal task

To adapt flexibly to a rapidly changing environment, humans must anticipate conflict and respond to surprising, unexpected events. To this end, the brain estimates upcoming conflict on the basis of prior experience and computes unsigned prediction error (UPE). Although much work implicates catecholamines in cognitive control, little is known about how pharmacological manipulation of catecholamines affects the neural processes underlying conflict anticipation and UPE computation. We addressed this issue by imaging 24 healthy young adults who received a 45 mg oral dose of methylphenidate (MPH) and 62 matched controls who did not receive MPH prior to performing the stop-signal task. We used a Bayesian Dynamic Belief Model to make trial-by-trial estimates of conflict and UPE during task performance. Replicating previous research, the control group showed anticipation-related activation in the presupplementary motor area and deactivation in the ventromedial prefrontal cortex and parahippocampal gyrus, as well as UPE-related activations in the dorsal anterior cingulate, insula, and inferior parietal lobule. In group comparison, MPH increased anticipation activity in the bilateral caudate head and decreased UPE activity in each of the aforementioned regions. These findings highlight distinct effects of catecholamines on the neural mechanisms underlying conflict anticipation and UPE, signals critical to learning and adaptive behavior.