Initial uncertainty in Pavlovian reward prediction persistently elevates incentive salience and extends sign-tracking to normally unattractive cues

Variation in the effectiveness of reinforcement and nonreinforcement in generating different conditioned behaviors

Rat autoshaping procedures generate two readily measurable conditioned responses: During lever presentations that have previously signaled food, rats approach the food well (called goal-tracking) and interact with the lever itself (called sign-tracking). We investigated how reinforced and nonreinforced trials affect the overall and temporal distributions of these two responses across 10-second lever presentations. In two experiments, reinforced trials generated more goal-tracking than sign-tracking, and nonreinforced trials resulted in a larger reduction in goal-tracking than sign-tracking. The effect of reinforced trials was evident as an increase in goal-tracking and reduction in sign-tracking across the duration of the lever presentations, and nonreinforced trials resulted in this pattern transiently reversing and then becoming less evident with further training. These dissociations are consistent with a recent elaboration of the Rescorla-Wagner model, HeiDI (Honey, R.C., Dwyer, D.M., & Iliescu, A.F. (2020a). HeiDI: A model for Pavlovian learning and performance with reciprocal associations. Psychological Review, 127, 829-852.), a model in which responses related to the nature of the unconditioned stimulus (e.g., goal-tracking) have a different origin than those related to the nature of the conditioned stimulus (e.g., sign-tracking).

Decomposition of Reinforcement Learning Deficits in Disordered Gambling via Drift Diffusion Modeling and Functional Magnetic Resonance Imaging

Gambling disorder is associated with deficits in reward-based learning, but the underlying computational mechanisms are still poorly understood. Here, we examined this issue using a stationary reinforcement learning task in combination with computational modeling and functional resonance imaging (fMRI) in individuals that regular participate in gambling (n = 23, seven fulfilled one to three DSM 5 criteria for gambling disorder, sixteen fulfilled four or more) and matched controls (n = 23). As predicted, the gambling group exhibited substantially reduced accuracy, whereas overall response times (RTs) were not reliably different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance deficit. In both groups, an RLDDM in which both non-decision time and decision threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter and model recovery, and posterior predictive checks revealed that, in both groups, the model accurately reproduced the evolution of accuracies and RTs over time. Modeling revealed that, compared to controls, the learning impairment in the gambling group was linked to a more rapid reduction in decision thresholds over time, and a reduced impact of value-differences on the drift rate. The gambling group also showed shorter non-decision times. FMRI analyses replicated effects of prediction error coding in the ventral striatum and value coding in the ventro-medial prefrontal cortex, but there was no credible evidence for group differences in these effects. Taken together, our findings show that reinforcement learning impairments in disordered gambling are linked to both maladaptive decision threshold adjustments and a reduced consideration of option values in the choice process.

Evidence Accumulation and Neural Correlates of Uncertainty in Obsessive-Compulsive Disorder

BACKGROUND

Decision making is frequently associated with risk taking under uncertainty. Elevated intolerance of uncertainty is suggested to be a critical feature of obsessive-compulsive disorder (OCD). However, impairments of latent constructs of uncertainty processing and its neural correlates remain unclear in OCD.

METHODS

In 83 participants (24 OCD patients treated with capsulotomy, 28 OCD control participants, and 31 healthy control participants), we performed magnetic resonance imaging using a card gambling task in which participants made decisions whether to bet or not that the next card would be larger than the current one. A hierarchical drift diffusion model was used to dissociate speed and amount of evidence accumulated before a decisional threshold (i.e., betting or no betting) was reached.

RESULTS

High uncertainty was characterized by a smaller amount of evidence accumulation (lower thresholds), thus dissociating uncertainty from conflict tasks and highlighting the specificity of this task to test value-based uncertainty. OCD patients exhibited greater caution with poor performance and greater evidence accumulation overall along with slower speed of accumulation, particularly under low uncertainty. Bilateral dorsal anterior cingulate and anterior insula distinguished high- and low-uncertainty decision processes in healthy control participants but not in the OCD groups, indicating impairments in anticipation of differences in outcome variance and salience network activity. There were no behavioral or imaging differences relating to capsulotomy despite improvements in OCD symptoms.

CONCLUSIONS

Our findings highlight greater impairments particularly in more certain trials in the OCD groups along with impaired neural differentiation of high and low uncertainty and suggest uncertainty processing as a trait cognitive endophenotype rather than a state-specific factor.

Unpredictable, intermittent access to sucrose or water promotes increased reward pursuit in rats

Reward uncertainty can sensitize reward pathways, promoting increased reward-seeking and -taking behaviours. This is relevant to human conditions such as pathological gambling, eating disorders and drug addiction. In the context of addiction, preclinical self-administration procedures have been developed to model the intermittency of human drug use. These intermittent-access (IntA) procedures involve intermittent but predictable access to drug during self-administration sessions. However, human drug use typically involves intermittent and unpredictable drug access. We introduce a new procedure modeling unpredictable, intermittent access (UIntA) to a reinforcer, and we compare it to procedures that provide predictable reinforcer availability; continuous (ContA) or intermittent (IntA) access. Female rats self-administered water or liquid sucrose in daily hour-long sessions. IntA and ContA rats had access to a fixed volume of water or sucrose (0.1 ml), under a fixed ratio 3 schedule of reinforcement. IntA rats had predictable 5-min reinforcer ON and 25-min reinforcer OFF periods. ContA rats had 60 min of reinforcer access during each session. For UIntA rats, variation in the length of ON and OFF periods (1, 5 or 9 min/period), response requirement (variable ratio 3 schedule of reinforcement), reinforcer probability (50%) and quantity (0, 0.1 or 0.2 ml) introduced reward uncertainty. Following 14 daily self-administration sessions, UIntA rats showed the highest levels of responding for water or sucrose under progressive ratio conditions, responding under extinction conditions, and cue-induced reinstatement of sucrose seeking. Thus, unpredictable, intermittent reward access promotes increased reward pursuit. This has implications for modeling addiction and other disorders of increased reward seeking.

Quantifying the instrumental and noninstrumental underpinnings of Pavlovian responding with the Price equation

The Price equation is a mathematical expression of selectionist and non-selectionist pressures on biological, cultural, and behavioral change. We use it here to specify instrumental and noninstrumental behaviors as they arise within the context of the Pavlovian autoshaping procedure, for rats trained under reward certainty and reward uncertainty. The point of departure for this endeavor is that some portion of autoshaped behavior referred to as goal-tracking appears instrumental-a function of resource attainment (the individual approaches the location where the unconditioned stimulus is to be delivered). By contrast, some other portion of autoshaped behavior referred to as sign-tracking is noninstrumental-irrelevant to making contact with the to-be-delivered unconditioned stimulus. A Price equation model is proposed that unifies our understanding of Pavlovian autoshaping behavior by isolating operant and respondent influences on goal-tracking (instrumental) and sign-tracking (noninstrumental) behavior.

An operant ethanol self-administration paradigm that discriminates between appetitive and consummatory behaviors reveals distinct behavioral phenotypes in commonly used rat strains

Alcohol use disorder (AUD) constitutes a major burden to global health. Recently, the translational success of animal models of AUD has come under increased scrutiny. Efforts to refine models to gain a more precise understanding of the neurobiology of addiction are warranted. Appetitive responding for ethanol (seeking) and its consumption (taking) are governed by distinct neurobiological mechanisms. However, consumption is often inferred from appetitive responding in operant ethanol self-administration paradigms, preventing identification of distinct experimental effects on seeking and taking. In the present study, male Long-Evans, Wistar, and Sprague-Dawley rats were trained to lever press for ethanol using a lickometer-equipped system that precisely measures both appetitive and consummatory behavior. Three distinct operant phenotypes emerged during training: 1) Drinkers, who lever press and consume ethanol; 2) Responders, who lever press but consume little to no ethanol; and 3) Non-responders, who do not lever press. While the prevalence of each phenotype differed across strains, appetitive and consummatory behavior was similar across strains within each phenotype. Appetitive and consummatory behaviors were significantly correlated in Drinkers, but not Responders. Analysis of drinking microstructure showed that greater consumption in Drinkers relative to Responders is due to increased incentive for ethanol rather than increased palatability. Importantly, withdrawal from chronic ethanol exposure resulted in a significant increase in appetitive responding in both Drinkers and Responders, but only Drinkers exhibited a concomitant increase in ethanol consumption. Together, these data reveal important strain differences in appetitive and consummatory responding for ethanol and uncover the presence of distinct operant phenotypes.

Effort-motivated behavior resolves paradoxes in appetitive conditioning

Motivated behavior has long been studied by psychologists, ethologists, and neuroscientists. To date, many scientists agree with the view that cue and reward attraction is the product of a dopamine-dependent unconscious process called incentive salience or "wanting". This process allows the influence of multiple factors such as hunger and odors on motivational attraction. In some cases, however, the resulting motivated behavior differs from what the incentive salience hypothesis would predict. I argue that seeking behavior under reward uncertainty illustrates this situation: Organisms do not just "want" (appetite-based attraction) cues that are inconsistent or associated with reward occasionally, they "hope" that those cues will consistently predict reward procurement in the ongoing trial. Said otherwise, they become motivated to invest time and energy to find consistent cue-reward associations despite no guarantee of success (effort-based attraction). A multi-test comparison of performance between individuals trained under uncertainty and certainty reveals behavioral paradoxes suggesting that the concept of incentive salience cannot fully account for responding to inconsistent cues. A mathematical model explains how appetite-based and effort-based attractions might combine their effects.

Individual variation in the vigor and form of Pavlovian conditioned responses: Analysis of a model system

Pavlovian conditioning results in individual variation in the vigor and form of acquired behaviors. Here, we describe a general-process model of associative learning (HeiDI; How excitation and inhibition determine ideo-motion) that provides an analysis for such variation together with a range of other important group-level phenomena. The model takes as its starting point the idea that pairings of a conditioned stimulus (CS) and an unconditioned stimulus (US) result in the formation of reciprocal associations between their central representations. The asymptotic values of these associations and the rate at which these are reached are held to be influenced by the perceived salience of the CS (αCS) and US (βUS). Importantly, whether this associative knowledge is exhibited in behavior that reflects the properties of the CS (e.g., sign-tracking) or US (e.g., goal-tracking) is also influenced by the relative values of αCS and βUS. In this way, HeiDI provides an analysis for both quantitative and qualitative individual differences generated by Pavlovian conditioning procedures.

Effects of partial reinforcement on autoshaping in inbred Roman high- and low-avoidance rats

Individuals trained under partial reinforcement (PR) typically show a greater resistance to extinction than individuals exposed to continuous reinforcement (CR). This phenomenon is referred to as the PR extinction effect (PREE) and is interpreted as a consequence of uncertainty-induced frustration counterconditioning. In this study, we assessed the effects of PR and CR in acquisition and extinction in two strains of rats, the inbred Roman high- and low-avoidance (RHA and RLA, respectively) rats. These two strains mainly differ in the expression of anxiety, the RLA rats showing more anxiety-related behaviors (hence, more sensitive to frustration) than the RHA rats. At a neurobiological level, mild stress is known to elevate corticosterone in RLA rats and dopamine in RHA rats. We tested four groups of rats (RHA/CR, RHA/PR, RLA/CR, and RLA/PR) in two successive acquisition-extinction phases to try to consolidate the behavioral effects. Animals received training in a Pavlovian autoshaping procedure with retractable levers as the conditioned stimulus, food pellets as the unconditioned stimulus, and lever presses as the conditioned response. In Phase 1, we observed a PREE in lever pressing in both strains, but this effect was larger and longer lasting in RHA/PR than in RLA/PR rats. In Phase 2, reacquisition was fast and the PREE persisted in both strains, although the two PR groups no longer differed in lever pressing. The results are discussed in terms of frustration theory and of uncertainty-induced sensitization of dopaminergic neurons.

Unpredictability as a modulator of drug self-administration: Relevance for substance-use disorders

Drug self-administration has been regarded as a gold-standard preclinical model of addiction and substance-use disorder (SUD). However, investigators are becoming increasingly aware, that certain aspects of addiction or SUDs experienced by humans are not accurately captured in our preclinical self-administration models. The current review will focus on two such aspects of current preclinical drug self-administration models: 1) Predictable vs. unpredictable drug access in terms of the time and effort put into obtaining drugs (i.e., response requirement) and drug quality (i.e., amount) and 2) rich vs. lean access to drugs. Some behavioral and neurobiological mechanisms that could contribute to excessive allocation of behavior toward drug-seeking and drug-taking at the expense of engaging in nondrug-related activities are discussed, and some directions for future research are identified. Based on the experiments reviewed, lean and unpredictable drug access could worsen drug-seeking and drug-taking behavior in individuals with SUDs. Once more fully explored, this area of research will help determine whether and how unpredictable and lean cost requirements affect drug self-administration in preclinical laboratory studies with nonhuman subjects and will help determine whether incorporating these conditions in current self-administration models will increase their predictive validity.

A direct and conceptual replication of post-loss speeding when gambling

To investigate the response to suboptimal outcomes, Verbuggen et al. (Verbruggen F, Chambers CD, Lawrence NS, McLaren IPL. 2017 Winning and losing: effects on impulsive action. J. Exp. Psychol.: Hum. Percept. Perform. 43, 147. (doi:10.1037/xhp0000284)) conducted a study in which participants chose between a gamble and a non-gamble option. The non-gamble option was a guaranteed amount of points, whereas the gamble option was associated with a higher amount but a lower probability of winning. The authors observed that participants initiated the next trial faster after a loss compared to wins or non-gambles. In the present study, we directly replicated these findings in the laboratory and online. We also designed another task controlling for the number of trials per outcome. In this task, participants guessed where a reward was hidden. They won points if they selected the correct location, but lost points if they selected the incorrect location. We included neutral trials as a baseline. Again, participants sped up after a loss relative to wins and neutral trials (but only with a response choice in neutral trials and a large sample size). These findings appear inconsistent with cognitive-control frameworks, which assume that suboptimal outcomes typically lead to slower responses; instead, they suggest that suboptimal outcomes can invigorate behaviour, consistent with accounts of frustrative non-reward and impulsive action.

From sign-tracking to attentional bias: Implications for gambling and substance use disorders

Sign-tracking behavior in Pavlovian autoshaping is known to be a relevant index of the incentive salience attributed to reward-related cues. Evidence has accumulated to suggest that animals that exhibit a sign-tracker phenotype are especially vulnerable to addiction and relapse due to their proneness to attribute incentive salience to drug cues, and their relatively weak cognitive and attentional control over their behavior. Interestingly, sign-tracking is also influenced by reward uncertainty in a way that may promote gambling disorder. Research indicates that reward uncertainty sensitizes sign-tracking responses and favors the development of a sign-tracker phenotype, compatible with the conditioned attractiveness of lights and sounds in casinos for problem gamblers. The study of attentional biases in humans (an effect akin to sign-tracking in animals) leads to similar observations, notably that the propensity to develop attraction for conditioned stimuli (CSs) is predictive of addictive behavior. Here we review the literature on drug addiction and gambling disorder, highlighting the similarities between studies of sign-tracking and attentional biases.

Dopaminergic signaling of uncertainty and the aetiology of gambling addiction

Although there is increasing clinical recognition of behavioral addictions, of which gambling disorder is the prototype example, there is a limited understanding of the psychological properties of (non-substance-related) behaviors that enable them to become 'addictive' in a way that is comparable to drugs of abuse. According to an influential application of reinforcement learning to substance addictions, the direct effects of drugs to release dopamine can create a perpetual escalation of incentive salience. This article focusses on reward uncertainty, which is proposed to be the core feature of gambling that creates the capacity for addiction. We describe the neuro-dynamics of the dopamine response to uncertainty that may allow a similar escalation of incentive salience, and its relevance to behavioral addictions. We review translational evidence from both preclinical animal models and human clinical research, including studies in people with gambling disorder. Further, we describe the evidence for 1) the effects of the omission of expected reward as a stressor and to promote sensitization, 2) the effect of the resolution of reward uncertainty as a source of value, 3) structural characteristics of modern Electronic Gaming Machines (EGMs) in leveraging these mechanisms, 4) analogies to the aberrant salience hypothesis of psychosis for creating and maintaining gambling-related cognitive distortions. This neurobiologically-inspired model has implications for harm profiling of other putative behavioral addictions, as well as offering avenues for enhancing neurological, pharmacological and psychological treatments for gambling disorder, and harm reduction strategies for EGM design.

Maladaptive consequences of repeated intermittent exposure to uncertainty

Recently we reported that nucleus accumbens (NAcc) dopamine (DA) tracks uncertainty during operant responding for non-caloric saccharin. We also showed that repeated intermittent exposure to this uncertainty, like exposure to drugs of abuse, leads to sensitization of the locomotor and NAcc DA effects of amphetamine and promotes the subsequent self-administration of the drug. Here we review these findings together with others showing that NAcc glutamate signaling is similarly affected by uncertainty. Extracellular levels of glutamate in this site also track uncertainty in a task in which nose poking for saccharin on an escalating variable ratio schedule of reinforcement is associated with progressively increasing variance between performance of the operant and payout. Furthermore, sensitized behavioral responding to and for amphetamine following exposure to uncertainty is accompanied by increased levels of Ca2+/calmodulin-dependent protein kinase II (CaMKII) and protein kinase C (PKC) phosphorylation as well as altered protein levels of the transcription factor ∆FosB (increased) and glutamate transporter 1 (GLT1; decreased) in NAcc tissues. Notably, phosphorylation by CaMKII and PKC regulates AMPA receptor trafficking and function in this site, is elevated following psychostimulant exposure, and is necessary for the expression of enhanced drug taking. Increased ∆FosB and decreased GLT1 levels are observed following psychostimulant exposure, are associated with increased drug taking and seeking, and are known to modulate AMPA receptors and extracellular glutamate levels respectively. These adaptations in glutamate transmission as well as those observed with DA following repeated intermittent exposure to uncertainty are similar to those produced by exposure to abused drugs. Together, they point to the recruitment of both DA and glutamate signaling pathways in the NAcc in both drug and behavioral addictions. As uncertainty is central to games of chance, these findings have particular relevance for gambling disorders known to exhibit comorbidity with drug abuse.

Reward uncertainty attributes incentive value to reward proximal cues, while amphetamine sensitization reverts attention to more predictive reward distal cues

Slot-machine gambling incorporates numerous audiovisual cues prior to and during reward delivery (e.g. spinning wheels, flashing lights, celebratory sounds). Over time, these cues may motivate playing and even elicit cravings and relapse in those affected by gambling disorder. Animal studies suggest a heightened attraction to these cues despite diminished predictive ability under reward uncertainty, as evidenced by sign-tracking behavior in rats. Repeated amphetamine administration may also enhance the incentive value attributed to cues. Here, we explored the impact of reward uncertainty and prior amphetamine sensitization on the relative attractiveness and conditioned reinforcing properties of serial Pavlovian cues with different degrees of predictive and incentive value in rats. Animals were sensitized through repeated injections of amphetamine (1-4 mg/kg) or saline and then trained in a Pavlovian autoshaping task involving two sequential lever-auditory cue combinations (CS1, CS2) under Certain (100%-1) or Uncertain (50%-1-2-3) reward conditions. Subsequently, we evaluated the impact of acute amphetamine exposure on cue attraction. Our results suggest that Uncertainty alone enhanced attraction towards the reward-proximal CS2. However, combined sensitization and Uncertainty reversed cue preference relative to Uncertainty alone, enhancing attraction towards the more predictive reward-distal CS1. Both cues acquired conditioned reinforcing properties, despite the CS2 being otherwise ignored in all groups besides Uncertainty. However, combined sensitization and Uncertainty increased the reinforcing value of both cues and doubled the amount of interaction with the CS1 lever per presentation. Our results imply competitive mechanisms for attributing incentive value to gambling-related cues between reward uncertainty, prior amphetamine sensitization, and acute amphetamine administration.

Occasion setters attain incentive motivational value: implications for contextual influences on reward-seeking

The context in which reward-paired cues are encountered can resolve ambiguity and set the occasion for appropriate reward-seeking. The psychological processes by which contexts regulate reward-seeking remain unclear as contexts are diffuse and difficult to isolate from other stimuli. To overcome this, we modeled a context as a phasic and discrete event—an occasion setter (OS)—which allowed for control over its presentation and influence on cue-driven reward-seeking. This allowed us to directly assess how OSs regulate the predictive and motivational significance of Pavlovian cues. Male rats (n = 50) were trained in a Pavlovian procedure where the presentation of an ambiguous conditioned stimulus (CS) was reinforced only if preceded by an occasion setting cue. We assessed the motivational value of the OS and CS alone or in combination using tests of conditioned reinforcement. Rats enhanced conditioned approach to the reward port during the CS when it was preceded by the OS. When allowed the opportunity, rats responded more to obtain presentations of the CS in combination with the OS than the CS alone. Critically, rats also worked to obtain presentations of the OS alone more than the CS alone, and this was resistant to manipulations of the value of the OS. We conclude that occasion setting can act via incentive motivational mechanisms and that, apart from resolving predictive information about ambiguous reward-paired cues, OSs themselves generate states of appetitive motivation that can facilitate reward-seeking.

Cued for risk: Evidence for an incentive sensitization framework to explain the interplay between stress and anxiety, substance abuse, and reward uncertainty in disordered gambling behavior

Gambling disorder is an impairing condition confounded by psychiatric co-morbidity, particularly with substance use and anxiety disorders. Yet, our knowledge of the mechanisms that cause these disorders to coalesce remains limited. The Incentive Sensitization Theory suggests that sensitization of neural "wanting" pathways, which attribute incentive salience to rewards and their cues, is responsible for the excessive desire for drugs and cue-triggered craving. The resulting hyper-reactivity of the "wanting' system is believed to heavily influence compulsive drug use and relapse. Notably, evidence for sensitization of the mesolimbic dopamine pathway has been seen across gambling and substance use, as well as anxiety and stress-related pathology, with stress playing a major role in relapse. Together, this evidence highlights a phenomenon known as cross-sensitization, whereby sensitization to stress, drugs, or gambling behaviors enhance the sensitivity and dopaminergic response to any of those stimuli. Here, we review the literature on how cue attraction and reward uncertainty may underlie gambling pathology, and examine how this framework may advance our understanding of co-mordidity with substance-use disorders (e.g., alcohol, nicotine) and anxiety disorders. We argue that reward uncertainty, as seen in slot machines and games of chance, increases dopaminergic activity in the mesolimbic pathway and enhances the incentive value of reward cues. We propose that incentive sensitization by reward uncertainty may interact with and predispose individuals to drug abuse and stress, creating a mechanism through which co-mordidity of these disorders may emerge.

Exposure to conditions of uncertainty promotes the pursuit of amphetamine

Prior exposure to abused drugs leads to long-lasting neuroadaptations culminating in excessive drug intake. Given the comorbidity between substance use and gambling disorders, surprisingly little is known about the effects of exposure to reinforcement contingencies experienced during games of chance. As it is a central feature of these games, we characterized the effects of exposure to uncertainty on biochemical and behavioral effects normally observed in rats exposed to amphetamine. Rats in different groups were trained to nose-poke for saccharin under certain [fixed-ratio (FR)] or uncertain conditions [variable-ratio (VR)] for 55 1-h sessions. Ratios were escalated on successive sessions and rats maintained on the last ratio (FR/VR 20) for 20-25 days. Two to three weeks later, rats were tested for their locomotor or nucleus accumbens dopamine (NAcc DA) response to amphetamine or self-administration of the drug using a lever press operant. NAcc DA overflow was also assessed in additional rats during the saccharin sessions. Rats exposed to uncertainty subsequently showed a higher locomotor and NAcc DA response to amphetamine and self-administered more drug infusions relative to rats exposed to predictable reinforcement. NAcc DA levels during the saccharin sessions tracked the variance of the scheduled ratios (a measure of uncertainty). VR rats showed escalating DA overflow with increasing ratios. Exposure to uncertainty triggered neuroadaptations similar to those produced by exposure to abused drugs. As these were produced in drug naive rats both during and after exposure to uncertainty, they provide a novel common pathway to drug and behavioral addictions.

Sign-tracking is an expectancy-mediated behavior that relies on prediction error mechanisms

When discrete localizable stimuli are used during appetitive Pavlovian conditioning, "sign-tracking" and "goal-tracking" responses emerge. Sign-tracking is observed when conditioned responding is directed toward the CS, whereas goal-tracking manifests as responding directed to the site of expected reward delivery. These behaviors seem to rely on distinct, though overlapping neural circuitries, and, possibly, distinct psychological processes as well, and are thought to be related to addiction vulnerability. One currently popular view is that sign-tracking reflects an incentive motivational process, whereas goal-tracking reflects the influence of more top-down cognitive processes. To test these ideas, we used illness-induced outcome-devaluation and Kamin blocking procedures to determine whether these behaviors rely on similar or distinct underlying associative mechanisms. In Experiments 1 and 2 we showed that outcome-devaluation reduced sign-tracking responses, demonstrating that sign-tracking is controlled by reward expectancies. We also observed that post-CS goal-tracking in these animals is also devaluation sensitive. To test whether these two types of behaviors rely on similar or different prediction error mechanisms, we next tested whether Kamin blocking effects could be observed across these two classes of behaviors. In Experiment 3 we asked if sign-tracking to a lever CS could block the development of goal-tracking to a tone CS; whereas in Experiment 4, we examined whether goal-tracking to a tone CS could block sign-tracking to a lever CS. In both experiments blocking effects were observed suggesting that both sign- and goal-tracking emerge via a common prediction error mechanism. Collectively, the studies reported here suggest that the psychological mechanisms mediating sign- and goal-tracking are more similar than is commonly acknowledged.

Dynamic Encoding of Incentive Salience in the Ventral Pallidum: Dependence on the Form of the Reward Cue

Some rats are especially prone to attribute incentive salience to a cue (conditioned stimulus, CS) paired with food reward (sign-trackers, STs), but the extent they do so varies as a function of the form of the CS. Other rats respond primarily to the predictive value of a cue (goal-trackers, GTs), regardless of its form. Sign-tracking is associated with greater cue-induced activation of mesolimbic structures than goal-tracking; however, it is unclear how the form of the CS itself influences activity in neural systems involved in incentive salience attribution. Thus, our goal was to determine how different cue modalities affect neural activity in the ventral pallidum (VP), which is known to encode incentive salience attribution, as rats performed a two-CS Pavlovian conditioned approach task in which both a lever-CS and a tone-CS predicted identical food reward. The lever-CS elicited sign-tracking in some rats (STs) and goal-tracking in others (GTs), whereas the tone-CS elicited only goal-tracking in all rats. The lever-CS elicited robust changes in neural activity (sustained tonic increases or decreases in firing) throughout the VP in STs, relative to GTs. These changes were not seen when STs were exposed to the tone-CS, and in GTs there were no differences in firing between the lever-CS and tone-CS. We conclude that neural activity throughout the VP encodes incentive signals and is especially responsive when a cue is of a form that promotes the attribution of incentive salience to it, especially in predisposed individuals.

How foraging works: Uncertainty magnifies food-seeking motivation

Food uncertainty has the effect of invigorating food-related responses. Psychologists have noted that mammals and birds respond more to a conditioned stimulus that unreliably predicts food delivery, and ecologists have shown that animals (especially small passerines) consume and/or hoard more food and can get fatter when access to that resource is unpredictable. Are these phenomena related? We think they are. Psychologists have proposed several mechanistic interpretations, while ecologists have suggested a functional interpretation: The effect of unpredictability on fat reserves and hoarding behavior is an evolutionary strategy acting against the risk of starvation when food is in short supply. Both perspectives are complementary, and we argue that the psychology of incentive motivational processes can shed some light on the causal mechanisms leading animals to seek and consume more food under uncertainty in the wild. Our theoretical approach is in agreement with neuroscientific data relating to the role of dopamine, a neurotransmitter strongly involved in incentive motivation, and its plausibility has received some explanatory and predictive value with respect to Pavlovian phenomena. Overall, we argue that the occasional and unavoidable absence of food rewards has motivational effects (called incentive hope) that facilitate foraging effort. We show that this hypothesis is computationally tenable, leading foragers in an unpredictable environment to consume more food items and to have higher long-term energy storage than foragers in a predictable environment.

Multiple reward-cue contingencies favor expectancy over uncertainty in shaping the reward-cue attentional salience

Reward-predicting cues attract attention because of their motivational value. A debated question regards the conditions under which the cue's attentional salience is governed more by reward expectancy rather than by reward uncertainty. To help shedding light on this relevant issue, here, we manipulated expectancy and uncertainty using three levels of reward-cue contingency, so that, for example, a high level of reward expectancy (p = .8) was compared with the highest level of reward uncertainty (p = .5). In Experiment 1, the best reward-cue during conditioning was preferentially attended in a subsequent visual search task. This result was replicated in Experiment 2, in which the cues were matched in terms of response history. In Experiment 3, we implemented a hybrid procedure consisting of two phases: an omission contingency procedure during conditioning, followed by a visual search task as in the previous experiments. Crucially, during both phases, the reward-cues were never task relevant. Results confirmed that, when multiple reward-cue contingencies are explored by a human observer, expectancy is the major factor controlling both the attentional and the oculomotor salience of the reward-cue.

Foraging motivation favors the occurrence of Lévy walks

Lévy walks are a property of random movements often observed among foraging animals (and humans), and they might confer some advantages for survival in an unpredictable environment, in comparison with Brownian walks. In animals with a nervous system, specific neurotransmitters associated with some psychological states could play a crucial role in controlling the occurrence of Lévy walks. We argue that incentive motivation, a dopamine-dependent process that in vertebrates makes rewards and their predictive conditioned stimuli attractive, has behavioral effects that may favor their occurrence: incentive motivation is higher when food is unpredictable and it strongly underpins foraging activity. An individual-based computer model is used to determine whether changes in incentive motivation can influence the probability that Lévy walks occur among foraging agents. Our results suggest that they are produced more often under an unpredictable than a predictable food access, and more often in strongly rather than weakly motivated foragers exposed to an unpredictable food access. Also, our motivational framework indicates that the occurrence of Lévy walks are correlated with, but not causally linked to, the number of food items consumed and the ability to store fat reserves. We conclude that Lévy walks can confer some advantages for survival in an unpredictable environment, provided that they appear in foragers with a high motivation to seek food.

Reward Prediction Errors in Drug Addiction and Parkinson's Disease: from Neurophysiology to Neuroimaging

PURPOSE OF REVIEW

Surprises are important sources of learning. Cognitive scientists often refer to surprises as "reward prediction errors," a parameter that captures discrepancies between expectations and actual outcomes. Here, we integrate neurophysiological and functional magnetic resonance imaging (fMRI) results addressing the processing of reward prediction errors and how they might be altered in drug addiction and Parkinson's disease.

RECENT FINDINGS

By increasing phasic dopamine responses, drugs might accentuate prediction error signals, causing increases in fMRI activity in mesolimbic areas in response to drugs. Chronic substance dependence, by contrast, has been linked with compromised dopaminergic function, which might be associated with blunted fMRI responses to pleasant non-drug stimuli in mesocorticolimbic areas. In Parkinson's disease, dopamine replacement therapies seem to induce impairments in learning from negative outcomes. The present review provides a holistic overview of reward prediction errors across different pathologies and might inform future clinical strategies targeting impulsive/compulsive disorders.

Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making

Stimuli that are more predictive of subsequent reward also function as better conditioned reinforcers. Moreover, stimuli attributed with incentive salience function as more robust conditioned reinforcers. Some theories have suggested that conditioned reinforcement plays an important role in promoting suboptimal choice behavior, like gambling. The present experiments examined how different stimuli, those attributed with incentive salience versus those without, can function in tandem with stimulus-reward predictive utility to promote maladaptive decision-making in rats. One group of rats had lights associated with goal-tracking as the reward-predictive stimuli and another had levers associated with sign-tracking as the reward-predictive stimuli. All rats were first trained on a choice procedure in which the expected value across both alternatives was equivalent but differed in their stimulus-reward predictive utility. Next, the expected value across both alternatives was systematically changed so that the alternative with greater stimulus-reward predictive utility was suboptimal in regard to primary reinforcement. The results demonstrate that in order to obtain suboptimal choice behavior, incentive salience alongside strong stimulus-reward predictive utility may be necessary; thus, maladaptive decision-making can be driven more by the value attributed to stimuli imbued with incentive salience that reliably predict a reward rather than the reward itself.

Learning not to respond: Role of the hippocampus in withholding responses during omission training

Autoshaping is a Pavlovian learning paradigm in which rats experience pairings of a CS and a US independently of their behavior. When the CS is a lever inserted into the test cage and the US is food delivered to an adjacent magazine, many rats acquire a lever-pressing response called 'sign-tracking' even though that response has no effect on the occurrence of either the CS or the US. Since these lever presses are always followed by the US, it has been suggested that sign-tracking could be due to unintended reinforcement of the response. To eliminate the possibility of such instrumental learning the omission schedule, in which a response to the CS cancels the US, was introduced. Previous research has shown that training rats on autoshaping and switching them to an omission schedule generally reduces but does not eliminate sign-tracking, suggesting that it may be due to both Pavlovian and instrumental learning. In the present study naive rats trained on an omission schedule sign-tracked less than a control group exposed to random, unpaired CS and US presentations, suggesting that they learned to withhold the lever press response because of the negative contingency between that response and the US. In a second experiment rats with dorsal hippocampus lesions sign-tracked more than sham-lesioned rats on omission schedules, suggesting that this case of learning not to respond is hippocampus-based. This conclusion is consistent with many previous findings on the inability of hippocampal rats to withhold or suppress responding, and with studies suggesting that one form of extinction of learned responses in normal rats is due to competition from hippocampus-based learning not to respond.

Winning and losing: Effects on impulsive action

In the present study, we examined the effect of wins and losses on impulsive action in gambling (Experiments 1-3) and nongambling tasks (Experiments 4-5). In each experiment, subjects performed a simple task in which they had to win points. On each trial, they had to choose between a gamble and a nongamble. The gamble was always associated with a higher amount but a lower probability of winning than the nongamble. After subjects indicated their choice (i.e., gamble or not), feedback was presented. They had to press a key to start the next trial. Experiments 1-3 showed that, compared to the nongambling baseline, subjects were faster to initiate the next trial after a gambled loss, indicating that losses can induce impulsive actions. In Experiments 4 and 5, subjects alternated between the gambling task and a neutral decision-making task in which they could not win or lose points. Subjects were faster in the neutral decision-making task if they had just lost in the gambling task, suggesting that losses have a general effect on action. Our results challenge the dominant idea that humans become more cautious after suboptimal outcomes. Instead, they indicate that losses in the context of potential rewards are emotional events that increase impulsivity. (PsycINFO Database Record

Deficits in context-dependent adaptive coding of reward in schizophrenia

Theoretical principles of information processing and empirical findings suggest that to efficiently represent all possible rewards in the natural environment, reward-sensitive neurons have to adapt their coding range dynamically to the current reward context. Adaptation ensures that the reward system is most sensitive for the most likely rewards, enabling the system to efficiently represent a potentially infinite range of reward information. A deficit in neural adaptation would prevent precise representation of rewards and could have detrimental effects for an organism’s ability to optimally engage with its environment. In schizophrenia, reward processing is known to be impaired and has been linked to different symptom dimensions. However, despite the fundamental significance of coding reward adaptively, no study has elucidated whether adaptive reward processing is impaired in schizophrenia. We therefore studied patients with schizophrenia (n=27) and healthy controls (n=25), using functional magnetic resonance imaging in combination with a variant of the monetary incentive delay task. Compared with healthy controls, patients with schizophrenia showed less efficient neural adaptation to the current reward context, which leads to imprecise neural representation of reward. Importantly, the deficit correlated with total symptom severity. Our results suggest that some of the deficits in reward processing in schizophrenia might be due to inefficient neural adaptation to the current reward context. Furthermore, because adaptive coding is a ubiquitous feature of the brain, we believe that our findings provide an avenue in defining a general impairment in neural information processing underlying this debilitating disorder.

Pavlovian Conditioned Approach Training in Rats

Cues that are contingently paired with unconditioned, rewarding stimuli can acquire rewarding properties themselves through a process known as the attribution of incentive salience, or the transformation of neutral stimuli into attractive, "wanted' stimuli capable of motivating behavior. Pavlovian conditioned approach (PCA) develops after the response-independent presentation of a conditioned stimulus (CS; e.g., a lever) that predicts the delivery of an unconditioned stimulus (US; e.g., a food pellet) and can be used to measure incentive salience. During training, three patterns of conditioned responses (CRs) can develop: sign-tracking behavior (CS-directed CR), goal-tracking behavior (US-directed CR), and an intermediate response (both CRs). Sign-trackers attribute incentive salience to reward-related cues and are more vulnerable to cue-induced reinstatement of drug-seeking as well as other addiction-related behaviors, making PCA a potentially valuable procedure for studying addiction vulnerability. Here, we describe materials and methods used to elicit PCA behavior from rats as well as analyze and interpret PCA behavior in individual experiments.

Roles of "Wanting" and "Liking" in Motivating Behavior: Gambling, Food, and Drug Addictions

The motivation to seek out and consume rewards has evolutionarily been driven by the urge to fulfill physiological needs. However in a modern society dominated more by plenty than scarcity, we tend to think of motivation as fueled by the search for pleasure. Here, we argue that two separate but interconnected subcortical and unconscious processes direct motivation: "wanting" and "liking." These two psychological and neuronal processes and their related brain structures typically work together, but can become dissociated, particularly in cases of addiction. In drug addiction, for example, repeated consumption of addictive drugs sensitizes the mesolimbic dopamine system, the primary component of the "wanting" system, resulting in excessive "wanting" for drugs and their cues. This sensitizing process is long-lasting and occurs independently of the "liking" system, which typically remains unchanged or may develop a blunted pleasure response to the drug. The result is excessive drug-taking despite minimal pleasure and intense cue-triggered craving that may promote relapse long after detoxification. Here, we describe the roles of "liking" and "wanting" in general motivation and review recent evidence for a dissociation of "liking" and "wanting" in drug addiction, known as the incentive sensitization theory (Robinson and Berridge 1993). We also make the case that sensitization of the "wanting" system and the resulting dissociation of "liking" and "wanting" occurs in both gambling disorder and food addiction.

Rats that sign-track are resistant to Pavlovian but not instrumental extinction

Individuals vary in the extent to which they attribute incentive salience to a discrete cue (conditioned stimulus; CS) that predicts reward delivery (unconditioned stimulus; US), which results in some individuals approaching and interacting with the CS (sign-trackers; STs) more than others (goal-trackers; GTs). Here we asked how periods of non-reinforcement influence conditioned responding in STs vs. GTs, in both Pavlovian and instrumental tasks. After classifying rats as STs or GTs by pairing a retractable lever (the CS) with the delivery of a food pellet (US), we introduced periods of non-reinforcement, first by simply withholding the US (i.e., extinction training; experiment 1), then by signaling alternating periods of reward (R) and non-reward (NR) within the same session (experiments 2 and 3). We also examined how alternating R and NR periods influenced instrumental responding for food (experiment 4). STs and GTs did not differ in their ability to discriminate between R and NR periods in the instrumental task. However, in Pavlovian settings STs and GTs responded to periods of non-reward very differently. Relative to STs, GTs very rapidly modified their behavior in response to periods of non-reward, showing much faster extinction and better and faster discrimination between R and NR conditions. These results highlight differences between Pavlovian and instrumental extinction learning, and suggest that if a Pavlovian CS is strongly attributed with incentive salience, as in STs, it may continue to bias attention toward it, and to facilitate persistent and relatively inflexible responding, even when it is no longer followed by reward.

Amphetamine-induced sensitization and reward uncertainty similarly enhance incentive salience for conditioned cues

Amphetamine and stress can sensitize mesolimbic dopamine-related systems. In Pavlovian autoshaping, repeated exposure to uncertainty of reward prediction can enhance motivated sign-tracking or attraction to a discrete reward-predicting cue (lever-conditioned stimulus; CS+), as well as produce cross-sensitization to amphetamine. However, it remains unknown how amphetamine sensitization or repeated restraint stress interact with uncertainty in controlling CS+ incentive salience attribution reflected in sign-tracking. Here rats were tested in 3 successive phases. First, different groups underwent either induction of amphetamine sensitization or repeated restraint stress, or else were not sensitized or stressed as control groups (either saline injections only, or no stress or injection at all). All next received Pavlovian autoshaping training under either certainty conditions (100% CS-UCS association) or uncertainty conditions (50% CS-UCS association and uncertain reward magnitude). During training, rats were assessed for sign-tracking to the CS+ lever versus goal-tracking to the sucrose dish. Finally, all groups were tested for psychomotor sensitization of locomotion revealed by an amphetamine challenge. Our results confirm that reward uncertainty enhanced sign-tracking attraction toward the predictive CS+ lever, at the expense of goal-tracking. We also reported that amphetamine sensitization promoted sign-tracking even in rats trained under CS-UCS certainty conditions, raising them to sign-tracking levels equivalent to the uncertainty group. Combining amphetamine sensitization and uncertainty conditions did not add together to elevate sign-tracking further above the relatively high levels induced by either manipulation alone. In contrast, repeated restraint stress enhanced subsequent amphetamine-elicited locomotion, but did not enhance CS+ attraction.

Relative reward effects on operant behavior: Incentive contrast, induction and variety effects

Comparing different rewards automatically produces dynamic relative outcome effects on behavior. Each new outcome exposure is to an updated version evaluated relative to alternatives. Relative reward effects include incentive contrast, positive induction and variety effects. The present study utilized a novel behavioral design to examine relative reward effects on a chain of operant behavior using auditory cues. Incentive contrast is the most often examined effect and focuses on increases or decreases in behavioral performance after value upshifts (positive) or downshifts (negative) relative to another outcome. We examined the impact of comparing two reward outcomes in a repeated measures design with three sessions: a single outcome and a mixed outcome and a final single outcome session. Relative reward effects should be apparent when comparing trials for the identical outcome between the single and mixed session types. An auditory cue triggered a series of operant responses (nosepoke-leverpress-food retrieval), and we measured possible contrast effects for different reward magnitude combinations. We found positive contrast for trials with the greatest magnitude differential but positive induction or variety effects in other combinations. This behavioral task could be useful for analyzing environmental or neurobiological factors involved in reward comparisons, decision-making and choice during instrumental, goal-directed action.

Incentive salience attribution under reward uncertainty: A Pavlovian model

There is a vast literature on the behavioural effects of partial reinforcement in Pavlovian conditioning. Compared with animals receiving continuous reinforcement, partially rewarded animals typically show (a) a slower development of the conditioned response (CR) early in training and (b) a higher asymptotic level of the CR later in training. This phenomenon is known as the partial reinforcement acquisition effect (PRAE). Learning models of Pavlovian conditioning fail to account for it. In accordance with the incentive salience hypothesis, it is here argued that incentive motivation (or 'wanting') plays a more direct role in controlling behaviour than does learning, and reward uncertainty is shown to have an excitatory effect on incentive motivation. The psychological origin of that effect is discussed and a computational model integrating this new interpretation is developed. Many features of CRs under partial reinforcement emerge from this model.

The form of a conditioned stimulus can influence the degree to which it acquires incentive motivational properties

There is considerable individual variation in the extent to which food- and drug-associated cues (conditioned stimuli, CSs) acquire incentive salience, as indicated by whether they elicit approach towards them, and/or act as conditioned reinforcers. Here we asked whether this variation is influenced by properties of the CS itself. In rats, we assessed both the attractiveness and conditioned reinforcing properties of two CSs: a manipulable lever CS versus an auditory (tone) CS. There was considerable individual variation in the extent to which a lever CS acquired incentive motivational properties, as indicated by whether it became attractive (evoked a sign-tracking or goal-tracking conditioned response) or acted as a conditioned reinforcer. However, with a tone CS all rats learned a goal-tracking response, and the tone CS was an equally effective conditioned reinforcer in sign-trackers and goal-trackers. Even when presented in compound (a lever-tone CS), the two elements of the compound differentially acquired motivational properties. In contrast, amphetamine and stress potentiated the conditioned reinforcing properties of both visual and auditory CSs similarly in rats that primarily sign-tracked or goal-tracked. We conclude that variation in the to the ability of CSs to acquire incentive salience, and thus their ability to act as incentive stimuli capable of motivating behavior, is determined in part by properties of the CS itself.