A transient dopamine signal encodes subjective value and causally influences demand in an economic context

Effects of psychedelic, DOI, on nucleus accumbens dopamine signaling to predictable rewards and cues in rats

Psychedelics produce lasting therapeutic responses in neuropsychiatric diseases suggesting they may disrupt entrenched associations and catalyze learning. Here, we examine psychedelic 5-HT2A/2C agonist, DOI, effects on dopamine signaling in the nucleus accumbens (NAc) core, a region extensively linked to reward learning, motivation, and drug-seeking. We measure phasic dopamine transients following acute DOI administration in rats during well learned Pavlovian tasks in which sequential cues predict rewards. We find that DOI (0.0-1.2 mg/kg, i.p.) increases dopamine signals, photometrically measured using GRABDA optical sensor, to rewards and proximal reward cues, but not to the distal cues that predict these events. We determine that the elevated dopamine produced by DOI to reward cues occurs independently of DOI-induced changes in reward value. The increased dopamine associated with predictable reward cues and rewards supports DOI-induced increases in prediction error signaling. These findings lay a foundation for developing psychedelic strategies aimed at engaging error-driven learning mechanisms to disrupt entrenched associations or produce new associations.

Striatal dopamine integrates cost, benefit, and motivation

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Effort cost of harvest affects decisions and movement vigor of marmosets during foraging

Our decisions are guided by how we perceive the value of an option, but this evaluation also affects how we move to acquire that option. Why should economic variables such as reward and effort alter the vigor of our movements? In theory, both the option that we choose and the vigor with which we move contribute to a measure of fitness in which the objective is to maximize rewards minus efforts, divided by time. To explore this idea, we engaged marmosets in a foraging task in which on each trial they decided whether to work by making saccades to visual targets, thus accumulating food, or to harvest by licking what they had earned. We varied the effort cost of harvest by moving the food tube with respect to the mouth. Theory predicted that the subjects should respond to the increased effort costs by choosing to work longer, stockpiling food before commencing harvest, but reduce their movement vigor to conserve energy. Indeed, in response to an increased effort cost of harvest, marmosets extended their work duration, but slowed their movements. These changes in decisions and movements coincided with changes in pupil size. As the effort cost of harvest declined, work duration decreased, the pupils dilated, and the vigor of licks and saccades increased. Thus, when acquisition of reward became effortful, the pupils constricted, the decisions exhibited delayed gratification, and the movements displayed reduced vigor.

Effort cost of harvest affects decisions and movement vigor of marmosets during foraging

Our decisions are guided by how we perceive the value of an option, but this evaluation also affects how we move to acquire that option. Why should economic variables such as reward and effort alter the vigor of our movements? In theory, both the option that we choose and the vigor with which we move contribute to a measure of fitness in which the objective is to maximize rewards minus efforts, divided by time. To explore this idea, we engaged marmosets in a foraging task in which on each trial they decided whether to work by making saccades to visual targets, thus accumulating food, or to harvest by licking what they had earned. We varied the effort cost of harvest by moving the food tube with respect to the mouth. Theory predicted that the subjects should respond to the increased effort costs by choosing to work longer, stockpiling food before commencing harvest, but reduce their movement vigor to conserve energy. Indeed, in response to an increased effort cost of harvest, marmosets extended their work duration, but slowed their movements. These changes in decisions and movements coincided with changes in pupil size. As the effort cost of harvest declined, work duration decreased, the pupils dilated, and the vigor of licks and saccades increased. Thus, when acquisition of reward became effortful, the pupils constricted, the decisions exhibited delayed gratification, and the movements displayed reduced vigor.

Significance statement

Our results suggest that as the brainstem neuromodulatory circuits that control pupil size respond to effort costs, they alter computations in the brain regions that control decisions, encouraging work and delaying gratification, and the brain regions that control movements, reducing vigor and suppressing energy expenditure. This coordinated response suggests that decisions and actions are part of a single control policy that aims to maximize a variable relevant to fitness: the capture rate.

Neural Signature of Buying Decisions in Real-World Online Shopping Scenarios – An Exploratory Electroencephalography Study Series

The neural underpinnings of decision-making are critical to understanding and predicting human behavior. However, findings from decision neuroscience are limited in their practical applicability due to the gap between experimental decision-making paradigms and real-world choices. The present manuscript investigates the neural markers of buying decisions in a fully natural purchase setting: participants are asked to use their favorite online shopping applications to buy common goods they are currently in need of. Their electroencephalography (EEG) is recorded while they view the product page for each item. EEG responses to pages for products that are eventually bought are compared to those that are discarded. Study 1 repeats this procedure in three batches with different participants, product types, and time periods. In an explorative analysis, two neural markers for buying compared to no-buying decisions are discovered over all three batches: frontal alpha asymmetry peak and frontal theta power peak. Occipital alpha power at alpha asymmetry peaks differs in only one of the three batches. No further significant markers are found. Study 2 compares the natural product search to a design in which subjects are told which product pages to view. In both settings, the frontal alpha asymmetry peak is increased for buying decisions. Frontal theta peak increase is replicated only when subjects search through product pages by themselves. The present study series represents an attempt to find neural markers of real-world decisions in a fully natural environment and explore how those markers can change due to small adjustments for the sake of experimental control. Limitations and practical applicability of the real-world approach to studying decision-making are discussed.

Increased elasticity of sucrose demand during hyperdopaminergic states in rats

RATIONALE

Deficits in cost-benefit decision-making are a core feature of several psychiatric disorders, including substance addiction, eating disorders and bipolar disorder. Mesocorticolimbic dopamine signalling has been implicated in various processes related to cognition and reward, but its precise role in reward valuation and cost-benefit trade-off decisions remains incompletely understood.

OBJECTIVES

We assessed the role of mesocorticolimbic dopamine signalling in the relationship between price and consumption of sucrose, to better understand its role in cost-benefit decisions.

METHODS

Dopamine neurons in the ventral tegmental area (VTA) were chemogenetically activated in rats, and a behavioural economics approach was used to quantify the relationship between price and consumption of sucrose. Motivation for sucrose was also assessed under a progressive ratio (PR) schedule of reinforcement. To further gauge the role of dopamine in cost-benefit trade-offs for sucrose, the effects of treatment with D-amphetamine and the dopamine receptor antagonist alpha-flupentixol were assessed.

RESULTS

Chemogenetic activation of VTA dopamine neurons increased demand elasticity, while responding for sucrose under a PR schedule of reinforcement was augmented upon stimulation of VTA dopamine neurons. Treatment with amphetamine partially replicated the effects of chemogenetic dopamine neuron activation, whereas treatment with alpha-flupentixol reduced free consumption of sucrose and had mixed effects on demand elasticity.

CONCLUSIONS

Stimulation of mesocorticolimbic dopaminergic neurotransmission altered cost-benefit trade-offs in a complex manner. It reduced the essential value of palatable food, increased incentive motivation and left free consumption unaltered. Together, these findings imply that mesocorticolimbic dopamine signalling differentially influences distinct components of cost expenditure processes aimed at obtaining rewards.

Syn3 Gene Knockout Negatively Impacts Aspects of Reversal Learning Performance

Behavioral flexibility enables the ability to adaptively respond to changes in contingency requirements to maintain access to desired outcomes, and deficits in behavioral flexibility have been documented in many psychiatric disorders. Previous research has shown a correlation between behavioral flexibility measured in a reversal learning test and Syn3, the gene encoding synapsin III, which negatively regulates phasic dopamine release. Syn3 expression in the hippocampus, striatum, and neocortex is reported to be negatively correlated with reversal learning performance, so here, we used a global knock-out line to investigate reversal learning in mice homozygous wild type, heterozygous null, and homozygous null for the Syn3 gene. Compared with wild-type animals, we found a reversal-specific effect of genetic Syn3 deficiency that resulted in a greater proportional increase in trials required to reach a preset performance criterion during contingency reversal, despite no observed genotype effects on the ability to acquire the initial discrimination. Behavioral flexibility scores, which quantified the likelihood of switching subsequent choice behavior following positive or negative feedback, became significantly more negative in reversal only for Syn3 homozygous-null mice, suggesting a substantial increase in perseverative behavior in the reversal phase. Syn3 ablation reduced the number of anticipatory responses made per trial, often interpreted as a measure of waiting impulsivity. Overall, Syn3 expression negatively affected behavioral flexibility in a reversal-specific manner but may have reduced waiting impulsivity.

Chronic Augmentation of Endocannabinoid Levels Persistently Increases Dopaminergic Encoding of Reward Cost and Motivation

Motivational deficits characterized by an unwillingness to overcome effortful costs are a common feature of neuropsychiatric and neurologic disorders that are insufficiently understood and treated. Dopamine (DA) signaling in the nucleus accumbens (NAc) facilitates goal-seeking, but how NAc DA release encodes motivationally salient stimuli to influence effortful investment is not clear. Using fast-scan cyclic voltammetry in male and female mice, we find that NAc DA release diametrically responds to cues signaling increasing cost of reward, while DA release to the reward itself is unaffected by its cost. Because endocannabinoid (eCB) signaling facilitates goal seeking and NAc DA release, we further investigated whether repeated augmentation of the eCB 2-arachidonoylglycerol with a low dose of a monoacylglycerol lipase (MAGL) inhibitor facilitates motivation and DA signaling without the development of tolerance. We find that chronic MAGL treatment stably facilitates goal seeking and DA encoding of prior reward cost, providing critical insight into the neurobiological mechanisms of a viable treatment for motivational deficits.SIGNIFICANCE STATEMENT Decades of work has established a fundamental role for dopamine neurotransmission in motivated behavior and cue-reward learning, but how dopaminergic encoding of cues associates with motivated action has remained unclear. Specifically, how dopamine neurons signal future and prior reward cost, and whether this can be modified to influence motivational set points is not known. The current study provides important insight into how dopamine neurons encode motivationally relevant stimuli to influence goal-directed action and supports cannabinoid-based therapies for treatment of motivational disorders.

A mosaic of cost-benefit control over cortico-striatal circuitry

Dopamine contributes to cognitive control through well-established effects in both the striatum and cortex. Although earlier work suggests that dopamine affects cognitive control capacity, more recent work suggests that striatal dopamine may also impact on cognitive motivation. We consider the emerging perspective that striatal dopamine boosts control by making people more sensitive to the benefits versus the costs of cognitive effort, and we discuss how this sensitivity shapes competition between controlled and prepotent actions. We propose that dopamine signaling in distinct cortico-striatal subregions mediates different types of cost-benefit tradeoffs, and also discuss mechanisms for the local control of dopamine release, enabling selectivity among cortico-striatal circuits. In so doing, we show how this cost-benefit mosaic can reconcile seemingly conflicting findings about the impact of dopamine signaling on cognitive control.

Cannabinoid Modulation of Dopamine Release During Motivation, Periodic Reinforcement, Exploratory Behavior, Habit Formation, and Attention

Motivational and attentional processes energize action sequences to facilitate evolutionary competition and promote behavioral fitness. Decades of neuropharmacology, electrophysiology and electrochemistry research indicate that the mesocorticolimbic DA pathway modulates both motivation and attention. More recently, it was realized that mesocorticolimbic DA function is tightly regulated by the brain's endocannabinoid system and greatly influenced by exogenous cannabinoids-which have been harnessed by humanity for medicinal, ritualistic, and recreational uses for 12,000 years. Exogenous cannabinoids, like the primary psychoactive component of cannabis, delta-9-tetrahydrocannabinol, produce their effects by acting at binding sites for naturally occurring endocannabinoids. The brain's endocannabinoid system consists of two G-protein coupled receptors, endogenous lipid ligands for these receptor targets, and several synthetic and metabolic enzymes involved in their production and degradation. Emerging evidence indicates that the endocannabinoid 2-arachidonoylglycerol is necessary to observe concurrent increases in DA release and motivated behavior. And the historical pharmacology literature indicates a role for cannabinoid signaling in both motivational and attentional processes. While both types of behaviors have been scrutinized under manipulation by either DA or cannabinoid agents, there is considerably less insight into prospective interactions between these two important signaling systems. This review attempts to summate the relevance of cannabinoid modulation of DA release during operant tasks designed to investigate either motivational or attentional control of behavior. We first describe how cannabinoids influence DA release and goal-directed action under a variety of reinforcement contingencies. Then we consider the role that endocannabinoids might play in switching an animal's motivation from a goal-directed action to the search for an alternative outcome, in addition to the formation of long-term habits. Finally, dissociable features of attentional behavior using both the 5-choice serial reaction time task and the attentional set-shifting task are discussed along with their distinct influences by DA and cannabinoids. We end with discussing potential targets for further research regarding DA-cannabinoid interactions within key substrates involved in motivation and attention.

Hof P. Understanding Emotions: Origins and Roles of the Amygdala

Emotions arise from activations of specialized neuronal populations in several parts of the cerebral cortex, notably the anterior cingulate, insula, ventromedial prefrontal, and subcortical structures, such as the amygdala, ventral striatum, putamen, caudate nucleus, and ventral tegmental area. Feelings are conscious, emotional experiences of these activations that contribute to neuronal networks mediating thoughts, language, and behavior, thus enhancing the ability to predict, learn, and reappraise stimuli and situations in the environment based on previous experiences. Contemporary theories of emotion converge around the key role of the amygdala as the central subcortical emotional brain structure that constantly evaluates and integrates a variety of sensory information from the surroundings and assigns them appropriate values of emotional dimensions, such as valence, intensity, and approachability. The amygdala participates in the regulation of autonomic and endocrine functions, decision-making and adaptations of instinctive and motivational behaviors to changes in the environment through implicit associative learning, changes in short- and long-term synaptic plasticity, and activation of the fight-or-flight response via efferent projections from its central nucleus to cortical and subcortical structures.

Endocannabinoid modulation of dopamine release during reward seeking, interval timing, and avoidance

Endocannabinoids (eCBs) are neuromodulators that influence a wide range of neural systems and behaviors. In the current review, we describe our recent research showing how eCBs, particularly 2-arachidonoylglycerol (2-AG), concurrently shape mesolimbic dopamine (DA) release and associated behavior. We will restrict our discussion by emphasizing three distinct behaviors: reward seeking, interval timing, and active avoidance. During reward seeking we find that 2-AG is necessary to observe cue-evoked DA release events that are thought to represent the value of a rewarding outcome. We then describe data showing that 2-AG modulates unique patterns of DA release and behavior observed under conditions of periodic reinforcement. These data are discussed within the context of interval timing and adjunctive behavior. eCB modulation of DA release is also implicated in defensive behavior, including the avoidance of harm. As in reward seeking, our data suggest that the concentration of DA that is evoked by a warning signal can represent the value of an avoidance outcome. And, disrupting eCB signaling concomitantly reduces the concentration of the avoidance value signal and active avoidance. Disruptions in reward seeking, interval timing, and defensive behavior are commonly observed in a variety of movement disorders (e.g., Parkinson's and Huntington's disease) and disorders of motivation (e.g., addiction). We believe our data on eCB-DA interactions have implications for the development of novel pharmacotherapies to treat these disorders. Thus, we conclude by discussing how eCB pharmacology might be harnessed to treat disorders of movement and motivation.

Precis of Vigor: Neuroeconomics of movement control

Why do we run toward people we love, but only walk toward others? Why do people in New York seem to walk faster than other cities? Why do our eyes linger longer on things we value more? There is a link between how the brain assigns value to things, and how it controls our movements. This link is an ancient one, developed through shared neural circuits that on one hand teach us how to value things, and on the other hand control the vigor with which we move. As a result, when there is damage to systems that signal reward, like dopamine and serotonin, that damage not only affects our mood and patterns of decision making, but how we move. In this book, we first ask why in principle evolution should have developed a shared system of control between valuation and vigor. We then focus on the neural basis of vigor, synthesizing results from experiments that have measured activity in various brain structures and neuromodulators, during tasks in which animals decide how patiently they should wait for reward, and how vigorously they should move to acquire it. Thus, the way we move unmasks one of our well-guarded secrets: how much we value the thing we are moving toward.

Cue-Evoked Dopamine Neuron Activity Helps Maintain but Does Not Encode Expected Value

Cue-evoked midbrain dopamine (DA) neuron activity reflects expected value, but its influence on reward assessment is unclear. In mice performing a trial-based operant task, we test if bidirectional manipulations of cue or operant-associated DA neuron activity drive learning as a result of under- or overexpectation of reward value. We target optogenetic manipulations to different components of forced trials, when only one lever is presented, and assess lever biases on choice trials in the absence of photomanipulation. Although lever biases are demonstrated to be flexible and sensitive to changes in expected value, augmentation of cue or operant-associated DA signaling does not significantly alter choice behavior, and blunting DA signaling during any component of the forced trials reduces choice trial responses on the associated lever. These data suggest cue-evoked DA helps maintain cue-value associations but does not encode expected value as to set the benchmark against which received reward is judged.

Buprenorphine is a weak dopamine releaser relative to heroin, but its pretreatment attenuates heroin-evoked dopamine release in rats

AIMS

The United States of America is currently in an opioid epidemic. Heroin remains the most lethal opioid option with its death rate increasing by over 500% in the last decade. The rewarding and reinforcing effects of heroin are thought to be mediated by its ability to increase dopamine concentration in the nucleus accumbens shell. By activating Gi/o-coupled μ-opioid receptors, opioids are thought to indirectly excite midbrain dopamine neurons by removing an inhibitory GABAergic tone. The partial μ-opioid receptor agonist buprenorphine is a substitution-based therapy for heroin dependence that is thought to produce a steady-state level of μ-opioid receptor activation. But it remains unclear how buprenorphine alters dopamine release relative to heroin and how buprenorphine alters the dopamine-releasing effects of heroin. Because buprenorphine is a partial agonist at the μ-opioid receptor and heroin is a full agonist, we predicted that buprenorphine would function as a weak dopamine releaser relative to heroin, while functioning as a competitive antagonist if administered in advance of heroin.

METHODS

We performed fast-scan cyclic voltammetry in awake and behaving rats to measure how heroin, buprenorphine HCl, and their combination affect transient dopamine release events in the nucleus accumbens shell. We also performed a complimentary pharmacokinetic analysis comparing opioid plasma levels at time points correlated to our neurochemical findings.

RESULTS

Both buprenorphine and heroin produced changes in the frequency of transient dopamine release events, although the effect of buprenorphine was weak and only observed at a low dose. In comparison with vehicle, the frequency of dopamine release events maximally increased by ~25% following buprenorphine treatment and by ~60% following heroin treatment. Distinct neuropharmacological effects were observed in the high-dose range. The frequency of dopamine release events increased linearly with heroin dose but biphasically with buprenorphine dose. We also found that buprenorphine pretreatment occluded the dopamine-releasing effects of heroin, but plasma levels of buprenorphine had returned to baseline at this time point.

CONCLUSION

These findings support the notion that low-dose buprenorphine is a weak dopamine releaser relative to heroin and that buprenorphine pretreatment can block the dopamine-releasing effects of heroin. The finding that high-dose buprenorphine fails to increase dopamine release might explain its relatively low abuse potential among opioid-dependent populations. Because high-dose buprenorphine decreased dopamine release before occluding heroin-evoked dopamine release, and buprenorphine was no longer detected in plasma, we conclude that the mechanisms through which buprenorphine blocks heroin-evoked dopamine release involve multifaceted pharmacokinetic and pharmacodynamic interactions.

"Stopping for knowledge": The sense of beauty in the perception-action cycle

According to a millennial-old philosophical debate, aesthetic emotions have been connected to knowledge acquisition. Recent scientific evidence, collected across different disciplinary domains, confirms this link, but also reveals that motor inhibition plays a crucial role in the process. In this review, we discuss multidisciplinary results and propose an original account of aesthetic appreciation (the stopping for knowledge hypothesis) framed within the predictive coding theory. We discuss evidence showing that aesthetic emotions emerge in correspondence with an inhibition of motor behavior (i.e., minimizing action), promoting a simultaneous perceptual processing enhancement, at the level of sensory cortices (i.e., optimizing learning). Accordingly, we suggest that aesthetic appreciation may represent a hedonic feedback over learning progresses, motivating the individual to inhibit motor routines to seek further knowledge acquisition. Furthermore, the neuroimaging and neuropsychological studies we review reveal the presence of a strong association between aesthetic appreciation and the activation of the dopaminergic reward-related circuits. Finally, we propose a number of possible applications of the stopping for knowledge hypothesis in the clinical and education domains.

Accumbal Dopamine Release Tracks the Expectation of Dopamine Neuron-Mediated Reinforcement

Dopamine (DA) transmission in the nucleus accumbens (NAc) facilitates cue-reward associations and appetitive action. Reward-related accumbal DA release dynamics are traditionally ascribed to ventral tegmental area (VTA) DA neurons. Activation of VTA to NAc DA signaling is thought to reinforce action and transfer reward-related information to predictive cues, allowing cues to guide behavior and elicit dopaminergic activity. Here, we use optogenetics to control DA neuron activity and voltammetry to simultaneously record accumbal DA release in order to quantify how reinforcer-evoked dopaminergic activity shapes conditioned mesolimbic DA transmission. We find that cues predicting access to DA neuron self-stimulation elicit conditioned responding and NAc DA release. However, cue-evoked DA release does not reflect the cost or magnitude of DA neuron activation. Accordingly, conditioned accumbal DA release selectively tracks the expected availability of DA-neuron-mediated reinforcement. This work provides insight into how mesolimbic DA transmission drives and encodes appetitive action.

Age-related variability in decision-making: Insights from neurochemistry

Despite dopamine's significant role in models of value-based decision-making and findings demonstrating loss of dopamine function in aging, evidence of systematic changes in decision-making over the life span remains elusive. Previous studies attempting to resolve the neural basis of age-related alteration in decision-making have typically focused on physical age, which can be a poor proxy for age-related effects on neural systems. There is growing appreciation that aging has heterogeneous effects on distinct components of the dopamine system within subject in addition to substantial variability between subjects. We propose that some of the conflicting findings in age-related effects on decision-making may be reconciled if we can observe the underlying dopamine components within individuals. This can be achieved by incorporating in vivo imaging techniques including positron emission tomography (PET) and neuromelanin-sensitive MR. Further, we discuss how affective factors may contribute to individual differences in decision-making performance among older adults. Specifically, we propose that age-related shifts in affective attention ("positivity effect") can, in some cases, counteract the impact of altered dopamine function on specific decision-making processes, contributing to variability in findings. In an effort to provide clarity to the field and advance productive hypothesis testing, we propose ways in which in vivo dopamine imaging can be leveraged to disambiguate dopaminergic influences on decision-making, and suggest strategies for assessing individual differences in the contribution of affective attentional focus.

Catecholaminergic modulation of the cost of cognitive control in healthy older adults

Catecholamines have long been associated with cognitive control and value-based decision-making. More recently, we have shown that catecholamines also modulate value-based decision-making about whether or not to engage in cognitive control. Yet it is unclear whether catecholamines influence these decisions by altering the subjective value of control. Thus, we tested whether tyrosine, a catecholamine precursor altered the subjective value of performing a demanding working memory task among healthy older adults (60-75 years). Contrary to our prediction, tyrosine administration did not significantly increase the subjective value of conducting an N-back task for reward, as a main effect. Instead, in line with our previous study, exploratory analyses indicated that drug effects varied as a function of participants' trait impulsivity scores. Specifically, tyrosine increased the subjective value of conducting an N-back task in low impulsive participants, while reducing its value in more impulsive participants. One implication of these findings is that the over-the-counter tyrosine supplements may be accompanied by an undermining effect on the motivation to perform demanding cognitive tasks, at least in certain older adults. Taken together, these findings indicate that catecholamines can alter cognitive control by modulating motivation (rather than just the ability) to exert cognitive control.

Nucleus accumbens shell dopamine mediates outcome value, but not predicted value, in a magnitude decision-making task

Effective decision-making depends on an animal's ability to predict and select the outcome of greatest value, and the nucleus accumbens (NAc) and its dopaminergic input play a key role in this process. We previously reported that rapid dopamine release in the NAc shell preferentially tracks the "preferred" (i.e., large reward) option during cues that predict the ability to respond for rewards of different sizes, as well as during reward delivery itself. The present study assessed whether shell dopamine release at these discrete times selectively mediated choice behavior for rewards of different magnitudes using optogenetics. Here, using Long Evans TH:Cre^± rats we employed selective optogenetic stimulation of dopamine terminals in the NAc shell during either reward-predictive cues (experiment 1) or reward delivery (experiment 2) in a magnitude-based decision-making task. We found that in TH:Cre^± rats, but not littermate controls, optical stimulation during low-magnitude reward delivery during forced choice trials was sufficient to bias preference for this option when given a choice. In contrast, optical stimulation of shell dopamine terminals during low-magnitude reward-predictive cues in forced choice trials did not shift free choice behavior in TH:Cre^± rats or controls. The findings indicate that preferential dopamine signaling in the NAc shell during reward outcome (delivery), but not reward-predictive cues are sufficient to influence choice behavior in our task supporting a causal role of dopamine in the NAc shell in reward outcome value, but not value-based predictive strategies.

The impact of cocaine and heroin drug history on motivation and cue sensitivity in a rat model of polydrug abuse

RATIONALE

Comorbid use of heroin and cocaine is highly prevalent among drug users and can greatly increase addiction risk. Nonetheless, little is known regarding how a multi-drug history impacts motivation and cue responsivity to individual drugs.

OBJECTIVE

We used behavioral-economic procedures to examine motivation to maintain drug consumption and tests of drug-seeking to drug-associated cues to assess sensitivity to heroin and cocaine-associated cues in rats that had a self-administration history of heroin, cocaine, or both drugs.

RESULTS

Unexpectedly, we found that groups with a polydrug history of heroin and cocaine did not have higher levels of motivation or cue-induced reinstatement of drug-seeking for either cocaine or heroin compared to single drug groups. Nonetheless, we did find drug-specific differences in both economic price and cue sensitivity. Specifically, demand elasticity was lower for cocaine compared to heroin in animals with a single drug history, but not with polydrug groups. In addition, cocaine demand was predictive of the degree of cue-induced reinstatement of drug-seeking for cocaine following extinction, whereas heroin demand was predictive of the degree of reactivity to a heroin-associated cue. Furthermore, although cue reactivity following the initial self-administration phase did not differ across cues and drug history, reactivity to both heroin and cocaine cues was greater during subsequent heroin use compared to cocaine use, and this enhanced reactivity to heroin cues persisted during forced abstinence.

CONCLUSIONS

These results indicate that there is a greater motivation to maintain cocaine consumption, but higher sensitivity to drug-associated cues with a history of heroin use, suggesting that cocaine and heroin may drive continued drug use through different behavioral processes.

Mesolimbic dopamine D2 receptors and neural representations of subjective value

The process by which the value of delayed rewards is discounted varies from person to person. It has been suggested that these individual differences in subjective valuation of delayed rewards are supported by mesolimbic dopamine D2-like receptors (D2Rs) in the ventral striatum. However, no study to date has documented an association between direct measures of dopamine receptors and neural representations of subjective value in humans. Here, we examined whether individual differences in D2R availability were related to neural subjective value signals during decision making. Human participants completed a monetary delay discounting task during an fMRI scan and on a separate visit completed a PET scan with the high affinity D2R tracer [18 F]fallypride. Region-of-interest analyses revealed that D2R availability in the ventral striatum was positively correlated with subjective value-related activity in the ventromedial prefrontal cortex and midbrain but not with choice behavior. Whole-brain analyses revealed a positive correlation between ventral striatum D2R availability and subjective value-related activity in the left inferior frontal gyrus and superior insula. These findings identify a link between a direct measure of mesolimbic dopamine function and subjective value representation in humans and suggest a mechanism by which individuals vary in neural representation of discounted subjective value.

Recent advances in understanding the role of phasic dopamine activity

The latest animal neurophysiology has revealed that the dopamine reward prediction error signal drives neuronal learning in addition to behavioral learning and reflects subjective reward representations beyond explicit contingency. The signal complies with formal economic concepts and functions in real-world consumer choice and social interaction. An early response component is influenced by physical impact, reward environment, and novelty but does not fully code prediction error. Some dopamine neurons are activated by aversive stimuli, which may reflect physical stimulus impact or true aversiveness, but they do not seem to code general negative value or aversive prediction error. The reward prediction error signal is complemented by distinct, heterogeneous, smaller and slower changes reflecting sensory and motor contributors to behavioral activation, such as substantial movement (as opposed to precise motor control), reward expectation, spatial choice, vigor, and motivation. The different dopamine signals seem to defy a simple unifying concept and should be distinguished to better understand phasic dopamine functions.

Anhedonia following mild traumatic brain injury in rats: A behavioral economic analysis of positive and negative reinforcement

Psychiatric disorders affect nearly 50% of individuals who have experienced a traumatic brain injury (TBI). Anhedonia is a major symptom of numerous psychiatric disorders and is a diagnostic criterion for depression. It has recently been appreciated that reinforcement may be separated into consummatory (hedonic), motivational and decisional components, all of which may be affected differently in disease. Although anhedonia is typically assessed using positive reinforcement, the importance of stress in psychopathology suggests the study of negative reinforcement (removal or avoidance of aversive events) may be equally important. The present study investigated positive and negative reinforcement following a rat model of mild TBI (mTBI) using lateral fluid percussion. Hedonic value and motivation for reinforcement was determined by behavioral economic analyses. Following mTBI, the hedonic value of avoiding foot shock was reduced. In contrast, the hedonic value of escaping foot shock or obtaining a sucrose pellet was not altered by mTBI. Moreover, motivation to avoid or escape foot shock or to acquire sucrose was not altered by mTBI. Our results suggest that individuals experiencing mTBI find avoidance of aversive events less reinforcing, and therefore are less apt to utilize proactive control of stress.

Rewards interact with repetition-dependent learning to enhance long-term retention of motor memories

The combination of behavioral experiences that enhance long-term retention remains largely unknown. Informed by neurophysiological lines of work, this study tested the hypothesis that performance-contingent monetary rewards potentiate repetition-dependent forms of learning, as induced by extensive practice at asymptote, to enhance long-term retention of motor memories. To this end, six groups of 14 participants (n = 84) acquired novel motor behaviors by adapting to a gradual visuomotor rotation while these factors were manipulated. Retention was assessed 24 h later. While all groups similarly acquired the novel motor behaviors, results from the retention session revealed an interaction indicating that rewards enhanced long-term retention, but only when practice was extended to asymptote. Specifically, the interaction indicated that this effect selectively occurred when rewards were intermittently available (i.e., 50%), but not when they were absent (i.e., 0%) or continuously available (i.e., 100%) during acquisition. This suggests that the influence of rewards on extensive practice and long-term retention is nonlinear, as continuous rewards did not further enhance retention as compared with intermittent rewards. One possibility is that rewards' intermittent availability allows to maintain their subjective value during acquisition, which may be key to potentiate long-term retention.

Individual Differences in Dopamine Are Associated with Reward Discounting in Clinical Groups But Not in Healthy Adults

Some people are more willing to make immediate, risky, or costly reward-focused choices than others, which has been hypothesized to be associated with individual differences in dopamine (DA) function. In two studies using PET imaging, one empirical (Study 1: N = 144 males and females across 3 samples) and one meta-analytic (Study 2: N = 307 across 12 samples), we sought to characterize associations between individual differences in DA and time, probability, and physical effort discounting in human adults. Study 1 demonstrated that individual differences in DA D2-like receptors were not associated with time or probability discounting of monetary rewards in healthy humans, and associations with physical effort discounting were inconsistent across adults of different ages. Meta-analytic results for temporal discounting corroborated our empirical finding for minimal effect of DA measures on discounting in healthy individuals but suggested that associations between individual differences in DA and reward discounting depend on clinical features. Addictions were characterized by negative correlations between DA and discounting, but other clinical conditions, such as Parkinson's disease, obesity, and attention-deficit/hyperactivity disorder, were characterized by positive correlations between DA and discounting. Together, the results suggest that trait differences in discounting in healthy adults do not appear to be strongly associated with individual differences in D2-like receptors. The difference in meta-analytic correlation effects between healthy controls and individuals with psychopathology suggests that individual difference findings related to DA and reward discounting in clinical samples may not be reliably generalized to healthy controls, and vice versa.SIGNIFICANCE STATEMENT Decisions to forgo large rewards for smaller ones due to increasing time delays, uncertainty, or physical effort have been linked to differences in dopamine (DA) function, which is disrupted in some forms of psychopathology. It remains unclear whether alterations in DA function associated with psychopathology also extend to explaining associations between DA function and decision making in healthy individuals. We show that individual differences in DA D2 receptor availability are not consistently related to monetary discounting of time, probability, or physical effort in healthy individuals across a broad age range. By contrast, we suggest that psychopathology accounts for observed inconsistencies in the relationship between measures of DA function and reward discounting behavior.

The power of price compels you: Behavioral economic insights into dopamine-based valuation of rewarding and aversively motivated behavior

The mesocorticolimbic dopamine pathway is generally considered to be a reward pathway. While shortsighted, there is a strong basis for this view of dopamine function. Here, we first describe the role of phasic dopamine release events in reward seeking. We then explain why these release events are being reconsidered as value signals and how we applied behavioral economics to confirm they play a causal role in the valuation of reward. Just because dopamine release can function as a dopamine reward value signal however, does not imply that dopamine is solely a reward molecule. Rather, mesocorticolimbic dopamine appears to mediate many adaptive behaviors, including: reward seeking, avoidance, escape and fear-associated conditioned freezing. While more studies are needed before a consensus is reached on when, where and how dopamine mediates aversively-motivated behavior, its involvement is almost irrefutable. Thus, we next describe the role dopamine plays in these ethologically-relevant defensive behaviors. We conclude by describing our recent behavioral economics results that reveal a causal role for dopamine in the valuation of avoidance.

Granulocyte Colony Stimulating Factor Enhances Reward Learning through Potentiation of Mesolimbic Dopamine System Function

Deficits in motivation and cognition are hallmark symptoms of multiple psychiatric diseases. These symptoms are disruptive to quality of life and often do not improve with available medications. In recent years there has been increased interest in the role of the immune system in neuropsychiatric illness, but to date no immune-related treatment strategies have come to fruition. The cytokine granulocyte-colony stimulating factor (G-CSF) is known to have trophic and neuroprotective properties in the brain, and we recently identified it as a modulator of neuronal and behavioral plasticity. By combining operant tasks that assess discrete aspects of motivated behavior and decision-making in male mice and rats with subsecond dopamine monitoring via fast-scan cyclic voltammetry, we defined the role of G-CSF in these processes as well as the neural mechanism by which it modulates dopamine function to exert these effects. G-CSF enhanced motivation for sucrose as well as cognitive flexibility as measured by reversal learning. These behavioral outcomes were driven by mesolimbic dopamine system plasticity, as systemically administered G-CSF increased evoked dopamine release in the nucleus accumbens independent of clearance mechanisms. Importantly, sustained increases in G-CSF were required for these effects as acute exposure did not enhance behavioral outcomes and decreased dopamine release. These effects seem to be a result of the ability of G-CSF to alter local inflammatory signaling cascades, particularly tumor necrosis factor α. Together, these data show G-CSF as a potent modulator of the mesolimbic dopamine circuit and its ability to appropriately attend to salient stimuli.SIGNIFICANCE STATEMENT Emerging evidence has highlighted the importance of the immune system in psychiatric diseases states. However, the effects of peripheral cytokines on motivation and cognitive function are largely unknown. Here, we report that granulocyte-colony stimulating factor (G-CSF), a pleiotropic cytokine with known trophic and neuroprotective properties in the brain, acts directly on dopaminergic circuits to enhance their function. These changes in dopaminergic dynamics enhance reward learning and motivation for natural stimuli. Together, these results suggest that targeting immune factors may provide a new avenue for therapeutic intervention in the multiple psychiatric disorders that are characterized by motivational and cognitive deficits.

Heterogeneous dopamine signals support distinct features of motivated actions: implications for learning and addiction

Despite decades of research, investigations into effective neural and pharmacological therapies for many drugs of abuse, such as cocaine, have produced no FDA-approved approaches. This difficulty derives from the complexity of substance use disorders, which encompass a variety of behavioral, psychological, and neural circuit-based changes that occur as a result of repeated experience with the drug. Dopamine signaling has been demonstrated to play a key role in several aspects of drug abuse—from mediating its reinforcing properties and drug-seeking to triggering relapse—while also mediating a number of important aspects of normal (nondrug related) motivated behaviors and actions. Real-time recording methods such as in vivo voltammetry, electrophysiology, and calcium imaging demonstrate that the signaling properties of dopamine for motivationally relevant stimuli are highly dynamic and spatiotemporally circumscribed within afferent target regions. In this review, we identify the origins and functional consequences of heterogeneous dopamine release in the limbic system, and how these properties are persistently altered in the drug-experienced brain. We propose that these spatiotemporally parallel dopaminergic signals are simultaneously available to the animal, but that these circuits are impaired following prolonged drug experience by disrupting the location and content of dopamine signals in afferent target regions. These findings are discussed in the context of relapse and pathways to discovering new treatments for addiction disorders.

Beyond simple tests of value: measuring addiction as a heterogeneous disease of computation-specific valuation processes

Addiction is considered to be a neurobiological disorder of learning and memory because addiction is capable of producing lasting changes in the brain. Recovering addicts chronically struggle with making poor decisions that ultimately lead to relapse, suggesting a view of addiction also as a neurobiological disorder of decision-making information processing. How the brain makes decisions depends on how decision-making processes access information stored as memories in the brain. Advancements in circuit-dissection tools and recent theories in neuroeconomics suggest that neurally dissociable valuation processes access distinct memories differently, and thus are uniquely susceptible as the brain changes during addiction. If addiction is to be considered a neurobiological disorder of memory, and thus decision-making, the heterogeneity with which information is both stored and processed must be taken into account in addiction studies. Addiction etiology can vary widely from person to person. We propose that addiction is not a single disease, nor simply a disorder of learning and memory, but rather a collection of symptoms of heterogeneous neurobiological diseases of distinct circuit-computation-specific decision-making processes.

Mice learn to avoid regret

Regret can be defined as the subjective experience of recognizing that one has made a mistake and that a better alternative could have been selected. The experience of regret is thought to carry negative utility. This typically takes two distinct forms: augmenting immediate postregret valuations to make up for losses, and augmenting long-term changes in decision-making strategies to avoid future instances of regret altogether. While the short-term changes in valuation have been studied in human psychology, economics, neuroscience, and even recently in nonhuman-primate and rodent neurophysiology, the latter long-term process has received far less attention, with no reports of regret avoidance in nonhuman decision-making paradigms. We trained 31 mice in a novel variant of the Restaurant Row economic decision-making task, in which mice make decisions of whether to spend time from a limited budget to achieve food rewards of varying costs (delays). Importantly, we tested mice longitudinally for 70 consecutive days, during which the task provided their only source of food. Thus, decision strategies were interdependent across both trials and days. We separated principal commitment decisions from secondary reevaluation decisions across space and time and found evidence for regret-like behaviors following change-of-mind decisions that corrected prior economically disadvantageous choices. Immediately following change-of-mind events, subsequent decisions appeared to make up for lost effort by altering willingness to wait, decision speed, and pellet consumption speed, consistent with past reports of regret in rodents. As mice were exposed to an increasingly reward-scarce environment, we found they adapted and refined distinct economic decision-making strategies over the course of weeks to maximize reinforcement rate. However, we also found that even without changes in reinforcement rate, mice transitioned from an early strategy rooted in foraging to a strategy rooted in deliberation and planning that prevented future regret-inducing change-of-mind episodes from occurring. These data suggest that mice are learning to avoid future regret, independent of and separate from reinforcement rate maximization.

Regret describes a unique postdecision phenomenon in which losses are realized as a fault of one’s own actions. Regret is often hypothesized to have an inherent negative utility, and humans will often incur costs so as to avoid the risk of future regret. However, current models of nonhuman decision-making are based on reward maximization hypotheses. We recently found that rats express regret behaviorally and neurophysiologically on neuroeconomic foraging tasks; however, it remains unknown whether nonhuman animals will change strategies so as to avoid regret, even in the absence of changes in the achieved rate of reinforcement. Here, we provide the first evidence that mice change strategies to avoid future regret, independent of and separate from reinforcement rate maximization. Our data suggest mice accomplish this by shifting from a foraging decision-making strategy that produces change-of-mind decisions after investment mistakes to one rooted in deliberation that learns to plan ahead.

A Transient Dopamine Signal Represents Avoidance Value and Causally Influences the Demand to Avoid

While an extensive literature supports the notion that mesocorticolimbic dopamine plays a role in negative reinforcement, recent evidence suggests that dopamine exclusively encodes the value of positive reinforcement. In the present study, we employed a behavioral economics approach to investigate whether dopamine plays a role in the valuation of negative reinforcement. Using rats as subjects, we first applied fast-scan cyclic voltammetry (FSCV) to determine that dopamine concentration decreases with the number of lever presses required to avoid electrical footshock (i.e., the economic price of avoidance). Analysis of the rate of decay of avoidance demand curves, which depict an inverse relationship between avoidance and increasing price, allows for inference of the worth an animal places on avoidance outcomes. Rapidly decaying demand curves indicate increased price sensitivity, or low worth placed on avoidance outcomes, while slow rates of decay indicate reduced price sensitivity, or greater worth placed on avoidance outcomes. We therefore used optogenetics to assess how inducing dopamine release causally modifies the demand to avoid electrical footshock in an economic setting. Increasing release at an avoidance predictive cue made animals more sensitive to price, consistent with a negative reward prediction error (i.e., the animal perceives they received a worse outcome than expected). Increasing release at avoidance made animals less sensitive to price, consistent with a positive reward prediction error (i.e., the animal perceives they received a better outcome than expected). These data demonstrate that transient dopamine release events represent the value of avoidance outcomes and can predictably modify the demand to avoid.