A neural computational model of incentive salience

Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction

BACKGROUND

Drugs like opioids are potent reinforcers thought to co-opt value-based decisions by overshadowing other rewarding outcomes, but how this happens at a neurocomputational level remains elusive. Range adaptation is a canonical process of fine-tuning representations of value based on reward context. Here, we tested whether recent opioid exposure impacts range adaptation in opioid use disorder, potentially explaining why shifting decision making away from drug taking during this vulnerable period is so difficult.

METHODS

Participants who had recently (<90 days) used opioids (n = 34) or who had abstained from opioid use for ≥ 90 days (n = 20) and comparison control participants (n = 44) completed a reinforcement learning task designed to induce robust contextual modulation of value. Two models were used to assess the latent process that participants engaged while making their decisions: 1) a Range model that dynamically tracks context and 2) a standard Absolute model that assumes stationary, objective encoding of value.

RESULTS

Control participants and ≥90-days-abstinent participants with opioid use disorder exhibited choice patterns consistent with range-adapted valuation. In contrast, participants with recent opioid use were more prone to learn and encode value on an absolute scale. Computational modeling confirmed the behavior of most control participants and ≥90-days-abstinent participants with opioid use disorder (75%), but a minority in the recent use group (38%), was better fit by the Range model than the Absolute model. Furthermore, the degree to which participants relied on range adaptation correlated with duration of continuous abstinence and subjective craving/withdrawal.

CONCLUSIONS

Reduced context adaptation to available rewards could explain difficulty deciding about smaller (typically nondrug) rewards in the aftermath of drug exposure.

Striatal dopamine integrates cost, benefit, and motivation

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Craving money? Evidence from the laboratory and the field

Continuing to gamble despite harmful consequences has plagued human life in many ways, from loss-chasing in problem gamblers to reckless investing during stock market bubbles. Here, we propose that these anomalies in human behavior can sometimes reflect Pavlovian perturbations on instrumental behavior. To show this, we combined key elements of Pavlovian psychology literature and standard economic theory into a single model. In it, when a gambling cue such as a gaming machine or a financial asset repeatedly delivers a good outcome, the agent may start engaging with the cue even when the expected value is negative. Next, we transported the theoretical framework into an experimental task and found that participants behaved like the agent in our model. Last, we applied the model to the domain of real-world financial trading and discovered an asset-pricing anomaly suggesting that market participants are susceptible to the purported Pavlovian bias.

Contexts facilitate dynamic value encoding in the mesolimbic dopamine system

Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.

Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping

The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.

Reward prediction-errors weighted by cue salience produces addictive behaviours in simulations, with asymmetrical learning and steeper delay discounting

Dysfunction in learning and motivational systems are thought to contribute to addictive behaviours. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviours through pharmacological manipulations that provide excess dopaminergic signalling towards these learning and motivational systems. Redish (2004) suggested a role based on dopaminergic signals of value prediction error, while (Zhang et al., 2009) suggested a role based on dopaminergic signals of motivation. However, both models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviours, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviours. Here, we propose a novel mathematical definition of salience, that combines aspects of dopamine's role in both learning and motivation within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviours that the (Zhang et al., 2009; Redish, 2004) models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviours and aspects of behavioural/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviours by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.

Separating desire from prediction of outcome value

Individuals typically want what they expect to like, often based on memories of previous positive experiences. However, in some situations desire can decouple completely from memories and from learned predictions of outcome value. The potential for desire to separate from prediction arises from independent operating rules that control motivational incentive salience. Incentive salience, or 'wanting', is a type of mesolimbic desire that evolved for adaptive goals, but can also generate maladaptive addictions. Two proof-of-principle examples are presented here to show how motivational 'wanting' can soar above memory-based predictions of outcome value: (i) 'wanting what is remembered to be disgusting', and (ii) 'wanting what is predicted to hurt'. Consequently, even outcomes remembered and predicted to be negatively aversive can become positively 'wanted'. Similarly, in human addictions, people may experience powerful cue-triggered cravings for outcomes that are not predicted to be enjoyable.

Inhibition of dopamine neurons prevents incentive value encoding of a reward cue: With revelations from deep phenotyping

The survival of an organism is dependent on their ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the nucleus accumbens is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCut revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient towards and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.

Significance Statement

Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCut revealed that cue-directed behaviors do not emerge without VTA dopamine. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of VTA dopamine during cue presentation to encode the incentive value of reward cues.

Doesn't everybody jaywalk? On codified rules that are seldom followed and selectively punished

Rules are meant to apply equally to all within their jurisdiction. However, some rules are frequently broken without consequence for most. These rules are only occasionally enforced, often at the discretion of a third-party observer. We propose that these rules-whose violations are frequent, and enforcement is rare-constitute a unique subclass of explicitly codified rules, which we call 'phantom rules' (e.g., proscribing jaywalking). Their apparent punishability is ambiguous and particularly susceptible to third-party motives. Across six experiments, (N = 1440) we validated the existence of phantom rules and found evidence for their motivated enforcement. First, people played a modified Dictator Game with a novel frequently broken and rarely enforced rule (i.e., a phantom rule). People enforced this rule more often when the "dictator" was selfish (vs. fair) even though the rule only proscribed fractional offers (not selfishness). Then we turned to third person judgments of the U.S. legal system. We found these violations are recognizable to participants as both illegal and commonplace (Experiment 2), differentiable from violations of prototypical laws (Experiments 3) and enforced in a motivated way (Experiments 4a and 4b). Phantom rule violations (but not prototypical legal violations) are seen as more justifiably punished when the rule violator has also violated a social norm (vs. rule violation alone)-unless the motivation to punish has been satiated (Experiment 5). Phantom rules are frequently broken, codified rules. Consequently, their apparent punishability is ambiguous, and their enforcement is particularly susceptible to third party motives.

Behavioral regulation relies on interacting forces and predictive models

OBJECTIVE

We discuss how our recent neural network model of personality and motivation can explain many aspects of the regulation of behavior.

METHOD

Contrary to approaches that focus on a goal-corrected, set-point, and discrepancy-reducing mechanism, we argue that many aspects of regulation can be understood in terms of two other mechanisms. First, many aspects of the stability and coherence of personality, as well as the dynamics of personality, can be understood in terms of the interaction of forces within organized motivational systems, and their interaction with features of the environment and interoceptive states, that identify an individual's current needs. This has been described as a settling point or equilibrium of forces model, rather than a set-point architecture. Second, regulation has been shown to depend also on the use of predictive models of the world, either learned or innate. Such models can be thought of as feedforward models, in contrast to the feedback models characteristic of set-point, goal-corrected systems.

RESULTS AND CONCLUSIONS

We describe a neural network model of these processes that simulates the behavior over time and situations of an individual and shows how important regulatory processes can operate through a process of interactive forces and predictive models of the world.

Flexible control of Pavlovian-instrumental transfer based on expected reward value

The Pavlovian-instrumental transfer (PIT) paradigm is widely used to assay the motivational influence of reward-predictive cues, reflected by their ability to invigorate instrumental behavior. Leading theories assume that a cue's motivational properties are tied to predicted reward value. We outline an alternative view that recognizes that reward-predictive cues may suppress rather than motivate instrumental behavior under certain conditions, an effect termed positive conditioned suppression. We posit that cues signaling imminent reward delivery tend to inhibit instrumental behavior, which is exploratory by nature, in order to facilitate efficient retrieval of the expected reward. According to this view, the motivation to engage in instrumental behavior during a cue should be inversely related to the value of the predicted reward, since there is more to lose by failing to secure a high-value reward than a low-value reward. We tested this hypothesis in rats using a PIT protocol known to induce positive conditioned suppression. In Experiment 1, cues signaling different reward magnitudes elicited distinct response patterns. Whereas the one-pellet cue increased instrumental behavior, cues signaling three or nine pellets suppressed instrumental behavior and elicited high levels of food-port activity. Experiment 2 found that reward-predictive cues suppressed instrumental behavior and increased food-port activity in a flexible manner that was disrupted by post-training reward devaluation. Further analyses suggest that these findings were not driven by overt competition between the instrumental and food-port responses. We discuss how the PIT task may provide a useful tool for studying cognitive control over cue-motivated behavior in rodents. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

"Wanting" versus "needing" related value: An fMRI meta-analysis

Consumption and its excesses are sometimes explained by imbalance of need or lack of control over "wanting." "Wanting" assigns value to cues that predict rewards, whereas "needing" assigns value to biologically significant stimuli that one is deprived of. Here we aimed at studying how the brain activation patterns related to value of "wanted" stimuli differs from that of "needed" stimuli using activation likelihood estimation neuroimaging meta-analysis approaches. We used the perception of a cue predicting a reward for "wanting" related value and the perception of food stimuli in a hungry state as a model for "needing" related value. We carried out separate, contrasts, and conjunction meta-analyses to identify differences and similarities between "wanting" and "needing" values. Our overall results for "wanting" related value show consistent activation of the ventral tegmental area, striatum, and pallidum, regions that both activate behavior and direct choice, while for "needing" related value, we found an overall consistent activation of the middle insula and to some extent the caudal-ventral putamen, regions that only direct choice. Our study suggests that wanting has more control on consumption and behavioral activation.

Chemogenetic Disconnection between the Orbitofrontal Cortex and the Rostromedial Caudate Nucleus Disrupts Motivational Control of Goal-Directed Action

The orbitofrontal cortex (OFC) and its major downstream target within the basal ganglia-the rostromedial caudate nucleus (rmCD)-are involved in reward-value processing and goal-directed behavior. However, a causal contribution of the pathway linking these two structures to goal-directed behavior has not been established. Using the chemogenetic technology of designer receptors exclusively activated by designer drugs with a crossed inactivation design, we functionally and reversibly disrupted interactions between the OFC and rmCD in two male macaque monkeys. We injected an adeno-associated virus vector expressing an inhibitory designer receptor, hM4Di, into the OFC and contralateral rmCD, the expression of which was visualized in vivo by positron emission tomography and confirmed by postmortem immunohistochemistry. Functional disconnection of the OFC and rmCD resulted in a significant and reproducible loss of sensitivity to the cued reward value for goal-directed action. This decreased sensitivity was most prominent when monkeys had accumulated a certain amount of reward. These results provide causal evidence that the interaction between the OFC and the rmCD is needed for motivational control of action on the basis of the relative reward value and internal drive. This finding extends the current understanding of the physiological basis of psychiatric disorders in which goal-directed behavior is affected, such as obsessive-compulsive disorder.SIGNIFICANCE STATEMENT In daily life, we routinely adjust the speed and accuracy of our actions on the basis of the value of expected reward. Abnormalities in these kinds of motivational adjustments might be related to behaviors seen in psychiatric disorders such as obsessive-compulsive disorder. In the current study, we show that the connection from the orbitofrontal cortex to the rostromedial caudate nucleus is essential for motivational control of action in monkeys. This finding expands our knowledge about how the primate brain controls motivation and behavior and provides a particular insight into disorders like obsessive-compulsive disorder in which altered connectivity between the orbitofrontal cortex and the striatum has been implicated.

Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite

Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.

Food wanting is mediated by transient activation of dopaminergic signaling in the honey bee brain

The biological bases of wanting have been characterized in mammals, but whether an equivalent wanting system exists in insects remains unknown. In this study, we focused on honey bees, which perform intensive foraging activities to satisfy colony needs, and sought to determine whether foragers leave the hive driven by specific expectations about reward and whether they recollect these expectations during their waggle dances. We monitored foraging and dance behavior and simultaneously quantified and interfered with biogenic amine signaling in the bee brain. We show that a dopamine-dependent wanting system is activated transiently in the bee brain by increased appetite and individual recollection of profitable food sources, both en route to the goal and during waggle dances. Our results show that insects share with mammals common neural mechanisms for encoding wanting of stimuli with positive hedonic value.

A review of mathematical modeling of addiction regarding both (neuro-) psychological processes and the social contagion perspectives

Addiction is a complex biopsychosocial phenomenon, impacted by biological predispositions, psychological processes, and the social environment. Using mathematical and computational models that allow for surrogative reasoning may be a promising avenue for gaining a deeper understanding of this complex behavior. This paper reviews and classifies a selection of formal models of addiction focusing on the intra- and inter-individual dynamics, i.e., (neuro) psychological models and social models. We find that these modeling approaches to addiction are too disjoint and argue that in order to unravel the complexities of biopsychosocial processes of addiction, models should integrate intra- and inter-individual factors.

A mathematical model of reward-mediated learning in drug addiction

Substances of abuse are known to activate and disrupt neuronal circuits in the brain reward system. We propose a simple and easily interpretable dynamical systems model to describe the neurobiology of drug addiction that incorporates the psychiatric concepts of reward prediction error, drug-induced incentive salience, and opponent process theory. Drug-induced dopamine releases activate a biphasic reward response with pleasurable, positive "a-processes" (euphoria, rush) followed by unpleasant, negative "b-processes" (cravings, withdrawal). Neuroadaptive processes triggered by successive intakes enhance the negative component of the reward response, which the user compensates for by increasing drug dose and/or intake frequency. This positive feedback between physiological changes and drug self-administration leads to habituation, tolerance, and, eventually, to full addiction. Our model gives rise to qualitatively different pathways to addiction that can represent a diverse set of user profiles (genetics, age) and drug potencies. We find that users who have, or neuroadaptively develop, a strong b-process response to drug consumption are most at risk for addiction. Finally, we include possible mechanisms to mitigate withdrawal symptoms, such as through the use of methadone or other auxiliary drugs used in detoxification.

A Computational View on the Nature of Reward and Value in Anhedonia

Anhedonia - a common feature of depression and other neuropsychiatric disorders - encompasses a reduction in the subjective experience and anticipation of rewarding events, and a reduction in the motivation to seek out such events. The presence of anhedonia often predicts or accompanies treatment resistance, and as such better interventions and treatments are important. Yet the mechanisms giving rise to anhedonia are not well understood. In this chapter, we briefly review existing computational conceptualisations of anhedonia. We argue that they are mostly descriptive and fail to provide an explanatory account of why anhedonia may occur. Working within the framework of reinforcement learning, we examine two potential computational mechanisms that could give rise to anhedonic phenomena. First, we show how anhedonia can arise in multi-dimensional drive-reduction settings through a trade-off between different rewards or needs. We then generalise this in terms of model-based value inference and identify a key role for associational belief structure. We close with a brief discussion of treatment implications of both of these conceptualisations. In summary, computational accounts of anhedonia have provided a useful descriptive framework. Recent advances in reinforcement learning suggest promising avenues by which the mechanisms underlying anhedonia may be teased apart, potentially motivating novel approaches to treatment.

Quantifying the instrumental and noninstrumental underpinnings of Pavlovian responding with the Price equation

The Price equation is a mathematical expression of selectionist and non-selectionist pressures on biological, cultural, and behavioral change. We use it here to specify instrumental and noninstrumental behaviors as they arise within the context of the Pavlovian autoshaping procedure, for rats trained under reward certainty and reward uncertainty. The point of departure for this endeavor is that some portion of autoshaped behavior referred to as goal-tracking appears instrumental-a function of resource attainment (the individual approaches the location where the unconditioned stimulus is to be delivered). By contrast, some other portion of autoshaped behavior referred to as sign-tracking is noninstrumental-irrelevant to making contact with the to-be-delivered unconditioned stimulus. A Price equation model is proposed that unifies our understanding of Pavlovian autoshaping behavior by isolating operant and respondent influences on goal-tracking (instrumental) and sign-tracking (noninstrumental) behavior.

Modeling incentive salience in Pavlovian learning more parsimoniously using a multiple attribute model

We present a multi-attribute incentive salience (MAIS) model as a computational account of incentive salience in model-based Pavlovian learning. A model of incentive salience as a joint function of reward value and physiological state has been previously proposed by Zhang et al. (2009). In that model, the function takes additive or multiplicative forms depending on whether a preference shifts from positive to negative or vice versa. We demonstrate that arbitrarily varying this function is unnecessary to explain observed data. A multiplicative function is sufficient if one takes into account empirical data suggesting the incentive salience function for an incentive is comprised of multiple physiological signals. We compare our model to the previously proposed model on two datasets. We find the MAIS model predicts the outcomes equally well, fits empirical data describing multiple sensory representations of a single stimulus, better approximates the dual-structure appetitive-aversive nature of the reward system, is compatible with existing knowledge about incentive salience in Pavlovian learning, and better describes revaluation in Pavlovian learning. This model addresses a call (Dayan & Berridge, 2014) for algorithmic and computational models of model-based Pavlovian learning that consistently and fully explain empirical observations. Because a multi-attribute model is relevant even for simple Pavlovian associations, it should be useful in a wide variety of decision-making contexts, including agent modeling and addiction research.

Range-Adaptive Value Representation in Different Stages of Schizophrenia: A Proof of Concept Study

Amotivation is related to value representation. A comprehensive account of amotivation requires a mechanistic understanding of how the brain exploits external information to represent value. To achieve maximal value discriminability, brain valuation system will dynamically adapt its coding sensitivity to the range of values available in any given condition, so-called range adaptive coding. We administered an experimental task to 30 patients with chronic schizophrenia (C-SCZ), 30 first-episode schizophrenia (FE-SCZ), 34 individuals with high social anhedonia (HSoA), and their paired controls to assess range adaptation ability. C-SCZ patients exhibited over-adaptation and their performances were negatively correlated with avolition symptoms and positive symptoms and positively correlated with blunted-affect symptoms and self-reported consummatory interpersonal pleasure scores, though the results were non-significant. FE-SCZ patients exhibited reduced adaptation, which was significantly and negatively correlated with avolition symptoms and positively correlated with the overall proportion of choosing to exert more effort. Although HSoA participants exhibited comparable range adaptation to controls, their performances were significantly and negatively correlated with the proportion of choosing to exert more effort under the lowest value condition. Our results suggest that different stages of schizophrenia spectrum showed distinct range adaptation patterns. Range adaptation impairments may index a possible underlying mechanism for amotivation symptoms in FE-SCZ and more complicated and pervasive effects on clinical symptoms in C-SCZ.

Effort-motivated behavior resolves paradoxes in appetitive conditioning

Motivated behavior has long been studied by psychologists, ethologists, and neuroscientists. To date, many scientists agree with the view that cue and reward attraction is the product of a dopamine-dependent unconscious process called incentive salience or "wanting". This process allows the influence of multiple factors such as hunger and odors on motivational attraction. In some cases, however, the resulting motivated behavior differs from what the incentive salience hypothesis would predict. I argue that seeking behavior under reward uncertainty illustrates this situation: Organisms do not just "want" (appetite-based attraction) cues that are inconsistent or associated with reward occasionally, they "hope" that those cues will consistently predict reward procurement in the ongoing trial. Said otherwise, they become motivated to invest time and energy to find consistent cue-reward associations despite no guarantee of success (effort-based attraction). A multi-test comparison of performance between individuals trained under uncertainty and certainty reveals behavioral paradoxes suggesting that the concept of incentive salience cannot fully account for responding to inconsistent cues. A mathematical model explains how appetite-based and effort-based attractions might combine their effects.

Single caudate neurons encode temporally discounted value for formulating motivation for action

The term 'temporal discounting' describes both choice preferences and motivation for delayed rewards. Here we show that neuronal activity in the dorsal part of the primate caudate head (dCDh) signals the temporally discounted value needed to compute the motivation for delayed rewards. Macaque monkeys performed an instrumental task, in which visual cues indicated the forthcoming size and delay duration before reward. Single dCDh neurons represented the temporally discounted value without reflecting changes in the animal's physiological state. Bilateral pharmacological or chemogenetic inactivation of dCDh markedly distorted the normal task performance based on the integration of reward size and delay, but did not affect the task performance for different reward sizes without delay. These results suggest that dCDh is involved in encoding the integrated multi-dimensional information critical for motivation.

The interpretation of behavior-model correlations in unidentified cognitive models

The rise of computational modeling in the past decade has led to a substantial increase in the number of papers that report parameter estimates of computational cognitive models. A common application of computational cognitive models is to quantify individual differences in behavior by estimating how these are expressed in differences in parameters. For these inferences to hold, models need to be identified, meaning that one set of parameters is most likely, given the behavior under consideration. For many models, model identification can be achieved up to a scaling constraint, which means that under the assumption that one parameter has a specific value, all remaining parameters are identified. In the current note, we argue that this scaling constraint implies a strong assumption about the cognitive process that the model is intended to explain, and warn against an overinterpretation of the associative relations found in this way. We will illustrate these points using signal detection theory, reinforcement learning models, and the linear ballistic accumulator model, and provide suggestions for a clearer interpretation of modeling results.

Incentive motivation: 'wanting' roles of central amygdala circuitry

The central nucleus of amygdala (CeA) mediates positively-valenced reward motivation as well as negatively-valenced fear. Optogenetic or neurochemical stimulation of CeA circuitry can generate intense incentive motivation to pursue and consume a paired natural food, sex, or addictive drug reward, and even create maladaptive 'wanting what hurts' such as attraction to a shock rod. Evidence indicates CeA stimulations selectively amplify incentive motivation ('wanting') but not hedonic impact ('liking') of the same reward. Further, valence flips can occur for CeA contributions to motivational salience. That is, CeA stimulation can promote either incentive motivation or fearful motivation, even in the same individual, depending on situation. These findings may carry implications for understanding CeA roles in neuropsychiatric disorders involving aberrant motivational salience, ranging from addiction to paranoia and anxiety disorders.

Alterations in reward network functional connectivity are associated with increased food addiction in obese individuals

Functional neuroimaging studies in obesity have identified alterations in the connectivity within the reward network leading to decreased homeostatic control of ingestive behavior. However, the neural mechanisms underlying sex differences in the prevalence of food addiction in obesity is unknown. The aim of the study was to identify functional connectivity alterations associated with: (1) Food addiction, (2) Sex- differences in food addiction, (3) Ingestive behaviors. 150 participants (females: N = 103, males: N = 47; food addiction: N = 40, no food addiction: N = 110) with high BMI ≥ 25 kg/m2 underwent functional resting state MRIs. Participants were administered the Yale Food Addiction Scale (YFAS), to determine diagnostic criteria for food addiction (YFAS Symptom Count ≥ 3 with clinically significant impairment or distress), and completed ingestive behavior questionnaires. Connectivity differences were analyzed using a general linear model in the CONN Toolbox and images were segmented using the Schaefer 400, Harvard-Oxford Subcortical, and Ascending Arousal Network atlases. Significant connectivities and clinical variables were correlated. Statistical significance was corrected for multiple comparisons at q < .05. (1) Individuals with food addiction had greater connectivity between brainstem regions and the orbital frontal gyrus compared to individuals with no food addiction. (2) Females with food addiction had greater connectivity in the salience and emotional regulation networks and lowered connectivity between the default mode network and central executive network compared to males with food addiction. (3) Increased connectivity between regions of the reward network was positively associated with scores on the General Food Cravings Questionnaire-Trait, indicative of greater food cravings in individuals with food addiction. Individuals with food addiction showed greater connectivity between regions of the reward network suggesting dysregulation of the dopaminergic pathway. Additionally, greater connectivity in the locus coeruleus could indicate that the maladaptive food behaviors displayed by individuals with food addiction serve as a coping mechanism in response to pathological anxiety and stress. Sex differences in functional connectivity suggest that females with food addiction engage more in emotional overeating and less cognitive control and homeostatic processing compared to males. These mechanistic pathways may have clinical implications for understanding the sex-dependent variability in response to diet interventions.

Neural Networks With Motivation

Animals rely on internal motivational states to make decisions. The role of motivational salience in decision making is in early stages of mathematical understanding. Here, we propose a reinforcement learning framework that relies on neural networks to learn optimal ongoing behavior for dynamically changing motivation values. First, we show that neural networks implementing Q-learning with motivational salience can navigate in environment with dynamic rewards without adjustments in synaptic strengths when the needs of an agent shift. In this setting, our networks may display elements of addictive behaviors. Second, we use a similar framework in hierarchical manager-agent system to implement a reinforcement learning algorithm with motivation that both infers motivational states and behaves. Finally, we show that, when trained in the Pavlovian conditioning setting, the responses of the neurons in our model resemble previously published neuronal recordings in the ventral pallidum, a basal ganglia structure involved in motivated behaviors. We conclude that motivation allows Q-learning networks to quickly adapt their behavior to conditions when expected reward is modulated by agent's dynamic needs. Our approach addresses the algorithmic rationale of motivation and makes a step toward better interpretability of behavioral data via inference of motivational dynamics in the brain.

Momentary subjective well-being depends on learning and not reward

Subjective well-being or happiness is often associated with wealth. Recent studies suggest that momentary happiness is associated with reward prediction error, the difference between experienced and predicted reward, a key component of adaptive behaviour. We tested subjects in a reinforcement learning task in which reward size and probability were uncorrelated, allowing us to dissociate between the contributions of reward and learning to happiness. Using computational modelling, we found convergent evidence across stable and volatile learning tasks that happiness, like behaviour, is sensitive to learning-relevant variables (i.e. probability prediction error). Unlike behaviour, happiness is not sensitive to learning-irrelevant variables (i.e. reward prediction error). Increasing volatility reduces how many past trials influence behaviour but not happiness. Finally, depressive symptoms reduce happiness more in volatile than stable environments. Our results suggest that how we learn about our world may be more important for how we feel than the rewards we actually receive.

Many people believe they would be happier if only they had more money. And events such as winning the lottery or receiving a large pay rise do make people happy, at least temporarily. But recent studies suggest that the main factor driving happiness on such occasions is not the size of the reward received. Instead, it is how well that reward matches up with expectations. Receiving a 10% pay rise when you were expecting 1% will make you feel happier than receiving 10% when you had been expecting 20%.

This difference between an expected and an actual reward is referred to as a reward prediction error. Reward prediction errors have a key role in learning. They motivate people to repeat behaviours that led to unexpectedly large rewards. But they also enable people to update their beliefs about the world, which is rewarding in itself. Could it be that reward prediction errors are associated with happiness mainly because they help us understand the world a little better than before?

To test this idea, Blain and Rutledge designed a task in which the likelihood of receiving a reward was unrelated to the size of the reward. This study design makes it possible to separate out the contributions of learning versus reward to moment-by-moment happiness.

In the task, volunteers had to decide which of two cars would win a race. In the ‘stable’ condition, one of the cars always had an 80% chance of winning. In the ‘volatile’ condition, one car had an 80% chance of winning for the first 20 trials. The other car then had an 80% chance of winning for the next 20 trials. The volunteers were not told these probabilities in advance, but had to work them out by playing the game. However, on every trial, the volunteers were shown the reward they would receive if they chose either of the cars and that car went on to win. The size of the rewards varied at random and was unrelated to the likelihood of a car winning.

Every few trials, the volunteers were asked to indicate their current level of happiness on a scale. The results showed that volunteers were happier after winning than after losing. On average they were also happier in the stable condition than in the volatile condition. This was especially true for volunteers with pre-existing symptoms of depression. Moreover, happiness after wins did not depend on how large the reward they got was, but instead simply on how surprised they were to win.

These results suggest that how we learn about the world around us can be more important for how we feel than rewards we receive directly. Measuring happiness in various types of environment could help us understand factors affecting mental health. The current results suggest, for example, that uncertain environments may be especially unpleasant for people with depression. Further research is needed to understand why this might be the case. In the real world, rewards are often uncertain and infrequent, but learning may nevertheless have the potential to boost happiness.

Reward activity in ventral pallidum tracks satiety-sensitive preference and drives choice behavior

A key function of the nervous system is producing adaptive behavior across changing conditions, like physiological state. Although states like thirst and hunger are known to impact decision-making, the neurobiology of this phenomenon has been studied minimally. Here, we tracked evolving preference for sucrose and water as rats proceeded from a thirsty to sated state. As rats shifted from water choices to sucrose choices across the session, the activity of a majority of neurons in the ventral pallidum, a region crucial for reward-related behaviors, closely matched the evolving behavioral preference. The timing of this signal followed the pattern of a reward prediction error, occurring at the cue or the reward depending on when reward identity was revealed. Additionally, optogenetic stimulation of ventral pallidum neurons at the time of reward was able to reverse behavioral preference. Our results suggest that ventral pallidum neurons guide reward-related decisions across changing physiological states.

Neuroanatomical Abnormalities in Older Depressed Adults With Apathy: A Systematic Review

OBJECTIVE

Apathy is a common phenomenon in late-life depression and is associated with poor outcomes. Apathy is often unrecognized in older depressed adults, and efficacious treatment options are lacking. This review provides a systematic review of the neuroanatomical abnormalities associated with apathy in late-life depression. In addition, the review summarizes the neuroimaging findings from studies of neurodegenerative and focal brain injury conditions that frequently present with apathy. The goal is to elucidate cerebral network abnormalities that give rise to apathy in older adults with mood disturbances and to inform future treatment targets.

METHOD

Systematic literature review.

RESULTS

The few studies that have directly examined the neuroanatomical abnormalities of apathy in late-life depression suggest disturbances in the anterior cingulate cortex, insula, orbital and dorsal prefrontal cortex, striatum, and limbic structures (ie, amygdala, thalamus, and hippocampus). Studies examining the neuroanatomical correlates of apathy in other aging populations are consistent with the pattern observed in late-life depression.

CONCLUSIONS

Apathy in late-life depression appears to be accompanied by neuroanatomical abnormalities in the salience and reward networks. These network findings are consistent with that observed in individuals presenting with apathy in other aging-related conditions. These findings may inform future treatments that target apathy.

Ventral Caudate and Anterior Insula Recruitment During Value Estimation of Passionate Rewarding Cues

“Wanting”, a component of reward processing, is a motivational property that guides decision making in goal-oriented behavior. This includes behavior aiming at supporting relational bonds, even at the group level. Accordingly, group belongingness works as this motivational property, which is fundamentally different from romantic or maternal love. While primary rewards (or learned associations, such as money) have been largely used to study the conceptual framework associated with “wanting,” other cues triggering behavior, such as passionate motives, are less well-studied. We investigated the neural correlates of value estimation of a passion-driven incentive in neuropsychologically defined football fans. We asked the participants (n = 57) to compute the value of football tickets (the cues that trigger passionate behavior in this “tribal love” context). The trials were all different, comprising tickets for different matches. The participants had no restrictions on the amount to be introduced. This enabled a parametric functional magnetic resonance imaging design based on the explicit estimated value given by the participants in a trial-by-trial approach. Using a whole-brain approach (to prevent biased focus on value-related regions), only the activity in the ventral caudate and left anterior insula showed a critical relationship with the reported value. Higher normalized values led to more activity in the striatum and left insula. The parametric map shows that these regions encode the magnitude of incentive by indexing self-relevant value. Other regions were involved in value computation, such as the ventromedial prefrontal cortex, lateral orbitofrontal cortex, and dorsolateral prefrontal cortex, but did not exhibit parametric patterns. The involvement of the nucleus accumbens in value estimation was only found in region of interest -based analysis, which emphasizes the role of the ventral caudate for the presently studied social “reinforcer” cue.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Subjective reward value of visual sexual stimuli is coded in human striatum and orbitofrontal cortex

Human neuroimaging research suggests the existence of one core network for the subjective valuation of rewards, including the striatum and orbitofrontal cortex. However, there is little research on the neural representation of subjective reward values of visual sexual stimuli (VSS) and on the role of these subjective valuations in the development of related addictive behaviors. Here, we investigate how neural reactivity to VSS is connected to individual preference using functional magnetic resonance imaging (fMRI). During the fMRI scan, 72 men viewed different VSS film clips. Ratings regarding valence and sexual arousal were collected and used as parametric modulators in the fMRI analysis. Subjects also filled out questionnaires on self-reported symptoms of problematic pornography use (PPU). Firstly, we found that neural reactivity towards VSS clips in the nucleus accumbens, caudate nucleus and orbitofrontal cortex was positively correlated with individual ratings of the respective VSS in all subjects. Second, the strength of the association between neural activity and sexual arousal ratings was positively correlated with self-reported symptoms of PPU. The first result suggests a precise appraisal of VSS according to individual preferences in established reward valuation regions. Secondly, stronger neural differentiation based on preference in participants with more PPU symptoms indicates an increased importance of VSS/preference fit in these individuals. This heightened correspondence between individual liking and neural activity may facilitate PPU development by increased signaling of incentive salience, thus boosting motivation to seek out and respond to these preferred stimuli.

Suboptimal choice by pigeons is eliminated when key-pecking behavior is replaced by treadle-pressing

In the study of suboptimal choice, a reliable result is that pigeons strongly prefer an alternative that signals whether a reinforcer will be delivered or not over another alternative without that information even if the first provides a lower probability of reinforcement. In the aforementioned research, key pecking has been the operant response and illuminated keys the discriminative stimuli. In the present study we modified both of these aspects of the procedure in order to analyze the generality of suboptimal preferences of pigeons and to investigate the effect of changes in the incentive salience of the discriminative stimuli. To accomplish this, we presented pigeons a choice situation with the same parameters of reinforcement than previous research, but with treadle pressing as the choice response and ambient lights as discriminative stimuli. Under these conditions, most of the pigeons showed optimal behavior and a high degree of discrimination of the stimuli associated with the discriminative alternative. A control condition with key pecking as choice response and keylights as discriminative stimuli showed that the same pigeons turned to be suboptimal, a result that discards the possibility that the optimality found in the main condition was a consequence of a particular characteristic of our sample of subjects or of our procedure. We discuss the influence that the attribution of incentive salience to the discriminative stimuli has on suboptimal choice in both pigeons and rats.

The central amygdala recruits mesocorticolimbic circuitry for pursuit of reward or pain

How do brain mechanisms create maladaptive attractions? Here intense maladaptive attractions are created in laboratory rats by pairing optogenetic channelrhodopsin (ChR2) stimulation of central nucleus of amygdala (CeA) in rats with encountering either sucrose, cocaine, or a painful shock-delivering object. We find that pairings make the respective rats pursue either sucrose exclusively, or cocaine exclusively, or repeatedly self-inflict shocks. CeA-induced maladaptive attractions, even to the painful shock-rod, recruit mesocorticolimbic incentive-related circuitry. Shock-associated cues also gain positive incentive value and are pursued. Yet the motivational effects of paired CeA stimulation can be reversed to negative valence in a Pavlovian fear learning situation, where CeA ChR2 pairing increases defensive reactions. Finally, CeA ChR2 valence can be switched to neutral by pairing with innocuous stimuli. These results reveal valence plasticity and multiple modes for motivation via mesocorticolimbic circuitry under the control of CeA activation.

Modeling the effects of motivation on choice and learning in the basal ganglia

Decision making relies on adequately evaluating the consequences of actions on the basis of past experience and the current physiological state. A key role in this process is played by the basal ganglia, where neural activity and plasticity are modulated by dopaminergic input from the midbrain. Internal physiological factors, such as hunger, scale signals encoded by dopaminergic neurons and thus they alter the motivation for taking actions and learning. However, to our knowledge, no formal mathematical formulation exists for how a physiological state affects learning and action selection in the basal ganglia. We developed a framework for modelling the effect of motivation on choice and learning. The framework defines the motivation to obtain a particular resource as the difference between the desired and the current level of this resource, and proposes how the utility of reinforcements depends on the motivation. To account for dopaminergic activity previously recorded in different physiological states, the paper argues that the prediction error encoded in the dopaminergic activity needs to be redefined as the difference between utility and expected utility, which depends on both the objective reinforcement and the motivation. We also demonstrate a possible mechanism by which the evaluation and learning of utility of actions can be implemented in the basal ganglia network. The presented theory brings together models of learning in the basal ganglia with the incentive salience theory in a single simple framework, and it provides a mechanistic insight into how decision processes and learning in the basal ganglia are modulated by the motivation. Moreover, this theory is also consistent with data on neural underpinnings of overeating and obesity, and makes further experimental predictions.

The role of the gut microbiome in opioid use

Although the gut and brain are separate organs, they communicate with each other via trillions of intestinal bacteria that collectively make up one's gut microbiome. Findings from both humans and animals support a critical role of gut microbes in regulating brain function, mood, and behavior. Gut bacteria influence neural circuits that are notably affected in addiction-related behaviors. These include circuits involved in stress, reward, and motivation, with substance use influencing gut microbial abnormalities, suggesting significant gut-brain interactions in drug addiction. Given the overwhelming rates of opioid overdose deaths driven by abuse and addiction, it is essential to characterize mechanisms mediating the abuse potential of opioids. We discuss in this review the role of gut microbiota in factors that influence opioid addiction, including incentive salience, reward, tolerance, withdrawal, stress, and compromised executive function. We present clinical and preclinical evidence supporting a bidirectional relationship between gut microbiota and opioid-related behaviors by highlighting the effects of opioid use on gut bacteria, and the effects of gut bacteria on behavioral responses to opioids. Further, we discuss possible mechanisms of this gut-brain communication influencing opioid use. By clarifying the relationship between the gut microbiome and opioid-related behaviors, we improve understanding on mechanisms mediating reward-, motivation-, and stress-related behaviors and disorders, which may contribute to the development of effective, targeted therapeutic interventions in opioid dependence and addiction.

Neural circuits linking sleep and addiction: Animal models to understand why select individuals are more vulnerable to substance use disorders after sleep deprivation

Individuals differ widely in their drug-craving behaviors. One reason for these differences involves sleep. Sleep disturbances lead to an increased risk of substance use disorders and relapse in only some individuals. While animal studies have examined the impact of sleep on reward circuitry, few have addressed the role of individual differences in the effects of altered sleep. There does, however, exist a rodent model of individual differences in reward-seeking behavior: the sign/goal-tracker model of Pavlovian conditioned approach. In this model, only some rats show the key behavioral traits associated with addiction, including impulsivity and poor attentional control, making this an ideal model system to examine individually distinct sleep-reward interactions. Here, we describe how the limbic neural circuits responsible for individual differences in incentive motivation overlap with those involved in sleep-wake regulation, and how this model can elucidate the common underlying mechanisms. Consideration of individual differences in preclinical models would improve our understanding of how sleep interacts with motivational systems, and why sleep deprivation contributes to addiction in only select individuals.

Predicting Behavior With Implicit Measures: Disillusioning Findings, Reasonable Explanations, and Sophisticated Solutions

Two decades ago, the introduction of the Implicit Association Test (IAT) sparked enthusiastic reactions. With implicit measures like the IAT, researchers hoped to finally be able to bridge the gap between self-reported attitudes on one hand and behavior on the other. Twenty years of research and several meta-analyses later, however, we have to conclude that neither the IAT nor its derivatives have fulfilled these expectations. Their predictive value for behavioral criteria is weak and their incremental validity over and above self-report measures is negligible. In our review, we present an overview of explanations for these unsatisfactory findings and delineate promising ways forward. Over the years, several reasons for the IAT's weak predictive validity have been proposed. They point to four potentially problematic features: First, the IAT is by no means a pure measure of individual differences in associations but suffers from extraneous influences like recoding. Hence, the predictive validity of IAT-scores should not be confused with the predictive validity of associations. Second, with the IAT, we usually aim to measure evaluation ("liking") instead of motivation ("wanting"). Yet, behavior might be determined much more often by the latter than the former. Third, the IAT focuses on measuring associations instead of propositional beliefs and thus taps into a construct that might be too unspecific to account for behavior. Finally, studies on predictive validity are often characterized by a mismatch between predictor and criterion (e.g., while behavior is highly context-specific, the IAT usually takes into account neither the situation nor the domain). Recent research, however, also revealed advances addressing each of these problems, namely (1) procedural and analytical advances to control for recoding in the IAT, (2) measurement procedures to assess implicit wanting, (3) measurement procedures to assess implicit beliefs, and (4) approaches to increase the fit between implicit measures and behavioral criteria (e.g., by incorporating contextual information). Implicit measures like the IAT hold an enormous potential. In order to allow them to fulfill this potential, however, we have to refine our understanding of these measures, and we should incorporate recent conceptual and methodological advancements. This review provides specific recommendations on how to do so.

Wait and you shall see: sexual delay discounting in hypersexual Parkinson's disease

Patients with Parkinson's disease may develop impulse control disorders under dopaminergic treatments. Impulse control disorders include a wide spectrum of behaviours, such as hypersexuality, pathological gambling or compulsive shopping. Yet, the neural systems engaged in specific impulse control disorders remain poorly characterized. Here, using model-based functional MRI, we aimed to determine the brain systems involved during delay-discounting of erotic rewards in hypersexual patients with Parkinson's disease (PD+HS), patients with Parkinson's disease without hypersexuality (PD - HS) and controls. Patients with Parkinson's disease were evaluated ON and OFF levodopa (counterbalanced). Participants had to decide between two options: (i) wait for 1.5 s to briefly view an erotic image; or (ii) wait longer to see the erotic image for a longer period of time. At the time of decision-making, we investigated which brain regions were engaged with the subjective valuation of the delayed erotic reward. At the time of the rewarded outcome, we searched for the brain regions responding more robustly after waiting longer to view the erotic image. PD+HS patients showed reduced discounting of erotic delayed rewards, compared to both patients with Parkinson's disease and controls, suggesting that they accepted waiting longer to view erotic images for a longer period of time. Thus, when using erotic stimuli that motivate PD+HS, these patients were less impulsive for the immediate reward. At the brain system level, this effect was paralleled by the fact that PD+HS, as compared to controls and PD - HS, showed a negative correlation between subjective value of the delayed reward and activity of medial prefrontal cortex and ventral striatum. Consistent with the incentive salience hypothesis combining learned cue-reward associations with current relevant physiological state, dopaminergic treatment in PD+HS boosted excessive 'wanting' of rewards and heightened activity in the anterior medial prefrontal cortex and the posterior cingulate cortex, as reflected by higher correlation with subjective value of the option associated to the delayed reward when ON medication as compared to the OFF medication state. At the time of outcome, the anterior medial prefrontal/rostral anterior cingulate cortex showed an interaction between group (PD+HS versus PD - HS) and medication (ON versus OFF), suggesting that dopaminergic treatment boosted activity of this brain region in PD+HS when viewing erotic images after waiting for longer periods of time. Our findings point to reduced delay discounting of erotic rewards in PD+HS, both at the behavioural and brain system levels, and abnormal reinforcing effect of levodopa when PD+HS patients are confronted with erotic stimuli.10.1093/brain/awy298_video1awy298media15983845074001.

Opposite environmental gating of the experienced utility ('liking') and decision utility ('wanting') of heroin versus cocaine in animals and humans: implications for computational neuroscience

BACKGROUND

In this paper, we reviewed translational studies concerned with environmental influences on the rewarding effects of heroin versus cocaine in rats and humans with substance use disorder. These studies show that both experienced utility ('liking') and decision utility ('wanting') of heroin and cocaine shift in opposite directions as a function of the setting in which these drugs were used. Briefly, rats and humans prefer using heroin at home but cocaine outside the home. These findings appear to challenge prevailing theories of drug reward, which focus on the notion of shared substrate of action for drug of abuse, and in particular on their shared ability to facilitate dopaminergic transmission.

AIMS

Thus, in the second part of the paper, we verified whether our findings could be accounted for by available computational models of reward. To account for our findings, a model must include a component that could mediate the substance-specific influence of setting on drug reward RESULTS: It appears of the extant models that none is fully compatible with the results of our studies.

CONCLUSIONS

We hope that this paper will serve as stimulus to design computational models more attuned to the complex mechanisms responsible for the rewarding effects of drugs in real-world contexts.

Roux-en-Y gastric bypass in rat reduces mu-opioid receptor levels in brain regions associated with stress and energy regulation

Roux-en-Y gastric bypass surgery (RYGB) is the most common and effective weight loss procedure for severe obesity. However, a significant increase in addictive behaviors and new-onset substance use disorder (SUD) are sometimes observed post-surgery. The endogenous opioid system is known to play a major role in motivated behavior and reward, as well as the abuse of substances, including alcohol, tobacco, opioids and highly palatable foods. Here, we examined the effects of RYGB on mu-opioid receptor levels in the brain. Male Sprague-Dawley rats were assigned to one of four groups: standard diet with sham surgery (control), ad libitum high-energy high-fat (HF) diet with sham surgery, calorie restricted HF diet with sham surgery (Sham-FR), or HF diet with RYGB surgery. Control and HF groups were fed their respective diets for 8 weeks, with surgery performed on the eighth week. After 9 weeks on their respective diets post-surgery, animals were sacrificed for mu-opioid receptor autoradiography using the [³H] [D-Ala2,N-Me-Phe4-Gly5-ol]- enkephalin (DAMGO) ligand. Rats with RYGB showed reduced DAMGO binding in the central amygdala compared to sham-operated HF diet controls, and in the hypothalamus compared to high-fat fed Sham-FR. Diet alone did not change [³H] DAMGO binding in any region. These findings show that RYGB surgery, independent of diet or caloric restriction, decreases mu opioid signaling in specific regions important for stress and energy regulation. Thus, RYGB surgery may lead to greater stress sensitivity via downregulated mu opioid signaling in the central amygdala, which may contribute to the observed increased risk in some subjects for addictive behavior.

Cued for risk: Evidence for an incentive sensitization framework to explain the interplay between stress and anxiety, substance abuse, and reward uncertainty in disordered gambling behavior

Gambling disorder is an impairing condition confounded by psychiatric co-morbidity, particularly with substance use and anxiety disorders. Yet, our knowledge of the mechanisms that cause these disorders to coalesce remains limited. The Incentive Sensitization Theory suggests that sensitization of neural "wanting" pathways, which attribute incentive salience to rewards and their cues, is responsible for the excessive desire for drugs and cue-triggered craving. The resulting hyper-reactivity of the "wanting' system is believed to heavily influence compulsive drug use and relapse. Notably, evidence for sensitization of the mesolimbic dopamine pathway has been seen across gambling and substance use, as well as anxiety and stress-related pathology, with stress playing a major role in relapse. Together, this evidence highlights a phenomenon known as cross-sensitization, whereby sensitization to stress, drugs, or gambling behaviors enhance the sensitivity and dopaminergic response to any of those stimuli. Here, we review the literature on how cue attraction and reward uncertainty may underlie gambling pathology, and examine how this framework may advance our understanding of co-mordidity with substance-use disorders (e.g., alcohol, nicotine) and anxiety disorders. We argue that reward uncertainty, as seen in slot machines and games of chance, increases dopaminergic activity in the mesolimbic pathway and enhances the incentive value of reward cues. We propose that incentive sensitization by reward uncertainty may interact with and predispose individuals to drug abuse and stress, creating a mechanism through which co-mordidity of these disorders may emerge.

Signaling Incentive and Drive in the Primate Ventral Pallidum for Motivational Control of Goal-Directed Action

Processing incentive and drive is essential for control of goal-directed behavior. The limbic part of the basal ganglia has been emphasized in these processes, yet the exact neuronal mechanism has remained elusive. In this study, we examined the neuronal activity of the ventral pallidum (VP) and its upstream area, the rostromedial caudate (rmCD), while two male macaque monkeys performed an instrumental lever release task in which a visual cue indicated the forthcoming reward size. We found that the activity of some neurons in VP and rmCD reflected the expected reward size transiently following the cue. Reward size coding appeared earlier and stronger in VP than in rmCD. We also found that the activity in these areas was modulated by the satiation level of monkeys, which also occurred more frequently in VP than in rmCD. The information regarding reward size and satiation level was independently signaled in the neuronal populations of these areas. The data thus highlighted the neuronal coding of key variables for goal-directed behavior in VP. Furthermore, pharmacological inactivation of VP induced more severe deficit of goal-directed behavior than inactivation of rmCD, which was indicated by abnormal error repetition and diminished satiation effect on the performance. These results suggest that VP encodes incentive value and internal drive and plays a pivotal role in the control of motivation to promote goal-directed behavior.

SIGNIFICANCE STATEMENT The limbic part of the basal ganglia has been implicated in the motivational control of goal-directed action. Here, we investigated how the ventral pallidum (VP) and the rostromedial caudate (rmCD) encode incentive value and internal drive and control goal-directed behavior. Neuronal recording and subsequent pharmacological inactivation revealed that VP had stronger coding of reward size and satiation level than rmCD. Reward size and satiation level were independently encoded in the neuronal population of these areas. Furthermore, VP inactivation impaired goal-directed behavior more severely than rmCD inactivation. These results highlight the central role of VP in the motivational control of goal-directed action.