Inter-individual discount factor differences in reward prediction are topographically associated with caudate activation

Reward prediction-errors weighted by cue salience produces addictive behaviours in simulations, with asymmetrical learning and steeper delay discounting

Dysfunction in learning and motivational systems are thought to contribute to addictive behaviours. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviours through pharmacological manipulations that provide excess dopaminergic signalling towards these learning and motivational systems. Redish (2004) suggested a role based on dopaminergic signals of value prediction error, while (Zhang et al., 2009) suggested a role based on dopaminergic signals of motivation. However, both models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviours, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviours. Here, we propose a novel mathematical definition of salience, that combines aspects of dopamine's role in both learning and motivation within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviours that the (Zhang et al., 2009; Redish, 2004) models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviours and aspects of behavioural/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviours by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.

Environmental uncertainty and the advantage of impulsive choice strategies

Choice impulsivity is characterized by the choice of immediate, smaller reward options over future, larger reward options, and is often thought to be associated with negative life outcomes. However, some environments make future rewards more uncertain, and in these environments impulsive choices can be beneficial. Here we examined the conditions under which impulsive vs. non-impulsive decision strategies would be advantageous. We used Markov Decision Processes (MDPs) to model three common decision-making tasks: Temporal Discounting, Information Sampling, and an Explore-Exploit task. We manipulated environmental variables to create circumstances where future outcomes were relatively uncertain. We then manipulated the discount factor of an MDP agent, which affects the value of immediate versus future rewards, to model impulsive and non-impulsive behavior. This allowed us to examine the performance of impulsive and non-impulsive agents in more or less predictable environments. In Temporal Discounting, we manipulated the transition probability to delayed rewards and found that the agent with the lower discount factor (i.e. the impulsive agent) collected more average reward than the agent with a higher discount factor (the non-impulsive agent) by selecting immediate reward options when the probability of receiving the future reward was low. In the Information Sampling task, we manipulated the amount of information obtained with each sample. When sampling led to small information gains, the impulsive MDP agent collected more average reward than the non-impulsive agent. Third, in the Explore-Exploit task, we manipulated the substitution rate for novel options. When the substitution rate was high, the impulsive agent again performed better than the non-impulsive agent, as it explored the novel options less and instead exploited options with known reward values. The results of these analyses show that impulsivity can be advantageous in environments that are unexpectedly uncertain.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

The anterior caudate nucleus supports impulsive choices triggered by pramipexole

BACKGROUND

Pramipexole is a dopamine agonist used as a treatment in PD and restless legs syndrome to reduce motor symptoms, but it often induces impulse control disorders. In particular, patients with impulse control disorders tend to make more impulsive choices in the delay discounting task, that is, they choose small immediate rewards over larger delayed ones more often than patients without impulse control disorders and healthy subjects do. Yet the site of action of pramipexole that produces these impulsive choices remains unknown. Based on the heterogeneity of corticostriatal projections and the massive dopamine innervation of the striatum, we hypothesized that impulsive choices triggered by dopamine treatments may be supported by a specific striatal territory.

OBJECTIVES

This study aims to determine by which anteriorstriatum territory the Pramipexole trigger impulsive choices; the caudate nucleus, the ventral striatum or the putamen.

METHODS

We compared pramipexole intramuscular injections to intracerebral microinjections within the three striatal territories in healthy monkeys trained to execute the delay discounting task, a behavioral paradigm typically used to evaluate impulsive choices.

RESULTS

We found that pramipexole intramuscular injections induced impulsive choices in all monkeys. Local microinjections were performed inside the anterior caudate nucleus, ventral striatum, and anterior putamen and reproduced those impulsive choices when pramipexole was directly injected into the caudate nucleus, whereas injections into the ventral striatum or putamen had no effect on monkeys' choices.

CONCLUSIONS

These results, consistent with clinical studies, suggest that impulsive choices triggered by pramipexole are supported by the caudate nucleus, allowing us to emphasize the importance of dopamine modulation inside this striatal territory in decision processes underlying impulsive behaviors. © 2019 International Parkinson and Movement Disorder Society.

The role of the dorsal striatum in choice impulsivity

It has long been recognized that the dorsal striatum is an essential brain region for control of action selection based on action-outcome contingency learning, particularly when the available actions are bound to rewarding outcomes. In principle, intertemporal choice in the delay-discounting task-a validated measure of choice impulsivity-involves reward-associated actions that require the recruitment of the dorsal striatum. Here, we conjecture about ways the dorsal striatum is involved in choice impulsivity. Based on a selective body of studies, we begin with a brief history of research on choice impulsivity and the dorsal striatum, and then provide a comprehensive summary of contemporary studies utilizing human neuroimaging and animal models to search for links between choice impulsivity and the dorsal striatum. In particular, we discuss in-depth the converging evidence for the associations of choice impulsivity with the reward valuation coded by the caudate, a ventral-to-dorsal gradient in the dorsal striatum, the origins of striatal afferents, and developmental maturation of frontostriatal connectivity during adolescence.

An adaptive, individualized fMRI delay discounting procedure to increase flexibility and optimize scanner time

Research on the rate at which people discount the value of future rewards has become increasingly prevalent as discount rate has been shown to be associated with many unhealthy patterns of behavior such as drug abuse, gambling, and overeating. fMRI research points to a fronto-parietal-limbic pathway that is active during decisions between smaller amounts of money now and larger amounts available after a delay. Researchers in this area have used different variants of delay discounting tasks and reported various contrasts between choice trials of different types from these tasks. For instance, researchers have compared 1) choices of delayed monetary amounts to choices of the immediate monetary amounts, 2) 'hard' choices made near one's point of indifference to 'easy' choices that require little thought, and 3) trials where an immediate choice is available versus trials where one is unavailable, regardless of actual eventual choice. These differences in procedure and analysis make comparison of results across studies difficult. In the present experiment, we designed a delay discounting task with the intended capability of being able to construct contrasts of all three comparisons listed above while optimizing scanning time to reduce costs and avoid participant fatigue. This was accomplished with an algorithm that customized the choice trials presented to each participant with the goal of equalizing choice trials of each type. We compared this task, which we refer to here as the individualized discounting task (IDT), to two other delay discounting tasks previously reported in the literature (McClure et al., 2004; Amlung et al., 2014) in 18 participants. Results show that the IDT can examine each of the three contrasts mentioned above, while yielding a similar degree of activation as the reference tasks. This suggests that this new task could be used in delay discounting fMRI studies to allow researchers to more easily compare their results to a majority of previous research while minimizing scanning duration.

Response in taste circuitry is not modulated by hunger and satiety in women remitted from bulimia nervosa

Individuals with bulimia nervosa (BN) engage in episodes of binge eating, marked by loss of control and eating despite fullness. Does altered reward and metabolic state contribute to BN pathophysiology? Normally, hunger increases (and satiety decreases) reward salience to regulate eating. We investigated whether BN is associated with an abnormal response in a neural circuit involved in translating taste signals into motivated behavior, when hungry and fed. Twenty-six women remitted from BN (RBN) and 22 control women (CW) were administered water and sucrose during 2 counterbalanced fMRI visits, following a 16-hr fast or a standardized breakfast. Significant Group × Condition interactions were found in the left putamen, insula, and amygdala. Post hoc analyses revealed CW were significantly more responsive to taste stimuli when hungry versus fed in the left putamen and amygdala. In contrast, RBN response did not differ between conditions. Further, RBN had greater activation in the left amygdala compared with CW when fed. Findings suggest that RBN neural response to rewarding stimuli may not be modulated by metabolic state. Data raise the possibility that disinhibited eating in BN could result from a failure to devalue food reward when fed, resulting in an exaggerated response. (PsycINFO Database Record

Entrepreneurial and parental love-are they the same?

Here we tested the hypothesis that entrepreneurs' emotional experience and brain responses toward their own firm resemble those of parents toward their own children. Using fMRI, we measured the brain activity while male entrepreneurs viewed pictures of their own and of a familiar firm, and while fathers viewed pictures of their own and of a familiar child. The entrepreneurs who self-rated as being very closely attached with their venture showed a similar suppression of activity in the posterior cingulate cortex, temporoparietal junction, and dorsomedial prefrontal cortex as fathers during viewing pictures of their own children versus familiar children. In addition, individual differences in the confidence trait influenced the neural encoding of both paternal and entrepreneurial processing. For underconfident fathers, a picture of one's own child was associated with stronger activation and for overconfident fathers with weaker activation in the amygdala and in caudate nucleus, a brain structure associated with processing of rewards. Similar association with activation, yet more widespread in the emotional processing network, was observed in entrepreneurs suggesting a similar neural basis for increased sensitivity to threats and potential risks concerning one's venture and child. In conclusion, both entrepreneurial and parental love seem to be supported by brain structures associated with reward and emotional processing as well as social understanding. Hum Brain Mapp 38:2923-2938, 2017. © 2017 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.

Attention Diversion Improves Response Inhibition of Immediate Reward, But Only When it Is Beneficial: An fMRI Study

Deficits of self-control are associated with a number of mental state disorders. The ability to direct attention away from an alluring stimulus appears to aid inhibition of an impulsive response. However, further functional imaging research is required to assess the impact of shifts in attention on self-regulating processes. We varied the level of attentional disengagement in an functional magnetic resonance imaging (fMRI)-based Go/No-go task to probe whether diversion of attention away from alluring stimuli facilitates response inhibition. We used the attention-grabbing characteristic of faces to exogenously direct attention away from stimuli and investigated the relative importance of attention and response inhibition mechanisms under different delayed reward scenarios [i.e., where forgoing an immediate reward ($1) led to a higher ($10) or no payoff in the future]. We found that diverting attention improved response inhibition performance, but only when resistance to an alluring stimulus led to delayed reward. Region of interest analyses indicated significant increased activity in posterior right inferior frontal gyrus during successful No-go trials for delayed reward trials compared to no delayed reward trials, and significant reduction in activity in the superior temporal gyri and left caudate in contexts of high attentional diversion. Our findings imply that strategies that increase the perceived benefits of response inhibition might assist individuals in abstaining from problematic impulsive behaviors.

Insulin, Central Dopamine D2 Receptors, and Monetary Reward Discounting in Obesity

Animal research finds that insulin regulates dopamine signaling and reward behavior, but similar research in humans is lacking. We investigated whether individual differences in body mass index, percent body fat, pancreatic β-cell function, and dopamine D2 receptor binding were related to reward discounting in obese and non-obese adult men and women. Obese (n = 27; body mass index>30) and non-obese (n = 20; body mass index<30) adults were assessed for percent body fat with dual-energy X-ray absorptiometry and for β-cell function using disposition index. Choice of larger, but delayed or less certain, monetary rewards relative to immediate, certain smaller monetary rewards was measured using delayed and probabilistic reward discounting tasks. Positron emission tomography using a non-displaceable D2-specific radioligand, [¹¹C](N-methyl)benperidol quantified striatal D2 receptor binding. Groups differed in body mass index, percent body fat, and disposition index, but not in striatal D2 receptor specific binding or reward discounting. Higher percent body fat in non-obese women related to preference for a smaller, certain reward over a larger, less likely one (greater probabilistic discounting). Lower β-cell function in the total sample and lower insulin sensitivity in obese related to stronger preference for an immediate and smaller monetary reward over delayed receipt of a larger one (greater delay discounting). In obese adults, higher striatal D2 receptor binding related to greater delay discounting. Interestingly, striatal D2 receptor binding was not significantly related to body mass index, percent body fat, or β-cell function in either group. Our findings indicate that individual differences in percent body fat, β-cell function, and striatal D2 receptor binding may each contribute to altered reward discounting behavior in non-obese and obese individuals. These results raise interesting questions about whether and how striatal D2 receptor binding and metabolic factors, including β-cell function, interact to affect reward discounting in humans.

Sex difference in the relationship between salivary testosterone and inter-temporal choice

Humans often prefer a small immediate reward to large reward in the future. This myopic tendency in inter-temporal choice is termed delay discounting, and has been the focus of intensive research in the past decades. Recent studies indicate that the neural regions underlying delay discounting are influenced by the gonadal steroids. However, the specific relationship between the testosterone levels and delay discounting is unclear at this point, especially in females. The present study investigated the relationship between salivary testosterone concentrations and discounting rates in delay- and probability-discounting tasks with healthy males and females. The results revealed a positive correlation between testosterone concentrations and delay-discounting rates in females and a negative correlation in males. Testosterone concentrations were unrelated to probability-discounting rates. Although causal effects of testosterone cannot be certain in this correlational study, if testosterone directly influenced this behavior, observed sex differences in delay discounting may be evidence of a curvilinear effect of testosterone. Alternatively, the findings may reflect inverse pattern of responsiveness to testosterone between male and female neural systems, or basic sex-difference in the neural mechanism underlying delay-discounting independent of testosterone itself.

Dopaminergic function and intertemporal choice

The discounting of delayed rewards, also known as temporal or delay discounting, is intrinsic to everyday decisions and can be impaired in pathological states such as addiction disorders. Preclinical and human studies suggest a role for dopaminergic function in temporal discounting but this relationship has not yet been verified using molecular imaging of the living human brain. Here, we evaluated dopaminergic function in temporal discounting using positron emission tomography (PET) with two different dopaminergic ligands assessing three populations in whom temporal discounting has been shown to be impaired. First, we show using [11C]raclopride PET that in pathological gamblers, greater temporal discounting correlates with decreased ventral striatal binding potential, convergent with translational findings of lower nucleus accumbens D2/D3 receptor density in high-impulsive rodents. Temporal discounting also correlates with lower ventral striatal dopamine release in response to high-reward magnitude suggesting that dopamine-mediated devaluation of larger delayed rewards may drive choice preferences. Second, we show using [18F]fluorodopa PET that in Parkinson's disease, temporal discounting correlates with greater left caudate dopaminergic terminal function. Finally, in subjects with Parkinson's disease and dopamine medication-induced behavioral addictions, temporal discounting is further correlated with greater dopaminergic terminal function in the anterior putamen. These findings provide insights into the relationship between striatal dopamine function and temporal discounting, and its potential role in pathological disorders and mechanisms underlying treatment interventions.

Are Extremes of Consumption in Eating Disorders Related to an Altered Balance between Reward and Inhibition?

The primary defining characteristic of a diagnosis of an eating disorder (ED) is the "disturbance of eating or eating-related behavior that results in the altered consumption or absorption of food" (DSM V; American Psychiatric Association, 2013). There is a spectrum, ranging from those who severely restrict eating and become emaciated on one end to those who binge and overconsume, usually accompanied by some form of compensatory behaviors, on the other. How can we understand reasons for such extremes of food consummatory behaviors? Recent work on obesity and substance use disorders has identified behaviors and neural pathways that play a powerful role in human consummatory behaviors. That is, corticostriatal limbic and dorsal cognitive neural circuitry can make drugs and food rewarding, but also engage self-control mechanisms that may inhibit their use. Importantly, there is considerable evidence that alterations of these systems also occur in ED. This paper explores the hypothesis that an altered balance of reward and inhibition contributes to altered extremes of response to salient stimuli, such as food. We will review recent studies that show altered sensitivity to reward and punishment in ED, with evidence of altered activity in corticostriatal and insula processes with respect to monetary gains or losses, and tastes of palatable foods. We will also discuss evidence for a spectrum of extremes of inhibition and dysregulation behaviors in ED supported by studies suggesting that this is related to top-down self-control mechanisms. The lack of a mechanistic understanding of ED has thwarted efforts for evidence-based approaches to develop interventions. Understanding how ED behavior is encoded in neural circuits would provide a foundation for developing more specific and effective treatment approaches.

Is there a neuroanatomical basis of the vulnerability to suicidal behavior? A coordinate-based meta-analysis of structural and functional MRI studies

Objective: We conducted meta-analyses of functional and structural neuroimaging studies comparing adolescent and adult individuals with a history of suicidal behavior and a psychiatric disorder to psychiatric controls in order to objectify changes in brain structure and function in association with a vulnerability to suicidal behavior.

Methods: Magnetic resonance imaging studies published up to July 2013 investigating structural or functional brain correlates of suicidal behavior were identified through computerized and manual literature searches. Activation foci from 12 studies encompassing 475 individuals, i.e., 213 suicide attempters and 262 psychiatric controls were subjected to meta-analytical study using anatomic or activation likelihood estimation (ALE).

Result: Activation likelihood estimation revealed structural deficits and functional changes in association with a history of suicidal behavior. Structural findings included reduced volumes of the rectal gyrus, superior temporal gyrus and caudate nucleus. Functional differences between study groups included an increased reactivity of the anterior and posterior cingulate cortices.

Discussion: A history of suicidal behavior appears to be associated with (probably interrelated) structural deficits and functional overactivation in brain areas, which contribute to a decision-making network. The findings suggest that a vulnerability to suicidal behavior can be defined in terms of a reduced motivational control over the intentional behavioral reaction to salient negative stimuli.

Personality influences temporal discounting preferences: behavioral and brain evidence

Personality traits are stable predictors of many life outcomes that are associated with important decisions that involve tradeoffs over time. Therefore, a fundamental question is how tradeoffs over time vary from person to person in relation to stable personality traits. We investigated the influence of personality, as measured by the Five-Factor Model, on time preferences and on neural activity engaged by intertemporal choice. During functional magnetic resonance imaging (fMRI), participants made choices between smaller-sooner and larger-later monetary rewards. For each participant, we estimated a constant-sensitivity discount function that dissociates impatience (devaluation of future consequences) from time sensitivity (consistency with rational, exponential discounting). Overall, higher neuroticism was associated with a relatively greater preference for immediate rewards and higher conscientiousness with a relatively greater preference for delayed rewards. Specifically, higher conscientiousness correlated positively with lower short-term impatience and more exponential time preferences, whereas higher neuroticism (lower emotional stability) correlated positively with higher short-term impatience and less exponential time preferences. Cognitive-control and reward brain regions were more activated when higher conscientiousness participants selected a smaller-sooner reward and, conversely, when higher neuroticism participants selected a larger-later reward. The greater activations that occurred when choosing rewards that contradicted personality predispositions may reflect the greater recruitment of mental resources needed to override those predispositions. These findings reveal that stable personality traits fundamentally influence how rewards are chosen over time.

Remember the future II: meta-analyses and functional overlap of working memory and delay discounting

Previously we showed that working memory training decreased the discounting of future rewards in stimulant addicts without affecting a go/no-go task. While a relationship between delay discounting and working memory is consistent with other studies, the unique brain regions of plausible causality between these two abilities have yet to be determined. Activation likelihood estimation meta-analyses were performed on foci from studies of delay discounting (DD = 449), working memory (WM = 452), finger tapping (finger tapping = 450), and response inhibition (RI = 450). Activity maps from relatively less (finger tapping) and more (RI) demanding executive tasks were contrasted with maps of DD and WM. Overlap analysis identified unique functional coincidence between DD and WM. The anterior cingulate cortex was engaged by all tasks. Finger tapping largely engaged motor-related brain areas. In addition to motor-related areas, RI engaged frontal brain regions. The right lateral prefrontal cortex was engaged by RI, DD, and WM and was contrasted out of overlap maps. A functional cluster in the posterior portion of the left lateral prefrontal cortex emerged as the largest location of unique overlap between DD and WM. A portion of the left lateral prefrontal cortex is a unique location where delay discounting and working memory processes overlap in the brain. This area, therefore, represents a therapeutic target for improving behaviors that rely on the integration of the recent past with the foreseeable future.

Self-choice enhances value in reward-seeking in primates

When an individual chooses one item from two or more alternatives, they compare the values of the expected outcomes. The outcome value can be determined by the associated reward amount, the probability of reward, and the workload required to earn the reward. Rational choice theory states that choices are made to maximize rewards over time, and that the same outcome values lead to an equal likelihood of choices. However, the theory does not distinguish between conditions with the same reward value, even when acquired under different circumstances, and does not always accurately describe real behavior. We have found that allowing a monkey to choose a reward schedule endows the schedule with extra value when compared to performance in an identical schedule that is chosen by another agent (a computer here). This behavior is not consistent with pure rational choice theory. Theoretical analysis using a modified temporal-difference learning model showed an enhanced schedule state value by self-choice. These results suggest that an increased reward value underlies the improved performances by self-choice during reward-seeking behavior.

Caudate responses to reward anticipation associated with delay discounting behavior in healthy youth

•

Greater discounting of future rewards may be a marker for vulnerability for substance abuse.

•

We tested for an association between discounting and brain responses to reward in healthy youth.

•

Left ventromedial caudate activation during anticipation of potential reward was associated with the willingness to delay gratification.

Background

Choices requiring delay of gratification made during adolescence can have significant impact on life trajectory. Willingness to delay gratification can be measured using delay discounting tasks that require a choice between a smaller immediate reward and a larger delayed reward. Individual differences in the subjective value of delayed rewards are associated with risk for development of psychopathology including substance abuse. The neurobiological underpinnings related to these individual differences early in life are not fully understood. Using functional magnetic resonance imaging (fMRI), we tested the hypothesis that individual differences in delay discounting behavior in healthy youth are related to differences in responsiveness to potential reward.

Method

Nineteen 10–14 year-olds performed a monetary incentive delay task to assess neural sensitivity to potential reward and a questionnaire to measure discounting of future monetary rewards.

Results

Left ventromedial caudate activation during anticipation of potential reward was negatively correlated with delay discounting behavior. There were no regions where brain responses during notification of reward outcome were associated with discounting behavior.

Conclusions

Brain activation during anticipation of potential reward may serve as a marker for individual differences in ability or willingness to delay gratification in healthy youth.