Quantitative analysis of the effect of lesions of the subthalamic nucleus on intertemporal choice: further evidence for enhancement of the incentive value of food reinforcers

Not all discounts are created equal: Regional activity and brain networks in temporal and effort discounting

Reward outcomes associated with costs like time delay and effort investment are generally discounted in decision-making. Standard economic models predict rewards associated with different types of costs are devalued in a similar manner. However, our review of rodent lesion studies indicated partial dissociations between brain regions supporting temporal- and effort-based decision-making. Another debate is whether options involving low and high costs are processed in different brain substrates (dual-system) or in the same regions (single-system). This research addressed these issues using coordinate-based, connectivity-based, and activation network-based meta-analyses to identify overlapping and separable neural systems supporting temporal (39 studies) and effort (20 studies) discounting. Coordinate-based activation likelihood estimation and resting-state connectivity analyses showed immediate-small reward and delayed-large reward choices engaged distinct regions with unique connectivity profiles, but their activation network mapping was found to engage the default mode network. For effort discounting, salience and sensorimotor networks supported low-effort choices, while the frontoparietal network supported high-effort choices. There was little overlap between the temporal and effort networks. Our findings underscore the importance of differentiating different types of costs in decision-making and understanding discounting at both regional and network levels.

In search of a definition of reinforcer value: Some successes and failures of the multiplicative hyperbolic model

The concept of 'value' has enjoyed a central position in many theoretical accounts of choice behaviour. Several definitions of 'value' are contrasted in this paper, and one particular approach is defended, whereby value is defined as a dimensionless intervening variable. This definition is a cornerstone of the multiplicative hyperbolic model of choice (MHM), which was proposed twenty years ago as a modification of Mazur's (1987) hyperbolic model of delay discounting. This paper reviews some of the merits and shortcomings of MHM, and suggests some ways in which MHM might be extended and improved. A formal link between 'value' and the related concept of 'response strength' is suggested, and revisions of the model are proposed which may enable it to accommodate several behavioural phenomena not considered in the original formulation. Broadening the scope of MHM comes at the cost of adding to its burden of free parameters, and it is emphasised that addition of any new parameters needs empirical justification. The status of value as a dimensionless intervening variable is upheld; however it is noted that a growing body of empirical evidence for links between neurobiological phenomena and value suggests that interpretation of value as a hypothetical construct may be warranted.

Dissecting drug effects in preclinical models of impulsive choice: emphasis on glutamatergic compounds

RATIONALE

Impulsive choice is often measured with delay discounting paradigms. Because there are multiple discounting procedures, as well as different statistical analyses that can be applied to data generated from these paradigms, there are some inconsistencies in the literature regarding drug effects on impulsive choice.

OBJECTIVES

The goal of the current paper is to review the methodological and analytic approaches used to measure discounting and to discuss how these differences can account for differential drug effects observed across studies.

RESULTS

Because some procedures/analyses use a single data point as the dependent variable, changes in this value following pharmacological treatment may be interpreted as alterations in sensitivity to delayed reinforcement, but when other procedures/analyses are used, no changes in behavior are observed. Even when multiple data points are included, some studies show that the statistical analysis (e.g., ANOVA on raw proportion of responses vs. using hyperbolic/exponential functions) can lead to different interpretations. Finally, procedural differences (e.g., delay presentation order, signaling the delay to reinforcement, etc.) in the same discounting paradigm can alter how drugs affect sensitivity to delayed reinforcement.

CONCLUSIONS

Future studies should utilize paradigms that allow one to observe alterations in responding at each delay (e.g., concurrent-chains schedules). Concerning statistical analyses, using parameter estimates derived from nonlinear functions or incorporating the generalized matching law can allow one to determine if drugs affect sensitivity to delayed reinforcement or impair discrimination of the large and small magnitude reinforcers. Using these approaches can help further our understanding of the neurochemical underpinnings of delay discounting.

Effects of intra-accumbal administration of dopamine and ionotropic glutamate receptor drugs on delay discounting performance in rats

Nucleus accumbens core (NAcc) has been implicated in impulsive choice, as measured in delay discounting. The role of dopamine (DA) in impulsive choice has received considerable attention, whereas glutamate (Glu) has recently been shown to be an important mediator of discounting. However, research has not examined how DA or Glu receptors in NAcc mediate different aspects of delay discounting performance, that is, (a) sensitivity to reinforcer magnitude and (b) sensitivity to delayed reinforcement. Adult male Sprague-Dawley rats were first trained in a delay discounting task, in which the delay to a large magnitude food reinforcer increased across blocks of trials. Following behavioral training, rats received bilateral implantation of guide cannulas into NAcc. Half of the rats (n = 12) received infusions of the DA-selective ligands SKF 38393 (D1-like agonist: 0.03 or 0.1 μg), SCH 23390 (D1-like antagonist: 0.3 or 1.0 μg), quinpirole (D2-like agonist: 0.3 or 1.0 μg), and eticlopride (D2-like antagonist: 0.3 or 1.0 μg). The other half received infusions of the ionotropic Glu ligands MK-801 (NMDA uncompetitive antagonist: 0.3 or 1.0 μg), AP-5 (NMDA competitive antagonist: 0.3 or 1.0 μg), ifenprodil (noncompetitive antagonist at NR2B-containing NMDA receptors: 0.3 or 1.0 μg), and CNQX (AMPA competitive antagonist: 0.2 or 0.5 μg). Results showed that SCH 23390 (0.3 μg) decreased sensitivity to reinforcer magnitude without altering impulsive choice, whereas ifenprodil (1.0 μg) decreased sensitivity to delayed reinforcement (i.e., impulsive choice). The current results show that DA and NMDA receptors in NAcc mediate distinct aspects of discounting performance. (PsycINFO Database Record

Inhibiting subthalamic nucleus decreases cocaine demand and relapse: therapeutic potential

Preclinical evidence indicates that inactivation of subthalamic nucleus (STN) may be effective for treating cocaine addiction, and therapies that target STN, e.g. deep brain stimulation, are available indicating that this may have clinical promise. Here, we assessed the therapeutic potential of STN inactivation using a translationally relevant economic approach that quantitatively describes drug-taking behavior, and tested these results with drug-seeking tasks. Economic demand for cocaine was assessed in rats (n = 11) using a within-session threshold procedure in which cocaine price (responses/mg cocaine) was sequentially increased throughout the session. Cocaine demand was assessed in this manner immediately after bilateral microinfusions into STN of either vehicle (artificial cerebrospinal fluid) or the GABAA receptor agonist muscimol. A separate group of animals (n = 8) was tested for changes in cocaine seeking either during extinction or in response to cocaine-associated cues. Muscimol-induced inhibition of STN significantly attenuated cocaine consumption at high prices, drug seeking during extinction and cued reinstatement of cocaine seeking. In contrast, STN inhibition did not reduce cocaine consumption at low prices or locomotor activity. Thus, STN inactivation reduced economic demand for cocaine and multiple measures of drug seeking during extinction. In view of the association between economic demand and addiction severity in both rat and human, these results indicate that STN inactivation has substantial clinical potential for treatment of cocaine addiction.

Effects of N-methyl-D-aspartate receptor ligands on sensitivity to reinforcer magnitude and delayed reinforcement in a delay-discounting procedure

RATIONALE

The N-methyl-D-aspartate (NMDA) receptor has been recently identified as an important mediator of impulsive choice, as assessed in delay discounting. Although discounting is independently influenced by sensitivity to reinforcer magnitude and delayed reinforcement, few studies have examined how NMDA receptor ligands differentially affect these parameters.

OBJECTIVES

The current study examined the effects of various NMDA receptor ligands on sensitivity to reinforcer magnitude and delayed reinforcement in a delay-discounting procedure.

METHODS

Following behavioral training, rats received treatments of the following NMDA receptor ligands: the uncompetitive antagonists ketamine (0, 1.0, 5.0, or 10.0 mg/kg; i.p.), MK-801 (0, 0.003, 0.01, or 0.03 mg/kg; s.c.), and memantine (0, 2.5, 5.0, or 10.0 mg/kg; i.p.), the competitive antagonist CGS 19755 (0, 5.0, 10.0, or 20.0 mg/kg; s.c.), the non-competitive NR2B subunit-selective antagonist ifenprodil (0, 1.0, 3.0, or 10.0 mg/kg; i.p), and the partial agonist D-cycloserine (0, 3.25, 15.0, or 30.0 mg/kg; s.c.).

RESULTS

When an exponential model was used to describe discounting, CGS 19755 (5.0 mg/kg) increased impulsive choice without altering sensitivity to reinforcer magnitude. Conversely, ketamine (10.0 mg/kg), memantine (5.0 mg/kg), and ifenprodil (10.0 mg/kg) decreased sensitivity to reinforcer magnitude without altering impulsive choice. MK-801 and D-cycloserine did not alter delay-discounting performance, although two-way ANOVA analyses indicated D-cycloserine (15.0 mg/kg) decreased impulsive choice.

CONCLUSIONS

The behavioral changes observed in delay discounting following administration of NMDA receptor antagonists do not always reflect an alteration in impulsive choice. These results emphasize the utility in employing quantitative methods to assess drug effects in delay discounting.

Deep brain stimulation of the subthalamic nucleus modulates sensitivity to decision outcome value in Parkinson's disease

Deep brain stimulation (DBS) of the subthalamic nucleus in Parkinson’s disease is known to cause a subtle but important adverse impact on behaviour, with impulsivity its most widely reported manifestation. However, precisely which computational components of the decision process are modulated is not fully understood. Here we probe a number of distinct subprocesses, including temporal discount, outcome utility, instrumental learning rate, instrumental outcome sensitivity, reward-loss trade-offs, and perseveration. We tested 22 Parkinson’s Disease patients both on and off subthalamic nucleus deep brain stimulation (STN-DBS), while they performed an instrumental learning task involving financial rewards and losses, and an inter-temporal choice task for financial rewards. We found that instrumental learning performance was significantly worse following stimulation, due to modulation of instrumental outcome sensitivity. Specifically, patients became less sensitive to decision values for both rewards and losses, but without any change to the learning rate or reward-loss trade-offs. However, we found no evidence that DBS modulated different components of temporal impulsivity. In conclusion, our results implicate the subthalamic nucleus in a modulation of outcome value in experience-based learning and decision-making in Parkinson’s disease, suggesting a more pervasive role of the subthalamic nucleus in the control of human decision-making than previously thought.

Neural substrates underlying effort, time, and risk-based decision making in motivated behavior

All mobile organisms rely on adaptive motivated behavior to overcome the challenges of living in an environment in which essential resources may be limited. A variety of influences ranging from an organism's environment, experiential history, and physiological state all influence a cost-benefit analysis which allows motivation to energize behavior and direct it toward specific goals. Here we review the substantial amount of research aimed at discovering the interconnected neural circuits which allow organisms to carry-out the cost-benefit computations which allow them to behave in adaptive ways. We specifically focus on how the brain deals with different types of costs, including effort requirements, delays to reward and payoff riskiness. An examination of this broad literature highlights the importance of the extended neural circuits which enable organisms to make decisions about these different types of costs. This involves Cortical Structures, including the Anterior Cingulate Cortex (ACC), the Orbital Frontal Cortex (OFC), the Infralimbic Cortex (IL), and prelimbic Cortex (PL), as well as the Baso-Lateral Amygdala (BLA), the Nucleus Accumbens (NAcc), the Ventral Pallidal (VP), the Sub Thalamic Nucleus (STN) among others. Some regions are involved in multiple aspects of cost-benefit computations while the involvement of other regions is restricted to information relating to specific types of costs.

Neural and neurochemical basis of reinforcement-guided decision making

Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcement-guided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decision-related information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making.

Impulsivity and sensitivity to amount and delay of reinforcement in an animal model of ADHD

Previous research has been inconclusive about the degree of impulsivity displayed by spontaneously hypertensive rats (SHR), an animal model of Attention Deficit Hyperactivity Disorder (ADHD). In the present set of experiments, concurrent-chains schedules were employed in order to explore SHR's impulsivity, sensitivity to delay, and sensitivity to amount of reinforcement; Wistar rats (WIS) were used as comparison group. In the three experiments - performed with different subjects - non-independent variable interval 30s schedules were presented in the initial links; the difference between experiments was in the terminal links. For exploring impulsivity, one of the terminal links (SS) was associated to a short delay (2s) and a small reinforcer (1 pellet), whereas the other terminal link (LL) was associated to a longer delay (28s) and a larger reinforcer (4 pellets). The results indicated a remarkably higher impulsivity in SHR. Because this impulsivity may have as potential mechanisms an increased sensitivity to delay and/or a decreased sensitivity to the amount of reinforcement, in experiments 2 and 3 these possibilities were examined. For assessing sensitivity to delay, the following pairs of fixed interval (FI) schedules were used in the terminal links in five conditions: 2-28, 6-24, 15-15, 24-6, 28-2s; the magnitude of reinforcement was 1 pellet in all conditions for both alternatives. For assessing sensitivity to amount, in five conditions the alternatives were associated with different magnitudes of reinforcement: 1-5 pellets, 2-4, 3-3, 4-2 and 5-1 in left-right alternatives, respectively; the delay to reinforcement was controlled by a FI 15s in all conditions and for both alternatives. The sensitivity to delay and the sensitivity to amount were calculated according to the Generalized Matching Law. The results indicated a higher sensitivity to delay in SHR, and the same sensitivity to amount in SHR and WIS rats. These results suggest that the increased sensitivity to delay influences the high level of impulsivity observed in SHR.

Do the adjusting-delay and increasing-delay tasks measure the same construct: delay discounting?

Delay discounting describes the subjective devaluation of a reward when it is delayed. In animals, the adjusting-delay (AD) and increasing-delay (ID) tasks often are used to assess individual differences in, and drug effects on, delay discounting. No study to date, however, has compared systematically the measures of discounting produced in these tasks. The current study examined the correlation between measures of delay discounting derived from AD and ID procedures. Twenty rats completed 30 sessions under each task (order counterbalanced across rats). Quantitative measures of delay discounting produced by the two tasks were positively correlated, suggesting that the AD and ID tasks measure the same underlying facet of impulsive choice (i.e. individual or conjoint sensitivities to reward delay and magnitude). The measures derived from either task, however, depended on the sequences in which the tasks were experienced. That is, pre-exposure to one task decreased discounting of delayed rewards in the second task. Consistent with other published findings, exposure to delayed consequences during the initial discounting assessment might explain this effect. Despite the observed correlation between ID and AD indifference delays, we suggest that the ID procedure might be a more appropriate procedure for pharmacological studies.

Recent Translational Findings on Impulsivity in Relation to Drug Abuse

Impulsive behavior is strongly implicated in drug abuse, as both a cause and a consequence of drug use. To understand how impulsive behaviors lead to and result from drug use, translational evidence from both human and non-human animal studies is needed. Here, we review recent (2009 or later) studies that have investigated two major components of impulsive behavior, inhibitory control and impulsive choice, across preclinical and clinical studies. We concentrate on the stop-signal task as the measure of inhibitory control and delay discounting as the measure of impulsive choice. Consistent with previous reports, recent studies show greater impulsive behavior in drug users compared with non-users. Additionally, new evidence supports the prospective role of impulsive behavior in drug abuse, and has begun to identify the neurobiological mechanisms underlying impulsive behavior. We focus on the commonalities and differences in findings between preclinical and clinical studies, and suggest future directions for translational research.

An examination of the effects of subthalamic nucleus inhibition or μ-opioid receptor stimulation on food-directed motivation in the non-deprived rat

The subthalamic nucleus (STN) serves important functions in regulating movement, cognition, and motivation and is connected with cortical and basal ganglia circuits that process reward and reinforcement. In order to further examine the role of the STN on motivation toward food in non-deprived rats, these experiments studied the effects of pharmacological inhibition or μ-opioid receptor stimulation of the STN on the 2-h intake of a sweetened fat diet, the amount of work exerted to earn sucrose on a progressive ratio 2 (PR-2) schedule of reinforcement, and performance on a differential reinforcement of low-rate responding (DRL) schedule for sucrose reward. Separate behavioral groups (N=6-9) were tested following bilateral inhibition of the STN with the GABA(A) receptor agonist muscimol (at 0-5 ng/0.5 μl/side) or following μ-opioid receptor stimulation with the agonist D-Ala², N-MePhe⁴, Gly-ol-enkephalin (DAMGO; at 0, 0.025 or 0.25 μg/0.5 μl/side). Although STN inhibition increased ambulatory behavior during 2-h feeding sessions, it did not significantly alter intake of the sweetened fat diet. STN inhibition also did not affect the breakpoint for sucrose pellets during a 1-h PR-2 reinforcement schedule or impact the number of reinforcers earned on a 1-h DRL-20s reinforcement schedule in non-deprived rats. In contrast, STN μ-opioid receptor stimulation significantly increased feeding on the palatable diet and reduced the reinforcers earned on a DRL-20 schedule, although DAMGO microinfusions had no effect on PR-2 performance. These data suggest that STN inhibition does not enhance incentive motivation for food in the absence of food restriction and that STN μ-opioid receptors play an important and unique role in motivational processes.

Nucleus accumbens and delay discounting in rats: evidence from a new quantitative protocol for analysing inter-temporal choice

RATIONALE

There is evidence that the core of the nucleus accumbens (AcbC) is involved in inter-temporal choice behaviour.

OBJECTIVE

A new behavioural protocol was used to examine the effect of destruction of the AcbC on delay discounting in inter-temporal choice schedules in rats.

METHOD

Rats with excitotoxic lesions of the AcbC or sham lesions made repeated choices on an adjusting-delay schedule between a smaller reinforcer (A) that was delivered immediately and a larger reinforcer (B) that was delivered after a delay which increased or decreased depending on the subject's choices. In two phases of the experiment, reinforcer sizes were selected which enabled theoretical parameters expressing delay discounting and sensitivity to reinforcer size to be estimated from the ratio of the indifference delays (i.e. the quasi-stable values of the adjusting delay seen after extended training) obtained in the two phases.

RESULTS

In both groups, indifference delays were shorter when the sizes of A and B were 14 and 25 μl than when they were 25 and 100 μl of a 0.6 M sucrose solution. Indifference delays were shorter in AcbC-lesioned than in sham-lesioned rats. Estimates of delay discounting rate based on the ratio of the indifference delays were lower in the AcbC-lesioned than in the sham-lesioned rats. The size sensitivity parameter did not differ between the groups. Adjusting delays in successive blocks of trials were analysed using Fourier transform. The period corresponding to the dominant frequency of the power spectrum and power within the dominant frequency band did not differ between the groups.

CONCLUSIONS

Destruction of the AcbC increased the rate of delay discounting.

Is there an inhibitory-response-control system in the rat? Evidence from anatomical and pharmacological studies of behavioral inhibition

Many common psychiatric conditions, such as attention deficit/hyperactivity disorder (ADHD), obsessive-compulsive disorder (OCD), Parkinson's disease, addiction and pathological gambling are linked by a failure in the mechanisms that control, or inhibit, inappropriate behavior. Models of rat behavioral inhibition permit us to study in detail the anatomical and pharmacological bases of inhibitory failure, using methods that translate directly with patient assessment in the clinic. This review updates current ideas relating to behavioral inhibition based on two significant lines of evidence from rat studies: (1) To integrate new findings from the stop-signal task into existing models of behavioral inhibition, in particular relating to 'impulsive action' control. The stop-signal task has been used for a number of years to evaluate psychiatric conditions and has recently been translated for use in the rat, bringing a wealth of new information to behavioral inhibition research. (2) To consider the importance of the subthalamic nucleus (STN) in the neural circuitry of behavioral inhibition. This function of this nucleus is central to a number of 'disinhibitory' disorders such as Parkinson's disease and OCD, and their therapies, but its role in behavioral inhibition is still undervalued, and often not considered in preclinical models of behavioral control. Integration of these findings has pinpointed the orbitofrontal cortex (OF), dorsomedial striatum (DMStr) and STN within a network that normally inhibits many forms of behavior, including both impulsive and compulsive forms. However, there are distinct differences between behavioral subtypes in their neurochemical modulation. This review brings new light to the classical view of the mechanisms that inhibit behavior, in particular suggesting a far more prominent role for the STN, a structure that is usually omitted from conventional behavioral-inhibition networks. The OF-DMStr-STN circuitry may form the basis of a control network that defines behavioral inhibition and that acts to suppress or countermand many forms of inappropriate or maladaptive behavior.