Validation and extension of the reward-mountain model

Dopamine and Beyond: Implications of Psychophysical Studies of Intracranial Self-Stimulation for the Treatment of Depression

Major depressive disorder is a leading cause of disability and suicide worldwide. Consecutive rounds of conventional interventions are ineffective in a significant sub-group of patients whose disorder is classified as treatment-resistant depression. Significant progress in managing this severe form of depression has been achieved through the use of deep brain stimulation of the medial forebrain bundle (MFB). The beneficial effect of such stimulation appears strong, safe, and enduring. The proposed neural substrate for this promising clinical finding includes midbrain dopamine neurons and a subset of their cortical afferents. Here, we aim to broaden the discussion of the candidate circuitry by exploring potential implications of a new “convergence” model of brain reward circuitry in rodents. We chart the evolution of the new model from its predecessors, which held that midbrain dopamine neurons constituted an obligatory stage of the final common path for reward seeking. In contrast, the new model includes a directly activated, non-dopaminergic pathway whose output ultimately converges with that of the dopaminergic neurons. On the basis of the new model and the relative ineffectiveness of dopamine agonists in the treatment of depression, we ask whether non-dopaminergic circuitry may contribute to the clinical efficacy of deep brain stimulation of the MFB.

The Convergence Model of Brain Reward Circuitry: Implications for Relief of Treatment-Resistant Depression by Deep-Brain Stimulation of the Medial Forebrain Bundle

Deep-brain stimulation of the medial forebrain bundle (MFB) can provide effective, enduring relief of treatment-resistant depression. Panksepp provided an explanatory framework: the MFB constitutes the core of the neural circuitry subserving the anticipation and pursuit of rewards: the “SEEKING” system. On that view, the SEEKING system is hypoactive in depressed individuals; background electrical stimulation of the MFB alleviates symptoms by normalizing activity. Panksepp attributed intracranial self-stimulation to excitation of the SEEKING system in which the ascending projections of midbrain dopamine neurons are an essential component. In parallel with Panksepp’s qualitative work, intracranial self-stimulation has long been studied quantitatively by psychophysical means. That work argues that the predominant directly stimulated substrate for MFB self-stimulation are myelinated, non-dopaminergic fibers, more readily excited by brief electrical current pulses than the thin, unmyelinated axons of the midbrain dopamine neurons. The series-circuit hypothesis reconciles this view with the evidence implicating dopamine in MFB self-stimulation as follows: direct activation of myelinated MFB fibers is rewarding due to their trans-synaptic activation of midbrain dopamine neurons. A recent study in which rats worked for optogenetic stimulation of midbrain dopamine neurons challenges the series-circuit hypothesis and provides a new model of intracranial self-stimulation in which the myelinated non-dopaminergic neurons and the midbrain dopamine projections access the behavioral final common path for reward seeking via separate, converging routes. We explore the potential implications of this convergence model for the interpretation of the antidepressant effect of MFB stimulation. We also discuss the consistent finding that psychomotor stimulants, which boost dopaminergic neurotransmission, fail to provide a monotherapy for depression. We propose that non-dopaminergic MFB components may contribute to the therapeutic effect in parallel to, in synergy with, or even instead of, a dopaminergic component.

Dissociating the effects of dopamine D2 receptors on effort-based versus value-based decision making using a novel behavioral approach

Cost-benefit decision making is essential for organisms to adapt to their ever-changing environment. Most studies of cost-benefit decision making involve choice conditions in which effort and value are varied simultaneously. This prevents identification of the aspects of cost-benefit decision making that are affected by experimental manipulations. We developed operant assays to isolate the individual impacts of effort and value manipulations on cost-benefit decision making. In the concurrent effort choice (CEC) task, mice choose between exerting two distinct types of effort: the number of responses and the duration of a response, to earn the same reward. By parametrically varying response cost, psychometric functions are obtained that reflect how the two types of effort scale against one another. Direct manipulations of effort shift the functions. Because reward value is held constant in this task, differences in scaling of the two response types must be related to the effort manipulations. In the concurrent value choice (CVC) task, mice make the same type of response to earn rewards of different value (e.g., pellets vs. sucrose solutions). Here the effort required to earn one reward type is parametrically varied to obtain the psychometric function that scales the value of the two rewards into the number of responses subjects will pay to earn one reward over the other. Direct value manipulations shift these functions. We tested the effect of the dopamine D2 receptor antagonist, haloperidol, on performance in the CEC and CVC assays and found that D2R signaling is important for effort-based, but not value-based decision making. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Dopamine neurons do not constitute an obligatory stage in the final common path for the evaluation and pursuit of brain stimulation reward

The neurobiological study of reward was launched by the discovery of intracranial self-stimulation (ICSS). Subsequent investigation of this phenomenon provided the initial link between reward-seeking behavior and dopaminergic neurotransmission. We re-evaluated this relationship by psychophysical, pharmacological, optogenetic, and computational means. In rats working for direct, optical activation of midbrain dopamine neurons, we varied the strength and opportunity cost of the stimulation and measured time allocation, the proportion of trial time devoted to reward pursuit. We found that the dependence of time allocation on the strength and cost of stimulation was similar formally to that observed when electrical stimulation of the medial forebrain bundle served as the reward. When the stimulation is strong and cheap, the rats devote almost all their time to reward pursuit; time allocation falls off as stimulation strength is decreased and/or its opportunity cost is increased. A 3D plot of time allocation versus stimulation strength and cost produces a surface resembling the corner of a plateau (the “reward mountain”). We show that dopamine-transporter blockade shifts the mountain along both the strength and cost axes in rats working for optical activation of midbrain dopamine neurons. In contrast, the same drug shifted the mountain uniquely along the opportunity-cost axis when rats worked for electrical MFB stimulation in a prior study. Dopamine neurons are an obligatory stage in the dominant model of ICSS, which positions them at a key nexus in the final common path for reward seeking. This model fails to provide a cogent account for the differential effect of dopamine transporter blockade on the reward mountain. Instead, we propose that midbrain dopamine neurons and neurons with non-dopaminergic, MFB axons constitute parallel limbs of brain-reward circuitry that ultimately converge on the final-common path for the evaluation and pursuit of rewards.

Cocaine-associated decision-making: Toward isolating preference

Ever-increasing evidence suggests that substance use disorder is mediated by decision-making processes, and as such, providing nondrug alternatives can shift maladaptive preferences away from drug reinforcers, such as cocaine. Of note, a recent hypothesis suggests that preference for cocaine is simply a byproduct of cocaine intake, such that the 'direct' effects of cocaine weaken the impact of non-drug alternatives while measuring choice. Conversely, existing quantitative theories of decision-making suggest preference is determined by various dimensions of concurrent reinforcers that in turn determine the relative value of available alternatives. Toward teasing apart the conflicting theories above, we developed a novel drug-choice procedure to control for reinforcer frequency and magnitude (two reinforcer dimensions well known to influence preference) that consequently controls for overall cocaine intake. As predicted by quantitative choice theory, results suggest that cocaine intake and preference are dissociable while measuring choice, with reinforcer frequency and magnitude having independent influence on the relative value of choice alternatives. Furthermore, we demonstrate that the choice procedure is sensitive to various manipulations known to alter cocaine reinforcement, all while keeping cocaine intake constant. Finally, the results point to the process of economic substitution as an important avenue of future neurobehavioral investigation toward the improvement of behavioral and pharmacological therapies for substance use disorders. Overall, the proposed choice procedure will allow for improved isolation of the neurobehavioral processes that mediate drug-associated decision-making in future studies.

Learning to use past evidence in a sophisticated world model

Humans and other animals are able to discover underlying statistical structure in their environments and exploit it to achieve efficient and effective performance. However, such structure is often difficult to learn and use because it is obscure, involving long-range temporal dependencies. Here, we analysed behavioural data from an extended experiment with rats, showing that the subjects learned the underlying statistical structure, albeit suffering at times from immediate inferential imperfections as to their current state within it. We accounted for their behaviour using a Hidden Markov Model, in which recent observations are integrated with evidence from the past. We found that over the course of training, subjects came to track their progress through the task more accurately, a change that our model largely attributed to improved integration of past evidence. This learning reflected the structure of the task, decreasing reliance on recent observations, which were potentially misleading.

Humans and other animals possess the remarkable ability to find and exploit patterns and structures in their experience of a complex and varied world. However, such structures are often temporally extended and latent or hidden, being only partially correlated with immediate observations of the world. This makes it essential to integrate current and historical information, and creates a challenging statistical and computational problem. Here, we examine the behaviour of rats facing a version of this challenge posed by a brain-stimulation reward task. We find that subjects learned the general structure of the task, but struggled when immediate observations were misleading. We captured this behaviour with a model in which subjects integrated evidence from recent observations together with evidence from the past. The subjects’ performance improved markedly over successive sessions, allowing them to overcome misleading observations. According to our model, this was made possible by more effective usage of past evidence to better determine the true state of the world.

Valuation of opportunity costs by rats working for rewarding electrical brain stimulation

Pursuit of one goal typically precludes simultaneous pursuit of another. Thus, each exclusive activity entails an “opportunity cost:” the forgone benefits from the next-best activity eschewed. The present experiment estimates, in laboratory rats, the function that maps objective opportunity costs into subjective ones. In an operant chamber, rewarding electrical brain stimulation was delivered when the cumulative time a lever had been depressed reached a criterion duration. The value of the activities forgone during this duration is the opportunity cost of the electrical reward. We determined which of four functions best describes how objective opportunity costs, expressed as the required duration of lever depression, are translated into their subjective equivalents. The simplest account is the identity function, which equates subjective and objective opportunity costs. A variant of this function called the “sigmoidal-slope function,” converges on the identity function at longer durations but deviates from it at shorter durations. The sigmoidal-slope function has the form of a hockey stick. The flat “blade” denotes a range over which opportunity costs are subjectively equivalent; these durations are too short to allow substitution of more beneficial activities. The blade extends into an upward-curving portion over which costs become discriminable and finally into the straight “handle,” over which objective and subjective costs match. The two remaining functions are based on hyperbolic and exponential temporal discounting, respectively. The results are best described by the sigmoidal-slope function. That this is so suggests that different principles of intertemporal choice are involved in the evaluation of time spent working for a reward or waiting for its delivery. The subjective opportunity-cost function plays a key role in the evaluation and selection of goals. An accurate description of its form and parameters is essential to successful modeling and prediction of instrumental performance and reward-related decision making.

Toward a systems-oriented approach to the role of the extended amygdala in adaptive responding

Research into the structure and function of the basal forebrain macrostructure called the extended amygdala (EA) has recently seen considerable growth. This paper reviews that work, with the objectives of identifying underlying themes and developing a common goal towards which investigators of EA function might work. The paper begins with a brief review of the structure and the ontological and phylogenetic origins of the EA. It continues with a review of research into the role of the EA in both aversive and appetitive states, noting that these two seemingly disparate avenues of research converge on the concept of reinforcement - either negative or positive - of adaptive responding. These reviews lead to a proposal as to where the EA may fit in the organization of the basal forebrain, and an invitation to investigators to place their findings in a unifying conceptual framework of the EA as a collection of neural ensembles that mediate adaptive responding.

Budget Constraints Affect Male Rats' Choices between Differently Priced Commodities

Demand theory can be applied to analyse how a human or animal consumer changes her selection of commodities within a certain budget in response to changes in price of those commodities. This change in consumption assessed over a range of prices is defined as demand elasticity. Previously, income-compensated and income-uncompensated price changes have been investigated using human and animal consumers, as demand theory predicts different elasticities for both conditions. However, in these studies, demand elasticity was only evaluated over the entirety of choices made from a budget. As compensating budgets changes the number of attainable commodities relative to uncompensated conditions, and thus the number of choices, it remained unclear whether budget compensation has a trivial effect on demand elasticity by simply sampling from a different total number of choices or has a direct effect on consumers’ sequential choice structure. If the budget context independently changes choices between commodities over and above price effects, this should become apparent when demand elasticity is assessed over choice sets of any reasonable size that are matched in choice opportunities between budget conditions. To gain more detailed insight in the sequential choice dynamics underlying differences in demand elasticity between budget conditions, we trained N=8 rat consumers to spend a daily budget by making a number of nosepokes to obtain two liquid commodities under different price regimes, in sessions with and without budget compensation. We confirmed that demand elasticity for both commodities differed between compensated and uncompensated budget conditions, also when the number of choices considered was matched, and showed that these elasticity differences emerge early in the sessions. These differences in demand elasticity were driven by a higher choice rate and an increased reselection bias for the preferred commodity in compensated compared to uncompensated budget conditions, suggesting a budget context effect on relative valuation.

Psychophysical inference of frequency-following fidelity in the neural substrate for brain stimulation reward

The rewarding effect of electrical brain stimulation has been studied extensively for 60 years, yet the identity of the underlying neural circuitry remains unknown. Previous experiments have characterized the directly stimulated ("first-stage") neurons implicated in self-stimulation of the medial forebrain bundle. Their properties are consistent with those of fine, myelinated axons, at least some of which project rostro-caudally. These properties do not match those of dopaminergic neurons. The present psychophysical experiment estimates an additional first-stage characteristic: maximum firing frequency. We test a frequency-following model that maps the experimenter-set pulse frequency into the frequency of firing induced in the directly stimulated neurons. As pulse frequency is increased, firing frequency initially increases at the same rate, then becomes probabilistic, and finally levels off. The frequency-following function is based on the counter model which holds that the rewarding effect of a pulse train is determined by the aggregate spike rate triggered in first-stage neurons during a given interval. In 7 self-stimulating rats, we measured current- vs. pulse-frequency trade-off functions. The trade-off data were well described by the frequency-following model, and its upper asymptote was approached at a median value of 360 Hz (IQR = 46 Hz). This value implies a highly excitable, non-dopaminergic population of first-stage neurons. Incorporating the frequency-following function and parameters in Shizgal's 3-dimensional reward-mountain model improves its accuracy and predictive power.

Contingency Management and Deliberative Decision-Making Processes

Contingency management is an effective treatment for drug addiction. The current explanation for its success is rooted in alternative reinforcement theory. We suggest that alternative reinforcement theory is inadequate to explain the success of contingency management and produce a model based on demand curves that show how little the monetary rewards offered in this treatment would affect drug use. Instead, we offer an explanation of its success based on the concept that it accesses deliberative decision-making processes. We suggest that contingency management is effective because it offers a concrete and immediate alternative to using drugs, which engages deliberative processes, improves the ability of those deliberative processes to attend to non-drug options, and offsets more automatic action-selection systems. This theory makes explicit predictions that can be tested, suggests which users will be most helped by contingency management, and suggests improvements in its implementation.

Mechanisms of motivation-cognition interaction: challenges and opportunities

Recent years have seen a rejuvenation of interest in studies of motivation-cognition interactions arising from many different areas of psychology and neuroscience. The present issue of Cognitive, Affective, & Behavioral Neuroscience provides a sampling of some of the latest research from a number of these different areas. In this introductory article, we provide an overview of the current state of the field, in terms of key research developments and candidate neural mechanisms receiving focused investigation as potential sources of motivation-cognition interaction. However, our primary goal is conceptual: to highlight the distinct perspectives taken by different research areas, in terms of how motivation is defined, the relevant dimensions and dissociations that are emphasized, and the theoretical questions being targeted. Together, these distinctions present both challenges and opportunities for efforts aiming toward a more unified and cross-disciplinary approach. We identify a set of pressing research questions calling for this sort of cross-disciplinary approach, with the explicit goal of encouraging integrative and collaborative investigations directed toward them.

The effect of probability discounting on reward seeking: a three-dimensional perspective

Rats will work for electrical stimulation of the medial forebrain bundle. The rewarding effect arises from the volleys of action potentials fired by the stimulation and subsequent spatio-temporal integration of their post-synpatic impact. The proportion of time allocated to self-stimulation depends on the intensity of the rewarding effect as well as on other key determinants of decision-making, such as subjective opportunity costs and reward probability. We have proposed that a 3D model relating time allocation to the intensity and cost of reward can distinguish manipulations acting prior to the output of the spatio-temporal integrator from those acting at or beyond it. Here, we test this proposition by varying reward probability, a variable that influences the computation of payoff in the 3D model downstream from the output of the integrator. On riskless trials, reward was delivered on every occasion that the rat held down the lever for a cumulative duration called the “price,” whereas on risky trials, reward was delivered with probability 0.75 or 0.50. According to the model, the 3D structure relating time allocation to reward intensity and price is shifted leftward along the price axis by reductions in reward probability; the magnitude of the shift estimates the change in subjective probability. The predictions were borne out: reducing reward probability shifted the 3D structure systematically along the price axis while producing only small, inconsistent displacements along the pulse-frequency axis. The results confirm that the model can accurately distinguish manipulations acting at or beyond the spatio-temporal integrator and strengthen the conclusions of previous studies showing similar shifts following dopaminergic manipulations. Subjective and objective reward probabilities appeared indistinguishable over the range of 0.5 ≤ p ≤ 1.0.

A new view of the effect of dopamine receptor antagonism on operant performance for rewarding brain stimulation in the rat

RATIONALE

Previous studies of neuroleptic challenges to intracranial self-stimulation (ICSS) employed two-dimensional (2D) measurements (curve shifts). Results so obtained are ambiguous with regard to the stage of neural processing at which the drug produces its performance-altering effect. We substituted a three-dimensional (3D) method that measures reward-seeking as a function of both the strength and cost of reward. This method reveals whether changes in reward seeking are due to drug action prior to the output of the circuitry that performs spatiotemporal integration of the stimulation-induced neural activity.

OBJECTIVES

The aim of this study was to obtain new information about the stage of neural processing at which pimozide acts to alter pursuit of brain stimulation reward (BSR).

METHODS

Following treatment with pimozide (0.1 mg/kg) or its vehicle, the proportion of trial time allocated to working for BSR was measured as a function of pulse frequency and opportunity cost. A surface defined by Shizgal's reward-mountain model was fitted to the drug and vehicle data.

RESULTS

Pimozide lowered the cost required to decrease performance for a maximal BSR to half its maximal level but did not alter the pulse-frequency required to produce a reward of half-maximal intensity.

CONCLUSIONS

Like indirect dopamine agonists, pimozide does not alter the sensitivity of brain reward circuity but changes reward-system gain, subjective effort costs, and/or the value of activities that compete with ICSS. The 3D method is more sensitive and informative than the 2D methods employed previously.