The effect of probability discounting on reward seeking: a three-dimensional perspective

The Convergence Model of Brain Reward Circuitry: Implications for Relief of Treatment-Resistant Depression by Deep-Brain Stimulation of the Medial Forebrain Bundle

Deep-brain stimulation of the medial forebrain bundle (MFB) can provide effective, enduring relief of treatment-resistant depression. Panksepp provided an explanatory framework: the MFB constitutes the core of the neural circuitry subserving the anticipation and pursuit of rewards: the “SEEKING” system. On that view, the SEEKING system is hypoactive in depressed individuals; background electrical stimulation of the MFB alleviates symptoms by normalizing activity. Panksepp attributed intracranial self-stimulation to excitation of the SEEKING system in which the ascending projections of midbrain dopamine neurons are an essential component. In parallel with Panksepp’s qualitative work, intracranial self-stimulation has long been studied quantitatively by psychophysical means. That work argues that the predominant directly stimulated substrate for MFB self-stimulation are myelinated, non-dopaminergic fibers, more readily excited by brief electrical current pulses than the thin, unmyelinated axons of the midbrain dopamine neurons. The series-circuit hypothesis reconciles this view with the evidence implicating dopamine in MFB self-stimulation as follows: direct activation of myelinated MFB fibers is rewarding due to their trans-synaptic activation of midbrain dopamine neurons. A recent study in which rats worked for optogenetic stimulation of midbrain dopamine neurons challenges the series-circuit hypothesis and provides a new model of intracranial self-stimulation in which the myelinated non-dopaminergic neurons and the midbrain dopamine projections access the behavioral final common path for reward seeking via separate, converging routes. We explore the potential implications of this convergence model for the interpretation of the antidepressant effect of MFB stimulation. We also discuss the consistent finding that psychomotor stimulants, which boost dopaminergic neurotransmission, fail to provide a monotherapy for depression. We propose that non-dopaminergic MFB components may contribute to the therapeutic effect in parallel to, in synergy with, or even instead of, a dopaminergic component.

Dopamine neurons do not constitute an obligatory stage in the final common path for the evaluation and pursuit of brain stimulation reward

The neurobiological study of reward was launched by the discovery of intracranial self-stimulation (ICSS). Subsequent investigation of this phenomenon provided the initial link between reward-seeking behavior and dopaminergic neurotransmission. We re-evaluated this relationship by psychophysical, pharmacological, optogenetic, and computational means. In rats working for direct, optical activation of midbrain dopamine neurons, we varied the strength and opportunity cost of the stimulation and measured time allocation, the proportion of trial time devoted to reward pursuit. We found that the dependence of time allocation on the strength and cost of stimulation was similar formally to that observed when electrical stimulation of the medial forebrain bundle served as the reward. When the stimulation is strong and cheap, the rats devote almost all their time to reward pursuit; time allocation falls off as stimulation strength is decreased and/or its opportunity cost is increased. A 3D plot of time allocation versus stimulation strength and cost produces a surface resembling the corner of a plateau (the “reward mountain”). We show that dopamine-transporter blockade shifts the mountain along both the strength and cost axes in rats working for optical activation of midbrain dopamine neurons. In contrast, the same drug shifted the mountain uniquely along the opportunity-cost axis when rats worked for electrical MFB stimulation in a prior study. Dopamine neurons are an obligatory stage in the dominant model of ICSS, which positions them at a key nexus in the final common path for reward seeking. This model fails to provide a cogent account for the differential effect of dopamine transporter blockade on the reward mountain. Instead, we propose that midbrain dopamine neurons and neurons with non-dopaminergic, MFB axons constitute parallel limbs of brain-reward circuitry that ultimately converge on the final-common path for the evaluation and pursuit of rewards.

In search of a definition of reinforcer value: Some successes and failures of the multiplicative hyperbolic model

The concept of 'value' has enjoyed a central position in many theoretical accounts of choice behaviour. Several definitions of 'value' are contrasted in this paper, and one particular approach is defended, whereby value is defined as a dimensionless intervening variable. This definition is a cornerstone of the multiplicative hyperbolic model of choice (MHM), which was proposed twenty years ago as a modification of Mazur's (1987) hyperbolic model of delay discounting. This paper reviews some of the merits and shortcomings of MHM, and suggests some ways in which MHM might be extended and improved. A formal link between 'value' and the related concept of 'response strength' is suggested, and revisions of the model are proposed which may enable it to accommodate several behavioural phenomena not considered in the original formulation. Broadening the scope of MHM comes at the cost of adding to its burden of free parameters, and it is emphasised that addition of any new parameters needs empirical justification. The status of value as a dimensionless intervening variable is upheld; however it is noted that a growing body of empirical evidence for links between neurobiological phenomena and value suggests that interpretation of value as a hypothetical construct may be warranted.

Valuation of opportunity costs by rats working for rewarding electrical brain stimulation

Pursuit of one goal typically precludes simultaneous pursuit of another. Thus, each exclusive activity entails an “opportunity cost:” the forgone benefits from the next-best activity eschewed. The present experiment estimates, in laboratory rats, the function that maps objective opportunity costs into subjective ones. In an operant chamber, rewarding electrical brain stimulation was delivered when the cumulative time a lever had been depressed reached a criterion duration. The value of the activities forgone during this duration is the opportunity cost of the electrical reward. We determined which of four functions best describes how objective opportunity costs, expressed as the required duration of lever depression, are translated into their subjective equivalents. The simplest account is the identity function, which equates subjective and objective opportunity costs. A variant of this function called the “sigmoidal-slope function,” converges on the identity function at longer durations but deviates from it at shorter durations. The sigmoidal-slope function has the form of a hockey stick. The flat “blade” denotes a range over which opportunity costs are subjectively equivalent; these durations are too short to allow substitution of more beneficial activities. The blade extends into an upward-curving portion over which costs become discriminable and finally into the straight “handle,” over which objective and subjective costs match. The two remaining functions are based on hyperbolic and exponential temporal discounting, respectively. The results are best described by the sigmoidal-slope function. That this is so suggests that different principles of intertemporal choice are involved in the evaluation of time spent working for a reward or waiting for its delivery. The subjective opportunity-cost function plays a key role in the evaluation and selection of goals. An accurate description of its form and parameters is essential to successful modeling and prediction of instrumental performance and reward-related decision making.

Toward a systems-oriented approach to the role of the extended amygdala in adaptive responding

Research into the structure and function of the basal forebrain macrostructure called the extended amygdala (EA) has recently seen considerable growth. This paper reviews that work, with the objectives of identifying underlying themes and developing a common goal towards which investigators of EA function might work. The paper begins with a brief review of the structure and the ontological and phylogenetic origins of the EA. It continues with a review of research into the role of the EA in both aversive and appetitive states, noting that these two seemingly disparate avenues of research converge on the concept of reinforcement - either negative or positive - of adaptive responding. These reviews lead to a proposal as to where the EA may fit in the organization of the basal forebrain, and an invitation to investigators to place their findings in a unifying conceptual framework of the EA as a collection of neural ensembles that mediate adaptive responding.

The basolateral amygdala in reward learning and addiction

Sophisticated behavioral paradigms partnered with the emergence of increasingly selective techniques to target the basolateral amygdala (BLA) have resulted in an enhanced understanding of the role of this nucleus in learning and using reward information. Due to the wide variety of behavioral approaches many questions remain on the circumscribed role of BLA in appetitive behavior. In this review, we integrate conclusions of BLA function in reward-related behavior using traditional interference techniques (lesion, pharmacological inactivation) with those using newer methodological approaches in experimental animals that allow in vivo manipulation of cell type-specific populations and neural recordings. Secondly, from a review of appetitive behavioral tasks in rodents and monkeys and recent computational models of reward procurement, we derive evidence for BLA as a neural integrator of reward value, history, and cost parameters. Taken together, BLA codes specific and temporally dynamic outcome representations in a distributed network to orchestrate adaptive responses. We provide evidence that experiences with opiates and psychostimulants alter these outcome representations in BLA, resulting in long-term modified action.

Psychophysical inference of frequency-following fidelity in the neural substrate for brain stimulation reward

The rewarding effect of electrical brain stimulation has been studied extensively for 60 years, yet the identity of the underlying neural circuitry remains unknown. Previous experiments have characterized the directly stimulated ("first-stage") neurons implicated in self-stimulation of the medial forebrain bundle. Their properties are consistent with those of fine, myelinated axons, at least some of which project rostro-caudally. These properties do not match those of dopaminergic neurons. The present psychophysical experiment estimates an additional first-stage characteristic: maximum firing frequency. We test a frequency-following model that maps the experimenter-set pulse frequency into the frequency of firing induced in the directly stimulated neurons. As pulse frequency is increased, firing frequency initially increases at the same rate, then becomes probabilistic, and finally levels off. The frequency-following function is based on the counter model which holds that the rewarding effect of a pulse train is determined by the aggregate spike rate triggered in first-stage neurons during a given interval. In 7 self-stimulating rats, we measured current- vs. pulse-frequency trade-off functions. The trade-off data were well described by the frequency-following model, and its upper asymptote was approached at a median value of 360 Hz (IQR = 46 Hz). This value implies a highly excitable, non-dopaminergic population of first-stage neurons. Incorporating the frequency-following function and parameters in Shizgal's 3-dimensional reward-mountain model improves its accuracy and predictive power.