Neurons in the macaque orbitofrontal cortex code relative preference of both rewarding and aversive outcomes

Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making

Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.

Laser stimulation of the skin for quantitative study of decision-making and motivation

Neuroeconomics studies how decision-making is guided by the value of rewards and punishments. But to date, little is known about how noxious experiences impact decisions. A challenge is the lack of an aversive stimulus that is dynamically adjustable in intensity and location, readily usable over many trials in a single experimental session, and compatible with multiple ways to measure neuronal activity. We show that skin laser stimulation used in human studies of aversion can be used for this purpose in several key animal models. We then use laser stimulation to study how neurons in the orbitofrontal cortex (OFC), an area whose many roles include guiding decisions among different rewards, encode the value of rewards and punishments. We show that some OFC neurons integrated the positive value of rewards with the negative value of aversive laser stimulation, suggesting that the OFC can play a role in more complex choices than previously appreciated.

Amygdala connectivity and implications for social cognition and disorders

The amygdala is a hub of subcortical region that is crucial in a wide array of affective and motivation-related behaviors. While early research contributed significantly to our understanding of this region's extensive connections to other subcortical and cortical regions, recent methodological advances have enabled researchers to better understand the details of these circuits and their behavioral contributions. Much of this work has focused specifically on investigating the role of amygdala circuits in social cognition. In this chapter, we review both long-standing knowledge and novel research on the amygdala's structure, function, and involvement in social cognition. We focus specifically on the amygdala's circuits with the medial prefrontal cortex, the orbitofrontal cortex, and the hippocampus, as these regions share extensive anatomic and functional connections with the amygdala. Furthermore, we discuss how dysfunction in the amygdala may contribute to social deficits in clinical disorders including autism spectrum disorder, social anxiety disorder, and Williams syndrome. We conclude that social functions mediated by the amygdala are orchestrated through multiple intricate interactions between the amygdala and its interconnected brain regions, endorsing the importance of understanding the amygdala from network perspectives.

Task- and Rest-based Functional Brain Connectivity in Food-related Reward Processes among Healthy Adolescents

It is known that the nucleus accumbens, orbitofrontal cortex and insula play a role in food-related reward processes. Although their interconnectedness would be an ideal topic for understanding food intake mechanisms, it nevertheless remains unclear especially in adolescent. Therefore, this study aims to investigate the effect of hunger on functional connectivity in healthy adolescents using task- and rest-based imaging. Fifteen participants underwent two MRI sessions, pre-lunch (hunger) and post-lunch (satiety), including food cue task and resting-state. During task- and rest-based imaging, functional connectivity was greater when hungry as opposed to satiated between the right posterior insula/nucleus accumbens, suggesting involvement of salient interoceptive stimuli signals. During task-based imaging, an increase was observed in functional connectivity when hungry as opposed to satiated between the medial and lateral orbitofrontal cortex which contributes to the perception of food deprivation as a frustration. A decrease was identified when hungry as opposed to satiated in functional connectivity in the right anterior orbitofrontal/accumbens and posterior insula/medial orbitofrontal cortices reflecting suppression of the affective and sensorial information. Conversely, functional connectivity was increased during aversive stimuli between the right medial orbitofrontal cortex and right posterior insula when hungry as opposed to satiated. This suggests that the value of valence could occur in the shift in connectivity between these two regions. In addition, during rest-based imaging, a left-sided lateralization was reported (accumbens/lateral orbitofrontal and accumbens/posterior insula) when hungry as opposed to satiated which may represent changes in internal state due to focus on the benefit of an upcoming meal.

Evaluation of cognitive effort in rats is not critically dependent on ventrolateral orbitofrontal cortex

Organisms must frequently evaluate the amount of effort to invest in pursuing future rewards. Despite explicit awareness of the potential benefits of cognitive work, individuals vary in their willingness to attempt cognitively demanding tasks, regardless of intellectual ability. Such differences may suggest that the degree to which cognitive effort degrades perceived outcome value is a subjective, rather than objective, process, similar to risk and delay discounting. Although numerous studies suggest the orbitofrontal cortex (OFC) is important for allowing subjective value estimates to be updated and/or used in cost/benefit decision-making, the causal role of the OFC in valuations of mental effort has received scant investigation. We therefore trained 24 female Long-Evans rats on the rodent cognitive effort task (rCET) and assessed performance following temporary bilateral inactivation of the ventrolateral OFC (vlOFC). In the rCET, rats decide at trial outset whether to perform an easy or hard attentional challenge, namely to localize a brief visual stimulus to one of five possible locations. The difficulty of the challenge is determined by the stimulus duration (1.0 vs. 0.2s for easy vs. hard trials respectively), and success on hard trials results in double the sugar pellet rewards. Somewhat surprisingly, inactivations of the vlOFC did not affect rats' willingness or ability to exert cognitive effort for larger rewards, despite increasing omissions and motor impulsivity on-task. When considered with previous work, it appears the vlOFC plays a minimal role in cognitive effort allocation specifically, and in valuations of effort more generally.

Encoding prediction signals during appetitive and aversive Pavlovian conditioning in the primate lateral hypothalamus

The lateral hypothalamus (LH), which plays a role in homeostatic functions such as appetite regulation, is also linked to arousal and motivational behavior. However, little is known about how these components are encoded in the LH. Thus cynomolgus monkeys were conditioned with two distinct contexts, i.e., an appetitive context with available rewards and an aversive context with predicted air puffs. Different LH neuron groups encoded different degrees of expectation, predictability, and risks of rewards in a specific manner. A nearly equal number of one-third of the recorded LH neurons showed a positive or negative correlation between their response to visual conditioned stimuli (CS) that predicted the probabilistic delivery of rewards (0%, 50%, and 100%) and the associative values. For another one-third of recorded neurons, a nearly equal number showed a positive or negative correlation between their responses to rewards [appetitive unconditioned stimulus (US)] and reward predictability. Some neurons exhibited their highest or lowest trace-period responses in the 50% reward trials. These response modulations were represented independently and overlaid on a consistent excitatory or inhibitory response across the conditioning events. LH neurons also showed consistent responses in the aversive context. However, the responses to aversive conditioning events depending on the air puff value and predictability were less common. The multifaceted modulation of consistent activity related to outcome predictions may reflect motivational and arousal signals. Furthermore, it may underlie the role the LH plays in the integration and relay of signals to cortices for adaptive and goal-directed physiological and behavioral responses to environmental changes.

NEW & NOTEWORTHY The lateral hypothalamus (LH) is implicated in motivational and arousal behavior; however, the detailed information carried by single LH neurons remains unclear. We demonstrate that primate LH neurons encode multiple combinations of signals concerning different degrees of expectation, appreciation, and uncertainty of rewards in consistent responses across conditioning events and between different contexts. This multifaceted modulation of activity may underlie the role of the LH as a critical node integrating motivational signals with arousal signals.

Prefrontal Theta Oscillations Promote Selective Encoding of Behaviorally Relevant Events

The ability to capture the most relevant information from everyday experiences without constantly learning unimportant details is vital to survival and mental health. While decreased activity of the medial prefrontal cortex (mPFC) is associated with failed or inflexible encoding of relevant events in the hippocampus, mechanisms used by the mPFC to discern behavioral relevance of events are not clear. To address this question, we chemogenetically activated excitatory neurons in the mPFC of male rats and examined its impact on local network activity and differential associative learning dependent on the hippocampus. Rats were exposed to two neutral stimuli in two environments whose contingency with an aversive stimulus changed systematically across days. Over 2 weeks of differential and reversal learning, theta band activity began to ramp up toward the expected onset of the aversive stimulus, and this ramping activity tracked the subsequent shift of the set (stimulus modality to environment) predictive of the aversive stimulus. With chemogenetic mPFC activation, the ramping activity emerged within a few sessions of differential learning, which paralleled faster learning and stronger correlations between the ramping activity and conditioned responses. Chemogenetic mPFC activity, however, did not affect the adjustment of ramping activity or behavior during reversal learning or set-shifting, suggesting that the faster learning was not because of a general enhancement of attention, sensory, or motor processing. Thus, the dynamics of the mPFC network activation during events provide a relevance-signaling mechanism through which the mPFC exerts executive control over the encoding of those events in the hippocampus.

Gustatory responses in macaque monkeys revealed with fMRI: Comments on taste, taste preference, and internal state

Studies of the neural mechanisms underlying value-based decision making typically employ food or fluid rewards to motivate subjects to perform cognitive tasks. Rewards are often treated as interchangeable, but it is well known that the specific tastes of foods and fluids and the values associated with their taste sensations influence choices and contribute to overall levels of food consumption. Accordingly, we characterized the gustatory system in three macaque monkeys (Macaca mulatta) and examined whether gustatory responses were modulated by preferences and hydration status. To identify taste-responsive cortex, we delivered small quantities (0.1 ml) of sucrose (sweet), citric acid (sour), or distilled water in random order without any predictive cues while scanning monkeys using event-related fMRI. Neural effects were evaluated by using each session in each monkey as a data point in a second-level analysis. By contrasting BOLD responses to sweet and sour tastes with those from distilled water in a group level analysis, we identified taste responses in primary gustatory cortex area G, an adjacent portion of the anterior insular cortex, and prefrontal cortex area 12o. Choice tests administered outside the scanner revealed that all three monkeys strongly preferred sucrose to citric acid or water. BOLD responses in the ventral striatum, ventral pallidum, and amygdala reflected monkeys' preferences, with greater BOLD responses to sucrose than citric acid. Finally, we examined the influence of hydration level by contrasting BOLD responses to receipt of fluids when monkeys were thirsty and after ad libitum water consumption. BOLD responses in area G and area 12o in the left hemisphere were greater following full hydration. By contrast, BOLD responses in portions of medial frontal cortex were reduced after ad libitum water consumption. These findings highlight brain regions involved in representing taste, taste preference and internal state.

Dietary dopamine depletion blunts reward network sensitivity to face trustworthiness

Research demonstrating responsiveness of the neural reward network to face trustworthiness has not assessed whether the effects are mediated by dopaminergic function. We filled this gap in the literature by investigating whether dietary dopamine depletion would blunt the sensitivity of neural activation to faces varying in trustworthiness across reward regions as well as the sensitivity of behavioral responses to those faces. As prolactin release is negatively regulated by dopamine, peripheral prolactin levels confirmed the efficacy of our manipulation. The dopamine depletion manipulation moderated neural activation to face trustworthiness in the amygdala, medial orbital frontal cortex, and ventral medial prefrontal cortex. Control participants ( n=20) showed nonlinear and linear neural activation to face trustworthiness in the amygdala and ventral medial prefrontal cortex, and nonlinear activation in the medial orbital frontal cortex, while depleted participants ( n=20) showed only a linear effect in the amygdala. Controls also showed stronger amygdala activation to high trustworthy faces than depleted participants. In contrast to effects on neural activation, dopamine depletion did not blunt the sensitivity of behavioral ratings. While this is the first study to demonstrate that dopamine depletion blunts the sensitivity of the neural reward system to social stimuli, namely faces varying in trustworthiness, future research should investigate behavioral measures that may be more responsive to dopaminergic effects than face ratings. Such research would shed further light on the possibility that individual differences in dopaminergic function that were simulated by our manipulation influence social interactions with people who vary in facial trustworthiness.

Older adults' neural activation in the reward circuit is sensitive to face trustworthiness

We examined older adult (OA) and younger adult (YA) neural sensitivity to face trustworthiness in reward circuit regions, previously found to respond to trustworthiness in YA. Interactions of face trustworthiness with age revealed effects exclusive to OA in the amygdala and caudate, and an effect that was not moderated by age in the dorsal anterior cingulate cortex (dACC). OA, but not YA, showed a nonlinear amygdala response to face trustworthiness, with significantly stronger activation response to high than to medium trustworthy faces, and no difference between low and medium or high. This may explain why an earlier study investigating OA amygdala activation to trustworthiness failed to find a significant effect, since only the linear low versus high trustworthiness difference was assessed. OA, but not YA, also showed significantly stronger activation to high than to low trustworthy faces in the right caudate, indicating a positive linear effect, consistent with previous YA research, as well as significantly stronger activation to high than to medium but not low trustworthy faces in the left caudate, indicating a nonlinear effect. Activation in dACC across both age groups showed a positive linear effect consistent with previous YA research. Finally, OA rated the faces as more trustworthy than did YA across all levels of trustworthiness. Future research should examine whether the null effects for YA were due to our inclusion of older faces. Research also should investigate possible implications of our findings for more ecologically valid OA responses to people who vary in facial trustworthiness.

Opioid and orexin hedonic hotspots in rat orbitofrontal cortex and insula

Hedonic hotspots are brain sites where particular neurochemical stimulations causally amplify the hedonic impact of sensory rewards, such as "liking" for sweetness. Here, we report the mapping of two hedonic hotspots in cortex, where mu opioid or orexin stimulations enhance the hedonic impact of sucrose taste. One hedonic hotspot was found in anterior orbitofrontal cortex (OFC), and another was found in posterior insula. A suppressive hedonic coldspot was also found in the form of an intervening strip stretching from the posterior OFC through the anterior and middle insula, bracketed by the two cortical hotspots. Opioid/orexin stimulations in either cortical hotspot activated Fos throughout a distributed "hedonic circuit" involving cortical and subcortical structures. Conversely, cortical coldspot stimulation activated circuitry for "hedonic suppression." Finally, food intake was increased by stimulations at several prefrontal cortical sites, indicating that the anatomical substrates in cortex for enhancing the motivation to eat are discriminable from those for hedonic impact.

The Good and Bad Differentially Encoded within the Subthalamic Nucleus in Rats(1,2,3)

The subthalamic nucleus (STN) has only recently been added into the reward circuit. It has been shown to encode information regarding rewards (4% sucrose, 32% cocaine). To investigate the encoding of negative value, STN neurons were recorded in rats performing a task using discriminative stimuli predicting various rewards and especially during the replacement of a positive reinforcer (4% sucrose) by an aversive reinforcer (quinine). The results show that STN neurons encode information relative to both positive and aversive reinforcers via specialized subpopulations. The specialization is reset when the context is modified (change from a favorable context (4% vs 32% sucrose) to an unfavorable context (quinine vs 32% sucrose). An excitatory response to the cue light predicting the reward seems to be associated with the preferred situation, suggesting that STN plays a role in encoding the relative value of rewards. STN also seems to play a critical role in the encoding of execution error. Indeed, various subpopulations of neurons responding exclusively at early (i.e., "oops neurons") or at correct lever release were identified. The oops neurons respond mostly when the preferred reward (32% sucrose) is missed. Furthermore, STN neurons respond to reward omission, suggesting a role in reward prediction error. These properties of STN neurons strengthen its position in the reward circuit as a key cerebral structure through which reward-related processes are mediated. It is particularly important given the fact that STN is the target of surgical treatment for Parkinson's disease and obsessive compulsive disorders, and has been suggested for the treatment of addiction as well.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

Appetitive and aversive information coding in the primate dorsal raphé nucleus

Serotonin is known to play a key role in the regulation of emotional behavior. There have been conflicting hypotheses about whether the central serotonergic system is involved in positive or negative emotional information processing. To reveal whether and how such opposing information processing can be achieved by single neurons in the dorsal raphé nucleus (DRN), the major source of serotonin in the forebrain, we recorded the activity of DRN neurons while monkeys were conditioned in a Pavlovian procedure with two distinct contexts: an appetitive block where a reward was available; and an aversive one where an airpuff was delivered. We found that single DRN neurons were involved in several aspects of both appetitive and aversive information processing. First, more than half of the recorded DRN neurons discriminated between appetitive and aversive contexts by tonic changes in their activity. In the appetitive context, they then kept track of the expected reward value indicated by the conditioned stimuli. Some of them also encoded an error between the obtained and expected values. In the aversive context, the same neurons maintained tonic modulation in their activity throughout the block. However, modulation of their responses to aversive task events depending on airpuff probability was less common. Together, these results indicate that single DRN neurons encode both appetitive and aversive information, but over differing time scales: relatively shorter for appetitive, and longer for aversive. Such temporally distinct processes of value coding in the DRN may provide the neural basis of emotional information processing in different contexts.

Neurophysiology of Reward-Guided Behavior: Correlates Related to Predictions, Value, Motivation, Errors, Attention, and Action

Many brain areas are activated by the possibility and receipt of reward. Are all of these brain areas reporting the same information about reward? Or are these signals related to other functions that accompany reward-guided learning and decision-making? Through carefully controlled behavioral studies, it has been shown that reward-related activity can represent reward expectations related to future outcomes, errors in those expectations, motivation, and signals related to goal- and habit-driven behaviors. These dissociations have been accomplished by manipulating the predictability of positively and negatively valued events. Here, we review single neuron recordings in behaving animals that have addressed this issue. We describe data showing that several brain areas, including orbitofrontal cortex, anterior cingulate, and basolateral amygdala signal reward prediction. In addition, anterior cingulate, basolateral amygdala, and dopamine neurons also signal errors in reward prediction, but in different ways. For these areas, we will describe how unexpected manipulations of positive and negative value can dissociate signed from unsigned reward prediction errors. All of these signals feed into striatum to modify signals that motivate behavior in ventral striatum and guide responding via associative encoding in dorsolateral striatum.

Impact of appetitive and aversive outcomes on brain responses: linking the animal and human literatures

Decision-making is motivated by the possibility of obtaining reward and/or avoiding punishment. Though many have investigated behavior associated with appetitive or aversive outcomes, few have examined behaviors that rely on both. Fewer still have addressed questions related to how anticipated appetitive and aversive outcomes interact to alter neural signals related to expected value, motivation, and salience. Here we review recent rodent, monkey, and human research that address these issues. Further development of this area will be fundamental to understanding the etiology behind human psychiatric diseases and cultivating more effective treatments.

Medial-lateral organization of the orbitofrontal cortex

Emerging evidence suggests that specific cognitive functions localize to different subregions of OFC, but the nature of these functional distinctions remains unclear. One prominent theory, derived from human neuroimaging, proposes that different stimulus valences are processed in separate orbital regions, with medial and lateral OFC processing positive and negative stimuli, respectively. Thus far, neurophysiology data have not supported this theory. We attempted to reconcile these accounts by recording neural activity from the full medial-lateral extent of the orbital surface in monkeys receiving rewards and punishments via gain or loss of secondary reinforcement. We found no convincing evidence for valence selectivity in any orbital region. Instead, we report differences between neurons in central OFC and those on the inferior-lateral orbital convexity, in that they encoded different sources of value information provided by the behavioral task. Neurons in inferior convexity encoded the value of external stimuli, whereas those in OFC encoded value information derived from the structure of the behavioral task. We interpret these results in light of recent theories of OFC function and propose that these distinctions, not valence selectivity, may shed light on a fundamental organizing principle for value processing in orbital cortex.

Contextual and social influences on valuation and choice

To survive in our complex environment, we have to adapt to changing contexts. Prior research that investigated how contextual changes are processed in the human brain has demonstrated important modulatory influences on multiple cognitive processes underlying decision-making, including perceptual judgments, working memory, as well as cognitive and attentional control. However, in everyday life, the importance of context is even more obvious during economic and social interactions, which often have implicit rule sets that need to be recognized by a decision-maker. Here, we review recent evidence from an increasing number of studies in the fields of Neuroeconomics and Social Neuroscience that investigate the neurobiological basis of contextual effects on valuation and social choice. Contrary to the assumptions of rational choice theory, multiple contextual factors, such as the availability of alternative choice options, shifts in reference point, and social context, have been shown to modulate behavior, as well as signals in task-relevant neural networks. A consistent picture that emerges from neurobiological results is that valuation-related activity in striatum and ventromedial prefrontal cortex is highly context dependent during both social and nonsocial choice. Alternative approaches to model and explain choice behavior, such as comparison-based choice models, as well as implications for future research are discussed.

The importance of context: when relative relief renders pain pleasant

Context can influence the experience of any event. For instance, the thought that "it could be worse" can improve feelings towards a present misfortune. In this study we measured hedonic feelings, skin conductance, and brain activation patterns in 16 healthy volunteers who experienced moderate pain in two different contexts. In the "relative relief context," moderate pain represented the best outcome, since the alternative outcome was intense pain. However, in the control context, moderate pain represented the worst outcome and elicited negative hedonic feelings. The context manipulation resulted in a "hedonic flip," such that moderate pain elicited positive hedonics in the relative relief context. Somewhat surprisingly, moderate pain was even rated as pleasant in this context, despite being reported as painful in the control context. This "hedonic flip" was corroborated by physiological and functional neuroimaging data. When moderate pain was perceived as pleasant, skin conductance and activity in insula and dorsal anterior cingulate were significantly attenuated relative to the control moderate stimulus. "Pleasant pain" also increased activity in reward and valuation circuitry, including the medial orbitofrontal and ventromedial prefrontal cortices. Furthermore, the change in outcome hedonics correlated with activity in the periacqueductal grey (PAG) of the descending pain modulatory system (DPMS). The context manipulation also significantly increased functional connectivity between reward circuitry and the PAG, consistent with a functional change of the DPMS due to the altered motivational state. The findings of this study point to a role for brainstem and reward circuitry in a context-induced "hedonic flip" of pain.

Value normalization in decision making: theory and evidence

A sizable body of evidence has shown that the brain computes several types of value-related signals to guide decision making, such as stimulus values, outcome values, and prediction errors. A critical question for understanding decision-making mechanisms is whether these value signals are computed using an absolute or a normalized code. Under an absolute code, the neural response used to represent the value of a given stimulus does not depend on what other values might have been encountered. By contrast, under a normalized code, the neural response associated with a given value depends on its relative position in the distribution of values. This review provides a simple framework for thinking about value normalization, and uses it to evaluate the existing experimental evidence.

Representations of appetitive and aversive information in the primate orbitofrontal cortex

Individuals weigh information about both rewarding and aversive stimuli to make adaptive decisions. Most studies of the orbitofrontal cortex (OFC), an area where appetitive and aversive neural subsystems might interact, have focused only on reward. Using a classical conditioning task where novel stimuli are paired with a reward or an aversive air puff, we discovered that two groups of orbitofrontal neurons respond preferentially to conditioned stimuli associated with rewarding and aversive outcomes; however, information about appetitive and aversive stimuli converges on individual neurons from both populations. Therefore, neurons in the OFC might participate in appetitive and aversive networks that track the motivational significance of stimuli even when they vary in valence and sensory modality. Further, we show that these networks, which also extend to the amygdala, exhibit different rates of change during reversal learning. Thus, although both networks represent appetitive and aversive associations, their distinct temporal dynamics might indicate different roles in learning processes.

Neuronal signals for reward risk in frontal cortex

Rewards can be viewed as probability distributions of reward values. Besides expected (mean) value, a key parameter of such distributions is variance (or standard deviation), which constitutes a measure of risk. Single neurons in orbitofrontal cortex signal risk mostly separately from value. Comparable risk signals in human frontal cortex reflect risk attitudes of individual participants. Subjective outcome value constitutes the primary economic decision variable. The terms risk avoidance and risk taking suggest that risk affects subjective outcome value, a basic tenet of economic decision theories. Correspondingly, risk reduces neuronal value signals in frontal cortex of human risk avoiders and enhances value signals in risk takers. Behavioral contrast effects and reference-dependent valuation demonstrate flexible reward valuation. As a potential correlate, value signals in orbitofrontal neurons adjust reward discrimination to variance (risk). These neurophysiological mechanisms of reward risk on economic decisions inform and validate theories of economic decision making under uncertainty.

The orbitofrontal cortex and the computation of subjective value: consolidated concepts and new perspectives

Remarkable progress has been made in recent years toward understanding the functions of the orbitofrontal cortex (OFC). The finding that neurons in this area encode the subjective value monkeys assign to different goods while making choices has been confirmed and extended by numerous studies using both primate neurophysiology and human imaging. Moreover, new lesion studies demonstrated that subjective values computed in the OFC are causally and specifically related to choice behavior. Importantly, values in the OFC are attached to goods, not to actions or to spatial locations. Furthermore, subjective values appear to be computed in this area even if the situation does not require a choice. In the light of this growing body of work, we propose that the primary function of the OFC is the computation of good identities and subjective values in an abstract representation. In this view, OFC neurons compute the subjective value of a good whenever that good is behaviorally relevant.

Differential encoding of factors influencing predicted reward value in monkey rostral anterior cingulate cortex

Background

The value of a predicted reward can be estimated based on the conjunction of both the intrinsic reward value and the length of time to obtain it. The question we addressed is how the two aspects, reward size and proximity to reward, influence the responses of neurons in rostral anterior cingulate cortex (rACC), a brain region thought to play an important role in reward processing.

Methods and Findings

We recorded from single neurons while two monkeys performed a multi-trial reward schedule task. The monkeys performed 1–4 sequential color discrimination trials to obtain a reward of 1–3 liquid drops. There were two task conditions, a valid cue condition, where the number of trials and reward amount were associated with visual cues, and a random cue condition, where the cue was picked from the cue set at random. In the valid cue condition, the neuronal firing is strongly modulated by the predicted reward proximity during the trials. Information about the predicted reward amount is almost absent at those times. In substantial subpopulations, the neuronal responses decreased or increased gradually through schedule progress to the predicted outcome. These two gradually modulating signals could be used to calculate the effect of time on the perception of reward value. In the random cue condition, little information about the reward proximity or reward amount is encoded during the course of the trial before reward delivery, but when the reward is actually delivered the responses reflect both the reward proximity and reward amount.

Conclusions

Our results suggest that the rACC neurons encode information about reward proximity and amount in a manner that is dependent on utility of reward information. The manner in which the information is represented could be used in the moment-to-moment calculation of the effect of time and amount on predicted outcome value.

Challenges of Interpreting Frontal Neurons during Value-Based Decision-Making

The frontal cortex is crucial to sound decision-making, and the activity of frontal neurons correlates with many aspects of a choice, including the reward value of options and outcomes. However, rewards are of high motivational significance and have widespread effects on neural activity. As such, many neural signals not directly involved in the decision process can correlate with reward value. With correlative techniques such as electrophysiological recording or functional neuroimaging, it can be challenging to distinguish neural signals underlying value-based decision-making from other perceptual, cognitive, and motor processes. In the first part of the paper, we examine how different value-related computations can potentially be confused. In particular, error-related signals in the anterior cingulate cortex, generated when one discovers the consequences of an action, might actually represent violations of outcome expectation, rather than errors per se. Also, signals generated at the time of choice are typically interpreted as reflecting predictions regarding the outcomes associated with the different choice alternatives. However, these signals could instead reflect comparisons between the presented choice options and previously presented choice alternatives. In the second part of the paper, we examine how value signals have been successfully dissociated from saliency-related signals, such as attention, arousal, and motor preparation in studies employing outcomes with both positive and negative valence. We hope that highlighting these issues will prove useful for future studies aimed at disambiguating the contribution of different neuronal populations to choice behavior.

Identifying a network of brain regions involved in aversion-related processing: a cross-species translational investigation

The ability to detect and respond appropriately to aversive stimuli is essential for all organisms, from fruit flies to humans. This suggests the existence of a core neural network which mediates aversion-related processing. Human imaging studies on aversion have highlighted the involvement of various cortical regions, such as the prefrontal cortex, while animal studies have focused largely on subcortical regions like the periaqueductal gray and hypothalamus. However, whether and how these regions form a core neural network of aversion remains unclear. To help determine this, a translational cross-species investigation in humans (i.e., meta-analysis) and other animals (i.e., systematic review of functional neuroanatomy) was performed. Our results highlighted the recruitment of the anterior cingulate cortex, the anterior insula, and the amygdala as well as other subcortical (e.g., thalamus, midbrain) and cortical (e.g., orbitofrontal) regions in both animals and humans. Importantly, involvement of these regions remained independent of sensory modality. This study provides evidence for a core neural network mediating aversion in both animals and humans. This not only contributes to our understanding of the trans-species neural correlates of aversion but may also carry important implications for psychiatric disorders where abnormal aversive behavior can often be observed.

Decision making and reward in frontal cortex: complementary evidence from neurophysiological and neuropsychological studies

Patients with damage to the prefrontal cortex (PFC)—especially the ventral and medial parts of PFC—often show a marked inability to make choices that meet their needs and goals. These decision-making impairments often reflect both a deficit in learning concerning the consequences of a choice, as well as deficits in the ability to adapt future choices based on experienced value of the current choice. Thus, areas of PFC must support some value computations that are necessary for optimal choice. However, recent frameworks of decision making have highlighted that optimal and adaptive decision making does not simply rest on a single computation, but a number of different value computations may be necessary. Using this framework as a guide, we summarize evidence from both lesion studies and single-neuron physiology for the representation of different value computations across PFC areas.

Different time courses for learning-related changes in amygdala and orbitofrontal cortex

The orbitofrontal cortex (OFC) and amygdala are thought to participate in reversal learning, a process in which cue-outcome associations are switched. However, current theories disagree on whether OFC directs reversal learning in the amygdala. Here, we show that during reversal of cues' associations with rewarding and aversive outcomes, neurons that respond preferentially to stimuli predicting aversive events update more quickly in amygdala than OFC; meanwhile, OFC neurons that respond preferentially to reward-predicting stimuli update more quickly than those in the amygdala. After learning, however, OFC consistently differentiates between impending reinforcements with a shorter latency than the amygdala. Finally, analysis of local field potentials (LFPs) reveals a disproportionate influence of OFC on amygdala that emerges after learning. We propose that reversal learning is supported by complex interactions between neural circuits spanning the amygdala and OFC, rather than directed by any single structure.

Prefrontal cortex activity during the discrimination of relative distance

To compare with our previous findings on relative-duration discrimination, we studied prefrontal cortex activity as monkeys performed a relative-distance discrimination task. We wanted to know whether the same parts of the prefrontal cortex compare durations and distances and, if so, whether they use similar mechanisms. Two stimuli appeared sequentially on a video screen, one above a fixed reference point, the other below it by a different distance. After a delay period, the same two stimuli reappeared (as choice stimuli), and the monkeys' task was to choose the one that had appeared farther from the reference point during its initial presentation. We recorded from neurons in the dorsolateral prefrontal cortex (area 46) and the caudal prefrontal cortex (area 8). Although some prefrontal neurons encoded the absolute distance of a stimulus from the reference point, many more encoded relative distance. Categorical representations ("farther") predominated over parametric ones ("how much farther"). Relative-distance coding was most often abstract, coding the farther or closer stimulus to the same degree, independent of its position on the screen. During the delay period before the choice stimuli appeared, feature-based coding supplanted order-based coding, and position-based coding-always rare-decreased to chance levels. The present results closely resembled those for a duration-discrimination task in the same cortical areas. We conclude, therefore, that these areas contribute to decisions based on both spatial and temporal information.

Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs

How do addictive drugs hijack the brain's reward system? This review speculates how normal, physiological reward processes may be affected by addictive drugs. Addictive drugs affect acute responses and plasticity in dopamine neurons and postsynaptic structures. These effects reduce reward discrimination, increase the effects of reward prediction error signals, and enhance neuronal responses to reward-predicting stimuli, which may contribute to compulsion. Addictive drugs steepen neuronal temporal reward discounting and create temporal myopia that impairs the control of drug taking. Tonically enhanced dopamine levels may disturb working memory mechanisms necessary for assessing background rewards and thus may generate inaccurate neuronal reward predictions. Drug-induced working memory deficits may impair neuronal risk signaling, promote risky behaviors, and facilitate preaddictive drug use. Malfunctioning adaptive reward coding may lead to overvaluation of drug rewards. Many of these malfunctions may result in inadequate neuronal decision mechanisms and lead to choices biased toward drug rewards.

Reward magnitude coding in primate amygdala neurons

Animals assess the values of rewards to learn and choose the best possible outcomes. We studied how single neurons in the primate amygdala coded reward magnitude, an important variable determining the value of rewards. A single, Pavlovian-conditioned visual stimulus predicted fruit juice to be delivered with one of three equiprobable volumes (P = 1/3). A population of amygdala neurons showed increased activity after reward delivery, and almost one half of these responses covaried with reward magnitude in a monotonically increasing or decreasing fashion. A subset of the reward responding neurons were tested with two different probability distributions of reward magnitude; the reward responses in almost one half of them adapted to the predicted distribution and thus showed reference-dependent coding. These data suggest parametric reward value coding in the amygdala as a characteristic component of its function in reinforcement learning and economic decision making.

Neural activation in the "reward circuit" shows a nonlinear response to facial attractiveness

Positive behavioral responses to attractive faces have led neuroscientists to investigate underlying neural mechanisms in a "reward circuit" that includes brain regions innervated by dopamine pathways. Using male faces ranging from attractive to extremely unattractive, disfigured ones, this study is the first to demonstrate heightened responses to both rewarding and aversive faces in numerous areas of this putative reward circuit. Parametric analyses employing orthogonal linear and nonlinear regressors revealed positive nonlinear effects in anterior cingulate cortex, lateral orbital frontal cortex (LOFC), striatum (nucleus accumbens, caudate, putamen), and ventral tegmental area, in addition to replicating previously documented linear effects in medial orbital frontal cortex (MOFC) and LOFC and nonlinear effects in amygdala and MOFC. The widespread nonlinear responses are consistent with single cell recordings in animals showing responses to both rewarding and aversive stimuli, and with some human fMRI investigations of non-face stimuli. They indicate that the reward circuit does not process face valence with any simple dissociation of function across structures. Perceiver gender modulated some responses to our male faces: Women showed stronger linear effects, and men showed stronger nonlinear effects, which may have functional implications. Our discovery of nonlinear responses to attractiveness throughout the reward circuit echoes the history of amygdala research: Early work indicated a linear response to threatening stimuli, including faces; later work also revealed a nonlinear response with heightened activation to affectively salient stimuli regardless of valence. The challenge remains to determine how such dual coding influences feelings, such as pleasure and pain, and guides goal-related behavioral responses, such as approach and avoidance.

Adaptation of reward sensitivity in orbitofrontal neurons

Animals depend on a large variety of rewards but their brains have a limited dynamic coding range. When rewards are uncertain, neuronal coding needs to cover a wide range of possible rewards. However, when reward is likely to occur within a specific range, focusing the sensitivity on the predicted range would optimize the discrimination of small reward differences. One way to overcome the trade-off between wide coverage and optimal discrimination is to adapt reward sensitivity dynamically to the available rewards. We investigated how changes in reward distribution influenced the coding of reward in the orbitofrontal cortex. Animals performed an oculomotor task in which a fixation cue predicted the SD of the probability distribution of juice volumes, while the expected mean volume was kept constant. A subsequent cue specified the exact juice volume obtained for a correct saccade response. Population responses of orbitofrontal neurons that reflected the predicted juice volume showed adaptation to the reward distribution. Statistical tests on individual responses revealed that a quarter of value-coding neurons shifted the reward sensitivity slope significantly between two reward distributions, whereas the remaining neurons showed insignificant change or lack of adaptation. Adaptations became more prominent when reward distributions changed less frequently, indicating time constraints for assessing reward distributions and adjusting neuronal sensitivity. The observed neuronal adaptation would optimize discrimination and contribute to the efficient coding of a large variety of potential rewards by neurons with limited dynamic range.

Range-adapting representation of economic value in the orbitofrontal cortex

While making economic choices, individuals assign subjective values to the available options. Values computed in different behavioral conditions, however, can vary substantially. The same person might choose some times between goods worth a few dollars, and other times between goods worth thousands of dollars, or more. How does the brain system that computes values -- the "valuation system" -- handle this large variability? Here we show that the representation of value in the orbitofrontal cortex (OFC), an area implicated in value assignment during economic choice, adapts to the behavioral condition of choice and, more specifically, to the range of values available in any given condition. In the experiments, monkeys chose between different juices and their choice patterns provided a measure of subjective value. Value ranges were varied from session to session and, in each session, OFC neurons encoded values in a linear way. Across the population, the neuronal sensitivity (defined as the change in neuronal activity elicited by the increase in one value unit) was inversely proportional to the value range. Conversely, the neuronal activity range did not depend on the value range. This phenomenon of range adaptation complements that of menu invariance observed in a previous study. Indeed, the activity of each neuron adapts to the range values it encodes but does not depend on other available goods. Our results thus suggest that the representation of value in the OFC is at one time instantiative of preference transitivity (menu invariance) and computationally efficient (range adaptation).

The convergence of information about rewarding and aversive stimuli in single neurons

Neuroscientists, psychologists, clinicians, and economists have long been interested in how individuals weigh information about potential rewarding and aversive stimuli to make decisions and to regulate their emotions. However, we know relatively little about how appetitive and aversive systems interact in the brain, as most prior studies have investigated only one valence of reinforcement. Previous work has suggested that primate orbitofrontal cortex (OFC) represents information about the reward value of stimuli. We therefore investigated whether OFC also represents information about aversive stimuli, and, if so, whether individual neurons process information about both rewarding and aversive stimuli. Monkeys performed a trace conditioning task in which different novel abstract visual stimuli (conditioned stimuli, CSs) predicted the occurrence of one of three unconditioned stimuli (USs): a large liquid reward, a small liquid reward, or an aversive air-puff. Three lines of evidence suggest that information about rewarding and aversive stimuli converges in individual neurons in OFC. First, OFC neurons often responded to both rewarding and aversive USs, despite their different sensory features. Second, OFC neural responses to CSs often encoded information about both potential rewarding and aversive stimuli, even though these stimuli differed in both valence and sensory modality. Finally, OFC neural responses were correlated with monkeys' behavioral use of information about both rewarding and aversive CS-US associations. These data indicate that processing of appetitive and aversive stimuli converges at the single cell level in OFC, providing a possible substrate for executive and emotional processes that require using information from both appetitive and aversive systems.

Structure-function relationships in the processing of regret in the orbitofrontal cortex

The influence of counterfactual thinking and regret on choice behavior has been widely acknowledged in economic science (Bell in Oper Res 30:961-981, 1982; Kahneman and Tversky in Judgment under uncertainty: heuristics and biases. Cambridge University Press, Cambridge, pp 201-210, 1982; Loomes and Sugden in Econ J 92:805-824, 1982). Neuroimaging studies have only recently begun to explore the neural correlates of this psychological factor and orbitofrontal cortex (OFC) activity was observed in several of them depending of the exact characteristics of the employed paradigm. This selective OFC involvement and, moreover, a consistently found dissociation of medial and lateral OFC activity clusters allow inferences to the function of this structure in counterfactual thinking and regret. Vice versa, the differential contribution of OFC subregions to these processes also adds evidence to the current debate on the function of this cortical structure in decision-making that attracted increasing attention in recent years.

The role of human orbitofrontal cortex in value comparison for incommensurable objects

The human orbitofrontal cortex is strongly implicated in appetitive valuation. Whether its role extends to support comparative valuation necessary to explain probabilistic choice patterns for incommensurable goods is unknown. Using a binary choice paradigm, we derived the subjective values of different bundles of goods, under conditions of both gain and loss. We demonstrate that orbitofrontal activation reflects the difference in subjective value between available options, an effect evident across valuation for both gains and losses. In contrast, activation in dorsal striatum and supplementary motor areas reflects subjects' choice probabilities. These findings indicate that orbitofrontal cortex plays a pivotal role in valuation for incommensurable goods, a critical component process in human decision making.

Context-dependent representation of reinforcement in monkey amygdala

It has been reported that neurons in the amygdala respond to visual cues that predict reward and aversive outcomes. However, it remains unclear whether the representation of reinforcement in the amygdala depends on the relative preference for an outcome compared with another simultaneously available outcome. In this study, we introduced three reinforcements (juice, water, and electrical stimulus) and used two of them in one experimental block. Of 52 neurons that showed cue responses reflecting the outcome information, 23% of amygdala neurons coded preferred outcomes, whereas only one neuron coded nonpreferred outcomes. These proportions of amygdala were significantly different from those of the orbitofrontal cortex, which had both types of neurons.

The encoding of cocaine vs. natural rewards in the striatum of nonhuman primates: categories with different activations

The behavioral and motivational changes that result from use of abused substances depend upon activation of neuronal populations in the reward centers of the brain, located primarily in the corpus striatum in primates. To gain insight into the cellular mechanisms through which abused drugs reinforce behavior in the primate brain, changes in firing of neurons in the ventral (VStr, nucleus accumbens) and dorsal (DStr, caudate-putamen) striatum to "natural" (juice) vs. drug (i.v. cocaine) rewards were examined in four rhesus monkeys performing a visual Go-Nogo decision task. Task-related striatal neurons increased firing to one or more of the specific events that occurred within a trial represented by (1) Target stimuli (Go trials) or (2) Nogotarget stimuli (Nogo trials), and (3) Reward delivery for correct performance. These three cell populations were further subdivided into categories that reflected firing exclusively on one or the other type of signaled reward (juice or cocaine) trial (20%-30% of all cells), or, a second subpopulation that fired on both (cocaine and juice) types of rewarded trial (50%). Results show that neurons in the primate striatum encoded cocaine-rewarded trials similar to juice-rewarded trials, except for (1) increased firing on cocaine-rewarded trials, (2) prolonged activation during delivery of i.v. cocaine infusion, and (3) differential firing in ventral (VStr cells) vs. dorsal (DStr cells) striatum cocaine-rewarded trials. Reciprocal activations of antithetic subpopulations of cells during different temporal intervals within the same trial suggest a functional interaction between processes that encode drug and natural rewards in the primate brain.

Neural representation of behavioral outcomes in the orbitofrontal cortex

The orbitofrontal cortex (OFC) is important in processing rewards and other behavioral outcomes. Here, we review from a computational perspective recent progress in understanding this complex function. OFC neurons appear to represent abstract outcome values, which may facilitate the comparison of options, as well as concrete outcome attributes, such as flavor or location, which may enable predictive cues to access current outcome values in the face of dynamic modulation by internal state, context and learning. OFC can use reinforcement learning to generate outcome predictions; it can also generate outcome predictions using other mechanisms, including the evaluation of decision confidence or uncertainty. OFC neurons encode not only the mean expected outcome but also the variance, consistent with the idea that OFC uses a probabilistic population code to represent outcomes. We suggest that further attention to the nature of its representations and algorithms will be critical to further elucidating OFC function.

Behavioral and neural changes after gains and losses of conditioned reinforcers

Human behaviors can be more powerfully influenced by conditioned reinforcers, such as money, than by primary reinforcers. Moreover, people often change their behaviors to avoid monetary losses. However, the effect of removing conditioned reinforcers on choices has not been explored in animals, and the neural mechanisms mediating the behavioral effects of gains and losses are not well understood. To investigate the behavioral and neural effects of gaining and losing a conditioned reinforcer, we trained rhesus monkeys for a matching pennies task in which the positive and negative values of its payoff matrix were realized by the delivery and removal of a conditioned reinforcer. Consistent with the findings previously obtained with non-negative payoffs and primary rewards, the animal's choice behavior during this task was nearly optimal. Nevertheless, the gain and loss of a conditioned reinforcer significantly increased and decreased, respectively, the tendency for the animal to choose the same target in subsequent trials. We also found that the neurons in the dorsomedial frontal cortex, dorsal anterior cingulate cortex, and dorsolateral prefrontal cortex often changed their activity according to whether the animal earned or lost a conditioned reinforcer in the current or previous trial. Moreover, many neurons in the dorsomedial frontal cortex also signaled the gain or loss occurring as a result of choosing a particular action as well as changes in the animal's behaviors resulting from such gains or losses. Thus, primate medial frontal cortex might mediate the behavioral effects of conditioned reinforcers and their losses.