Partial Adaptation of Obtained and Observed Value Signals Preserves Information about Gains and Losses

Efficient value synthesis in the orbitofrontal cortex explains how loss aversion adapts to the ranges of gain and loss prospects

Is irrational behavior the incidental outcome of biological constraints imposed on neural information processing? In this work, we consider the paradigmatic case of gamble decisions, where gamble values integrate prospective gains and losses. Under the assumption that neurons have a limited firing response range, we show that mitigating the ensuing information loss within artificial neural networks that synthetize value involves a specific form of self-organized plasticity. We demonstrate that the ensuing efficient value synthesis mechanism induces value range adaptation. We also reveal how the ranges of prospective gains and/or losses eventually determine both the behavioral sensitivity to gains and losses and the information content of the network. We test these predictions on two fMRI datasets from the OpenNeuro.org initiative that probe gamble decision-making but differ in terms of the range of gain prospects. First, we show that peoples' loss aversion eventually adapts to the range of gain prospects they are exposed to. Second, we show that the strength with which the orbitofrontal cortex (in particular: Brodmann area 11) encodes gains and expected value also depends upon the range of gain prospects. Third, we show that, when fitted to participant's gambling choices, self-organizing artificial neural networks generalize across gain range contexts and predict the geometry of information content within the orbitofrontal cortex. Our results demonstrate how self-organizing plasticity aiming at mitigating information loss induced by neurons' limited response range may result in value range adaptation, eventually yielding irrational behavior.

Humans can infer social preferences from decision speed alone

Humans are known to be capable of inferring hidden preferences and beliefs of their conspecifics when observing their decisions. While observational learning based on choices has been explored extensively, the question of how response times (RT) impact our learning of others' social preferences has received little attention. Yet, while observing choices alone can inform us about the direction of preference, they reveal little about the strength of this preference. In contrast, RT provides a continuous measure of strength of preference with faster responses indicating stronger preferences and slower responses signaling hesitation or uncertainty. Here, we outline a preregistered orthogonal design to investigate the involvement of both choices and RT in learning and inferring other's social preferences. Participants observed other people's behavior in a social preferences task (Dictator Game), seeing either their choices, RT, both, or no information. By coupling behavioral analyses with computational modeling, we show that RT is predictive of social preferences and that observers were able to infer those preferences even when receiving only RT information. Based on these findings, we propose a novel observational reinforcement learning model that closely matches participants' inferences in all relevant conditions. In contrast to previous literature suggesting that, from a Bayesian perspective, people should be able to learn equally well from choices and RT, we show that observers' behavior substantially deviates from this prediction. Our study elucidates a hitherto unknown sophistication in human observational learning but also identifies important limitations to this ability.

Range adaptive value representations in schizophrenia and major depression

Anhedonia and amotivation are core symptoms of schizophrenia (SCZ) and major depressive disorder (MDD). Reward processing involves constructing and contrasting the representations for expected value (EV) and outcome value (OV) of a given stimulus, a phenomenon termed range adaptation. Impaired range adaptation can lead to anhedonia and amotivation. This study aimed to examine range adaptation in SCZ patients and MDD patients. Fifty SCZ, 46 MDD patients and 56 controls completed the Effort-based Pleasure Experience Task to measure EV and OV adaptation. SCZ and MDD patients showed altered range adaptation, albeit in different patterns. SCZ patients exhibited over-adaptation to OV and reduced adaptation to EV. By contrast, MDD patients exhibited diminished OV adaptation but intact EV adaptation. Both OV and EV adaptation were correlated with anhedonia and amotivation in SCZ and MDD. Taken together, our findings suggest that range adaptation is altered in both SCZ and MDD patients. Associations of OV and EV adaptation with anhedonia and amotivation were consistently found in SCZ and MDD patients. Impaired range adaptation in SCZ and MDD patients may be putative neural mechanisms and potential intervention targets for anhedonia and amotivation.

Electrophysiological population dynamics reveal context dependencies during decision making in human frontal cortex

Evidence from monkeys and humans suggests that the orbitofrontal cortex (OFC) encodes the subjective value of options under consideration during choice. Data from non-human primates suggests that these value signals are context-dependent, representing subjective value in a way influenced by the decision makers' recent experience. Using electrodes distributed throughout cortical and subcortical structures, human epilepsy patients performed an auction task where they repeatedly reported the subjective values they placed on snack food items. High-gamma activity in many cortical and subcortical sites including the OFC positively correlated with subjective value. Other OFC sites showed signals contextually modulated by the subjective value of previously offered goods-a context dependency predicted by theory but not previously observed in humans. These results suggest that value and value-context signals are simultaneously present but separately represented in human frontal cortical activity.

Worth the Work? Monkeys Discount Rewards by a Subjective Adapting Effort Cost

All life must solve how to allocate limited energy resources to maximize benefits from scarce opportunities. Economic theory posits decision makers optimize choice by maximizing the subjective benefit (utility) of reward minus the subjective cost (disutility) of the required effort. While successful in many settings, this model does not fully account for how experience can alter reward-effort trade-offs. Here, we test how well the subtractive model of effort disutility explains the behavior of two male nonhuman primates (Macaca mulatta) in a binary choice task in which reward quantity and physical effort to obtain were varied. Applying random utility modeling to independently estimate reward utility and effort disutility, we show the subtractive effort model better explains out-of-sample choice behavior when compared with parabolic and exponential effort discounting. Furthermore, we demonstrate that effort disutility depends on previous experience of effort: in analogy to work from behavioral labor economics, we develop a model of reference-dependent effort disutility to explain the increased willingness to expend effort following previous experience of effortful options in a session. The result of this analysis suggests that monkeys discount reward by an effort cost that is measured relative to an expected effort learned from previous trials. When this subjective cost of effort, a function of context and experience, is accounted for, trial-by-trial choices can be explained by the subtractive cost model of effort. Therefore, in searching for net utility signals that may underpin effort-based decision-making in the brain, careful measurement of subjective effort costs is an essential first step.SIGNIFICANCE STATEMENT All decision-makers need to consider how much effort they need to expend when evaluating potential options. Economic theories suggest that the optimal way to choose is by cost-benefit analysis of reward against effort. To be able to do this efficiently over many decision contexts, this needs to be done flexibly, with appropriate adaptation to context and experience. Therefore, in aiming to understand how this might be achieved in the brain, it is important to first carefully measure the subjective cost of effort. Here, we show monkeys make reward-effort cost-benefit decisions, subtracting the subjective cost of effort from the subjective value of rewards. Moreover, the subjective cost of effort is dependent on the monkeys' experience of effort in previous trials.

Effects of blocked versus interleaved training on relative value learning

In reinforcement learning tasks, people learn the values of options relative to other options in the local context. Prior research suggests that relative value learning is enhanced when choice contexts are temporally clustered in a blocked sequence compared to a randomly interleaved sequence. The present study was aimed at further investigating the effects of blocked versus interleaved training using a choice task that distinguishes among different contextual encoding models. Our results showed that the presentation format in which contexts are experienced can lead to qualitatively distinct forms of relative value learning. This conclusion was supported by a combination of model-free and model-based analyses. In the blocked condition, choice behavior was most consistent with a reference point model in which outcomes are encoded relative to a dynamic estimate of the contextual average reward. In contrast, the interleaved condition was best described by a range-frequency encoding model. We propose that blocked training makes it easier to track contextual outcome statistics, such as the average reward, which may then be used to relativize the values of experienced outcomes. When contexts are interleaved, range-frequency encoding may serve as a more efficient means of storing option values in memory for later retrieval.

Intrinsic rewards explain context-sensitive valuation in reinforcement learning

When observing the outcome of a choice, people are sensitive to the choice's context, such that the experienced value of an option depends on the alternatives: getting $1 when the possibilities were 0 or 1 feels much better than when the possibilities were 1 or 10. Context-sensitive valuation has been documented within reinforcement learning (RL) tasks, in which values are learned from experience through trial and error. Range adaptation, wherein options are rescaled according to the range of values yielded by available options, has been proposed to account for this phenomenon. However, we propose that other mechanisms-reflecting a different theoretical viewpoint-may also explain this phenomenon. Specifically, we theorize that internally defined goals play a crucial role in shaping the subjective value attributed to any given option. Motivated by this theory, we develop a new "intrinsically enhanced" RL model, which combines extrinsically provided rewards with internally generated signals of goal achievement as a teaching signal. Across 7 different studies (including previously published data sets as well as a novel, preregistered experiment with replication and control studies), we show that the intrinsically enhanced model can explain context-sensitive valuation as well as, or better than, range adaptation. Our findings indicate a more prominent role of intrinsic, goal-dependent rewards than previously recognized within formal models of human RL. By integrating internally generated signals of reward, standard RL theories should better account for human behavior, including context-sensitive valuation and beyond.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

Worth the work? Monkeys discount rewards by a subjective adapting effort cost

All life must solve how to allocate limited energy resources to maximise benefits from scarce opportunities. Economic theory posits decision makers optimise choice by maximising the subjective benefit (utility) of reward minus the subjective cost (disutility) of the required effort. While successful in many settings, this model does not fully account for how experience can alter reward-effort trade-offs. Here we test how well the subtractive model of effort disutility explains the behavior of two non-human primates ( Macaca mulatta ) in a binary choice task in which reward quantity and physical effort to obtain were varied.Applying random utility modelling to independently estimate reward utility and effort disutility, we show the subtractive effort model better explains out-of-sample choice behavior when compared to parabolic and exponential effort discounting. Furthermore, we demonstrate that effort disutility is dependent on previous experience of effort: in analogy to work from behavioral labour economics, we develop a model of reference-dependent effort disutility to explain the increased willingness to expend effort following previous experience of effortful options in a session. The result of this analysis suggests that monkeys discount reward by an effort cost that is measured relative to an expected effort learned from previous trials. When this subjective cost of effort, a function of context and experience, is accounted for, trial-by-trial choice behavior can be explained by the subtractive cost model of effort.Therefore, in searching for net utility signals that may underpin effort-based decision-making in the brain, careful measurement of subjective effort costs is an essential first step.

Significance

All decision-makers need to consider how much effort they need to expend when evaluating potential options. Economic theories suggest that the optimal way to choose is by cost-benefit analysis of reward against effort. To be able to do this efficiently over many decision contexts, this needs to be done flexibly, with appropriate adaptation to context and experience. Therefore, in aiming to understand how this might be achieved in the brain, it is important to first carefully measure the subjective cost of effort. Here we show monkeys make reward-effort cost-benefit decisions, subtracting the subjective cost of effort from the subjective value of rewards. Moreover, the subjective cost of effort is dependent on the monkeys’ experience of effort in previous trials.

Testing models of context-dependent outcome encoding in reinforcement learning

Previous studies of reinforcement learning (RL) have established that choice outcomes are encoded in a context-dependent fashion. Several computational models have been proposed to explain context-dependent encoding, including reference point centering and range adaptation models. The former assumes that outcomes are centered around a running estimate of the average reward in each choice context, while the latter assumes that outcomes are compared to the minimum reward and then scaled by an estimate of the range of outcomes in each choice context. However, there are other computational mechanisms that can explain context dependence in RL. In the present study, a frequency encoding model is introduced that assumes outcomes are evaluated based on their proportional rank within a sample of recently experienced outcomes from the local context. A range-frequency model is also considered that combines the range adaptation and frequency encoding mechanisms. We conducted two fully incentivized behavioral experiments using choice tasks for which the candidate models make divergent predictions. The results were most consistent with models that incorporate frequency or rank-based encoding. The findings from these experiments deepen our understanding of the underlying computational processes mediating context-dependent outcome encoding in human RL.

Adaptive coding occurs in object categorization and may not be associated with schizotypal personality traits

Processing more likely inputs with higher sensitivity (adaptive coding) enables the brain to represent the large range of inputs coming in from the world. Healthy individuals high in schizotypy show reduced adaptive coding in the reward domain but it is an open question whether these deficits extend to non-motivational domains, such as object categorization. Here, we develop a novel variant of a classic task to test range adaptation for face/house categorization in healthy participants on the psychosis spectrum. In each trial of this task, participants decide whether a presented image is a face or a house. Images vary on a face-house continuum and appear in both wide and narrow range blocks. The wide range block includes most of the face-house continuum (2.50-97.5% face), while the narrow range blocks limit inputs to a smaller section of the continuum (27.5-72.5% face). Adaptive coding corresponds to better performance for the overlapping smaller section of the continuum in the narrow range than in the wide range block. We find that participants show efficient use of the range in this task, with more accurate responses in the overlapping section for the narrow range blocks relative to the wide range blocks. However, we find little evidence that range adaptation in our object categorization task is reduced in healthy individuals scoring high on schizotypy. Thus, reduced range adaptation may not be a domain-general feature of schizotypy.

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

Comparing adaptive coding of reward in bipolar I disorder and schizophrenia

Deficits in neural processing of reward have been described in both bipolar disorder (BD) and schizophrenia (SZ), but it remains unclear to what extent these deficits are caused by similar mechanisms. Efficient reward processing relies on adaptive coding which allows representing large input spans by limited neuronal encoding ranges. Deficits in adaptive coding of reward have previously been observed across the SZ spectrum and correlated with total symptom severity. In the present work, we sought to establish whether adaptive coding is similarly affected in patients with BD. Twenty-five patients with BD, 27 patients with SZ and 25 healthy controls performed a variant of the Monetary Incentive Delay task during functional magnetic resonance imaging in two reward range conditions. Adaptive coding was impaired in the posterior part of the right caudate in BD and SZ (trend level). In contrast, BD did not show impaired adaptive coding in the anterior caudate and right precentral gyrus/insula, where SZ showed deficits compared to healthy controls. BD patients show adaptive coding deficits that are similar to those observed in SZ in the right posterior caudate. Adaptive coding in BD appeared more preserved as compared to SZ participants especially in the more anterior part of the right caudate and to a lesser extent also in the right precentral gyrus. Thus, dysfunctional adaptive coding could constitute a fundamental deficit in severe mental illnesses that extends beyond the SZ spectrum.

Adaptation of utility functions to reward distribution in rhesus monkeys

This study investigated how the experience of different reward distributions would shape the utility functions that can be inferred from economic choice. Despite the generally accepted notion that utility functions are not insensitive to external references, the exact way in which such changes take place remains largely unknown. Here we benefitted from the capacity to engage in thorough and prolonged empirical tests of economic choice by one of our evolutionary cousins, the rhesus macaque. We analyzed data from thousands of binary choices and found that the animals' preferences changed depending on the statistics of rewards experienced in the past (up to weeks) and that these changes could reflect monkeys' adapting their expectations of reward. The utility functions we elicited from their choices stretched and shifted over several months of sequential changes in the mean and range of rewards that the macaques experienced. However, this adaptation was usually incomplete, suggesting that - even after months - past experiences held weight when monkeys' assigned value to future rewards. Rather than having stable and fixed preferences assumed by normative economic models, our results demonstrate that rhesus macaques flexibly shape their preferences around the past and present statistics of their environment. That is, rather than relying on a singular reference-point, reference-dependent preferences are likely to capture a monkey's range of expectations.

Multi-scale neural decoding and analysis

Objective.

Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.

Approach.

We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.

Main results.

We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.

Significance.

This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

Two sides of the same coin: Beneficial and detrimental consequences of range adaptation in human reinforcement learning

Evidence suggests that economic values are rescaled as a function of the range of the available options. Although locally adaptive, range adaptation has been shown to lead to suboptimal choices, particularly notable in reinforcement learning (RL) situations when options are extrapolated from their original context to a new one. Range adaptation can be seen as the result of an adaptive coding process aiming at increasing the signal-to-noise ratio. However, this hypothesis leads to a counterintuitive prediction: Decreasing task difficulty should increase range adaptation and, consequently, extrapolation errors. Here, we tested the paradoxical relation between range adaptation and performance in a large sample of participants performing variants of an RL task, where we manipulated task difficulty. Results confirmed that range adaptation induces systematic extrapolation errors and is stronger when decreasing task difficulty. Last, we propose a range-adapting model and show that it is able to parsimoniously capture all the behavioral results.

The motivational cost of inequality: Opportunity gaps reduce the willingness to work

Factors beyond a person’s control, such as demographic characteristics at birth, often influence the availability of rewards an individual can expect for their efforts. We know surprisingly little how such differences in opportunities impact human motivation. To test this, we designed a study in which we arbitrarily varied the reward offered to each participant in a group for performing the same task. Participants then had to decide whether or not they were willing to exert effort to receive their reward. Across three experiments, we found that the unequal distribution of offers reduced participants’ motivation to pursue rewards even when their relative position in the distribution was high, and despite the decision being of no benefit to others and reducing the reward for oneself. Participants’ feelings partially mediated this relationship. In particular, a large disparity in rewards was associated with greater unhappiness, which was associated with lower willingness to work–even when controlling for absolute reward and its relative value, both of which also affected decisions to work. A model that incorporated a person’s relative position and unfairness of rewards in the group fit better to the data than other popular models describing the effects of inequality. Our findings suggest opportunity-gaps can trigger psychological dynamics that hurt productivity and well-being of all involved.

The Effect of Counterfactual Information on Outcome Value Coding in Medial Prefrontal and Cingulate Cortex: From an Absolute to a Relative Neural Code

Adaptive coding of stimuli is well documented in perception, where it supports efficient encoding over a broad range of possible percepts. Recently, a similar neural mechanism has been reported also in value-based decision, where it allows optimal encoding of vast ranges of values in PFC: neuronal response to value depends on the choice context (relative coding), rather than being invariant across contexts (absolute coding). Additionally, value learning is sensitive to the amount of feedback information: providing complete feedback (both obtained and forgone outcomes) instead of partial feedback (only obtained outcome) improves learning. However, it is unclear whether relative coding occurs in all PFC regions and how it is affected by feedback information. We systematically investigated univariate and multivariate feedback encoding in various mPFC regions and compared three modes of neural coding: absolute, partially-adaptive and fully-adaptive.Twenty-eight human participants (both sexes) performed a learning task while undergoing fMRI scanning. On each trial, they chose between two symbols associated with a certain outcome. Then, the decision outcome was revealed. Notably, in one-half of the trials participants received partial feedback, whereas in the other half they got complete feedback. We used univariate and multivariate analysis to explore value encoding in different feedback conditions.We found that both obtained and forgone outcomes were encoded in mPFC, but with opposite sign in its ventral and dorsal subdivisions. Moreover, we showed that increasing feedback information induced a switch from absolute to relative coding. Our results suggest that complete feedback information enhances context-dependent outcome encoding.SIGNIFICANCE STATEMENT This study offers a systematic investigation of the effect of the amount of feedback information (partial vs complete) on univariate and multivariate outcome value encoding, within multiple regions in mPFC and cingulate cortex that are critical for value-based decisions and behavioral adaptation. Moreover, we provide the first comparison of three possible models of neural coding (i.e., absolute, partially-adaptive, and fully-adaptive coding) of value signal in these regions, by using commensurable measures of prediction accuracy. Taken together, our results help build a more comprehensive picture of how the human brain encodes and processes outcome value. In particular, our results suggest that simultaneous presentation of obtained and foregone outcomes promotes relative value representation.

Partial Adaptation to the Value Range in the Macaque Orbitofrontal Cortex

Values available for choice in different behavioral contexts can vary immensely. To compensate for this variability, neuronal circuits underlying economic decisions undergo adaptation. In orbitofrontal cortex (OFC), neurons encode the subjective value of offered and chosen goods in a quasilinear way. Previous experiments found that the gain of the encoding is lower when the value range is wider. However, the parameters OFC neurons adapted to remained unclear. Furthermore, previous studies did not examine additive changes in neuronal responses. Computational considerations indicate that these factors can directly impact choice behavior. Here we investigated how OFC neurons adapt to changes in the value range. We recorded from two male rhesus monkeys during a juice choice task. Each session was divided into two blocks of trials. In each block, juices were offered within a set range of values, and ranges changed between blocks. Across blocks, neuronal responses adapted to both the maximum and the minimum value, but only partially. As a result, the minimum neural activity was elevated in some value ranges relative to others. Through simulation of a linear decision model, we showed that increasing the minimum response increases choice variability, lowering the expected payoff. This effect is modulated by the balance between cells with positive and negative encoding. The presence of these two populations induces a non-monotonic relationship between the value range and choice efficacy, such that the expected payoff is highest for decisions in an intermediate value range.SIGNIFICANCE STATEMENT Economic decisions are thought to rely on the orbitofrontal cortex (OFC). The values available for choice vary enormously in different contexts. Previous work showed that neurons in OFC encode values in a linear way, and that the gain of encoding is inversely related to the range of available values. However, the specific parameters driving adaptation remained unclear. Here we show that OFC neurons adapt to both the maximum and minimum value in the current context. However, adaptation is partial, leading to contextual changes in the response offset. Interestingly, increasing the activity offset negatively affects choices in a simulated network. Partial adaptation may allow the circuit to maintain information about context value at the cost of slightly reduced payoff.

Social, self, (situational), and affective processes in medial prefrontal cortex (MPFC): Causal, multivariate, and reverse inference evidence

The medial prefrontal cortex (MPFC) has been posited to serve a variety of social, affective, and cognitive functions. These conclusions have largely been driven by forward inference analyses (e.g. GLM fMRI studies and meta-analyses) that indicate where domain-specific tasks tend to produce activity but tell us little about what those regions do. Here, we take a multi-method, multi-domain approach to the functionality of MPFC subdivisions within Brodmann areas 9-11. We consider four methods that each have reverse inference or causal inference value: lesion work, transcranial magnetic stimulation, multivariate pattern analysis, and Neurosynth analyses. The Neurosynth analyses include multi-term reverse inference analyses that compare several domains of interest to one another at once. We examine the evidence supporting structure-function links in five domains: social cognition, self, value, emotional experience, and mental time travel. The evidence is considered for each of three MPFC subdivisions: dorsomedial prefrontal cortex (DMPFC), anteromedial prefrontal cortex (AMPFC), and ventromedial prefrontal cortex (VMPFC). Although there is evidentiary variability across methods, the results suggest that social processes are functionally linked to DMPFC (and somewhat surprisingly in VMPFC), self processes are linked to AMPFC, and affective processes are linked to AMPFC and VMPFC. There is also a relatively non-selective region of VMPFC that may support situational processing, a process key to each domain, but also independent of each.

The power of price compels you: Behavioral economic insights into dopamine-based valuation of rewarding and aversively motivated behavior

The mesocorticolimbic dopamine pathway is generally considered to be a reward pathway. While shortsighted, there is a strong basis for this view of dopamine function. Here, we first describe the role of phasic dopamine release events in reward seeking. We then explain why these release events are being reconsidered as value signals and how we applied behavioral economics to confirm they play a causal role in the valuation of reward. Just because dopamine release can function as a dopamine reward value signal however, does not imply that dopamine is solely a reward molecule. Rather, mesocorticolimbic dopamine appears to mediate many adaptive behaviors, including: reward seeking, avoidance, escape and fear-associated conditioned freezing. While more studies are needed before a consensus is reached on when, where and how dopamine mediates aversively-motivated behavior, its involvement is almost irrefutable. Thus, we next describe the role dopamine plays in these ethologically-relevant defensive behaviors. We conclude by describing our recent behavioral economics results that reveal a causal role for dopamine in the valuation of avoidance.

Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences

In economics and perceptual decision-making contextual effects are well documented, where decision weights are adjusted as a function of the distribution of stimuli. Yet, in reinforcement learning literature whether and how contextual information pertaining to decision states is integrated in learning algorithms has received comparably little attention. Here, we investigate reinforcement learning behavior and its computational substrates in a task where we orthogonally manipulate outcome valence and magnitude, resulting in systematic variations in state-values. Model comparison indicates that subjects' behavior is best accounted for by an algorithm which includes both reference point-dependence and range-adaptation-two crucial features of state-dependent valuation. In addition, we find that state-dependent outcome valuation progressively emerges, is favored by increasing outcome information and correlated with explicit understanding of the task structure. Finally, our data clearly show that, while being locally adaptive (for instance in negative valence and small magnitude contexts), state-dependent valuation comes at the cost of seemingly irrational choices, when options are extrapolated out from their original contexts.

Deficits in context-dependent adaptive coding in early psychosis and healthy individuals with schizotypal personality traits

Adaptive coding of information is a fundamental principle of brain functioning. It allows for efficient representation over a large range of inputs and thereby alleviates the limited coding range of neurons. In the present study, we investigated for the first time potential alterations in context-dependent reward adaptation and its association with symptom dimensions in the schizophrenia spectrum. We studied 27 patients with first-episode psychosis, 26 individuals with schizotypal personality traits and 25 healthy controls. We used functional MRI in combination with a variant of the monetary incentive delay task and assessed adaptive reward coding in two reward conditions with different reward ranges. Compared to healthy controls, patients with first-episode psychosis and healthy individuals with schizotypal personality traits showed a deficit in increasing the blood oxygen level-dependent response slope in the right caudate for the low reward range compared to the high reward range. In other words, the two groups showed inefficient neural adaptation to the current reward context. In addition, we found impaired adaptive coding of reward in the caudate nucleus and putamen to be associated with total symptom severity across the schizophrenia spectrum. Symptom severity was more strongly associated with neural deficits in adaptive coding than with the neural coding of absolute reward outcomes. Deficits in adaptive coding were prominent across the schizophrenia spectrum and even detectable in unmedicated (healthy) individuals with schizotypal personality traits. Furthermore, the association between total symptom severity and impaired adaptive coding in the right caudate and putamen suggests a dimensional mechanism underlying imprecise neural adaptation. Our findings support the idea that impaired adaptive coding may be a general information-processing deficit explaining disturbances within the schizophrenia spectrum over and above a simple model of blunted absolute reward signals.

A transient dopamine signal encodes subjective value and causally influences demand in an economic context

The mesolimbic dopamine system is strongly implicated in motivational processes. Currently accepted theories suggest that transient mesolimbic dopamine release events energize reward seeking and encode reward value. During the pursuit of reward, critical associations are formed between the reward and cues that predict its availability. Conditioned by these experiences, dopamine neurons begin to fire upon the earliest presentation of a cue, and again at the receipt of reward. The resulting dopamine concentration scales proportionally to the value of the reward. In this study, we used a behavioral economics approach to quantify how transient dopamine release events scale with price and causally alter price sensitivity. We presented sucrose to rats across a range of prices and modeled the resulting demand curves to estimate price sensitivity. Using fast-scan cyclic voltammetry, we determined that the concentration of accumbal dopamine time-locked to cue presentation decreased with price. These data confirm and extend the notion that dopamine release events originating in the ventral tegmental area encode subjective value. Using optogenetics to augment dopamine concentration, we found that enhancing dopamine release at cue made demand more sensitive to price and decreased dopamine concentration at reward delivery. From these observations, we infer that value is decreased because of a negative reward prediction error (i.e., the animal receives less than expected). Conversely, enhancing dopamine at reward made demand less sensitive to price. We attribute this finding to a positive reward prediction error, whereby the animal perceives they received a better value than anticipated.

Adaptive Value Normalization in the Prefrontal Cortex Is Reduced by Memory Load

Adaptation facilitates neural representation of a wide range of diverse inputs, including reward values. Adaptive value coding typically relies on contextual information either obtained from the environment or retrieved from and maintained in memory. However, it is unknown whether having to retrieve and maintain context information modulates the brain's capacity for value adaptation. To address this issue, we measured hemodynamic responses of the prefrontal cortex (PFC) in two studies on risky decision-making. In each trial, healthy human subjects chose between a risky and a safe alternative; half of the participants had to remember the risky alternatives, whereas for the other half they were presented visually. The value of safe alternatives varied across trials. PFC responses adapted to contextual risk information, with steeper coding of safe alternative value in lower-risk contexts. Importantly, this adaptation depended on working memory load, such that response functions relating PFC activity to safe values were steeper with presented versus remembered risk. An independent second study replicated the findings of the first study and showed that similar slope reductions also arose when memory maintenance demands were increased with a secondary working memory task. Formal model comparison showed that a divisive normalization model fitted effects of both risk context and working memory demands on PFC activity better than alternative models of value adaptation, and revealed that reduced suppression of background activity was the critical parameter impairing normalization with increased memory maintenance demand. Our findings suggest that mnemonic processes can constrain normalization of neural value representations.