MEASURING BELIEFS AND REWARDS: A NEUROECONOMIC APPROACH

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

How thin rational choice theory explains choices

The critics of rational choice theory (RCT) frequently build on the contrast between so-called thick and thin applications of RCT to argue that thin RCT lacks the potential to explain the choices of real-world agents. In this paper, I draw on often-cited RCT applications in several decision sciences to demonstrate that despite this prominent critique there are at least two different senses in which thin RCT can explain real-world agents' choices. I then defend this thesis against the most influential objections put forward by the critics of RCT. In doing so, I explicate the implications of my thesis for the ongoing philosophical debate concerning the explanatory potential of RCT and the comparative merits of widely endorsed accounts of explanation.

Neural Mechanisms of Reward Prediction Error in Autism Spectrum Disorder

Few studies have explored neural mechanisms of reward learning in ASD despite evidence of behavioral impairments of predictive abilities in ASD. To investigate the neural correlates of reward prediction errors in ASD, 16 adults with ASD and 14 typically developing controls performed a prediction error task during fMRI scanning. Results revealed greater activation in the ASD group in the left paracingulate gyrus during signed prediction errors and the left insula and right frontal pole during thresholded unsigned prediction errors. Findings support atypical neural processing of reward prediction errors in ASD in frontostriatal regions critical for prediction coding and reward learning. Results provide a neural basis for impairments in reward learning that may contribute to traits common in ASD (e.g., intolerance of unpredictability).

The Brain's Reward Response Occurs Even Without Actual Reward!

What if the brain's response to reward occurs even when there is no reward? Wouldn't that be a further concern for people prone to problem gambling and other forms of addiction, like those related to eating? Electroencephalography was employed to investigate this possibility using probabilistic feedback manipulations and measures of known event-related potentials (ERPs) related to reward processing. We tested the hypothesis-that reward-based ERPs would occur even in the absence of a tangible reward and when manipulations on expectation are implicit. The well-known P300 response potential was a key focus, and was assessed in non-gambling volunteer undergraduates on a task involving experimentally-manipulated probabilities of positive or negative feedback comprising three trial types-80, 50, or 20% positive feedback. A feedback stimulus (F1) followed a guess response between two possible outcomes (implicit win/loss), and then a second feedback stimulus (F2) was presented to confirm an alleged 'win' or 'loss' (explicit win/loss). Results revealed that amplitude of the P300 in F1-locked data (implicit manipulation) was larger (more positive) on average for feedback outcomes that were manipulated to be less likely than expected. The effect is pronounced after increased time on task (later trials), even though the majority of participants were not explicitly aware of our probability manipulations. For the explicit effects in F2-locked data, no meaningful or significant effects were observed. These findings point to the existence of proposed success-response mechanisms that operate not only explicitly but also with implicit manipulations that do not involve any direct indication of a win or loss, and are not associated with tangible rewards. Thus, there seems to be a non-explicit form of perception (we call 'implicit') associated with an internal experience of wins/losses (in the absence of actual rewards or losses) that can be measured in associated brain processes. The potential significance of these findings is discussed in terms of implications for problem gambling.

Neural and computational processes underlying dynamic changes in self-esteem

Self-esteem is shaped by the appraisals we receive from others. Here, we characterize neural and computational mechanisms underlying this form of social influence. We introduce a computational model that captures fluctuations in self-esteem engendered by prediction errors that quantify the difference between expected and received social feedback. Using functional MRI, we show these social prediction errors correlate with activity in ventral striatum/subgenual anterior cingulate cortex, while updates in self-esteem resulting from these errors co-varied with activity in ventromedial prefrontal cortex (vmPFC). We linked computational parameters to psychiatric symptoms using canonical correlation analysis to identify an 'interpersonal vulnerability' dimension. Vulnerability modulated the expression of prediction error responses in anterior insula and insula-vmPFC connectivity during self-esteem updates. Our findings indicate that updating of self-evaluative beliefs relies on learning mechanisms akin to those used in learning about others. Enhanced insula-vmPFC connectivity during updating of those beliefs may represent a marker for psychiatric vulnerability.

Association of Neural and Emotional Impacts of Reward Prediction Errors With Major Depression

IMPORTANCE

Major depressive disorder (MDD) is associated with deficits in representing reward prediction errors (RPEs), which are the difference between experienced and predicted reward. Reward prediction errors underlie learning of values in reinforcement learning models, are represented by phasic dopamine release, and are known to affect momentary mood.

OBJECTIVE

To combine functional neuroimaging, computational modeling, and smartphone-based large-scale data collection to test, in the absence of learning-related concerns, the hypothesis that depression attenuates the impact of RPEs.

DESIGN, SETTING, AND PARTICIPANTS

Functional magnetic resonance imaging (fMRI) data were collected on 32 individuals with moderate MDD and 20 control participants who performed a probabilistic reward task. A risky decision task with repeated happiness ratings as a measure of momentary mood was also tested in the laboratory in 74 participants and with a smartphone-based platform in 1833 participants. The study was conducted from November 20, 2012, to February 17, 2015.

MAIN OUTCOMES AND MEASURES

Blood oxygen level-dependent activity was measured in ventral striatum, a dopamine target area known to represent RPEs. Momentary mood was measured during risky decision making.

RESULTS

Of the 52 fMRI participants (mean [SD] age, 34.0 [9.1] years), 30 (58%) were women and 32 had MDD. Of the 74 participants in the laboratory risky decision task (mean age, 34.2 [10.3] years), 44 (59%) were women and 54 had MDD. Of the smartphone group, 543 (30%) had a depression history and 1290 (70%) had no depression history; 918 (50%) were women, and 593 (32%) were younger than 30 years. Contrary to previous results in reinforcement learning tasks, individuals with moderate depression showed intact RPE signals in ventral striatum (z = 3.16; P = .002) that did not differ significantly from controls (z = 0.91; P = .36). Symptom severity correlated with baseline mood parameters in laboratory (ρ = -0.54; P < 1 × 10-6) and smartphone (ρ = -0.30; P < 1 × 10-39) data. However, participants with depression showed an intact association between RPEs and happiness in a computational model of momentary mood dynamics (z = 4.55; P < .001) that was not attenuated compared with controls (z = -0.42; P = .67).

CONCLUSIONS AND RELEVANCE

The neural and emotional impact of RPEs is intact in major depression. These results suggest that depression does not affect the expression of dopaminergic RPEs and that attenuated RPEs in previous reports may reflect downstream effects more closely related to aberrant behavior. The correlation between symptom severity and baseline mood parameters supports an association between depression and momentary mood fluctuations during cognitive tasks. These results demonstrate a potential for smartphones in large-scale computational phenotyping, which is a goal for computational psychiatry.

Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans

Reward learning depends on accurate reward associations with potential choices. These associations can be attained with reinforcement learning mechanisms using a reward prediction error (RPE) signal (the difference between actual and expected rewards) for updating future reward expectations. Despite an extensive body of literature on the influence of RPE on learning, little has been done to investigate the potentially separate contributions of RPE valence (positive or negative) and surprise (absolute degree of deviation from expectations). Here, we coupled single-trial electroencephalography with simultaneously acquired fMRI, during a probabilistic reversal-learning task, to offer evidence of temporally overlapping but largely distinct spatial representations of RPE valence and surprise. Electrophysiological variability in RPE valence correlated with activity in regions of the human reward network promoting approach or avoidance learning. Electrophysiological variability in RPE surprise correlated primarily with activity in regions of the human attentional network controlling the speed of learning. Crucially, despite the largely separate spatial extend of these representations our EEG-informed fMRI approach uniquely revealed a linear superposition of the two RPE components in a smaller network encompassing visuo-mnemonic and reward areas. Activity in this network was further predictive of stimulus value updating indicating a comparable contribution of both signals to reward learning.

The social contingency of momentary subjective well-being

Although social comparison is a known determinant of overall life satisfaction, it is not clear how it affects moment-to-moment variation in subjective emotional state. Using a novel social decision task combined with computational modelling, we show that a participant's subjective emotional state reflects not only the impact of rewards they themselves receive, but also the rewards received by a social partner. Unequal outcomes, whether advantageous or disadvantageous, reduce average momentary happiness. Furthermore, the relative impacts of advantageous and disadvantageous inequality on momentary happiness at the individual level predict a subject's generosity in a separate dictator game. These findings demonstrate a powerful social influence upon subjective emotional state, where emotional reactivity to inequality is strongly predictive of altruism in an independent task domain.

Mood as Representation of Momentum

Experiences affect mood, which in turn affects subsequent experiences. Recent studies suggest two specific principles. First, mood depends on how recent reward outcomes differ from expectations. Second, mood biases the way we perceive outcomes (e.g., rewards), and this bias affects learning about those outcomes. We propose that this two-way interaction serves to mitigate inefficiencies in the application of reinforcement learning to real-world problems. Specifically, we propose that mood represents the overall momentum of recent outcomes, and its biasing influence on the perception of outcomes ‘corrects’ learning to account for environmental dependencies. We describe potential dysfunctions of this adaptive mechanism that might contribute to the symptoms of mood disorders.

With increasing use of computational models to understand human behavior, scientists have begun to model the dynamics of subjective states such as mood.

Recent data suggest that mood reflects the cumulative impact of differences between reward outcomes and expectations.

Behavioral and neural findings suggest that mood biases the perception of reward outcomes such that outcomes are perceived as better when one is in a good mood relative to when one is in a bad mood.

These two lines of research establish a bidirectional interaction between mood and reinforcement learning, which may play an important adaptive role in healthy behavior, and whose dysfunction might contribute to psychiatric disorders.

No unified reward prediction error in local field potentials from the human nucleus accumbens: evidence from epilepsy patients

Functional magnetic resonance imaging (fMRI), cyclic voltammetry, and single-unit electrophysiology studies suggest that signals measured in the nucleus accumbens (Nacc) during value-based decision making represent reward prediction errors (RPEs), the difference between actual and predicted rewards. Here, we studied the precise temporal and spectral pattern of reward-related signals in the human Nacc. We recorded local field potentials (LFPs) from the Nacc of six epilepsy patients during an economic decision-making task. On each trial, patients decided whether to accept or reject a gamble with equal probabilities of a monetary gain or loss. The behavior of four patients was consistent with choices being guided by value expectations. Expected value signals before outcome onset were observed in three of those patients, at varying latencies and with nonoverlapping spectral patterns. Signals after outcome onset were correlated with RPE regressors in all subjects. However, further analysis revealed that these signals were better explained as outcome valence rather than RPE signals, with gamble gains and losses differing in the power of beta oscillations and in evoked response amplitudes. Taken together, our results do not support the idea that postsynaptic potentials in the Nacc represent a RPE that unifies outcome magnitude and prior value expectation. We discuss the generalizability of our findings to healthy individuals and the relation of our results to measurements of RPE signals obtained from the Nacc with other methods.

Using Neural Data to Test A Theory of Investor Behavior: An Application to Realization Utility

We use measures of neural activity provided by functional magnetic resonance imaging (fMRI) to test the "realization utility" theory of investor behavior, which posits that people derive utility directly from the act of realizing gains and losses. Subjects traded stocks in an experimental market while we measured their brain activity. We find that all subjects exhibit a strong disposition effect in their trading, even though it is suboptimal. Consistent with the realization utility explanation for this behavior, we find that activity in the ventromedial prefrontal cortex, an area known to encode the value of options during choices, correlates with the capital gains of potential trades; that the neural measures of realization utility correlate across subjects with their individual tendency to exhibit a disposition effect; and that activity in the ventral striatum, an area known to encode information about changes in the present value of experienced utility, exhibits a positive response when subjects realize capital gains. These results provide support for the realization utility model and, more generally, demonstrate how neural data can be helpful in testing models of investor behavior.

Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term

Making predictions about the rewards associated with environmental stimuli and updating those predictions through feedback is an essential aspect of adaptive behavior. Theorists have argued that dopamine encodes a reward prediction error (RPE) signal that is used in such a reinforcement learning process. Recent work with fMRI has demonstrated that the BOLD signal in dopaminergic target areas meets both necessary and sufficient conditions of an axiomatic model of the RPE hypothesis. However, there has been no direct evidence that dopamine release itself also meets necessary and sufficient criteria for encoding an RPE signal. Further, the fact that dopamine neurons have low tonic firing rates that yield a limited dynamic range for encoding negative RPEs has led to significant debate about whether positive and negative prediction errors are encoded on a similar scale. To address both of these issues, we used fast-scan cyclic voltammetry to measure reward-evoked dopamine release at carbon fiber electrodes chronically implanted in the nucleus accumbens core of rats trained on a probabilistic decision-making task. We demonstrate that dopamine concentrations transmit a bidirectional RPE signal with symmetrical encoding of positive and negative RPEs. Our findings strengthen the case that changes in dopamine concentration alone are sufficient to encode the full range of RPEs necessary for reinforcement learning.

Deep and beautiful. The reward prediction error hypothesis of dopamine

According to the reward-prediction error hypothesis (RPEH) of dopamine, the phasic activity of dopaminergic neurons in the midbrain signals a discrepancy between the predicted and currently experienced reward of a particular event. It can be claimed that this hypothesis is deep, elegant and beautiful, representing one of the largest successes of computational neuroscience. This paper examines this claim, making two contributions to existing literature. First, it draws a comprehensive historical account of the main steps that led to the formulation and subsequent success of the RPEH. Second, in light of this historical account, it explains in which sense the RPEH is explanatory and under which conditions it can be justifiably deemed deeper than the incentive salience hypothesis of dopamine, which is arguably the most prominent contemporary alternative to the RPEH.

Pleasurable music affects reinforcement learning according to the listener

Mounting evidence links the enjoyment of music to brain areas implicated in emotion and the dopaminergic reward system. In particular, dopamine release in the ventral striatum seems to play a major role in the rewarding aspect of music listening. Striatal dopamine also influences reinforcement learning, such that subjects with greater dopamine efficacy learn better to approach rewards while those with lesser dopamine efficacy learn better to avoid punishments. In this study, we explored the practical implications of musical pleasure through its ability to facilitate reinforcement learning via non-pharmacological dopamine elicitation. Subjects from a wide variety of musical backgrounds chose a pleasurable and a neutral piece of music from an experimenter-compiled database, and then listened to one or both of these pieces (according to pseudo-random group assignment) as they performed a reinforcement learning task dependent on dopamine transmission. We assessed musical backgrounds as well as typical listening patterns with the new Helsinki Inventory of Music and Affective Behaviors (HIMAB), and separately investigated behavior for the training and test phases of the learning task. Subjects with more musical experience trained better with neutral music and tested better with pleasurable music, while those with less musical experience exhibited the opposite effect. HIMAB results regarding listening behaviors and subjective music ratings indicate that these effects arose from different listening styles: namely, more affective listening in non-musicians and more analytical listening in musicians. In conclusion, musical pleasure was able to influence task performance, and the shape of this effect depended on group and individual factors. These findings have implications in affective neuroscience, neuroaesthetics, learning, and music therapy.

Cannabinoids and value-based decision making: implications for neurodegenerative disorders

In recent years, disturbances in cognitive function have been increasingly recognized as important symptomatic phenomena in neurodegenerative diseases, including Parkinson's Disease (PD). Value-based decision making in particular is an important executive cognitive function that is not only impaired in patients with PD, but also shares neural substrates with PD in basal ganglia structures and the dopamine system. Interestingly, the endogenous cannabinoid system modulates dopamine function and subsequently value-based decision making. This review will provide an overview of the interdisciplinary research that has influenced our understanding of value-based decision making and the role of dopamine, particularly in the context of reinforcement learning theories, as well as recent animal and human studies that demonstrate the modulatory role of activation of cannabinoid receptors by exogenous agonists or their naturally occurring ligands. The implications of this research for the symptomatology of and potential treatments for PD are also discussed.

Positive interaction of social comparison and personal responsibility for outcomes

We formulate and test a model that allows sharp separation between two different ways in which environment affects evaluation of outcomes, by comparing social vs. private and personal responsibility vs. chance. In the experiment, subjects chose between two lotteries, one low-risk and one high-risk. They could then observe the outcomes. By varying the environment between private (they could observe the outcome of the chosen lottery and the outcome of the lottery they had not chosen) and social (they could observe the outcome of the lottery chosen by another subject) we can differentiate the response and brain activity following the feedback in social and private settings. The evidence suggests that envy and pride are significant motives driving decisions and outcomes evaluation, stronger than private emotions like regret and rejoice, with ventral striatum playing a key role. When we focus on the outcome evaluation stage we demonstrate that BOLD signal in ventral striatum is increasing in the difference between obtained and counterfactual payoffs. For a given difference in payoffs, striatal responses are more pronounced in social than in private environment. Moreover, a positive interaction (complementarity) between social comparison and personal responsibility is reflected in the pattern of activity in the ventral striatum. At decision stage we observe getting ahead of the Joneses effect in ventral striatum with subjective value of risk larger in social than in private environment.

Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis

A number of recent advances have been achieved in the study of midbrain dopaminergic neurons. Understanding these advances and how they relate to one another requires a deep understanding of the computational models that serve as an explanatory framework and guide ongoing experimental inquiry. This intertwining of theory and experiment now suggests very clearly that the phasic activity of the midbrain dopamine neurons provides a global mechanism for synaptic modification. These synaptic modifications, in turn, provide the mechanistic underpinning for a specific class of reinforcement learning mechanisms that now seem to underlie much of human and animal behavior. This review describes both the critical empirical findings that are at the root of this conclusion and the fantastic theoretical advances from which this conclusion is drawn.

Choice from non-choice: predicting consumer preferences from blood oxygenation level-dependent signals obtained during passive viewing

Decision-making is often viewed as a two-stage process, where subjective values are first assigned to each option and then the option of the highest value is selected. Converging evidence suggests that these subjective values are represented in the striatum and medial prefrontal cortex (MPFC). A separate line of evidence suggests that activation in the same areas represents the values of rewards even when choice is not required, as in classical conditioning tasks. However, it is unclear whether the same neural mechanism is engaged in both cases. To address this question we measured brain activation with functional magnetic resonance imaging while human subjects passively viewed individual consumer goods. We then sampled activation from predefined regions of interest and used it to predict subsequent choices between the same items made outside of the scanner. Our results show that activation in the striatum and MPFC in the absence of choice predicts subsequent choices, suggesting that these brain areas represent value in a similar manner whether or not choice is required.

Testing the reward prediction error hypothesis with an axiomatic model

Neuroimaging studies typically identify neural activity correlated with the predictions of highly parameterized models, like the many reward prediction error (RPE) models used to study reinforcement learning. Identified brain areas might encode RPEs or, alternatively, only have activity correlated with RPE model predictions. Here, we use an alternate axiomatic approach rooted in economic theory to formally test the entire class of RPE models on neural data. We show that measurements of human neural activity from the striatum, medial prefrontal cortex, amygdala, and posterior cingulate cortex satisfy necessary and sufficient conditions for the entire class of RPE models. However, activity measured from the anterior insula falsifies the axiomatic model, and therefore no RPE model can account for measured activity. Further analysis suggests the anterior insula might instead encode something related to the salience of an outcome. As cognitive neuroscience matures and models proliferate, formal approaches of this kind that assess entire model classes rather than specific model exemplars may take on increased significance.

Modeling the value of strategic actions in the superior colliculus

In learning models of strategic game play, an agent constructs a valuation (action value) over possible future choices as a function of past actions and rewards. Choices are then stochastic functions of these action values. Our goal is to uncover a neural signal that correlates with the action value posited by behavioral learning models. We measured activity from neurons in the superior colliculus (SC), a midbrain region involved in planning saccadic eye movements, while monkeys performed two saccade tasks. In the strategic task, monkeys competed against a computer in a saccade version of the mixed-strategy game ”matching-pennies”. In the instructed task, saccades were elicited through explicit instruction rather than free choices. In both tasks neuronal activity and behavior were shaped by past actions and rewards with more recent events exerting a larger influence. Further, SC activity predicted upcoming choices during the strategic task and upcoming reaction times during the instructed task. Finally, we found that neuronal activity in both tasks correlated with an established learning model, the Experience Weighted Attraction model of action valuation (Camerer and Ho, 1999). Collectively, our results provide evidence that action values hypothesized by learning models are represented in the motor planning regions of the brain in a manner that could be used to select strategic actions.