Adaptive properties of differential learning rates for positive and negative outcomes

A cognitive-computational account of mood swings in adolescence

Teenagers have a reputation for being fickle, in both their choices and their moods. This variability may help adolescents as they begin to independently navigate novel environments. Recently, however, adolescent moodiness has also been linked to psychopathology. Here, we consider adolescents' mood swings from a novel computational perspective, grounded in reinforcement learning (RL). This model proposes that mood is determined by surprises about outcomes in the environment, and how much we learn from these surprises. It additionally suggests that mood biases learning and choice in a bidirectional manner. Integrating independent lines of research, we sketch a cognitive-computational account of how adolescents' mood, learning, and choice dynamics influence each other, with implications for normative and psychopathological development.

Negative mood and optimism bias: An experimental investigation of sadness and belief updating

BACKGROUND AND OBJECTIVES

Understanding how individuals integrate new information to form beliefs under changing emotional conditions is crucial to describing decision-making processes. Previous research suggests that although most people demonstrate bias toward optimistic appraisals of new information when updating beliefs, individuals with dysphoric psychiatric conditions (e.g., major depression) do not demonstrate this same bias. Despite these findings, limited research has investigated the relationship between affective states and belief updating processes.

METHODS

We induced neutral and sad moods in participants and had them complete a belief-updating paradigm by estimating the likelihood of negative future events happening to them, viewing the actual likelihood, and then re-estimating their perceived likelihood.

RESULTS

We observed that individuals updated their beliefs more after receiving desirable information relative to undesirable information under neutral conditions. Further, we found that individuals did not demonstrate unrealistic optimism under negative affective conditions.

LIMITATIONS

This study incorporated a population of university students under laboratory conditions and would benefit from replication and extension in clinical populations and naturalistic settings.

CONCLUSIONS

These findings suggest that momentary fluctuations in mood affect how individuals integrate information to form beliefs.

Impaired flexible reward learning in ADHD patients is associated with blunted reinforcement sensitivity and neural signals in ventral striatum and parietal cortex

Reward-based learning and decision-making are prime candidates to understand symptoms of attention deficit hyperactivity disorder (ADHD). However, only limited evidence is available regarding the neurocomputational underpinnings of the alterations seen in ADHD. This concerns flexible behavioral adaption in dynamically changing environments, which is challenging for individuals with ADHD. One previous study points to elevated choice switching in adolescent ADHD, which was accompanied by disrupted learning signals in medial prefrontal cortex. Here, we investigated young adults with ADHD (n = 17) as compared to age- and sex-matched controls (n = 17) using a probabilistic reversal learning experiment during functional magnetic resonance imaging (fMRI). The task requires continuous learning to guide flexible behavioral adaptation to changing reward contingencies. To disentangle the neurocomputational underpinnings of the behavioral data, we used reinforcement learning (RL) models, which informed the analysis of fMRI data. ADHD patients performed worse than controls particularly in trials before reversals, i.e., when reward contingencies were stable. This pattern resulted from 'noisy' choice switching regardless of previous feedback. RL modelling showed decreased reinforcement sensitivity and enhanced learning rates for negative feedback in ADHD patients. At the neural level, this was reflected in a diminished representation of choice probability in the left posterior parietal cortex in ADHD. Moreover, modelling showed a marginal reduction of learning about the unchosen option, which was paralleled by a marginal reduction in learning signals incorporating the unchosen option in the left ventral striatum. Taken together, we show that impaired flexible behavior in ADHD is due to excessive choice switching ('hyper-flexibility'), which can be detrimental or beneficial depending on the learning environment. Computationally, this resulted from blunted sensitivity to reinforcement of which we detected neural correlates in the attention-control network, specifically in the parietal cortex. These neurocomputational findings remain preliminary due to the relatively small sample size.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Deficits in proactive avoidance and neural responses to drinking motives in problem drinkers

Physical pain and negative emotions represent two distinct drinking motives that contribute to harmful alcohol use. Proactive avoidance which can reduce problem drinking in response to these motives appears to be impaired in problem drinkers. However, proactive avoidance and its underlying neural deficits have not been assessed experimentally. How these deficits inter-relate with drinking motives to influence alcohol use also remains unclear. The current study leveraged neuroimaging data collected in forty-one problem and forty-one social drinkers who performed a probabilistic learning go/nogo task that involved proactive avoidance of painful outcomes. We characterized the regional brain responses to proactive avoidance and identified the neural correlates of drinking to avoid physical pain and negative emotions. Behavioral results confirmed problem drinkers' proactive avoidance deficits in learning rate and performance accuracy, both which were associated with greater alcohol use. Imaging findings in problem drinkers showed that negative emotions as a drinking motive predicted attenuated right insula activation during proactive avoidance. In contrast, physical pain motive predicted reduced right putamen response. These regions' activations as well as functional connectivity with the somatomotor cortex also demonstrated a negative relationship with drinking severity and positive relationship with proactive avoidance performance. Path modeling further delineated the pathways through which physical pain and negative emotions, along with alcohol use severity, influenced the neural and behavioral measures of proactive avoidance. Taken together, the current findings provide experimental evidence for proactive avoidance deficits in problem drinkers and establish the link between their neural underpinnings and alcohol misuse.

Reduction of Aversive Learning Rates in Pavlovian Conditioning by Angiotensin II Antagonist Losartan: A Randomized Controlled Trial

BACKGROUND

Angiotensin receptor blockade has been linked to aspects of aversive learning and memory formation and to the prevention of posttraumatic stress disorder symptom development.

METHODS

We investigated the influence of the angiotensin receptor blocker losartan on aversive Pavlovian conditioning using a probabilistic learning paradigm. In a double-blind, randomized, placebo-controlled design, we tested 45 (18 female) healthy volunteers during a baseline session, after application of losartan or placebo (drug session), and during a follow-up session. During each session, participants engaged in a task in which they had to predict the probability of an electrical stimulation on every trial while the true shock contingencies switched repeatedly between phases of high and low shock threat. Computational reinforcement learning models were used to investigate learning dynamics.

RESULTS

Acute administration of losartan significantly reduced participants' adjustment during both low-to-high and high-to-low threat changes. This was driven by reduced aversive learning rates in the losartan group during the drug session compared with baseline. The 50-mg drug dose did not induce reduction of blood pressure or change in reaction times, ruling out a general reduction in attention and engagement. Decreased adjustment of aversive expectations was maintained at a follow-up session 24 hours later.

CONCLUSIONS

This study shows that losartan acutely reduces Pavlovian learning in aversive environments, thereby highlighting a potential role of the renin-angiotensin system in anxiety development.

The Neural Correlates of Individual Differences in Reinforcement Learning during Pain Avoidance and Reward Seeking

Organisms learn to gain reward and avoid punishment through action-outcome associations. Reinforcement learning (RL) offers a critical framework to understand individual differences in this associative learning by assessing learning rate, action bias, pavlovian factor (i.e., the extent to which action values are influenced by stimulus values), and subjective impact of outcomes (i.e., motivation to seek reward and avoid punishment). Nevertheless, how these individual-level metrics are represented in the brain remains unclear. The current study leveraged fMRI in healthy humans and a probabilistic learning go/no-go task to characterize the neural correlates involved in learning to seek reward and avoid pain. Behaviorally, participants showed a higher learning rate during pain avoidance relative to reward seeking. Additionally, the subjective impact of outcomes was greater for reward trials and associated with lower response randomness. Our imaging findings showed that individual differences in learning rate and performance accuracy during avoidance learning were positively associated with activities of the dorsal anterior cingulate cortex, midcingulate cortex, and postcentral gyrus. In contrast, the pavlovian factor was represented in the precentral gyrus and superior frontal gyrus (SFG) during pain avoidance and reward seeking, respectively. Individual variation of the subjective impact of outcomes was positively predicted by activation of the left posterior cingulate cortex. Finally, action bias was represented by the supplementary motor area (SMA) and pre-SMA whereas the SFG played a role in restraining this action tendency. Together, these findings highlight for the first time the neural substrates of individual differences in the computational processes during RL.

Test-retest reliability of reinforcement learning parameters

It has recently been suggested that parameter estimates of computational models can be used to understand individual differences at the process level. One area of research in which this approach, called computational phenotyping, has taken hold is computational psychiatry. One requirement for successful computational phenotyping is that behavior and parameters are stable over time. Surprisingly, the test-retest reliability of behavior and model parameters remains unknown for most experimental tasks and models. The present study seeks to close this gap by investigating the test-retest reliability of canonical reinforcement learning models in the context of two often-used learning paradigms: a two-armed bandit and a reversal learning task. We tested independent cohorts for the two tasks (N = 69 and N = 47) via an online testing platform with a between-test interval of five weeks. Whereas reliability was high for personality and cognitive measures (with ICCs ranging from .67 to .93), it was generally poor for the parameter estimates of the reinforcement learning models (with ICCs ranging from .02 to .52 for the bandit task and from .01 to .71 for the reversal learning task). Given that simulations indicated that our procedures could detect high test-retest reliability, this suggests that a significant proportion of the variability must be ascribed to the participants themselves. In support of that hypothesis, we show that mood (stress and happiness) can partly explain within-participant variability. Taken together, these results are critical for current practices in computational phenotyping and suggest that individual variability should be taken into account in the future development of the field.

There is no belief update bias for neutral events: failure to replicate Burton et al. (2022)

In a recent paper, Burton et al. claim that individuals update beliefs to a greater extent when learning an event is less likely compared to more likely than expected. Here, we investigate Burton's et al.'s, findings. First, we show how Burton et al.'s data do not in fact support a belief update bias for neutral events. Next, in an attempt to replicate their findings, we collect a new data set employing the original belief update task design, but with neutral events. A belief update bias for neutral events is not observed. Finally, we highlight the statistical errors and confounds in Burton et al.'s design and analysis. This includes mis-specifying a reinforcement learning approach to model the data and failing to follow standard computational model fitting sanity checks such as parameter recovery, model comparison and out of sample prediction. Together, the results find little evidence for biased updating for neutral events.

Contextual influence of reinforcement learning performance of depression: evidence for a negativity bias?

BACKGROUNDS

Value-based decision-making impairment in depression is a complex phenomenon: while some studies did find evidence of blunted reward learning and reward-related signals in the brain, others indicate no effect. Here we test whether such reward sensitivity deficits are dependent on the overall value of the decision problem.

METHODS

We used a two-armed bandit task with two different contexts: one 'rich', one 'poor' where both options were associated with an overall positive, negative expected value, respectively. We tested patients (N = 30) undergoing a major depressive episode and age, gender and socio-economically matched controls (N = 26). Learning performance followed by a transfer phase, without feedback, were analyzed to distangle between a decision or a value-update process mechanism. Finally, we used computational model simulation and fitting to link behavioral patterns to learning biases.

RESULTS

Control subjects showed similar learning performance in the 'rich' and the 'poor' contexts, while patients displayed reduced learning in the 'poor' context. Analysis of the transfer phase showed that the context-dependent impairment in patients generalized, suggesting that the effect of depression has to be traced to the outcome encoding. Computational model-based results showed that patients displayed a higher learning rate for negative compared to positive outcomes (the opposite was true in controls).

CONCLUSIONS

Our results illustrate that reinforcement learning performances in depression depend on the value of the context. We show that depressive patients have a specific trouble in contexts with an overall negative state value, which in our task is consistent with a negativity bias at the learning rates level.

Exploring the effects of risk-taking, exploitation, and exploration on divergent thinking under group dynamics

This study examined the effects of risk-taking and exploitation/exploration trade-off on divergent thinking in individuals, dyads, and triads. We adopted a simple Q-learning model to estimate risk attitudes, exploitation, and exploration parameters. The results showed that risk-taking, exploitation, and exploration did not affect divergent thinking in dyads. Instead, loss aversion was negatively related to divergent thinking. In contrast, risk attitudes and the inverse temperature as a ratio between exploitation and exploration were significant but with contrasting effects in individuals and triads. For individuals, risk-taking, exploitation and loss aversion played a critical role in divergent thinking. For triads, risk aversion and exploration were significantly related to divergent thinking. However, the results also indicated that balancing risk with exploitation/exploration and loss aversion is critical in enhancing divergent thinking in individuals and triads when learning coherence emerges. These results could be interpreted consistently with related literature such as the odd-vs. even-numbered group dynamics, knowledge diversity in group creativity, and representational change theory in insight problem-solving.

Individual differences in naturalistic learning link negative emotionality to the development of anxiety

Organisms learn from prediction errors (PEs) to predict the future. Laboratory studies using small financial outcomes find that humans use PEs to update expectations and link individual differences in PE-based learning to internalizing disorders. Because of the low-stakes outcomes in most tasks, it is unclear whether PE learning emerges in naturalistic, high-stakes contexts and whether individual differences in PE learning predict psychopathology risk. Using experience sampling to assess 625 college students' expected exam grades, we found evidence of PE-based learning and a general tendency to discount negative PEs, an "optimism bias." However, individuals with elevated negative emotionality, a personality trait linked to the development of anxiety disorders, displayed a global pessimism and learning differences that impeded accurate expectations and predicted future anxiety symptoms. A sensitivity to PEs combined with an aversion to negative PEs may result in a pessimistic and inaccurate model of the world, leading to anxiety.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Interactions between attributions and beliefs at trial-by-trial level: Evidence from a novel computer game task

Inferring causes of the good and bad events that we experience is part of the process of building models of our own capabilities and of the world around us. Making such inferences can be difficult because of complex reciprocal relationships between attributions of the causes of particular events, and beliefs about the capabilities and skills that influence our role in bringing them about. Abnormal causal attributions have long been studied in connection with psychiatric disorders, notably depression and paranoia; however, the mechanisms behind attributional inferences and the way they can go awry are not fully understood. We administered a novel, challenging, game of skill to a substantial population of healthy online participants, and collected trial-by-trial time series of both their beliefs about skill and attributions about the causes of the success and failure of real experienced outcomes. We found reciprocal relationships that provide empirical confirmation of the attribution-self representation cycle theory. This highlights the dynamic nature of the processes involved in attribution, and validates a framework for developing and testing computational accounts of attribution-belief interactions.

As part of interpreting our experiences, we spontaneously make causal attributions and use them to update our beliefs about the world, ourselves and others. This has long been a topic of interest, particularly within psychiatry. Some theories assume that people have stable “attributional styles”, others focus on the changing nature of attribution-making and on the relationships between attributions and one’s beliefs about the self, suggesting that the two are mutually connected. In this area of research, people have traditionally been asked to imagine themselves experiencing various significant life events and report on how they would interpret those, or have been exposed to artificial and highly simplified situations in the lab. In this work, we introduce a new task to study relationships between causal attributions and beliefs: repeatedly playing an engaging and relatively complex game of skill. We show that we can detect mutual influences between attributions and beliefs at the level of individual wins and losses. This has implications for how everyday successes and failures impact our beliefs about ourselves and our well-being. It also could help understand how our interpretations of negative experiences can spiral out of control, affecting our mental health.

Anterior cingulate cortex causally supports flexible learning under motivationally challenging and cognitively demanding conditions

Anterior cingulate cortex (ACC) and striatum (STR) contain neurons encoding not only the expected values of actions, but also the value of stimulus features irrespective of actions. Values about stimulus features in ACC or STR might contribute to adaptive behavior by guiding fixational information sampling and biasing choices toward relevant objects, but they might also have indirect motivational functions by enabling subjects to estimate the value of putting effort into choosing objects. Here, we tested these possibilities by modulating neuronal activity in ACC and STR of nonhuman primates using transcranial ultrasound stimulation while subjects learned the relevance of objects in situations with varying motivational and cognitive demands. Motivational demand was indexed by varying gains and losses during learning, while cognitive demand was varied by increasing the uncertainty about which object features could be relevant during learning. We found that ultrasound stimulation of the ACC, but not the STR, reduced learning efficiency and prolonged information sampling when the task required averting losses and motivational demands were high. Reduced learning efficiency was particularly evident at higher cognitive demands and when subjects experienced loss of already attained tokens. These results suggest that the ACC supports flexible learning of feature values when loss experiences impose a motivational challenge and when uncertainty about the relevance of objects is high. Taken together, these findings provide causal evidence that the ACC facilitates resource allocation and improves visual information sampling during adaptive behavior.

Gains and Losses Affect Learning Differentially at Low and High Attentional Load

Prospective gains and losses influence cognitive processing, but it is unresolved how they modulate flexible learning in changing environments. The prospect of gains might enhance flexible learning through prioritized processing of reward-predicting stimuli, but it is unclear how far this learning benefit extends when task demands increase. Similarly, experiencing losses might facilitate learning when they trigger attentional reorienting away from loss-inducing stimuli, but losses may also impair learning by increasing motivational costs or when negative outcomes are overgeneralized. To clarify these divergent views, we tested how varying magnitudes of gains and losses affect the flexible learning of feature values in environments that varied attentional load by increasing the number of interfering object features. With this task design, we found that larger prospective gains improved learning efficacy and learning speed, but only when attentional load was low. In contrast, expecting losses impaired learning efficacy, and this impairment was larger at higher attentional load. These findings functionally dissociate the contributions of gains and losses on flexible learning, suggesting they operate via separate control mechanisms. One mechanism is triggered by experiencing loss and reduces the ability to reduce distractor interference, impairs assigning credit to specific loss-inducing features, and decreases efficient exploration during learning. The second mechanism is triggered by experiencing gains, which enhances prioritizing reward-predicting stimulus features as long as the interference of distracting features is limited. Taken together, these results support a rational theory of cognitive control during learning, suggesting that experiencing losses and experiencing distractor interference impose costs for learning.

Flexibility in valenced reinforcement learning computations across development

Optimal integration of positive and negative outcomes during learning varies depending on an environment's reward statistics. The present study investigated the extent to which children, adolescents, and adults (N = 142 8-25 year-olds, 55% female, 42% White, 31% Asian, 17% mixed race, and 8% Black; data collected in 2021) adapt their weighting of better-than-expected and worse-than-expected outcomes when learning from reinforcement. Participants made choices across two contexts: one in which weighting positive outcomes more heavily than negative outcomes led to better performance, and one in which the reverse was true. Reinforcement learning modeling revealed that across age, participants shifted their valence biases in accordance with environmental structure. Exploratory analyses revealed strengthening of context-dependent flexibility with increasing age.

Serotonin modulates asymmetric learning from reward and punishment in healthy human volunteers

Instrumental learning is driven by a history of outcome success and failure. Here, we examined the impact of serotonin on learning from positive and negative outcomes. Healthy human volunteers were assessed twice, once after acute (single-dose), and once after prolonged (week-long) daily administration of the SSRI citalopram or placebo. Using computational modelling, we show that prolonged boosting of serotonin enhances learning from punishment and reduces learning from reward. This valence-dependent learning asymmetry increases subjects’ tendency to avoid actions as a function of cumulative failure without leading to detrimental, or advantageous, outcomes. By contrast, no significant modulation of learning was observed following acute SSRI administration. However, differences between the effects of acute and prolonged administration were not significant. Overall, these findings may help explain how serotonergic agents impact on mood disorders.

Two factors can drive learning: punishment of failures and reward of successes. Serotonin induces a valence-dependent learning asymmetry, as revealed by prolonged administering of SSRIs to healthy participants in a gambling task.

Asymmetric and adaptive reward coding via normalized reinforcement learning

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.

Reinforcement learning models are widely used to characterize reward-driven learning in biological and computational agents. Standard reinforcement learning models use linear value functions, despite strong empirical evidence that biological value representations are nonlinear functions of external rewards. Here, we examine the properties of a biologically-based nonlinear reinforcement learning algorithm employing the canonical divisive normalization function, a neural computation commonly found in sensory, cognitive, and reward coding. We show that this normalized reinforcement learning algorithm implements a simple but powerful control of how reward learning reflects relative gains and losses. This property explains diverse behavioral and neural phenomena, and suggests the importance of using biologically valid value functions in computational models of learning and decision-making.

Assessing behavioural profiles following neutral, positive and negative feedback

Previous data suggest zero-value, neutral outcomes (draw) are subjectively assigned negative rather than positive valence. The combined observations of faster rather than slower reaction times, subsequent actions defined by shift rather than stay behaviour, reduced flexibility, and, larger rather than smaller deviations from optimal performance following draws all align with the consequences of explicitly negative outcomes such as losses. We further tested the relationships between neutral, positive and negative outcomes by manipulating value salience and observing their behavioural profiles. Despite speeded reaction times and a non-significant bias towards shift behaviour similar to losses when draws were assigned the value of 0 (Experiment 1), the degree of shift behaviour approached an approximation of optimal performance when the draw value was explicitly positive (+1). This was in contrast to when the draw value was explicitly negative (-1), which led to a significant increase in the degree of shift behaviour (Experiment 2). Similar modifications were absent when the same value manipulations were applied to win or lose trials (Experiment 3). Rather than viewing draws as neutral and valence-free outcomes, the processing cascade generated by draws produces a complex behavioural profile containing elements found in response to both explicitly positive and explicitly negative results.

The computational roots of positivity and confirmation biases in reinforcement learning

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.

Reinforcement learning and bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8–30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants’ mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

Pupil Correlates of Decision Variables in Mice Playing a Competitive Mixed-Strategy Game

In a competitive game involving an animal and an opponent, the outcome is contingent on the choices of both players. To succeed, the animal must continually adapt to competitive pressure, or else risk being exploited and lose out on rewards. In this study, we demonstrate that head-fixed male mice can be trained to play the iterative competitive game "matching pennies" against a virtual computer opponent. We find that the animals' performance is well described by a hybrid computational model that includes Q-learning and choice kernels. Comparing between matching pennies and a non-competitive two-armed bandit task, we show that the tasks encourage animals to operate at different regimes of reinforcement learning. To understand the involvement of neuromodulatory mechanisms, we measure fluctuations in pupil size and use multiple linear regression to relate the trial-by-trial transient pupil responses to decision-related variables. The analysis reveals that pupil responses are modulated by observable variables, including choice and outcome, as well as latent variables for value updating, but not action selection. Collectively, these results establish a paradigm for studying competitive decision-making in head-fixed mice and provide insights into the role of arousal-linked neuromodulation in the decision process.

Valence biases in reinforcement learning shift across adolescence and modulate subsequent memory

As individuals learn through trial and error, some are more influenced by good outcomes, while others weight bad outcomes more heavily. Such valence biases may also influence memory for past experiences. Here, we examined whether valence asymmetries in reinforcement learning change across adolescence, and whether individual learning asymmetries bias the content of subsequent memory. Participants ages 8-27 learned the values of 'point machines,' after which their memory for trial-unique images presented with choice outcomes was assessed. Relative to children and adults, adolescents overweighted worse-than-expected outcomes during learning. Individuals' valence biases modulated incidental memory, such that those who prioritized worse- (or better-) than-expected outcomes during learning were also more likely to remember images paired with these outcomes, an effect reproduced in an independent dataset. Collectively, these results highlight age-related changes in the computation of subjective value and demonstrate that a valence-asymmetric valuation process influences how information is prioritized in episodic memory.

The interpretation of computational model parameters depends on the context

Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8–30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.

A Normative Account of Confirmation Bias During Reinforcement Learning

Reinforcement learning involves updating estimates of the value of states and actions on the basis of experience. Previous work has shown that in humans, reinforcement learning exhibits a confirmatory bias: when the value of a chosen option is being updated, estimates are revised more radically following positive than negative reward prediction errors, but the converse is observed when updating the unchosen option value estimate. Here, we simulate performance on a multi-arm bandit task to examine the consequences of a confirmatory bias for reward harvesting. We report a paradoxical finding: that confirmatory biases allow the agent to maximize reward relative to an unbiased updating rule. This principle holds over a wide range of experimental settings and is most influential when decisions are corrupted by noise. We show that this occurs because on average, confirmatory biases lead to overestimating the value of more valuable bandits and underestimating the value of less valuable bandits, rendering decisions overall more robust in the face of noise. Our results show how apparently suboptimal learning rules can in fact be reward maximizing if decisions are made with finite computational precision.

A learning mechanism shaping risk preferences and a preliminary test of its relationship with psychopathic traits

People tend to avoid risk in the domain of gains but take risks in the domain of losses; this is called the reflection effect. Formal theories of decision-making have provided important perspectives on risk preferences, but how individuals acquire risk preferences through experiences remains unknown. In the present study, we used reinforcement learning (RL) models to examine the learning processes that can shape attitudes toward risk in both domains. In addition, relationships between learning parameters and personality traits were investigated. Fifty-one participants performed a learning task, and we examined learning parameters and risk preference in each domain. Our results revealed that an RL model that included a nonlinear subjective utility parameter and differential learning rates for positive and negative prediction errors exhibited better fit than other models and that these parameters independently predicted risk preferences and the reflection effect. Regarding personality traits, although the sample sizes may be too small to test personality traits, increased primary psychopathy scores could be linked with decreased learning rates for positive prediction error in loss conditions among participants who had low anxiety traits. The present findings not only contribute to understanding how decision-making in risky conditions is influenced by past experiences but also provide insights into certain psychiatric problems.

Reward prediction error in the ERP following unconditioned aversive stimuli

Reinforcement learning in humans and other animals is driven by reward prediction errors: deviations between the amount of reward or punishment initially expected and that which is obtained. Temporal difference methods of reinforcement learning generate this reward prediction error at the earliest time at which a revision in reward or punishment likelihood is signalled, for example by a conditioned stimulus. Midbrain dopamine neurons, believed to compute reward prediction errors, generate this signal in response to both conditioned and unconditioned stimuli, as predicted by temporal difference learning. Electroencephalographic recordings of human participants have suggested that a component named the feedback-related negativity (FRN) is generated when this signal is carried to the cortex. If this is so, the FRN should be expected to respond equivalently to conditioned and unconditioned stimuli. However, very few studies have attempted to measure the FRN's response to unconditioned stimuli. The present study attempted to elicit the FRN in response to a primary aversive stimulus (electric shock) using a design that varied reward prediction error while holding physical intensity constant. The FRN was strongly elicited, but earlier and more transiently than typically seen, suggesting that it may incorporate other processes than the midbrain dopamine system.

A Competition of Critics in Human Decision-Making

Recent experiments and theories of human decision-making suggest positive and negative errors are processed and encoded differently by serotonin and dopamine, with serotonin possibly serving to oppose dopamine and protect against risky decisions. We introduce a temporal difference (TD) model of human decision-making to account for these features. Our model involves two critics, an optimistic learning system and a pessimistic learning system, whose predictions are integrated in time to control how potential decisions compete to be selected. Our model predicts that human decision-making can be decomposed along two dimensions: the degree to which the individual is sensitive to (1) risk and (2) uncertainty. In addition, we demonstrate that the model can learn about the mean and standard deviation of rewards, and provide information about reaction time despite not modeling these variables directly. Lastly, we simulate a recent experiment to show how updates of the two learning systems could relate to dopamine and serotonin transients, thereby providing a mathematical formalism to serotonin's hypothesized role as an opponent to dopamine. This new model should be useful for future experiments on human decision-making.

Modeling changes in probabilistic reinforcement learning during adolescence

In the real world, many relationships between events are uncertain and probabilistic. Uncertainty is also likely to be a more common feature of daily experience for youth because they have less experience to draw from than adults. Some studies suggest probabilistic learning may be inefficient in youths compared to adults, while others suggest it may be more efficient in youths in mid adolescence. Here we used a probabilistic reinforcement learning task to test how youth age 8-17 (N = 187) and adults age 18-30 (N = 110) learn about stable probabilistic contingencies. Performance increased with age through early-twenties, then stabilized. Using hierarchical Bayesian methods to fit computational reinforcement learning models, we show that all participants' performance was better explained by models in which negative outcomes had minimal to no impact on learning. The performance increase over age was driven by 1) an increase in learning rate (i.e. decrease in integration time scale); 2) a decrease in noisy/exploratory choices. In mid-adolescence age 13-15, salivary testosterone and learning rate were positively related. We discuss our findings in the context of other studies and hypotheses about adolescent brain development.

Three heads are better than two: Comparing learning properties and performances across individuals, dyads, and triads through a computational approach

Although it is considered that two heads are better than one, related studies argued that groups rarely outperform their best members. This study examined not only whether two heads are better than one but also whether three heads are better than two or one in the context of two-armed bandit problems where learning plays an instrumental role in achieving high performance. This research revealed that a U-shaped correlation exists between performance and group size. The performance was highest for either individuals or triads, but the lowest for dyads. Moreover, this study estimated learning properties and determined that high inverse temperature (exploitation) accounted for high performance. In particular, it was shown that group effects regarding the inverse temperatures in dyads did not generate higher values to surpass the averages of their two group members. In contrast, triads gave rise to higher values of the inverse temperatures than their averages of their individual group members. These results were consistent with our proposed hypothesis that learning coherence is likely to emerge in individuals and triads, but not in dyads, which in turn leads to higher performance. This hypothesis is based on the classical argument by Simmel stating that while dyads are likely to involve more emotion and generate greater variability, triads are the smallest structure which tends to constrain emotions, reduce individuality, and generate behavioral convergences or uniformity because of the ‘‘two against one” social pressures. As a result, three heads or one head were better than two in our study.

The asymmetric learning rates of murine exploratory behavior in sparse reward environments

Goal-oriented behaviors of animals can be modeled by reinforcement learning algorithms. Such algorithms predict future outcomes of selected actions utilizing action values and updating those values in response to the positive and negative outcomes. In many models of animal behavior, the action values are updated symmetrically based on a common learning rate, that is, in the same way for both positive and negative outcomes. However, animals in environments with scarce rewards may have uneven learning rates. To investigate the asymmetry in learning rates in reward and non-reward, we analyzed the exploration behavior of mice in five-armed bandit tasks using a Q-learning model with differential learning rates for positive and negative outcomes. The positive learning rate was significantly higher in a scarce reward environment than in a rich reward environment, and conversely, the negative learning rate was significantly lower in the scarce environment. The positive to negative learning rate ratio was about 10 in the scarce environment and about 2 in the rich environment. This result suggests that when the reward probability was low, the mice tend to ignore failures and exploit the rare rewards. Computational modeling analysis revealed that the increased learning rates ratio could cause an overestimation of and perseveration on rare-rewarding events, increasing total reward acquisition in the scarce environment but disadvantaging impartial exploration.

The Event-Related Potentials Responding to Outcome Valence and Expectancy Violation during Feedback Processing

Feedback-related negativity (FRN) is believed to encode reward prediction error (RPE), a term describing whether the outcome is better or worse than expected. However, some studies suggest that it may reflect unsigned prediction error (UPE) instead. Some disagreement remains as to whether FRN is sensitive to the interaction of outcome valence and prediction error (PE) or merely responsive to the absolute size of PE. Moreover, few studies have compared FRN in appetitive and aversive domains to clarify the valence effect or examine PE's quantitative modulation. To investigate the impact of valence and parametrical PE on FRN, we varied the prediction and feedback magnitudes within a probabilistic learning task in valence (gain and loss domains, Experiment 1) and non-valence contexts (pure digits, Experiment 2). Experiment 3 was identical to Experiment 1 except that some blocks emphasized outcome valence, while others highlighted predictive accuracy. Experiments 1 and 2 revealed a UPE encoder; Experiment 3 found an RPE encoder when valence was emphasized and a UPE encoder when predictive accuracy was highlighted. In this investigation, we demonstrate that FRN is sensitive to outcome valence and expectancy violation, exhibiting a preferential response depending on the dimension that is emphasized.

Information about action outcomes differentially affects learning from self-determined versus imposed choices

The valence of new information influences learning rates in humans: good news tends to receive more weight than bad news. We investigated this learning bias in four experiments, by systematically manipulating the source of required action (free versus forced choices), outcome contingencies (low versus high reward) and motor requirements (go versus no-go choices). Analysis of model-estimated learning rates showed that the confirmation bias in learning rates was specific to free choices, but was independent of outcome contingencies. The bias was also unaffected by the motor requirements, thus suggesting that it operates in the representational space of decisions, rather than motoric actions. Finally, model simulations revealed that learning rates estimated from the choice-confirmation model had the effect of maximizing performance across low- and high-reward environments. We therefore suggest that choice-confirmation bias may be adaptive for efficient learning of action-outcome contingencies, above and beyond fostering person-level dispositions such as self-esteem.

Learning From Success or Failure? - Positivity Biases Revisited

The purpose of this study was to reexamine positivity learning biases through a Q learning computation model and relate them to behavioral characteristics of exploitation and exploration. It was found that while the positivity learning biases existed in the simple asymmetric Q learning model, they completely disappeared once the time-varying nature of learning rates was incorporated. In the time-varying model, learning rates depended on the magnitudes of success and failure. The corresponding positive and negative learning rates were related to high and low performance, respectively, indicating that successes and failures were accounted for by positive and negative learning rates. Moreover, these learning rates were related to both exploitation and exploration in somewhat balanced ways. In contrast, under the constant learning parameter model, positivity biases were associated only with exploration. Therefore, the results in the time-varying model are more intuitively appealing than the simple asymmetric model. However, the statistical tests indicated that participants eclectically selected between the asymmetric learning model and its time-varying version, a frequency of which differed across participants.

Biased belief updating and suboptimal choice in foraging decisions

Deciding which options to engage, and which to forego, requires developing accurate beliefs about the overall distribution of prospects. Here we adapt a classic prey selection task from foraging theory to examine how individuals keep track of an environment’s reward rate and adjust choices in response to its fluctuations. Preference shifts were most pronounced when the environment improved compared to when it deteriorated. This is best explained by a trial-by-trial learning model in which participants estimate the reward rate with upward vs. downward changes controlled by separate learning rates. A failure to adjust expectations sufficiently when an environment becomes worse leads to suboptimal choices: options that are valuable given the environmental conditions are rejected in the false expectation that better options will materialize. These findings offer a previously unappreciated parallel in the serial choice setting of observations of asymmetric updating and resulting biased (often overoptimistic) estimates in other domains.

In some types of decision-making, people must accept or forego an option without knowing what prospects might later be available. Here, the authors reveal how a key bias– asymmetric learning from negative versus positive outcomes – emerges in this type of decision.

Separating Probability and Reversal Learning in a Novel Probabilistic Reversal Learning Task for Mice

The exploration/exploitation tradeoff – pursuing a known reward vs. sampling from lesser known options in the hope of finding a better payoff – is a fundamental aspect of learning and decision making. In humans, this has been studied using multi-armed bandit tasks. The same processes have also been studied using simplified probabilistic reversal learning (PRL) tasks with binary choices. Our investigations suggest that protocols previously used to explore PRL in mice may prove beyond their cognitive capacities, with animals performing at a no-better-than-chance level. We sought a novel probabilistic learning task to improve behavioral responding in mice, whilst allowing the investigation of the exploration/exploitation tradeoff in decision making. To achieve this, we developed a two-lever operant chamber task with levers corresponding to different probabilities (high/low) of receiving a saccharin reward, reversing the reward contingencies associated with levers once animals reached a threshold of 80% responding at the high rewarding lever. We found that, unlike in existing PRL tasks, mice are able to learn and behave near optimally with 80% high/20% low reward probabilities. Altering the reward contingencies towards equality showed that some mice displayed preference for the high rewarding lever with probabilities as close as 60% high/40% low. Additionally, we show that animal choice behavior can be effectively modelled using reinforcement learning (RL) models incorporating learning rates for positive and negative prediction error, a perseveration parameter, and a noise parameter. This new decision task, coupled with RL analyses, advances access to investigate the neuroscience of the exploration/exploitation tradeoff in decision making.

Reinforcement learning across development: What insights can we draw from a decade of research?

The past decade has seen the emergence of the use of reinforcement learning models to study developmental change in value-based learning. It is unclear, however, whether these computational modeling studies, which have employed a wide variety of tasks and model variants, have reached convergent conclusions. In this review, we examine whether the tuning of model parameters that govern different aspects of learning and decision-making processes vary consistently as a function of age, and what neurocognitive developmental changes may account for differences in these parameter estimates across development. We explore whether patterns of developmental change in these estimates are better described by differences in the extent to which individuals adapt their learning processes to the statistics of different environments, or by more static learning biases that emerge across varied contexts. We focus specifically on learning rates and inverse temperature parameter estimates, and find evidence that from childhood to adulthood, individuals become better at optimally weighting recent outcomes during learning across diverse contexts and less exploratory in their value-based decision-making. We provide recommendations for how these two possibilities - and potential alternative accounts - can be tested more directly to build a cohesive body of research that yields greater insight into the development of core learning processes.

Behavioural and neural interactions between objective and subjective performance in a Matching Pennies game

To examine the behavioural and neural interactions between objective and subjective performance during competitive decision-making, participants completed a Matching Pennies game where win-rates were fixed within three conditions (win > lose, win = lose, win < lose) and outcomes were predicted at each trial. Using random behaviour as the hallmark of optimal performance, we observed item (heads), contingency (win-stay, lose-shift) and combinatorial (HH, HT, TH, TT) biases across all conditions. Higher-quality behaviour represented by a reduction in combinatorial bias was observed during high win-rate exposure. In contrast, over-optimism biases were observed only in conditions where win rates were equal to, or less than, loss rates. At a group level, a neural measure of outcome evaluation (feedback-related negativity; FRN) indexed the binary distinction between positive and negative outcome. At an individual level, increased belief in successful performance accentuated FRN amplitude differences between wins and losses. Taken together, the data suggest that objective experiences of, or, subjective beliefs in, the predominance of positive outcomes may be mutual attempts to self-regulate performance during competition. In this way, increased exposure to positive outcomes (real or imagined) may help to weight the output of the more diligent and analytic System 2, relative to the impulsive and intuitive System 1.

The Effect of Reduced Learning Ability on Avoidance in Psychopathy: A Computational Approach

Individuals with psychopathy often show deficits in learning, which often have negative consequences. Several theories have been proposed to explain psychopathic behaviors, but the learning mechanisms in psychopathy are still unclear. To clarify the learning anomalies in psychopathy, we fitted reinforcement learning (RL) models to behavioral data. We conducted two experiments to examine the effect of psychopathy as a group difference (Experiment 1) and as a continuum (Experiment 2). Forty-three undergraduates (in Experiment 1) and fifty-five undergraduate and graduate students (in Experiment 2) performed a go/no-go based learning task with accompanying rewards or punishments. Although we observed no differences in learning performance among the levels of psychopathic traits, the learning rate for the positive prediction error in the loss domain was lower for those with high-psychopathic trait than for those with low-psychopathic trait. This finding indicates that individuals with high-psychopathic traits update an action value less when they avoid a negative outcome. Our model can represent previous theories under a computational framework and provide a new perspective on impaired learning in psychopathy.

The effect of single-dose methylphenidate on the rate of error-driven learning in healthy males: a randomized controlled trial

RATIONALE AND OBJECTIVES

Norepinephrine mediates the adjustment of error-driven learning to match the rate of change of the environment, while phasic dopamine signals prediction errors. We tested the hypothesis that pharmacologic manipulation may modulate this process.

METHODS

We administered a single dose of methylphenidate, a norepinephrine/dopamine reuptake inhibitor, or placebo in double-blind randomized fashion to 20 healthy human males, who then performed a probabilistic learning task. Each subject was tested in two sessions, receiving methylphenidate in one session and placebo in the other, in randomized order. Task performance was quantified by the percentage of trials on which subjects chose the most likely option, while learning rate was measured using a computational model-based parameter as well as with a behavioral analogue of this parameter.

RESULTS

There was a substance-by-session interaction effect on behavioral learning rate and model-based learning rate, such that subjects receiving methylphenidate exhibited higher learning rates than those receiving placebo in session 1, with no difference observed in session 2, suggesting that subjects retained the increased learning rate across sessions. Higher behavioral learning rate was associated with both higher task performance and with the model-based learning rate. Higher learning rates were advantageous given the high rate of change on the task. Subjects receiving methylphenidate and placebo began the task in session 1 with a similar behavioral learning rate, but those receiving methylphenidate rapidly increased learning rate toward the optimal value, suggesting that methylphenidate accelerated the adaptation of learning rate based on the environment.

CONCLUSIONS

The results suggest that methylphenidate may improve disrupted probabilistic learning in disorders involving noradrenergic or dopaminergic dysfunction.

Why we stay with our social partners: Neural mechanisms of stay/leave decision-making

How do we decide to keep interacting (e.g., stay) with a social partner or to switch (e.g., leave) to another? This paper investigated the neural mechanisms of stay/leave decision-making. We hypothesized that these decisions fit within a framework of value-based decision-making, and explored four potential mechanisms underlying a hypothesized bias to stay. Twenty-six participants underwent functional Magnetic Resonance Imaging (fMRI) while completing social and nonsocial versions of a stay/leave decision-making task. On each trial, participants chose between four alternative options, after which they received a monetary reward. Crucially, in the social condition, reward magnitude was ostensibly determined by the generosity of social partners, whereas in the nonsocial condition, reward amounts were ostensibly determined in a pre-programmed manner. Results demonstrated that participants were more likely to stay with options of relatively high expected value, with these values updated through Reinforcement Learning mechanisms and represented neurally within ventromedial prefrontal cortex. Moreover, we demonstrated that greater brain activity in ventromedial prefrontal cortex, caudate nucleus, and septo-hypothalamic regions for social versus nonsocial decisions to stay may underlie a bias towards staying with social partners in particular. These findings complement existing social psychological theories by investigating the neural mechanisms of actual stay/leave decisions.

Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing

Previous studies suggest that factual learning, that is, learning from obtained outcomes, is biased, such that participants preferentially take into account positive, as compared to negative, prediction errors. However, whether or not the prediction error valence also affects counterfactual learning, that is, learning from forgone outcomes, is unknown. To address this question, we analysed the performance of two groups of participants on reinforcement learning tasks using a computational model that was adapted to test if prediction error valence influences learning. We carried out two experiments: in the factual learning experiment, participants learned from partial feedback (i.e., the outcome of the chosen option only); in the counterfactual learning experiment, participants learned from complete feedback information (i.e., the outcomes of both the chosen and unchosen option were displayed). In the factual learning experiment, we replicated previous findings of a valence-induced bias, whereby participants learned preferentially from positive, relative to negative, prediction errors. In contrast, for counterfactual learning, we found the opposite valence-induced bias: negative prediction errors were preferentially taken into account, relative to positive ones. When considering valence-induced bias in the context of both factual and counterfactual learning, it appears that people tend to preferentially take into account information that confirms their current choice.

While the investigation of decision-making biases has a long history in economics and psychology, learning biases have been much less systematically investigated. This is surprising as most of the choices we deal with in everyday life are recurrent, thus allowing learning to occur and therefore influencing future decision-making. Combining behavioural testing and computational modeling, here we show that the valence of an outcome biases both factual and counterfactual learning. When considering factual and counterfactual learning together, it appears that people tend to preferentially take into account information that confirms their current choice. Increasing our understanding of learning biases will enable the refinement of existing models of value-based decision-making.

Do learning rates adapt to the distribution of rewards?

Studies of reinforcement learning have shown that humans learn differently in response to positive and negative reward prediction errors, a phenomenon that can be captured computationally by positing asymmetric learning rates. This asymmetry, motivated by neurobiological and cognitive considerations, has been invoked to explain learning differences across the lifespan as well as a range of psychiatric disorders. Recent theoretical work, motivated by normative considerations, has hypothesized that the learning rate asymmetry should be modulated by the distribution of rewards across the available options. In particular, the learning rate for negative prediction errors should be higher than the learning rate for positive prediction errors when the average reward rate is high, and this relationship should reverse when the reward rate is low. We tested this hypothesis in a series of experiments. Contrary to the theoretical predictions, we found that the asymmetry was largely insensitive to the average reward rate; instead, the dominant pattern was a higher learning rate for negative than for positive prediction errors, possibly reflecting risk aversion.

Adolescent-specific patterns of behavior and neural activity during social reinforcement learning

Humans are sophisticated social beings. Social cues from others are exceptionally salient, particularly during adolescence. Understanding how adolescents interpret and learn from variable social signals can provide insight into the observed shift in social sensitivity during this period. The present study tested 120 participants between the ages of 8 and 25 years on a social reinforcement learning task where the probability of receiving positive social feedback was parametrically manipulated. Seventy-eight of these participants completed the task during fMRI scanning. Modeling trial-by-trial learning, children and adults showed higher positive learning rates than did adolescents, suggesting that adolescents demonstrated less differentiation in their reaction times for peers who provided more positive feedback. Forming expectations about receiving positive social reinforcement correlated with neural activity within the medial prefrontal cortex and ventral striatum across age. Adolescents, unlike children and adults, showed greater insular activity during positive prediction error learning and increased activity in the supplementary motor cortex and the putamen when receiving positive social feedback regardless of the expected outcome, suggesting that peer approval may motivate adolescents toward action. While different amounts of positive social reinforcement enhanced learning in children and adults, all positive social reinforcement equally motivated adolescents. Together, these findings indicate that sensitivity to peer approval during adolescence goes beyond simple reinforcement theory accounts and suggest possible explanations for how peers may motivate adolescent behavior.