Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices

Mechanisms underlying food devaluation after response inhibition to food

Multiple studies reveal that a requirement to stop a response to appetitive food stimuli causes devaluation of these stimuli. However, the mechanism underlying food devaluation after stopping is still under debate. The immediate-affect theory suggests that an increase in negative affect after stopping a response is the driving force for food devaluation. A competing value-updating theory presumes that food devaluation after stopping occurs through the need to align behavior with goals. The current study assessed how food devaluation after response inhibition is influenced by negative emotional reactivity and behavior-goal alignment on a trial-by-trial basis. The study included 60 healthy participants who completed a Food-Stop-Signal-Emotion task. Participants categorized high vs. low-calorie food stimuli and stopped their response upon encountering a stop signal. Subsequently, participants made subjective negativity ratings of negative- or neutral-valenced emotional images, and rated their desire to eat the previously depicted food. In contrast to predictions made by the immediate-affect account, food devaluation after stopping was not mediated nor moderated via changes in negative emotional reactivity after stopping. In support of the value-updating account, food devaluation was modulated by behavior-goal alignment, indicated by larger food devaluation after successful vs. failed stopping. In agreement with this theory, the findings indicate that devaluation occurs more strongly when performance aligns with the task requirement. This study sheds light on the mechanism that likely underlies food devaluation after stopping. Implications regarding applied use of food-inhibition trainings are discussed.

The influence of social feedback on reward learning in the Iowa gambling task

Learning, an important activity for both human and animals, has long been a focal point of research. During the learning process, subjects assimilate not only their own information but also information from others, a phenomenon known as social learning. While numerous studies have explored the impact of social feedback as a reward/punishment during learning, few studies have investigated whether social feedback facilitates or inhibits the learning of environmental rewards/punishments. This study aims to test the effects of social feedback on economic feedback and its cognitive processes by using the Iowa Gambling Task (IGT). One hundred ninety-two participants were recruited and categorized into one non-social feedback group and four social feedback groups. Participants in the social feedback groups were informed that after the outcome of each choice, they would also receive feedback from an online peer. This peer was a fictitious entity, with variations in identity (novice or expert) and feedback type (random or effective). The Outcome-Representation Learning model (ORL model) was used to quantify the cognitive components of learning. Behavioral results showed that both the identity of the peer and the type of feedback provided significantly influenced the deck selection, with effective social feedback increasing the ratio of chosen good decks. Results in the ORL model showed that the four social feedback groups exhibited lower learning rates for gain and loss compared to the nonsocial feedback group, which suggested, in the social feedback groups, the impact of the recent outcome on the update of value decreased. Parameters such as forgetfulness, win frequency, and deck perseverance in the expert-effective feedback group were significantly higher than those in the non-social feedback and expert-random feedback groups. These findings suggest that individuals proactively evaluate feedback providers and selectively adopt effective feedback to enhance learning.

Impaired Punishment Learning in Conduct Disorder

OBJECTIVE

Conduct disorder (CD) has been associated with deficits in the use of punishment to guide reinforcement learning (RL) and decision making. This may explain the poorly planned and often impulsive antisocial and aggressive behavior in affected youths. Here, we used a computational modeling approach to examine differences in RL abilities between CD youths and typically developing controls (TDCs). Specifically, we tested 2 competing hypotheses that RL deficits in CD reflect either reward dominance (also known as reward hypersensitivity) or punishment insensitivity (also known as punishment hyposensitivity).

METHOD

The study included 92 CD youths and 130 TDCs (aged 9-18 years, 48% girls) who completed a probabilistic RL task with reward, punishment, and neutral contingencies. Using computational modeling, we investigated the extent to which the 2 groups differed in their learning abilities to obtain reward and/or to avoid punishment.

RESULTS

RL model comparisons showed that a model with separate learning rates per contingency explained behavioral performance best. Importantly, CD youths showed lower learning rates than TDCs specifically for punishment, whereas learning rates for reward and neutral contingencies did not differ. Moreover, callous-unemotional (CU) traits did not correlate with learning rates in CD.

CONCLUSION

CD youths have a highly selective impairment in probabilistic punishment learning, regardless of their CU traits, whereas reward learning appears to be intact. In summary, our data suggest punishment insensitivity rather than reward dominance in CD. Clinically, the use of punishment-based intervention techniques to achieve effective discipline in patients with CD may be a less helpful strategy than reward-based techniques.

Common and distinct equity preferences in children and adults

Fairness plays a crucial role in children's social life and has garnered considerable attention. However, previous research and theories primarily examined the development of children's fairness behaviors in the conflict between self-interest motivation and fairness-complying motivation, neglecting the influence of advantage-seeking motivation. Moreover, despite the well-established role of gain/loss frame in human decision-making, it remains largely unclear whether the framing effect modulates fairness behaviors in children. It was hypothesized that children would exhibit advantage-seeking motivation resulting in more selfish behaviors in the loss context. To examine the hypothesis, we combined an adapted dictator game and computational modeling to investigate various motivations underlying fairness behaviors of children in both loss and gain contexts and to explore the developmental directions by contrasting children and adults. In addition, the current design enabled the dissociation between fairness knowledge and behaviors by asking participants to decide for themselves (the first-party role) or for others (the third-party role). This study recruited a total of 34 children (9-10 years, Mage = 9.82, SDage = 0.38, 16 females) and 31 college students (Mage = 19.81, SDage = 1.40, 17 females). The behavioral results indicated that children behaved more selfishly in first-party and more fairly in third-party than adults, without any significant framing effects. The computational results revealed that both children and adults exhibited aversion to advantageous and disadvantageous inequity in third-party. However, they showed distinct preferences for advantageous inequity in first-party, with advantage-seeking preferences among children and aversion to advantageous inequity among adults. These findings contribute to a deeper understanding of children's social preferences and their developmental directions.

Neural and behavioral evidence for oxytocin's facilitatory effects on learning in volatile and stable environments

Outcomes of past decisions profoundly shape our behavior. However, choice-outcome associations can become volatile and adaption to such changes is of importance. The present study combines pharmaco-electroencephalography with computational modeling to examine whether intranasal oxytocin can modulate reinforcement learning under a volatile vs. a stable association. Results show that oxytocin increases choice accuracy independent of learning context, which is paralleled by a larger N2pc and a smaller P300. Model-based analyses reveal that while oxytocin promotes learning by accelerating value update of outcomes in the volatile context, in the stable context it does so by improving choice consistency. These findings suggest that oxytocin's facilitatory effects on learning may be exerted via improving early attentional selection and late neural processing efficiency, although at the computational level oxytocin's actions are highly adaptive between learning contexts. Our findings provide proof of concept for oxytocin's therapeutic potential in mental disorders with adaptive learning dysfunction.

Neural Correlates of Social Decision-Making

Objective: Recent studies have utilized innovative techniques to investigate the neural mechanisms underlying social and individual decision-making, aiming to understand how individuals respond to the world. Method : In this review, we summarized current scientific evidence concerning the neural underpinnings of social decision-making and their impact on social behavior. Results: Critical brain regions involved in social cognition and decision-making are integral to the process of social decision-making. Notably, the medial prefrontal cortex (mPFC) and temporoparietal junction (TPJ) contribute to the comprehension of others' mental states. Similarly, the posterior superior temporal sulcus (pSTS) shows heightened activity when individuals observe faces and movements. On the lateral surface of the brain, the inferior frontal gyrus (IFG) and inferior parietal sulcus (IPS) play a role in social cognition. Furthermore, the medial surface of the brain, including the amygdala, anterior cingulate cortex (ACC), and anterior insula (AI), also participates in social cognition processes. Regarding decision-making, functional magnetic resonance imaging (fMRI) studies have illuminated the involvement of a network of brain regions, encompassing the ventromedial prefrontal cortex (vmPFC), ventral striatum (VS), and nucleus accumbens (NAcc). Conclusion: Dysfunction in specific subregions of the prefrontal cortex (PFC) has been linked to various psychiatric conditions. These subregions play pivotal roles in cognitive, emotional, and social processing, and their impairment can contribute to the development and manifestation of psychiatric symptoms. A comprehensive understanding of the unique contributions of these PFC subregions to psychiatric disorders has the potential to inform the development of targeted interventions and treatments for affected individuals.

The right superior temporal gyrus plays a role in semantic-rule learning: Evidence supporting a reinforcement learning model

In real-life communication, individuals use language that carries evident rewarding and punishing elements, such as praise and criticism. A common trend is to seek more praise while avoiding criticism. Furthermore, semantics is crucial for conveying information, but such semantic access to native and foreign languages is subtly distinct. To investigate how rule learning occurs in different languages and to highlight the importance of semantics in this process, we investigated both verbal and non-verbal rule learning in first (L1) and second (L2) languages using a reinforcement learning framework, including a semantic rule and a color rule. Our computational modeling on behavioral and brain imaging data revealed that individuals may be more motivated to learn and adhere to rules in an L1 compared to L2, with greater striatum activation during the outcome phase in the L1. Additionally, results on the learning rates and inverse temperature in the two rule learning tasks showed that individuals tend to be conservative and are reluctant to change their judgments regarding rule learning of semantic information. Moreover, the greater the prediction errors, the greater activation of the right superior temporal gyrus in the semantic-rule learning condition, demonstrating that such learning has differential neural correlates than symbolic rule learning. Overall, the findings provide insight into the neural mechanisms underlying rule learning in different languages, and indicate that rule learning involving verbal semantics is not a general symbolic learning that resembles a conditioned stimulus-response, but rather has its own specific characteristics.

A Brain-To-Brain Mechanism for Social Transmission of Threat Learning

Survival and adaptation in environments require swift and efficacious learning about what is dangerous. Across species, much of such threat learning is acquired socially, e.g., through the observation of others' ("demonstrators'") defensive behaviors. However, the specific neural mechanisms responsible for the integration of information shared between demonstrators and observers remain largely unknown. This dearth of knowledge is addressed by performing magnetoencephalography (MEG) neuroimaging in demonstrator-observer dyads. A set of stimuli are first shown to a demonstrator whose defensive responses are filmed and later presented to an observer, while neuronal activity is recorded sequentially from both individuals who never interacted directly. These results show that brain-to-brain coupling (BtBC) in the fronto-limbic circuit (including insula, ventromedial, and dorsolateral prefrontal cortex) within demonstrator-observer dyads predict subsequent expressions of learning in the observer. Importantly, the predictive power of BtBC magnifies when a threat is imminent to the demonstrator. Furthermore, BtBC depends on how observers perceive their social status relative to the demonstrator, likely driven by shared attention and emotion, as bolstered by dyadic pupillary coupling. Taken together, this study describes a brain-to-brain mechanism for social threat learning, involving BtBC, which reflects social relationships and predicts adaptive, learned behaviors.

Impaired social learning in patients with major depressive disorder revealed by a reinforcement learning model

Background/objective

Patients with major depressive disorder (MDD) have altered learning rates for rewards and losses in non-social learning paradigms. However, it is not well understood whether the ability to learn from social interactions is altered in MDD patients. Using reinforcement learning during the repeated Trust Game (rTG), we investigated how MDD patients learn to trust newly-met partners in MDD patients.

Method

Sixty-eight MDD patients and fifty-four controls each played as 'investor' and interacted with ten different partners. We manipulated both the level of trustworthiness by varying the chance of reciprocity (10, 30, 50, 70 and 90%) and reputation disclosure, where partners' reputation was either pre-disclosed or hidden.

Results

Our reinforcement learning model revealed that MDD patients had significantly higher learning rates for losses than the controls in both the reputation disclosure and non-disclosure condition. The difference was larger when reputation was not disclosed than disclosed. We observed no difference in learning rates for gains in either condition.

Conclusions

Our findings highlight that abnormal learning for losses underlies the social learning process in MDD patients. This abnormality is higher when situational unpredictability is high versus low. Our findings provide novel insights into social rehabilitation of MDD.

The dorsomedial prefrontal cortex represents subjective value across effort-based and risky decision-making

Decisions that require taking effort costs into account are ubiquitous in real life. The neural common currency theory hypothesizes that a particular neural network integrates different costs (e.g., risk) and rewards into a common scale to facilitate value comparison. Although there has been a surge of interest in the computational and neural basis of effort-related value integration, it is still under debate if effort-based decision-making relies on a domain-general valuation network as implicated in the neural common currency theory. Therefore, we comprehensively compared effort-based and risky decision-making using a combination of computational modeling, univariate and multivariate fMRI analyses, and data from two independent studies. We found that effort-based decision-making can be best described by a power discounting model that accounts for both the discounting rate and effort sensitivity. At the neural level, multivariate decoding analyses indicated that the neural patterns of the dorsomedial prefrontal cortex (dmPFC) represented subjective value across different decision-making tasks including either effort or risk costs, although univariate signals were more diverse. These findings suggest that multivariate dmPFC patterns play a critical role in computing subjective value in a task-independent manner and thus extend the scope of the neural common currency theory.

Reduced effects of social feedback on learning in Turner syndrome

Turner syndrome is a genetic condition caused by a complete or partial loss of one of the X chromosomes. Previous studies indicate that Turner syndrome is associated with challenges in social skills, but the underlying mechanisms remain largely unexplored. A possible mechanism is a reduced social influence on learning. The current study examined the impact of social and non-social feedback on learning in women with Turner syndrome (n = 35) and a sex- and age-matched control group (n = 37). Participants were instructed to earn points by repeatedly choosing between two stimuli with unequal probabilities of resulting in a reward. Mastering the task therefore required participants to learn through feedback which of the two stimuli was more likely to be rewarded. Data were analyzed using computational modeling and analyses of choice behavior. Social feedback led to a more explorative choice behavior in the control group, resulting in reduced learning compared to non-social feedback. No effects of social feedback on learning were found in Turner syndrome. The current study thus indicates that women with Turner syndrome may be less sensitive to social influences on reinforcement learning, than the general population.

Testosterone eliminates strategic prosocial behavior through impacting choice consistency in healthy males

Humans are strategically more prosocial when their actions are being watched by others than when they act alone. Using a psychopharmacogenetic approach, we investigated the endocrinological and computational mechanisms of such audience-driven prosociality. One hundred and ninety-two male participants received either a single dose of testosterone (150 mg) or a placebo and performed a prosocial and self-benefitting reinforcement learning task. Crucially, the task was performed either in private or when being watched. Rival theories suggest that the hormone might either diminish or strengthen audience-dependent prosociality. We show that exogenous testosterone fully eliminated strategic, i.e., feigned, prosociality and thus decreased submission to audience expectations. We next performed reinforcement-learning drift-diffusion computational modeling to elucidate which latent aspects of decision-making testosterone acted on. The modeling revealed that testosterone compared to placebo did not deteriorate reinforcement learning per se. Rather, when being watched, the hormone altered the degree to which the learned information on choice value translated to action selection. Taken together, our study provides novel evidence of testosterone's effects on implicit reward processing, through which it counteracts conformity and deceptive reputation strategies.

From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Advances in computational statistics and corresponding shifts in funding initiatives over the past few decades have led to a proliferation of neuroscientific measures being developed in the context of mental health research. Although such measures have undoubtedly deepened our understanding of neural mechanisms underlying cognitive, affective, and behavioral processes associated with various mental health conditions, the clinical utility of such measures remains underwhelming. Recent commentaries point toward the poor reliability of neuroscientific measures to partially explain this lack of clinical translation. Here, we provide a concise theoretical overview of how unreliability impedes clinical translation of neuroscientific measures; discuss how various modeling principles, including those from hierarchical and structural equation modeling frameworks, can help to improve reliability; and demonstrate how to combine principles of hierarchical and structural modeling within the generative modeling framework to achieve more reliable, generalizable measures of brain-behavior relationships for use in mental health research.

Intra- and inter-brain synchrony oscillations underlying social adjustment

Humans naturally synchronize their behavior with other people. However, although it happens almost automatically, adjusting behavior and conformity to others is a complex phenomenon whose neural mechanisms are still yet to be understood entirely. The present experiment aimed to study the oscillatory synchronization mechanisms underlying automatic dyadic convergence in an EEG hyperscanning experiment. Thirty-six people performed a cooperative decision-making task where dyads had to guess the correct position of a point on a line. A reinforcement learning algorithm was used to model different aspects of the participants' behavior and their expectations of their peers. Intra- and inter-connectivity among electrode sites were assessed using inter-site phase clustering in three main frequency bands (theta, alpha, beta) using a two-level Bayesian mixed-effects modeling approach. The results showed two oscillatory synchronization dynamics related to attention and executive functions in alpha and reinforcement learning in theta. In addition, inter-brain synchrony was mainly driven by beta oscillations. This study contributes preliminary evidence on the phase-coherence mechanism underlying inter-personal behavioral adjustment.

Blocking D2/D3 dopamine receptors in male participants increases volatility of beliefs when learning to trust others

The ability to learn about other people is crucial for human social functioning. Dopamine has been proposed to regulate the precision of beliefs, but direct behavioural evidence of this is lacking. In this study, we investigate how a high dose of the D2/D3 dopamine receptor antagonist sulpiride impacts learning about other people's prosocial attitudes in a repeated Trust game. Using a Bayesian model of belief updating, we show that in a sample of 76 male participants sulpiride increases the volatility of beliefs, which leads to higher precision weights on prediction errors. This effect is driven by participants with genetically conferred higher dopamine availability (Taq1a polymorphism) and remains even after controlling for working memory performance. Higher precision weights are reflected in higher reciprocal behaviour in the repeated Trust game but not in single-round Trust games. Our data provide evidence that the D2 receptors are pivotal in regulating prediction error-driven belief updating in a social context.

A neuro-computational social learning framework to facilitate transdiagnostic classification and treatment across psychiatric disorders

Social deficits are among the core and most striking psychiatric symptoms, present in most psychiatric disorders. Here, we introduce a novel social learning framework, which consists of neuro-computational models that combine reinforcement learning with various types of social knowledge structures. We outline how this social learning framework can help specify and quantify social psychopathology across disorders and provide an overview of the brain regions that may be involved in this type of social learning. We highlight how this framework can specify commonalities and differences in the social psychopathology of individuals with autism spectrum disorder (ASD), personality disorders (PD), and major depressive disorder (MDD) and improve treatments on an individual basis. We conjecture that individuals with psychiatric disorders rely on rigid social knowledge representations when learning about others, albeit the nature of their rigidity and the behavioral consequences can greatly differ. While non-clinical cohorts tend to efficiently adapt social knowledge representations to relevant environmental constraints, psychiatric cohorts may rigidly stick to their preconceived notions or overly coarse knowledge representations during learning.

A Fluid Self-Concept: How the Brain Maintains Coherence and Positivity across an Interconnected Self-Concept While Incorporating Social Feedback

People experience instances of social feedback as interdependent with potential implications for their entire self-concept. How do people maintain positivity and coherence across the self-concept while updating self-views from feedback? We present a network model describing how the brain represents the semantic dependency relations among traits and uses this information to avoid an overall loss of positivity and coherence. Both male and female human participants received social feedback during a self-evaluation task while undergoing functional magnetic resonance imaging. We modeled self-belief updating by incorporating a reinforcement learning model within the network structure. Participants learned more rapidly from positive than negative feedback and were less likely to change self-views for traits with more dependencies in the network. Further, participants back propagated feedback across network relations while retrieving prior feedback on the basis of network similarity to inform ongoing self-views. Activation in ventromedial prefrontal cortex (vmPFC) reflected the constrained updating process such that positive feedback led to higher activation and negative feedback to less activation for traits with more dependencies. Additionally, vmPFC was associated with the novelty of a trait relative to previously self-evaluated traits in the network, and angular gyrus was associated with greater certainty for self-beliefs given the relevance of prior feedback. We propose that neural computations that selectively enhance or attenuate social feedback and retrieve past relevant experiences to guide ongoing self-evaluations may support an overall positive and coherent self-concept.SIGNIFICANCE STATEMENT We humans experience social feedback throughout our lives, but we do not dispassionately incorporate feedback into our self-concept. The implications of feedback for our entire self-concept plays a role in how we either change or retain our prior self-beliefs. In a neuroimaging study, we find that people are less likely to change their beliefs from feedback when the feedback has broader implications for the self-concept. This resistance to change is reflected in processing in the ventromedial prefrontal cortex, a region that is central to self-referential and social cognition. These results are broadly applicable given the role that maintaining a positive and coherent self-concept plays in promoting mental health and development throughout the lifespan.

Modulation of ventromedial orbitofrontal cortical glutamatergic activity affects the explore-exploit balance and influences value-based decision-making

The balance between exploration and exploitation is essential for decision-making. The present study investigated the role of ventromedial orbitofrontal cortex (vmOFC) glutamate neurons in mediating value-based decision-making by first using optogenetics to manipulate vmOFC glutamate activity in rats during a probabilistic reversal learning (PRL) task. Rats that received vmOFC activation during informative feedback completed fewer reversals and exhibited reduced reward sensitivity relative to rats. Analysis with a Q-learning computational model revealed that increased vmOFC activity did not affect the learning rate but instead promoted maladaptive exploration. By contrast, vmOFC inhibition increased the number of completed reversals and increased exploitative behavior. In a separate group of animals, calcium activity of vmOFC glutamate neurons was recorded using fiber photometry. Complementing our results above, we found that suppression of vmOFC activity during the latter part of rewarded trials was associated with improved PRL performance, greater win-stay responding and selecting the correct choice on the next trial. These data demonstrate that excessive vmOFC activity during reward feedback disrupted value-based decision-making by increasing the maladaptive exploration of lower-valued options. Our findings support the premise that pharmacological interventions that normalize aberrant vmOFC glutamate activity during reward feedback processing may attenuate deficits in value-based decision-making.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

l-DOPA and oxytocin influence the neurocomputational mechanisms of self-benefitting and prosocial reinforcement learning

Humans learn through reinforcement, particularly when outcomes are unexpected. Recent research suggests similar mechanisms drive how we learn to benefit other people, that is, how we learn to be prosocial. Yet the neurochemical mechanisms underlying such prosocial computations remain poorly understood. Here, we investigated whether pharmacological manipulation of oxytocin and dopamine influence the neurocomputational mechanisms underlying self-benefitting and prosocial reinforcement learning. Using a double-blind placebo-controlled cross-over design, we administered intranasal oxytocin (24 IU), dopamine precursor l-DOPA (100 mg + 25 mg carbidopa), or placebo over three sessions. Participants performed a probabilistic reinforcement learning task with potential rewards for themselves, another participant, or no one, during functional magnetic resonance imaging. Computational models of reinforcement learning were used to calculate prediction errors (PEs) and learning rates. Participants behavior was best explained by a model with different learning rates for each recipient, but these were unaffected by either drug. On the neural level, however, both drugs blunted PE signaling in the ventral striatum and led to negative signaling of PEs in the anterior mid-cingulate cortex, dorsolateral prefrontal cortex, inferior parietal gyrus, and precentral gyrus, compared to placebo, and regardless of recipient. Oxytocin (versus placebo) administration was additionally associated with opposing tracking of self-benefitting versus prosocial PEs in dorsal anterior cingulate cortex, insula and superior temporal gyrus. These findings suggest that both l-DOPA and oxytocin induce a context-independent shift from positive towards negative tracking of PEs during learning. Moreover, oxytocin may have opposing effects on PE signaling when learning to benefit oneself versus another.

Aberrant uncertainty processing is linked to psychotic-like experiences, autistic traits, and is reflected in pupil dilation during probabilistic learning

Aberrant belief updating due to misestimation of uncertainty and an increased perception of the world as volatile (i.e., unstable) has been found in autism and psychotic disorders. Pupil dilation tracks events that warrant belief updating, likely reflecting the adjustment of neural gain. However, whether subclinical autistic or psychotic symptoms affect this adjustment and how they relate to learning in volatile environments remains to be unraveled. We investigated the relationship between behavioral and pupillometric markers of subjective volatility (i.e., experience of the world as unstable), autistic traits, and psychotic-like experiences in 52 neurotypical adults with a probabilistic reversal learning task. Computational modeling revealed that participants with higher psychotic-like experience scores overestimated volatility in low-volatile task periods. This was not the case for participants scoring high on autistic-like traits, who instead showed a diminished adaptation of choice-switching behavior in response to risk. Pupillometric data indicated that individuals with higher autistic- or psychotic-like trait and experience scores differentiated less between events that warrant belief updating and those that do not when volatility was high. These findings are in line with misestimation of uncertainty accounts of psychosis and autism spectrum disorders and indicate that aberrancies are already present at the subclinical level.

Neuro-computational mechanisms and individual biases in action-outcome learning under moral conflict

Learning to predict action outcomes in morally conflicting situations is essential for social decision-making but poorly understood. Here we tested which forms of Reinforcement Learning Theory capture how participants learn to choose between self-money and other-shocks, and how they adapt to changes in contingencies. We find choices were better described by a reinforcement learning model based on the current value of separately expected outcomes than by one based on the combined historical values of past outcomes. Participants track expected values of self-money and other-shocks separately, with the substantial individual difference in preference reflected in a valuation parameter balancing their relative weight. This valuation parameter also predicted choices in an independent costly helping task. The expectations of self-money and other-shocks were biased toward the favored outcome but fMRI revealed this bias to be reflected in the ventromedial prefrontal cortex while the pain-observation network represented pain prediction errors independently of individual preferences.

Angiotensin blockade enhances motivational reward learning via enhancing striatal prediction error signaling and frontostriatal communication

Adaptive human learning utilizes reward prediction errors (RPEs) that scale the differences between expected and actual outcomes to optimize future choices. Depression has been linked with biased RPE signaling and an exaggerated impact of negative outcomes on learning which may promote amotivation and anhedonia. The present proof-of-concept study combined computational modeling and multivariate decoding with neuroimaging to determine the influence of the selective competitive angiotensin II type 1 receptor antagonist losartan on learning from positive or negative outcomes and the underlying neural mechanisms in healthy humans. In a double-blind, between-subjects, placebo-controlled pharmaco-fMRI experiment, 61 healthy male participants (losartan, n = 30; placebo, n = 31) underwent a probabilistic selection reinforcement learning task incorporating a learning and transfer phase. Losartan improved choice accuracy for the hardest stimulus pair via increasing expected value sensitivity towards the rewarding stimulus relative to the placebo group during learning. Computational modeling revealed that losartan reduced the learning rate for negative outcomes and increased exploitatory choice behaviors while preserving learning for positive outcomes. These behavioral patterns were paralleled on the neural level by increased RPE signaling in orbitofrontal-striatal regions and enhanced positive outcome representations in the ventral striatum (VS) following losartan. In the transfer phase, losartan accelerated response times and enhanced VS functional connectivity with left dorsolateral prefrontal cortex when approaching maximum rewards. These findings elucidate the potential of losartan to reduce the impact of negative outcomes during learning and subsequently facilitate motivational approach towards maximum rewards in the transfer of learning. This may indicate a promising therapeutic mechanism to normalize distorted reward learning and fronto-striatal functioning in depression.

Social feedback enhances learning in Williams syndrome

Williams syndrome (WS) is a rare genetic condition characterized by high social interest and approach motivation as well as intellectual disability and anxiety. Despite the fact that social stimuli are believed to have an increased intrinsic reward value in WS, it is not known whether this translates to learning and decision making. Genes homozygously deleted in WS are linked to sociability in the general population, making it a potential model condition for understanding the social brain. Probabilistic reinforcement learning was studied with either social or non-social rewards for correct choices. Social feedback improved learning in individuals with Williams syndrome but not in typically developing controls or individuals with other intellectual disabilities. Computational modeling indicated that these effects on social feedback were mediated by a shift towards higher weight given to rewards relative to punishments and increased choice consistency. We conclude that reward learning in WS is characterized by high volatility and a tendency to learn how to avoid punishment rather than how to gain rewards. Social feedback can partly normalize this pattern and promote adaptive reward learning.

Comparing gratitude and pride: evidence from brain and behavior

Gratitude and pride are both positive emotions. Yet gratitude motivates people to help others and build up relationships, whereas pride motivates people to pursue achievements and build on self-esteem. Although these social outcomes are crucial for humans to be evolutionarily adaptive, no study so far has systematically compared gratitude and pride to understand why and how they can motivate humans differently. In this review, we compared gratitude and pride from their etymologies, cognitive prerequisites, motivational functions, and brain regions involved. By integrating the evidence from brain and behavior, we suggest that gratitude and pride share a common reward basis, yet gratitude is more related to theory of mind, while pride is more related to self-referential processing. Moreover, we proposed a cognitive neuroscientific model to explain the dynamics in gratitude and pride under a reinforcement learning framework.

Incorporating social knowledge structures into computational models

To navigate social interactions successfully, humans need to continuously learn about the personality traits of other people (e.g., how helpful or aggressive is the other person?). However, formal models that capture the complexities of social learning processes are currently lacking. In this study, we specify and test potential strategies that humans can employ for learning about others. Standard Rescorla-Wagner (RW) learning models only capture parts of the learning process because they neglect inherent knowledge structures and omit previously acquired knowledge. We therefore formalize two social knowledge structures and implement them in hybrid RW models to test their usefulness across multiple social learning tasks. We name these concepts granularity (knowledge structures about personality traits that can be utilized at different levels of detail during learning) and reference points (previous knowledge formalized into representations of average people within a social group). In five behavioural experiments, results from model comparisons and statistical analyses indicate that participants efficiently combine the concepts of granularity and reference points-with the specific combinations in models depending on the people and traits that participants learned about. Overall, our experiments demonstrate that variants of RW algorithms, which incorporate social knowledge structures, describe crucial aspects of the dynamics at play when people interact with each other.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Sustained MK-801 induced deficit in a novel probabilistic reversal learning task

Cognitive flexibility, the ability to adapt to unexpected changes, is critical for healthy environmental and social interactions, and thus to everyday functioning. In neuropsychiatric diseases, cognitive flexibility is often impaired and treatment options are lacking. Probabilistic reversal learning (PRL) is commonly used to measure cognitive flexibility in rodents and humans. In PRL tasks, subjects must sample choice options and, from probabilistic feedback, find the current best choice which then changes without warning. However, in rodents, pharmacological models of human cognitive impairment tend to disrupt only the first (or few) of several contingency reversals, making quantitative assessment of behavioral effects difficult. To address this limitation, we developed a novel rat PRL where reversals occur at relatively long intervals in time that demonstrates increased sensitivity to the non-competitive NMDA receptor antagonist MK-801. Here, we quantitively compare behavior in time-based PRL with a widely used task where reversals occur based on choice behavior. In time-based PRL, MK-801 induced sustained reversal learning deficits both in time and across reversal blocks but, at the same dose, only transient weak effects in performance-based PRL. Moreover, time-based PRL yielded better estimates of behavior and reinforcement learning model parameters, which opens meaningful pharmacological windows to efficiently test and develop novel drugs preclinically with the goal of improving cognitive impairment in human patients.

Learning from Ingroup Experiences Changes Intergroup Impressions

Humans form impressions toward individuals of their own social groups (ingroup members) and of different social groups (outgroup members). Outgroup-focused theories predict that intergroup impressions are mainly shaped by experiences with outgroup individuals, while ingroup-focused theories predict that ingroup experiences play a dominant role. Here we test predictions from these two psychological theories by estimating how intergroup impressions are dynamically shaped when people learn from both ingroup and outgroup experiences. While undergoing fMRI, male participants had identical experiences with different ingroup or outgroup members and rated their social closeness and impressions toward the ingroup and the outgroup. Behavioral results showed an initial ingroup bias in impression ratings which was significantly reduced over the course of learning, with larger effects in individuals with stronger ingroup identification. Computational learning models revealed that these changes in intergroup impressions were predicted by the weight given to ingroup prediction errors. Neurally, the individual weight for ingroup prediction errors was related to the coupling between the left inferior parietal lobule and the left anterior insula, which, in turn, predicted learning-related changes in intergroup impressions. Our findings provide computational and neural evidence for ingroup-focused theories, highlighting the importance of ingroup experiences in shaping social impressions in intergroup settings.SIGNIFICANCE STATEMENT Living in multicultural societies, humans interact with individuals of their own social groups (ingroup members) and of different social groups (outgroup members). However, little is known about how people learn from the mixture of ingroup and outgroup interactions, the most natural experiences in current societies. Here, participants had identical, intermixed experiences with different ingroup and outgroup individuals and rated their closeness and impressions toward the ingroup and the outgroup. Combining computational models and fMRI, we find that the weight given to ingroup experiences (ingroup prediction errors) is the main source of intergroup impression change, captured by changes in connectivity between the parietal lobe and insula. These findings highlight the importance of ingroup experiences in shaping intergroup impressions in complex social environments.

Neurocomputations of strategic behavior: From iterated to novel interactions

Strategic interactions, where an individual's payoff depends on the decisions of multiple intelligent agents, are ubiquitous among social animals. They span a variety of important social behaviors such as competition, cooperation, coordination, and communication, and often involve complex, intertwining cognitive operations ranging from basic reward processing to higher‐order mentalization. Here, we review the progress and challenges in probing the neural and cognitive mechanisms of strategic behavior of interacting individuals, drawing an analogy to recent developments in studies of reward‐seeking behavior, in particular, how research focuses in the field of strategic behavior have been expanded from adaptive behavior based on trial‐and‐error to flexible decisions based on limited prior experience. We highlight two important research questions in the field of strategic behavior: (i) How does the brain exploit past experience for learning to behave strategically? and (ii) How does the brain decide what to do in novel strategic situations in the absence of direct experience? For the former, we discuss the utility of learning models that have effectively connected various types of neural data with strategic learning behavior and helped elucidate the interplay among multiple learning processes. For the latter, we review the recent evidence and propose a neural generative mechanism by which the brain makes novel strategic choices through simulating others' goal‐directed actions according to rational or bounded‐rational principles obtained through indirect social knowledge.

This article is categorized under:

Economics > Interactive Decision‐Making

Psychology > Reasoning and Decision Making

Neuroscience > Cognition

Monetization of customer futures through machine learning and artificial intelligence based persuasive technologies

Purpose

The purpose of this study was to ascertain how real options investment perspective could be applied towards monetization of customer futures through the deployment of machine learning (ML) and artificial intelligence (AI)-based persuasive technologies.

Design/methodology/approach

The authors embarked on a theoretical treatise as advocated by scholars (Cornelissen, 2019; Barney, 2018; Cornelissen, 2017; Smithey Fulmer, 2012; Bacharach, 1989; Whetten, 1989; Weick,1989). Towards this end, theoretical argumentative logic was incrementally used to build an integrated perspective on the deployment of learning and AI-based persuasive technologies. This was carried out with strategic real options investment perspective to secure customer futures on m-commerce apps and e-commerce sites.

Findings

M-commerce apps and e-commerce sites have been deploying ML and AI-based tools (referred to as persuasive technologies), to nudge customers for increased and quicker purchase. The primary objective was to increase engagement time of customers (at an individual level), grow the number of customers (at market level) and increase firm revenue (at an organizational level). The deployment of any persuasive technology entailed increased investment (cash outflow) but was also expected to increase the level of revenue and margin (cash inflow). Given the dynamics of market and the emergent nature of persuasive technologies, ascertaining favourable cash flow was challenging. Real options strategy provided a robust theoretical perspective to time the persuasive technology-related investment in stages. This helped managers to be on time with loading customer purchase with increased temporal immediacy. A real options investment space involving six spaces has also been developed in this conceptual work. These were Never Invest, Immediately Investment, Present-day Investment Possibility, Possibly Invest Later, Invest Probably Later and Possibly Never Invest.

Research limitations/implications

The foundations of this study domain encompassed work done by an eclectic mix of scholars like from technology management (Siggelkow and Terwiesch, 2019a; Porter and Heppelmann, 2014), real options (Trigeorgis and Reuer, 2017; Luehrman, 1998a, 1998b), marketing intelligence and planning (Appel et al., 2020; Thaichon et al., 2019; Thaichon et al., 2020; Ye et al., 2019) and strategy from a demand positioning school of thought (Adner and Zemsky, 2006).

Practical implications

The findings would help managers to comprehend what level of investments need to be done in a staggered manner. The phased way of investing towards the deployment of ML and AI-based persuasive technologies would enable better monetization of customer futures. This would aid marketing managers for increased customer engagement at the individual level, fast monetization of customer futures and increased number of customers and consumption on m-commerce apps and e-commerce sites.

Originality/value

This was one of the first studies to apply real options investment perspective towards the deployment of ML and AI-based persuasive technologies for monetizing customer futures.

Spared performance but increased uncertainty in schizophrenia: Evidence from a probabilistic decision-making task

Aberrant attribution of salience to in fact little informative events might explain the emergence of positive symptoms in schizophrenia and has been linked to belief uncertainty. Uncertainty is thought to be encoded by neuromodulators, including norepinephrine. However, norepinephrinergic encoding of uncertainty, measured as task-related pupil dilation, has rarely been explored in schizophrenia. Here, we addressed this question by comparing individuals with a disorder from the schizophrenia spectrum to a non-psychiatric control group on behavioral and pupillometric measures in a probabilistic prediction task, where different levels of uncertainty were introduced. Behaviorally, patients performed similar to controls, but their belief uncertainty was higher, particularly when instability of the task environment was high, suggesting an increased sensitivity to this instability. Furthermore, while pupil dilation scaled positively with uncertainty, this was less the case for patients, suggesting aberrant neuromodulatory regulation of neural gain, which may hinder the reduction of uncertainty in the long run. Together, the findings point to abnormal uncertainty processing and norepinephrinergic signaling in schizophrenia, potentially informing future development of both psychopharmacological therapies and psychotherapeutic approaches that deal with the processing of uncertain information.

Learning About the Self: Motives for Coherence and Positivity Constrain Learning From Self-Relevant Social Feedback

People learn about themselves from social feedback, but desires for coherence and positivity constrain how feedback is incorporated into the self-concept. We developed a network-based model of the self-concept and embedded it in a reinforcement-learning framework to provide a computational account of how motivations shape self-learning from feedback. Participants (N = 46 adult university students) received feedback while evaluating themselves on traits drawn from a causal network of trait semantics. Network-defined communities were assigned different likelihoods of positive feedback. Participants learned from positive feedback but dismissed negative feedback, as reflected by asymmetries in computational parameters that represent the incorporation of positive versus negative outcomes. Furthermore, participants were constrained in how they incorporated feedback: Self-evaluations changed less for traits that have more implications and are thus more important to the coherence of the network. We provide a computational explanation of how motives for coherence and positivity jointly constrain learning about the self from feedback, an explanation that makes testable predictions for future clinical research.

Learning from gain and loss: Links to suicide risk

Despite preliminary evidence that people with suicide attempt histories demonstrate deficits in processing feedback, no studies have examined the interrelations of learning from feedback and emotional state on suicide risk. This study examined the influence of suicide risk and negative emotions on learning accuracy and rates among individuals with a range of borderline personality features (N = 145). Participants completed a reinforcement learning task after neutral and negative emotion inductions. Results revealed interactions between suicide risk and emotion condition, with elevated risk linked to greater increases in loss learning rate (training phase models) and gain learning rate (test phase models) post-negative emotion induction. Emotion-dependent fluctuations in learning performance may be markers of decision-making that are associated with greater suicide risk. This line of work has the potential to identify the contexts that confer greater risk for suicidal behaviors.

Adaptive Empathy: A Model for Learning Empathic Responses in Response to Feedback

Empathy is usually deployed in social interactions. Nevertheless, common measures and examinations of empathy study this construct in isolation from the person in distress. In this article we seek to extend the field of examination to include both empathizer and target to determine whether and how empathic responses are affected by feedback and learned through interaction. Building on computational approaches in feedback-based adaptations (e.g., no feedback, model-free and model-based learning), we propose a framework for understanding how empathic responses are learned on the basis of feedback. In this framework, adaptive empathy, defined as the ability to adapt one's empathic responses, is a central aspect of empathic skills and can provide a new dimension to the evaluation and investigation of empathy. By extending existing neural models of empathy, we suggest that adaptive empathy may be mediated by interactions between the neural circuits associated with valuation, shared distress, observation-execution, and mentalizing. Finally, we propose that adaptive empathy should be considered a prominent facet of empathic capabilities with the potential to explain empathic behavior in health and psychopathology.

Effects of dopamine D2/3 and opioid receptor antagonism on the trade-off between model-based and model-free behaviour in healthy volunteers

Human behaviour requires flexible arbitration between actions we do out of habit and actions that are directed towards a specific goal. Drugs that target opioid and dopamine receptors are notorious for inducing maladaptive habitual drug consumption; yet, how the opioidergic and dopaminergic neurotransmitter systems contribute to the arbitration between habitual and goal-directed behaviour is poorly understood. By combining pharmacological challenges with a well-established decision-making task and a novel computational model, we show that the administration of the dopamine D2/3 receptor antagonist amisulpride led to an increase in goal-directed or 'model-based' relative to habitual or 'model-free' behaviour, whereas the non-selective opioid receptor antagonist naltrexone had no appreciable effect. The effect of amisulpride on model-based/model-free behaviour did not scale with drug serum levels in the blood. Furthermore, participants with higher amisulpride serum levels showed higher explorative behaviour. These findings highlight the distinct functional contributions of dopamine and opioid receptors to goal-directed and habitual behaviour and support the notion that even small doses of amisulpride promote flexible application of cognitive control.

"Carpe Diem?": Disjunction Effect of Incidental Affect on Intertemporal Choice

Incidental affect has an important impact on intertemporal choice (IC). This research aimed to test how positive incidental affect influences IC and its underlying mechanisms. We assumed that positive incidental affect may have a disjunction effect on IC that includes or excludes immediate time. Moreover, we examined the role of time perception for the effect of affect on IC. In Study 1, after undergoing affect priming by video clips, participants completed the IC task using a multiple staircase paradigm. Using Hierarchical Bayesian Modeling, we estimated the discount rate parameter by distinguishing “immediate” and “non-immediate” conditions of IC. The participants’ time perception was also measured. In Study 2, apart from the choice preference of IC, we additionally investigated the differences in the participants’ attention to delay and reward attributes before decision making. The results of the two studies indicated that positive incidental affect leads to longer time perception (Study 1) and prior and more attention to the delay attribute of IC (Study 2), which leads individuals to prefer immediate options in the IC (Studies 1 and 2). Moreover, there is a disjunction effect of affect; in other words, the incidental affect did not influence IC excluding immediate time (Studies 1 and 2). This study improves our understanding of the disjunctive effect and its mechanism of inducing a positive incidental affect on IC and thus provides a new perspective on how related decision making can be improved.

The computational challenge of social learning

The complex reward structure of the social world and the uncertainty endemic to social contexts poses a challenge for modeling. For example, during social interactions, the actions of one person influence the internal states of another. These social dependencies make it difficult to formalize social learning problems in a mathematically tractable way. While it is tempting to dispense with these complexities, they are a defining feature of social life. Because the structure of social interactions challenges the simplifying assumptions often made in models, they make an ideal testbed for computational models of cognition. By adopting a framework that embeds existing social knowledge into the model, we can go beyond explaining behaviors in laboratory tasks to explaining those observed in the wild.

The goal-relevance of affective stimuli is dynamically represented in affective experience

Affect is a continuous and temporally dependent process that represents an individual's ongoing relationship with its environment. However, there is a lack of evidence on how factors defining the dynamic sensory environment modulate changes in momentary affective experience. Here, we show that goal-dependent relevance of stimuli is a key factor shaping momentary affect in a dynamic context. Participants (N = 83) viewed sequentially presented images and reported their momentary affective experience after every fourth stimulus. Relevance was manipulated through an attentional task that rendered each image either task-relevant or task-irrelevant. Computational models were fitted to trial-by-trial affective responses to capture the key dynamic parameters explaining momentary affective experience. The findings from statistical analyses and computational models showed that momentary affective experience was shaped by the temporal integration of the affective impact of recently encountered stimuli, and that task-relevant stimuli, independent of stimulus affect, prompted larger changes in experienced pleasantness compared with task-irrelevant stimuli. These findings clearly show that dynamics of affective experience reflect goal-relevance of stimuli in our surroundings.

Oscillatory brain activity links experience to expectancy during associative learning

Associating a novel situation with a specific outcome involves a cascade of cognitive processes, including selecting relevant stimuli, forming predictions regarding expected outcomes, and updating memorized predictions based on experience. The present manuscript uses computational modeling and machine learning to test the hypothesis that alpha-band (8-12 Hz) oscillations are involved in the updating of expectations based on experience. Participants learned that a visual cue predicted an aversive loud noise with a probability of 50%. The Rescorla-Wagner model of associative learning explained trial-wise changes in self-reported noise expectancy as well as alpha power changes. Experience in the past trial and self-reported expectancy for the subsequent trial were accurately decoded based on the topographical distribution of alpha power at specific latencies. Decodable information during initial association formation and contingency report recurred when viewing the conditioned cue. Findings support the idea that alpha oscillations have multiple, temporally specific, roles in the formation of associations between cues and outcomes.

Computational modelling of social cognition and behaviour-a reinforcement learning primer

Social neuroscience aims to describe the neural systems that underpin social cognition and behaviour. Over the past decade, researchers have begun to combine computational models with neuroimaging to link social computations to the brain. Inspired by approaches from reinforcement learning theory, which describes how decisions are driven by the unexpectedness of outcomes, accounts of the neural basis of prosocial learning, observational learning, mentalizing and impression formation have been developed. Here we provide an introduction for researchers who wish to use these models in their studies. We consider both theoretical and practical issues related to their implementation, with a focus on specific examples from the field.

Neurocomputational models of altruistic decision-making and social motives: Advances, pitfalls, and future directions

This article discusses insights from computational models and social neuroscience into motivations, precursors, and mechanisms of altruistic decision-making and other-regard. We introduce theoretical and methodological tools for researchers who wish to adopt a multilevel, computational approach to study behaviors that promote others' welfare. Using examples from recent studies, we outline multiple mental and neural processes relevant to altruism. To this end, we integrate evidence from neuroimaging, psychology, economics, and formalized mathematical models. We introduce basic mechanisms-pertinent to a broad range of value-based decisions-and social emotions and cognitions commonly recruited when our decisions involve other people. Regarding the latter, we discuss how decomposing distinct facets of social processes can advance altruistic models and the development of novel, targeted interventions. We propose that an accelerated synthesis of computational approaches and social neuroscience represents a critical step towards a more comprehensive understanding of altruistic decision-making. We discuss the utility of this approach to study lifespan differences in social preference in late adulthood, a crucial future direction in aging global populations. Finally, we review potential pitfalls and recommendations for researchers interested in applying a computational approach to their research. This article is categorized under: Economics > Interactive Decision-Making Psychology > Emotion and Motivation Neuroscience > Cognition Economics > Individual Decision-Making.

A causal role of estradiol in human reinforcement learning

The sex hormone estradiol is hypothesized to play a key role in human cognition, and reward processing specifically, via increased dopamine D1-receptor signalling. However, the effect of estradiol on reward processing in men has never been established. To fill this gap, we performed a double-blind placebo-controlled study in which men (N = 100) received either a single dose of estradiol (2 mg) or a placebo. Subjects performed a probabilistic reinforcement learning task where they had to choose between two options with varying reward probabilities to maximize monetary reward. Results showed that estradiol administration increased reward sensitivity compared to placebo. This effect was observed in subjects' choices, how much weight they assigned to their previous choices, and subjective reports about the reward probabilities. Furthermore, effects of estradiol were moderated by reward sensitivity, as measured through the BIS/BAS questionnaire. Using reinforcement learning models, we found that behavioral effects of estradiol were reflected in increased learning rates. These results demonstrate a causal role of estradiol within the framework of reinforcement learning, by enhancing reward sensitivity and learning. Furthermore, they provide preliminary evidence for dopamine-related genetic variants moderating the effect of estradiol on reward processing.

Adaptive Empathy: Empathic Response Selection as a Dynamic, Feedback-Based Learning Process

Empathy allows us to respond to the emotional state of another person. Considering that an empathic interaction may last beyond the initial response, learning mechanisms may be involved in dynamic adaptation of the reaction to the changing emotional state of the other person. However, traditionally, empathy is assessed through sets of isolated reactions to another's distress. Here we address this gap by focusing on adaptive empathy, defined as the ability to learn and adjust one's empathic responses based on feedback. For this purpose, we designed a novel paradigm of associative learning in which participants chose one of two empathic strategies (reappraisal or distraction) to attenuate the distress of a target person, where one strategy had a higher probability of relieving distress. After each choice, participants received feedback about the success of their chosen strategy in relieving the target person's distress, which they could use to inform their future decisions. The results show that the participants made more accurate choices in the adaptive empathy condition than in a non-social control condition, pointing to an advantage for learning from social feedback. We found a correlation between adaptive empathy and a trait measure of cognitive empathy. These findings indicate that the ability to learn about the effectiveness of empathic responses may benefit from incorporating mentalizing abilities. Our findings provide a lab-based model for studying adaptive empathy and point to the potential contribution of learning theory to enhancing our understanding of the dynamic nature of empathy.