The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning

Observational learning of exploration-exploitation strategies in bandit tasks

In decision-making scenarios, individuals often face the challenge of balancing between exploring new options and exploiting known ones-a dynamic known as the exploration-exploitation trade-off. In such situations, people frequently have the opportunity to observe others' actions. Yet little is known about when, how, and from whom individuals use observational learning in the exploration-exploitation dilemma. In two experiments, participants completed multiple nine-armed bandit tasks, either independently or while observing a fictitious agent using either an explorative or equally successful exploitative strategy. To analyze participants' behaviors, we used a reinforcement learning model (simplified Kalman Filter) to extract parameters for both copying and exploration at the individual level. Results showed that participants copied the observed agents' choices by adding a bonus to the individually estimated value of the observed action. While most participants appear to use an unconditional copying approach, a subset of participants adopted a copy-when-uncertain approach, that is copying more when uncertain about the optimal action based on their individually acquired knowledge. Further, participants adjusted their exploration strategies in alignment with those observed. We discuss, in how far this can be understood as a form of emulation. Results on participants' preferences to copy from explorative versus exploitative agents are ambiguous. Contrary to expectations, similarity or dissimilarity between participants' and agents' exploration tendencies had no impact on observational learning. These results shed light on humans' processing of social and non-social information in exploration scenarios and conditions of observational learning.

Adaptive mechanisms of social and asocial learning in immersive collective foraging

Human cognition is distinguished by our ability to adapt to different environments and circumstances. Yet the mechanisms driving adaptive behavior have predominantly been studied in separate asocial and social contexts, with an integrated framework remaining elusive. Here, we use a collective foraging task in a virtual Minecraft environment to integrate these two fields, by leveraging automated transcriptions of visual field data combined with high-resolution spatial trajectories. Our behavioral analyses capture both the structure and temporal dynamics of social interactions, which are then directly tested using computational models sequentially predicting each foraging decision. These results reveal that adaptation mechanisms of both asocial foraging and selective social learning are driven by individual foraging success (rather than social factors). Furthermore, it is the degree of adaptivity-of both asocial and social learning-that best predicts individual performance. These findings not only integrate theories across asocial and social domains, but also provide key insights into the adaptability of human decision-making in complex and dynamic social landscapes.

Self-utility distance as a computational approach to understanding self-concept clarity

Self-concept stability and cohesion are crucial for psychological functioning and well-being, yet the mechanisms that underpin this fundamental aspect of human cognition remain underexplored. Integrating insights from cognitive and personality psychology with reinforcement learning, we introduce Self-Utility Distance (SUD)-a metric quantifying the dissimilarities between individuals' self-concept attributes and their expected utility value. In Study 1 (n = 155), participants provided self- and expected utility ratings using a set of predefined adjectives. SUD showed a significant negative relationship with Self-Concept Clarity that persisted after accounting for individuals' Self-Esteem. In Study 2 (n = 323), we found that SUD provides incremental predictive accuracy over Ideal-Self and Ought-Self discrepancies in the prediction of Self-Concept Clarity. In Study 3 (n = 85), we investigated the mechanistic principles underlying Self-Utility Distance. Participants conducted a social learning task where they learned about trait utilities from a reference group. We formalized different computational models to investigate the strategies individuals use to adjust trait utility estimates in response to environmental feedback. Through Hierarchical Bayesian Inference, we found evidence that participants utilized their self-concept to modulate trait utility learning, effectively avoiding the maximization of Self-Utility Distance. Our findings provide insights into self-concept dynamics that might help understand the maintenance of adaptive and maladaptive traits.

When to stop social learning from a predecessor in an information-foraging task

Striking a balance between individual and social learning is one of the key capabilities that support adaptation under uncertainty. Although intergenerational transmission of information is ubiquitous, little is known about when and how newcomers switch from learning loyally from preceding models to exploring independently. Using a behavioural experiment, we investigated how social information available from a preceding demonstrator affects the timing of becoming independent and individual performance thereafter. Participants worked on a 30-armed bandit task for 100 trials. For the first 15 trials, participants simply observed the choices of a demonstrator who had accumulated more knowledge about the environment and passively received rewards from the demonstrator's choices. Thereafter, participants could switch to making independent choices at any time. We had three conditions differing in the social information available from the demonstrator: choice only, reward only or both. Results showed that both participants' strategies about when to stop observational learning and their behavioural patterns after independence depended on the available social information. Participants generally failed to make the best use of previously observed social information in their subsequent independent choices, suggesting the importance of direct communication beyond passive observation for better intergenerational transmission under uncertainty. Implications for cultural evolution are discussed.

Effects of described demonstrator ability on brain and behavior when learning from others

Observational learning enables us to make decisions by watching others' behaviors. The quality of such learning depends on the abilities of those we observe, but also on our beliefs about those abilities. We have previously demonstrated that observers learned better from demonstrators described as high vs. low in ability, regardless of their actual performance. The current study aimed to conceptually replicate these findings, and explore the neural mechanisms involved. Forty-five participants performed an observational learning task while undergoing functional magnetic resonance imaging (fMRI). We hypothesized that participants would perform better when demonstrators were described as having high vs. low ability. Unexpectedly, participants performed equally well regardless of described demonstrator ability. The behavioral effects of biased observational learning seem to be driven by mentalizing processes together with general learning and decision-making processes.

Experience and advice consequences shape information sharing strategies

Individuals often rely on the advice of more experienced peers to minimise uncertainty and increase success likelihood. In most domains where knowledge is acquired through experience, advisers are themselves continuously learning. Here we examine the way advising behaviour changes throughout the learning process, and the way individual traits and costs and benefits of giving advice shape this behaviour. We ran a series of experiments implementing a decision task within a reinforcement learning framework, where participants could decide to share their choices as advice to others. Participants were overall likely to share their choices as advice, even on the first trial before learning. Tendency to share advice and advice quality increased as advisers learned about the value of choices, and moved from exploratory to exploitative behaviour. The introduction of consequences to advising resulted in a shift of the overall tendency to give advice, lowering it when advising implicated monetary loss, and increasing it when advising held reputational value. Individual differences in social anxiety levels were associated with lower tendency to share exploratory decisions. Our results show that advisers tend to share choices that are backed by their own experience, but that this relationship can be altered by advice-consequences and individual traits.

Causal involvement of dorsomedial prefrontal cortex in learning the predictability of observable actions

Social learning is well established across species. While recent neuroimaging studies show that dorsomedial prefrontal cortex (DMPFC/preSMA) activation correlates with observational learning signals, the precise computations that are implemented by DMPFC/preSMA have remained unclear. To identify whether DMPFC/preSMA supports learning from observed outcomes or observed actions, or possibly encodes even a higher order factor (such as the reliability of the demonstrator), we downregulate DMPFC/preSMA excitability with continuous theta burst stimulation (cTBS) and assess different forms of observational learning. Relative to a vertex-cTBS control condition, DMPFC/preSMA downregulation decreases performance during action-based learning but has no effect on outcome-based learning. Computational modeling reveals that DMPFC/preSMA cTBS disrupts learning the predictability, a proxy of reliability, of the demonstrator and modulates the rate of learning from observed actions. Thus, our results suggest that the DMPFC is causally involved in observational action learning, mainly by adjusting the speed of learning about the predictability of the demonstrator.

Humans flexibly integrate social information despite interindividual differences in reward

There has been much progress in understanding human social learning, including recent studies integrating social information into the reinforcement learning framework. Yet previous studies often assume identical payoffs between observer and demonstrator, overlooking the diversity of social information in real-world interactions. We address this gap by introducing a socially correlated bandit task that accommodates payoff differences among participants, allowing for the study of social learning under more realistic conditions. Our Social Generalization (SG) model, tested through evolutionary simulations and two online experiments, outperforms existing models by incorporating social information into the generalization process, but treating it as noisier than individual observations. Our findings suggest that human social learning is more flexible than previously believed, with the SG model indicating a potential resource-rational trade-off where social learning partially replaces individual exploration. This research highlights the flexibility of humans' social learning, allowing us to integrate social information from others with different preferences, skills, or goals.

Transmission of social bias through observational learning

People often rely on social learning-learning by observing others' actions and outcomes-to form preferences in advance of their own direct experiences. Although typically adaptive, we investigated whether social learning may also contribute to the formation and spread of prejudice. In six experiments (n = 1550), we demonstrate that by merely observing interactions between a prejudiced actor and social group members, observers acquired the prejudices of the actor. Moreover, observers were unaware of the actors' bias, misattributing their acquired group preferences to the behavior of group members, despite identical behavior between groups. Computational modeling revealed that this effect was due to value shaping, whereby one's preferences are shaped by another's actions toward a target, in addition to the target's reward feedback. These findings identify social learning as a potent mechanism of prejudice formation that operates implicitly and supports the transmission of intergroup bias.

Humans can infer social preferences from decision speed alone

Humans are known to be capable of inferring hidden preferences and beliefs of their conspecifics when observing their decisions. While observational learning based on choices has been explored extensively, the question of how response times (RT) impact our learning of others' social preferences has received little attention. Yet, while observing choices alone can inform us about the direction of preference, they reveal little about the strength of this preference. In contrast, RT provides a continuous measure of strength of preference with faster responses indicating stronger preferences and slower responses signaling hesitation or uncertainty. Here, we outline a preregistered orthogonal design to investigate the involvement of both choices and RT in learning and inferring other's social preferences. Participants observed other people's behavior in a social preferences task (Dictator Game), seeing either their choices, RT, both, or no information. By coupling behavioral analyses with computational modeling, we show that RT is predictive of social preferences and that observers were able to infer those preferences even when receiving only RT information. Based on these findings, we propose a novel observational reinforcement learning model that closely matches participants' inferences in all relevant conditions. In contrast to previous literature suggesting that, from a Bayesian perspective, people should be able to learn equally well from choices and RT, we show that observers' behavior substantially deviates from this prediction. Our study elucidates a hitherto unknown sophistication in human observational learning but also identifies important limitations to this ability.

Social demonstration of colour preference improves the learning of associated demonstrated actions

We studied how different types of social demonstration improve house sparrows' (Passer domesticus) success in solving a foraging task that requires both operant learning (opening covers) and discrimination learning (preferring covers of the rewarding colour). We provided learners with either paired demonstration (of both cover opening and colour preference), action-only demonstration (of opening white covers only), or no demonstration (a companion bird eating without covers). We found that sparrows failed to learn the two tasks with no demonstration, and learned them best with a paired demonstration. Interestingly, the action of cover opening was learned faster with paired rather than action-only demonstration despite being equally demonstrated in both. We also found that only with paired demonstration, the speed of operant (action) learning was related to the demonstrator's level of activity. Colour preference (i.e. discrimination learning) was eventually acquired by all sparrows that learned to open covers, even without social demonstration of colour preference. Thus, adding a demonstration of colour preference was actually more important for operant learning, possibly as a result of increasing the similarity between the demonstrated and the learned tasks, thereby increasing the learner's attention to the actions of the demonstrator. Giving more attention to individuals in similar settings may be an adaptive strategy directing social learners to focus on ecologically relevant behaviours and on tasks that are likely to be learned successfully.

A Novel Obstacle Traversal Method for Multiple Robotic Fish Based on Cross-Modal Variational Autoencoders and Imitation Learning

Precision control of multiple robotic fish visual navigation in complex underwater environments has long been a challenging issue in the field of underwater robotics. To address this problem, this paper proposes a multi-robot fish obstacle traversal technique based on the combination of cross-modal variational autoencoder (CM-VAE) and imitation learning. Firstly, the overall framework of the robotic fish control system is introduced, where the first-person view of the robotic fish is encoded into a low-dimensional latent space using CM-VAE, and then different latent features in the space are mapped to the velocity commands of the robotic fish through imitation learning. Finally, to validate the effectiveness of the proposed method, experiments are conducted on linear, S-shaped, and circular gate frame trajectories with both single and multiple robotic fish. Analysis reveals that the visual navigation method proposed in this paper can stably traverse various types of gate frame trajectories. Compared to end-to-end learning and purely unsupervised image reconstruction, the proposed control strategy demonstrates superior performance, offering a new solution for the intelligent navigation of robotic fish in complex environments.

Kurvers RHJ. A Cognitive Computational Approach to Social and Collective Decision-Making

Collective dynamics play a key role in everyday decision-making. Whether social influence promotes the spread of accurate information and ultimately results in adaptive behavior or leads to false information cascades and maladaptive social contagion strongly depends on the cognitive mechanisms underlying social interactions. Here we argue that cognitive modeling, in tandem with experiments that allow collective dynamics to emerge, can mechanistically link cognitive processes at the individual and collective levels. We illustrate the strength of this cognitive computational approach with two highly successful cognitive models that have been applied to interactive group experiments: evidence-accumulation and reinforcement-learning models. We show how these approaches make it possible to simultaneously study (a) how individual cognition drives social systems, (b) how social systems drive individual cognition, and (c) the dynamic feedback processes between the two layers.

Disentangling the contribution of individual and social learning processes in human advice-taking behavior

The study of social learning examines how individuals learn from others by means of observation, imitation, or compliance with advice. However, it still remains largely unknown whether social learning processes have a distinct contribution to behavior, independent from non-social trial-and-error learning that often occurs simultaneously. 153 participants completed a reinforcement learning task, where they were asked to make choices to gain rewards. Advice from an artificial teacher was presented in 60% of the trials, allowing us to compare choice behavior with and without advice. Results showed a strong and reliable tendency to follow advice (test-retest reliability ~0.73). Computational modeling suggested a unique contribution of three distinct learning strategies: (a) individual learning (i.e., learning the value of actions, independent of advice), (b) informed advice-taking (i.e., learning the value of following advice), and (c) non-informed advice-taking (i.e., a constant bias to follow advice regardless of outcome history). Comparing artificial and empirical data provided specific behavioral regression signatures to both informed and non-informed advice taking processes. We discuss the theoretical implications of integrating internal and external information during the learning process.

Computational Social Psychology

Social psychologists attempt to explain how we interact by appealing to basic principles of how we think. To make good on this ambition, they are increasingly relying on an interconnected set of formal tools that model inference, attribution, value-guided decision making, and multi-agent interactions. By reviewing progress in each of these areas and highlighting the connections between them, we can better appreciate the structure of social thought and behavior, while also coming to understand when, why, and how formal tools can be useful for social psychologists.

Flexible social inference facilitates targeted social learning when rewards are not observable

Groups coordinate more effectively when individuals are able to learn from others' successes. But acquiring such knowledge is not always easy, especially in real-world environments where success is hidden from public view. We suggest that social inference capacities may help bridge this gap, allowing individuals to update their beliefs about others' underlying knowledge and success from observable trajectories of behaviour. We compared our social inference model against simpler heuristics in three studies of human behaviour in a collective-sensing task. Experiment 1 demonstrated that average performance improved as a function of group size at a rate greater than predicted by heuristic models. Experiment 2 introduced artificial agents to evaluate how individuals selectively rely on social information. Experiment 3 generalized these findings to a more complex reward landscape. Taken together, our findings provide insight into the relationship between individual social cognition and the flexibility of collective behaviour.

Prosocial learning: Model-based or model-free?

Prosocial learning involves the acquisition of knowledge and skills necessary for making decisions that benefit others. We asked if, in the context of value-based decision-making, there is any difference between learning strategies for oneself vs. for others. We implemented a 2-step reinforcement learning paradigm in which participants learned, in separate blocks, to make decisions for themselves or for a present other confederate who evaluated their performance. We replicated the canonical features of the model-based and model-free reinforcement learning in our results. The behaviour of the majority of participants was best explained by a mixture of the model-based and model-free control, while most participants relied more heavily on MB control, and this strategy enhanced their learning success. Regarding our key self-other hypothesis, we did not find any significant difference between the behavioural performances nor in the model-based parameters of learning when comparing self and other conditions.

How we learn social norms: a three-stage model for social norm learning

As social animals, humans are unique to make the world function well by developing, maintaining, and enforcing social norms. As a prerequisite among these norm-related processes, learning social norms can act as a basis that helps us quickly coordinate with others, which is beneficial to social inclusion when people enter into a new environment or experience certain sociocultural changes. Given the positive effects of learning social norms on social order and sociocultural adaptability in daily life, there is an urgent need to understand the underlying mechanisms of social norm learning. In this article, we review a set of works regarding social norms and highlight the specificity of social norm learning. We then propose an integrated model of social norm learning containing three stages, i.e., pre-learning, reinforcement learning, and internalization, map a potential brain network in processing social norm learning, and further discuss the potential influencing factors that modulate social norm learning. Finally, we outline a couple of future directions along this line, including theoretical (i.e., societal and individual differences in social norm learning), methodological (i.e., longitudinal research, experimental methods, neuroimaging studies), and practical issues.

Incorporating social knowledge structures into computational models

To navigate social interactions successfully, humans need to continuously learn about the personality traits of other people (e.g., how helpful or aggressive is the other person?). However, formal models that capture the complexities of social learning processes are currently lacking. In this study, we specify and test potential strategies that humans can employ for learning about others. Standard Rescorla-Wagner (RW) learning models only capture parts of the learning process because they neglect inherent knowledge structures and omit previously acquired knowledge. We therefore formalize two social knowledge structures and implement them in hybrid RW models to test their usefulness across multiple social learning tasks. We name these concepts granularity (knowledge structures about personality traits that can be utilized at different levels of detail during learning) and reference points (previous knowledge formalized into representations of average people within a social group). In five behavioural experiments, results from model comparisons and statistical analyses indicate that participants efficiently combine the concepts of granularity and reference points-with the specific combinations in models depending on the people and traits that participants learned about. Overall, our experiments demonstrate that variants of RW algorithms, which incorporate social knowledge structures, describe crucial aspects of the dynamics at play when people interact with each other.

Social learning across adolescence: A Bayesian neurocognitive perspective

Adolescence is a period of social re-orientation in which we are generally more prone to peer influence and the updating of our beliefs based on social information, also called social learning, than in any other stage of our life. However, how do we know when to use social information and whose information to use and how does this ability develop across adolescence? Here, we review the social learning literature from a behavioral, neural and computational viewpoint, focusing on the development of brain systems related to executive functioning, value-based decision-making and social cognition. We put forward a Bayesian reinforcement learning framework that incorporates social learning about value associated with particular behavior and uncertainty in our environment and experiences. We discuss how this framework can inform us about developmental changes in social learning, including how the assessment of uncertainty and the ability to adaptively discriminate between information from different social sources change across adolescence. By combining reward-based decision-making in the domains of both informational and normative influence, this framework explains both negative and positive social peer influence in adolescence.

Cultural diffusion dynamics depend on behavioural production rules

Culture is an outcome of both the acquisition of knowledge about behaviour through social transmission, and its subsequent production by individuals. Acquisition and production are often discussed or modelled interchangeably, yet to date no study has explored the consequences of their interaction for cultural diffusions. We present a generative model that integrates the two, and ask how variation in production rules might influence diffusion dynamics. Agents make behavioural choices that change as they learn from their productions. Their repertoires may also change, and the acquisition of behaviour is conditioned on its frequency. We analyse the diffusion of a novel behaviour through social networks, yielding generalizable predictions of how individual-level behavioural production rules influence population-level diffusion dynamics. We then investigate how linking acquisition and production might affect the performance of two commonly used inferential models for social learning; network-based diffusion analysis, and experience-weighted attraction models. We find that the influence that production rules have on diffusion dynamics has consequences for how inferential methods are applied to empirical data. Our model illuminates the differences between social learning and social influence, demonstrates the overlooked role of reinforcement learning in cultural diffusions, and allows for clearer discussions about social learning strategies.

Insights about the common generative rule underlying an information foraging task can be facilitated via collective search

Social learning is beneficial for efficient information search in unfamiliar environments ("within-task" learning). In the real world, however, possible search spaces are often so large that decision makers are incapable of covering all options, even if they pool their information collectively. One strategy to handle such overload is developing generalizable knowledge that extends to multiple related environments ("across-task" learning). However, it is unknown whether and how social information may facilitate such across-task learning. Here, we investigated participants' social learning processes across multiple laboratory foraging sessions in spatially correlated reward landscapes that were generated according to a common rule. The results showed that paired participants were able to improve efficiency in information search across sessions more than solo participants. Computational analysis of participants' choice-behaviors revealed that such improvement across sessions was related to better understanding of the common generative rule. Rule understanding was correlated within a pair, suggesting that social interaction is a key to the improvement of across-task learning.

Conformist social learning leads to self-organised prevention against adverse bias in risky decision making

Given the ubiquity of potentially adverse behavioural bias owing to myopic trial-and-error learning, it seems paradoxical that improvements in decision-making performance through conformist social learning, a process widely considered to be bias amplification, still prevail in animal collective behaviour. Here we show, through model analyses and large-scale interactive behavioural experiments with 585 human subjects, that conformist influence can indeed promote favourable risk taking in repeated experience-based decision making, even though many individuals are systematically biased towards adverse risk aversion. Although strong positive feedback conferred by copying the majority's behaviour could result in unfavourable informational cascades, our differential equation model of collective behavioural dynamics identified a key role for increasing exploration by negative feedback arising when a weak minority influence undermines the inherent behavioural bias. This 'collective behavioural rescue', emerging through coordination of positive and negative feedback, highlights a benefit of collective learning in a broader range of environmental conditions than previously assumed and resolves the ostensible paradox of adaptive collective behavioural flexibility under conformist influences.

Distinct neurocomputational mechanisms support informational and socially normative conformity

A change of mind in response to social influence could be driven by informational conformity to increase accuracy, or by normative conformity to comply with social norms such as reciprocity. Disentangling the behavioural, cognitive, and neurobiological underpinnings of informational and normative conformity have proven elusive. Here, participants underwent fMRI while performing a perceptual task that involved both advice-taking and advice-giving to human and computer partners. The concurrent inclusion of 2 different social roles and 2 different social partners revealed distinct behavioural and neural markers for informational and normative conformity. Dorsal anterior cingulate cortex (dACC) BOLD response tracked informational conformity towards both human and computer but tracked normative conformity only when interacting with humans. A network of brain areas (dorsomedial prefrontal cortex (dmPFC) and temporoparietal junction (TPJ)) that tracked normative conformity increased their functional coupling with the dACC when interacting with humans. These findings enable differentiating the neural mechanisms by which different types of conformity shape social changes of mind.

Trusting and learning from others: immediate and long-term effects of learning from observation and advice

Social learning underpins our species's extraordinary success. Learning through observation has been investigated in several species, but learning from advice-where information is intentionally broadcast-is less understood. We used a pre-registered, online experiment (n = 1492) combined with computational modelling to examine learning through observation and advice. Participants were more likely to immediately follow advice than to copy an observed choice, but this was dependent upon trust in the adviser: highly paranoid participants were less likely to follow advice in the short term. Reinforcement learning modelling revealed two distinct patterns regarding the long-term effects of social information: some individuals relied fully on social information, whereas others reverted to trial-and-error learning. This variation may affect the prevalence and fidelity of socially transmitted information. Our results highlight the privileged status of advice relative to observation and how the assimilation of intentionally broadcast information is affected by trust in others.

Reinforcement Learning With Human Advice: A Survey

In this paper, we provide an overview of the existing methods for integrating human advice into a reinforcement learning process. We first propose a taxonomy of the different forms of advice that can be provided to a learning agent. We then describe the methods that can be used for interpreting advice when its meaning is not determined beforehand. Finally, we review different approaches for integrating advice into the learning process.