A simple model for learning in volatile environments

Further perceptions of probability: Accurate, stepwise updating is contingent on prior information about the task and the response mode

To adapt to an uncertain world, humans must learn event probabilities. These probabilities may be stationary, such as that of rolling a 6 on a die, or changing over time, like the probability of rainfall over the year. Research on how people estimate and track changing probabilities has recently reopened an old epistemological issue. A small, mostly recent literature finds that people accurately track the probability and change their estimates only occasionally, resulting in staircase-shaped response patterns. This has been taken as evidence that people entertain beliefs about unknown, distal states of the world, which are tested against observations to produce discrete shifts between hypotheses. That idea stands in contrast to the claim that people learn by continuously updating associations between observed events. The purpose of this article is to investigate the generality and robustness of the accurate, staircase-shaped pattern. In two experiments, we find that the response pattern is contingent on the response mode and prior information about the generative process. Participants exist on continua of accuracy and staircase-ness and we only reproduce previous results when changing estimates is effortful and prior information is provided-the specific conditions of previous experiments. We conclude that explaining this solely through either hypotheses or associations is untenable. A complete theory of probability estimation requires the interaction of three components: (i) online tracking of observed data, (ii) beliefs about the unobserved "generative process," and (iii) a response updating process. Participants' overt estimates depend on how the specific task conditions jointly determine all three.

Differentiating Reinforcement Learning and Episodic Memory in Value-Based Decisions in Parkinson's Disease

Patients with Parkinson's disease (PD) are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in PD. Recent studies have found that even simple reward-based learning tasks rely on a combination of cognitive and computational strategies, including one-shot episodic memory. These findings raise questions about how incremental learning and episodic memory contribute to decision-making in PD. We tested healthy participants (n = 26; 14 males and 12 females) and patients with PD (n = 26; 16 males and 10 females), both on- and off-dopamine replacement medication, on a task designed to differentiate between the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that PD patients performed equally well as healthy controls when using episodic memory but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit and enhanced subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that while PD patients are impaired at learning about reward from trial-and-error, their ability to encode memories for the value of one-shot experiences is intact.

Dynamic prefrontal coupling coordinates adaptive decision-making

Adaptive decision-making requires flexibly maintaining or changing behavior in response to uncertainty. While the dorsomedial (dmPFC) and dorsolateral (dIPFC) prefrontal cortex are each essential for this ability, how they coordinate to drive adaptation remains unknown. Using intracranial EEG recordings from human participants performing a dynamic reward task, we identified distinct, frequency-specific computations: dmPFC high-gamma activity encoded uncertainty before stay decisions but transitioned to prediction error before switches, while theta activity shifted from uncertainty to value representation. In contrast, dIPFC theta activity signaled both value and uncertainty before stays, but predominantly value before switches. Crucially, these regions coordinated through two temporally specific coupling mechanisms that predicted behavioral changes: theta-theta amplitude coupling during feedback processing and theta-gamma phase coupling before decisions. Both coupling mechanisms strengthened before switches, suggesting that changing behavior requires greater dmPFC-dIPFC integration than maintaining. These findings reveal how the dorsal prefrontal cortex employs frequency-specific computations and precise temporal coordination to guide adaptive behavior.

Sample size matters when estimating test-retest reliability of behaviour

Intraclass correlation coefficients (ICCs) are a commonly used metric in test-retest reliability research to assess a measure's ability to quantify systematic between-subject differences. However, estimates of between-subject differences are also influenced by factors including within-subject variability, random errors, and measurement bias. Here, we use data collected from a large online sample (N = 150) to (1) quantify test-retest reliability of behavioural and computational measures of reversal learning using ICCs, and (2) use our dataset as the basis for a simulation study investigating the effects of sample size on variance component estimation and the association between estimates of variance components and ICC measures. In line with previously published work, we find reliable behavioural and computational measures of reversal learning, a commonly used assay of behavioural flexibility. Reliable estimates of between-subject, within-subject (across-session), and error variance components for behavioural and computational measures (with ± .05 precision and 80% confidence) required sample sizes ranging from 10 to over 300 (behavioural median N: between-subject = 167, within-subject = 34, error = 103; computational median N: between-subject = 68, within-subject = 20, error = 45). These sample sizes exceed those often used in reliability studies, suggesting that sample sizes larger than are commonly used for reliability studies (circa 30) are required to robustly estimate reliability of task performance measures. Additionally, we found that ICC estimates showed highly positive and highly negative correlations with between-subject and error variance components, respectively, as might be expected, which remained relatively stable across sample sizes. However, ICC estimates were weakly or not correlated with within-subject variance, providing evidence for the importance of variance decomposition for reliability studies.

Striatal arbitration between choice strategies guides few-shot adaptation

Animals often exhibit rapid action changes in context-switching environments. This study hypothesized that, compared to the expected outcome, an unexpected outcome leads to distinctly different action-selection strategies to guide rapid adaptation. We designed behavioral measures differentiating between trial-by-trial dynamics after expected and unexpected events. In various reversal learning data with different rodent species and task complexities, conventional learning models failed to replicate the choice behavior following an unexpected outcome. This discrepancy was resolved by the proposed model with two different decision variables contingent on outcome expectation: the support-stay and conflict-shift bias. Electrophysiological data analyses revealed that striatal neurons encode our model's key variables. Furthermore, the inactivation of striatal direct and indirect pathways neutralizes the effect of past expected and unexpected outcomes, respectively, on the action-selection strategy following an unexpected outcome. Our study suggests unique roles of the striatum in arbitrating between different action selection strategies for few-shot adaptation.

Distinct Computational Mechanisms of Uncertainty Processing Explain Opposing Exploratory Behaviors in Anxiety and Apathy

BACKGROUND

Decision making in uncertain environments can lead to varied outcomes, and how we process those outcomes may depend on our emotional state. Understanding how individuals interpret the sources of uncertainty is crucial for understanding adaptive behavior and mental well-being. Uncertainty can be broadly categorized into 2 components: volatility and stochasticity. Volatility describes how quickly conditions change. Stochasticity, on the other hand, refers to outcome randomness. We investigated how anxiety and apathy influenced people's perceptions of uncertainty and how uncertainty perception shaped explore-exploit decisions.

METHODS

Participants (N = 1001, nonclinical sample) completed a restless 3-armed bandit task that was analyzed using both latent state and process models.

RESULTS

Individuals with anxiety perceived uncertainty as resulting more from volatility, leading to increased exploration and learning rates, especially after reward omission. Conversely, individuals with apathy viewed uncertainty as more stochastic, resulting in decreased exploration and learning rates. The perceived volatility to stochasticity ratio mediated the anxiety-exploration relationship post adverse outcomes. Dimensionality reduction showed exploration and uncertainty estimation to be distinct but related latent factors shaping a manifold of adaptive behavior that is modulated by anxiety and apathy.

CONCLUSIONS

These findings reveal distinct computational mechanisms for how anxiety and apathy influence decision making, providing a framework for understanding cognitive and affective processes in neuropsychiatric disorders.

Stochastic decisions support optimal foraging of volatile environments, and are disrupted by anxiety

Adolescence is a developmental period of relative volatility, where the individual experiences significant changes to their physical and social environment. The ability to adapt to the volatility of one's surroundings is an important cognitive ability, particularly while foraging, a near-ubiquitous behaviour across the animal kingdom. As adolescents experience more volatility in their surroundings, we predicted that this age group would be more adept than adults at using exploration to adjust to volatility. We employed a foraging task with a well-validated computational model to characterise the mechanisms of exploration in volatile environments, preregistering the hypothesis that adolescents (aged 16-17; N = 91) would exhibit more optimal adaptation of their learning rate to changes in environmental volatility compared with adults (aged 24+; N = 90). However, surprisingly, both adolescents and adults exhibited suboptimal adjustment of their learning rate to environmental volatility. In contrast to the learning rate, it was instead participants' stochasticity (i.e., decision variability) that better resembled the adjustment to volatility made by the optimal RL agent. Although heightened stochasticity in the volatile environment led participants to more often trial different responses that facilitated discovery of changes to the environment, we also found that anxiety impaired this adaptive ability. The finding of heightened stochasticity in volatile environments contradicts expectations that the learning rate is responsible for successful adaptation and motivates future work on the deleterious role that anxiety plays when adolescents manage periods of transition.

Human hippocampus and dorsomedial prefrontal cortex infer and update latent causes during social interaction

Latent-cause inference is the process of identifying features of the environment that have caused an outcome. This problem is especially important in social settings where individuals may not make equal contributions to the outcomes they achieve together. Here, we designed a novel task in which participants inferred which of two characters was more likely to have been responsible for outcomes achieved by working together. Using computational modeling, univariate and multivariate analysis of human fMRI, and continuous theta-burst stimulation, we identified two brain regions that solved the task. Notably, as each outcome occurred, it was possible to decode the inference of its cause (the responsible character) from hippocampal activity. Activity in dorsomedial prefrontal cortex (dmPFC) updated estimates of association between cause-responsible character-and the outcome. Disruption of dmPFC activity impaired participants' ability to update their estimate as a function of inferred responsibility but spared their ability to infer responsibility.

Computational processes of simultaneous learning of stochasticity and volatility in humans

Making adaptive decisions requires predicting outcomes, and this in turn requires adapting to uncertain environments. This study explores computational challenges in distinguishing two types of noise influencing predictions: volatility and stochasticity. Volatility refers to diffusion noise in latent causes, requiring a higher learning rate, while stochasticity introduces moment-to-moment observation noise and reduces learning rate. Dissociating these effects is challenging as both increase the variance of observations. Previous research examined these factors mostly separately, but it remains unclear whether and how humans dissociate them when they are played off against one another. In two large-scale experiments, through a behavioral prediction task and computational modeling, we report evidence of humans dissociating volatility and stochasticity solely based on their observations. We observed contrasting effects of volatility and stochasticity on learning rates, consistent with statistical principles. These results are consistent with a computational model that estimates volatility and stochasticity by balancing their dueling effects.

Distinct computational mechanisms of uncertainty processing explain opposing exploratory behaviors in anxiety and apathy

Decision-making in uncertain environments often leads to varied outcomes. Understanding how individuals interpret the causes of unexpected feedback is crucial for adaptive behavior and mental well-being. Uncertainty can be broadly categorized into two components: volatility and stochasticity. Volatility is about how quickly conditions change, impacting results. Stochasticity, on the other hand, refers to outcomes affected by random chance or "luck". Understanding these factors enables individuals to have more effective environmental analysis and strategy implementation (explore or exploit) for future decisions. This study investigates how anxiety and apathy, two prevalent affective states, influence the perceptions of uncertainty and exploratory behavior. Participants (N = 1001) completed a restless three-armed bandit task that was analyzed using latent state models. Anxious individuals perceived uncertainty as more volatile, leading to increased exploration and learning rates, especially after reward omission. Conversely, apathetic individuals viewed uncertainty as more stochastic, resulting in decreased exploration and learning rates. The perceived volatility-to-stochasticity ratio mediated the anxiety-exploration relationship post-adverse outcomes. Dimensionality reduction showed exploration and uncertainty estimation to be distinct but related latent factors shaping a manifold of adaptive behavior that is modulated by anxiety and apathy. These findings reveal distinct computational mechanisms for how anxiety and apathy influence decision-making, providing a framework for understanding cognitive and affective processes in neuropsychiatric disorders.

Two routes to value-based decisions in Parkinson's disease: differentiating incremental reinforcement learning from episodic memory

Patients with Parkinson's disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson's. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson's disease. Here we addressed these questions by asking patients with Parkinson's disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson's patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson's patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson's disease.

Foraging in a non-foraging task: Fitness maximization explains human risk preference dynamics under changing environment

Changes in risk preference have been reported when making a series of independent risky choices or non-foraging economic decisions. Behavioral economics has put forward various explanations for specific changes in risk preference in non-foraging tasks, but a consensus regarding the general principle underlying these effects has not been reached. In contrast, recent studies have investigated human economic risky choices using tasks adapted from foraging theory, which require consideration of past choices and future opportunities to make optimal decisions. In these foraging tasks, human economic risky choices are explained by the ethological principle of fitness maximization, which naturally leads to dynamic risk preference. Here, we conducted two online experiments to investigate whether the principle of fitness maximization can explain risk preference dynamics in a non-foraging task. Participants were asked to make a series of independent risky economic decisions while the environmental richness changed. We found that participants' risk preferences were influenced by the current and past environments, making them more risk-averse during and after the rich environment compared to the poor environment. These changes in risk preference align with fitness maximization. Our findings suggest that the ethological principle of fitness maximization might serve as a generalizable principle for explaining dynamic preferences, including risk preference, in human economic decision-making.

Bayesian reinforcement learning: A basic overview

We and other animals learn because there is some aspect of the world about which we are uncertain. This uncertainty arises from initial ignorance, and from changes in the world that we do not perfectly know; the uncertainty often becomes evident when our predictions about the world are found to be erroneous. The Rescorla-Wagner learning rule, which specifies one way that prediction errors can occasion learning, has been hugely influential as a characterization of Pavlovian conditioning and, through its equivalence to the delta rule in engineering, in a much wider class of learning problems. Here, we review the embedding of the Rescorla-Wagner rule in a Bayesian context that is precise about the link between uncertainty and learning, and thereby discuss extensions to such suggestions as the Kalman filter, structure learning, and beyond, that collectively encompass a wider range of uncertainties and accommodate a wider assortment of phenomena in conditioning.

Specifying the timescale of early life unpredictability helps explain the development of internalising and externalising behaviours

Early life unpredictability is associated with both physical and mental health outcomes throughout the life course. Here, we classified adverse experiences based on the timescale on which they are likely to introduce variability in children's environments: variations unfolding over short time scales (e.g., hours, days, weeks) and labelled Stochasticity vs variations unfolding over longer time scales (e.g., months, years) and labelled Volatility and explored how they contribute to the development of problem behaviours. Results indicate that externalising behaviours at age 9 and 15 and internalising behaviours at age 15 were better accounted for by models that separated Stochasticity and Volatility measured at ages 3 to 5. Both externalising and internalising behaviours were specifically associated with Volatility, with larger effects for externalising behaviours. These findings are interpreted in light of evolutionary-developmental models of psychopathology and reinforcement learning models of learning under uncertainty.

The Computational and Neural Bases of Context-Dependent Learning

Flexible behavior requires the creation, updating, and expression of memories to depend on context. While the neural underpinnings of each of these processes have been intensively studied, recent advances in computational modeling revealed a key challenge in context-dependent learning that had been largely ignored previously: Under naturalistic conditions, context is typically uncertain, necessitating contextual inference. We review a theoretical approach to formalizing context-dependent learning in the face of contextual uncertainty and the core computations it requires. We show how this approach begins to organize a large body of disparate experimental observations, from multiple levels of brain organization (including circuits, systems, and behavior) and multiple brain regions (most prominently the prefrontal cortex, the hippocampus, and motor cortices), into a coherent framework. We argue that contextual inference may also be key to understanding continual learning in the brain. This theory-driven perspective places contextual inference as a core component of learning.

Hierarchical inference as a source of human biases

The finding that human decision-making is systematically biased continues to have an immense impact on both research and policymaking. Prevailing views ascribe biases to limited computational resources, which require humans to resort to less costly resource-rational heuristics. Here, we propose that many biases in fact arise due to a computationally costly way of coping with uncertainty-namely, hierarchical inference-which by nature incorporates information that can seem irrelevant. We show how, in uncertain situations, Bayesian inference may avail of the environment's hierarchical structure to reduce uncertainty at the cost of introducing bias. We illustrate how this account can explain a range of familiar biases, focusing in detail on the halo effect and on the neglect of base rates. In each case, we show how a hierarchical-inference account takes the characterization of a bias beyond phenomenological description by revealing the computations and assumptions it might reflect. Furthermore, we highlight new predictions entailed by our account concerning factors that could mitigate or exacerbate bias, some of which have already garnered empirical support. We conclude that a hierarchical inference account may inform scientists and policy makers with a richer understanding of the adaptive and maladaptive aspects of human decision-making.

Adaptive tuning of human learning and choice variability to unexpected uncertainty

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Acute stress alters probabilistic reversal learning in healthy male adults

Behavioural adaptation is a fundamental cognitive ability, ensuring survival by allowing for flexible adjustment to changing environments. In laboratory settings, behavioural adaptation can be measured with reversal learning paradigms requiring agents to adjust reward learning to stimulus-action-outcome contingency changes. Stress is found to alter flexibility of reward learning, but effect directionality is mixed across studies. Here, we used model-based functional MRI (fMRI) in a within-subjects design to investigate the effect of acute psychosocial stress on flexible behavioural adaptation. Healthy male volunteers (n = 28) did a reversal learning task during fMRI in two sessions, once after the Trier Social Stress Test (TSST), a validated psychosocial stress induction method, and once after a control condition. Stress effects on choice behaviour were investigated using multilevel generalized linear models and computational models describing different learning processes that potentially generated the data. Computational models were fitted using a hierarchical Bayesian approach, and model-derived reward prediction errors (RPE) were used as fMRI regressors. We found that acute psychosocial stress slightly increased correct response rates. Model comparison revealed that double-update learning with altered choice temperature under stress best explained the observed behaviour. In the brain, model-derived RPEs were correlated with BOLD signals in striatum and ventromedial prefrontal cortex (vmPFC). Striatal RPE signals for win trials were stronger during stress compared with the control condition. Our study suggests that acute psychosocial stress could enhance reversal learning and RPE brain responses in healthy male participants and provides a starting point to explore these effects further in a more diverse population.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

Contextual inference in learning and memory

Context is widely regarded as a major determinant of learning and memory across numerous domains, including classical and instrumental conditioning, episodic memory, economic decision-making, and motor learning. However, studies across these domains remain disconnected due to the lack of a unifying framework formalizing the concept of context and its role in learning. Here, we develop a unified vernacular allowing direct comparisons between different domains of contextual learning. This leads to a Bayesian model positing that context is unobserved and needs to be inferred. Contextual inference then controls the creation, expression, and updating of memories. This theoretical approach reveals two distinct components that underlie adaptation, proper and apparent learning, respectively referring to the creation and updating of memories versus time-varying adjustments in their expression. We review a number of extensions of the basic Bayesian model that allow it to account for increasingly complex forms of contextual learning.

Belief Updating in Subclinical and Clinical Delusions

Background and Hypothesis

Current frameworks propose that delusions result from aberrant belief updating due to altered prediction error (PE) signaling and misestimation of environmental volatility. We aimed to investigate whether behavioral and neural signatures of belief updating are specifically related to the presence of delusions or generally associated with manifest schizophrenia.

Methods

Our cross-sectional design includes human participants (n[female/male] = 66[25/41]), stratified into four groups: healthy participants with minimal (n = 22) or strong delusional-like ideation (n = 18), and participants with diagnosed schizophrenia with minimal (n = 13) or strong delusions (n = 13), resulting in a 2 × 2 design, which allows to test for the effects of delusion and diagnosis. Participants performed a reversal learning task with stable and volatile task contingencies during fMRI scanning. We formalized learning with a hierarchical Gaussian filter model and conducted model-based fMRI analysis regarding beliefs of outcome uncertainty and volatility, precision-weighted PEs of the outcome- and the volatility-belief.

Results

Patients with schizophrenia as compared to healthy controls showed lower accuracy and heightened choice switching, while delusional ideation did not affect these measures. Participants with delusions showed increased precision-weighted PE-related neural activation in fronto-striatal regions. People with diagnosed schizophrenia overestimated environmental volatility and showed an attenuated neural representation of volatility in the anterior insula, medial frontal and angular gyrus.

Conclusions

Delusional beliefs are associated with altered striatal PE-signals. Juxtaposing, the potentially unsettling belief that the environment is constantly changing and weaker neural encoding of this subjective volatility seems to be associated with manifest schizophrenia, but not with the presence of delusional ideation.

Pupil diameter tracked during motor adaptation in humans

Pupil diameter, under constant illumination, is known to reflect individuals' internal states, such as surprise about observation and environmental uncertainty. Despite the growing use of pupillometry in cognitive learning studies as an additional measure for examining internal states, few studies have used pupillometry in human motor learning studies. Here, we provide the first detailed characterization of pupil diameter changes in a short-term reach adaptation paradigm. We measured pupil changes in 121 human participants while they adapted to abrupt, gradual, or switching force field conditions. Sudden increases in movement error caused by the introduction/reversal of the force field resulted in strong phasic pupil dilation during movement accompanied by a transient increase in tonic premovement baseline pupil diameter in subsequent trials. In contrast, pupil responses were reduced when the force field was gradually introduced, indicating that large, unexpected errors drove the changes in pupil responses. Interestingly, however, error-induced pupil responses gradually became insensitive after experiencing multiple force field reversals. We also found an association between baseline pupil diameter and incidental knowledge of the gradually introduced perturbation. Finally, in all experiments, we found a strong co-occurrence of larger baseline pupil diameter with slower reaction and movement times after each rest break. Collectively, these results suggest that tonic baseline pupil diameter reflects one's belief about environmental uncertainty, whereas phasic pupil dilation during movement reflects surprise about a sensory outcome (i.e., movement error), and both effects are modulated by novelty. Our results provide a new approach for nonverbally assessing participants' internal states during motor learning.NEW & NOTEWORTHY Pupil diameter is known as a noninvasive window into individuals' internal states. Despite the growing use of pupillometry in cognitive learning studies, it receives little attention in motor learning studies. Here, we characterized the pupil responses in a short-term reach adaptation paradigm by measuring pupil diameter of human participants while they adapted to abrupt, gradual, or switching force field conditions. Our results demonstrate how surprise and uncertainty reflected in pupil diameter develop during motor adaptation.

Uncertainty about others' trustworthiness increases during adolescence and guides social information sampling

Adolescence is a key life phase for developing well-adjusted social behaviour. An essential component of well-adjusted social behaviour is the ability to update our beliefs about the trustworthiness of others based on gathered information. Here, we examined how adolescents (n = 157, 10-24 years) sequentially sampled information about the trustworthiness of peers and how they used this information to update their beliefs about others' trustworthiness. Our Bayesian computational modelling approach revealed an adolescence-emergent increase in uncertainty of prior beliefs about others' trustworthiness. As a consequence, early to mid-adolescents (ages 10-16) gradually relied less on their prior beliefs and more on the gathered evidence when deciding to sample more information, and when deciding to trust. We propose that these age-related differences could be adaptive to the rapidly changing social environment of early and mid-adolescents. Together, these findings contribute to the understanding of adolescent social development by revealing adolescent-emergent flexibility in prior beliefs about others that drives adolescents' information sampling and trust decisions.

Uncertainty-guided learning with scaled prediction errors in the basal ganglia

To accurately predict rewards associated with states or actions, the variability of observations has to be taken into account. In particular, when the observations are noisy, the individual rewards should have less influence on tracking of average reward, and the estimate of the mean reward should be updated to a smaller extent after each observation. However, it is not known how the magnitude of the observation noise might be tracked and used to control prediction updates in the brain reward system. Here, we introduce a new model that uses simple, tractable learning rules that track the mean and standard deviation of reward, and leverages prediction errors scaled by uncertainty as the central feedback signal. We show that the new model has an advantage over conventional reinforcement learning models in a value tracking task, and approaches a theoretic limit of performance provided by the Kalman filter. Further, we propose a possible biological implementation of the model in the basal ganglia circuit. In the proposed network, dopaminergic neurons encode reward prediction errors scaled by standard deviation of rewards. We show that such scaling may arise if the striatal neurons learn the standard deviation of rewards and modulate the activity of dopaminergic neurons. The model is consistent with experimental findings concerning dopamine prediction error scaling relative to reward magnitude, and with many features of striatal plasticity. Our results span across the levels of implementation, algorithm, and computation, and might have important implications for understanding the dopaminergic prediction error signal and its relation to adaptive and effective learning.

Dissociable roles for the striatal cholinergic system in different flexibility contexts

The production of behavioural flexibility requires the coordination and integration of information from across the brain, by the dorsal striatum. In particular, the striatal cholinergic system is thought to be important for the modulation of striatal activity. Research from animal literature has shown that chemical inactivation of the dorsal striatum leads to impairments in reversal learning. Furthermore, proton magnetic resonance spectroscopy work has shown that the striatal cholinergic system is also important for reversal learning in humans. Here, we aim to assess whether the state of the dorsal striatal cholinergic system at rest is related to serial reversal learning in humans. We provide preliminary results showing that variability in choline in the dorsal striatum is significantly related to both the number of perseverative and regressive errors that participants make, and their rate of learning from positive and negative prediction errors. These findings, in line with previous work, suggest the resting state of dorsal striatal cholinergic system has important implications for producing flexible behaviour. However, these results also suggest the system may have heterogeneous functionality across different types of tasks measuring behavioural flexibility. These findings provide a starting point for further interrogation into understanding the functional role of the striatal cholinergic system in flexibility.

•

Striatal acetylcholine is important for behavioural flexibility in rodents & primates.

•

Nascent evidence the striatal cholinergic system is important for human flexibility.

•

¹H-MRS, reversal learning and reinforcement learning used to interrogate relationship.

•

Striatal cholinergic system at rest is associated with direct and latent performance.

•

Results specific to concentrations of striatal choline, and not other metabolites.

Does Amount of Information Support Aesthetic Values?

Obtaining information from the world is important for survival. The brain, therefore, has special mechanisms to extract as much information as possible from sensory stimuli. Hence, given its importance, the amount of available information may underlie aesthetic values. Such information-based aesthetic values would be significant because they would compete with others to drive decision-making. In this article, we ask, "What is the evidence that amount of information support aesthetic values?" An important concept in the measurement of informational volume is entropy. Research on aesthetic values has thus used Shannon entropy to evaluate the contribution of quantity of information. We review here the concepts of information and aesthetic values, and research on the visual and auditory systems to probe whether the brain uses entropy or other relevant measures, specially, Fisher information, in aesthetic decisions. We conclude that information measures contribute to these decisions in two ways: first, the absolute quantity of information can modulate aesthetic preferences for certain sensory patterns. However, the preference for volume of information is highly individualized, with information-measures competing with organizing principles, such as rhythm and symmetry. In addition, people tend to be resistant to too much entropy, but not necessarily, high amounts of Fisher information. We show that this resistance may stem in part from the distribution of amount of information in natural sensory stimuli. Second, the measurement of entropic-like quantities over time reveal that they can modulate aesthetic decisions by varying degrees of surprise given temporally integrated expectations. We propose that amount of information underpins complex aesthetic values, possibly informing the brain on the allocation of resources or the situational appropriateness of some cognitive models.

Models of Dynamic Belief Updating in Psychosis-A Review Across Different Computational Approaches

To understand the dysfunctional mechanisms underlying maladaptive reasoning of psychosis, computational models of decision making have widely been applied over the past decade. Thereby, a particular focus has been on the degree to which beliefs are updated based on new evidence, expressed by the learning rate in computational models. Higher order beliefs about the stability of the environment can determine the attribution of meaningfulness to events that deviate from existing beliefs by interpreting these either as noise or as true systematic changes (volatility). Both, the inappropriate downplaying of important changes as noise (belief update too low) as well as the overly flexible adaptation to random events (belief update too high) were theoretically and empirically linked to symptoms of psychosis. Whereas models with fixed learning rates fail to adjust learning in reaction to dynamic changes, increasingly complex learning models have been adopted in samples with clinical and subclinical psychosis lately. These ranged from advanced reinforcement learning models, over fully Bayesian belief updating models to approximations of fully Bayesian models with hierarchical learning or change point detection algorithms. It remains difficult to draw comparisons across findings of learning alterations in psychosis modeled by different approaches e.g., the Hierarchical Gaussian Filter and change point detection. Therefore, this review aims to summarize and compare computational definitions and findings of dynamic belief updating without perceptual ambiguity in (sub)clinical psychosis across these different mathematical approaches. There was strong heterogeneity in tasks and samples. Overall, individuals with schizophrenia and delusion-proneness showed lower behavioral performance linked to failed differentiation between uninformative noise and environmental change. This was indicated by increased belief updating and an overestimation of volatility, which was associated with cognitive deficits. Correlational evidence for computational mechanisms and positive symptoms is still sparse and might diverge from the group finding of instable beliefs. Based on the reviewed studies, we highlight some aspects to be considered to advance the field with regard to task design, modeling approach, and inclusion of participants across the psychosis spectrum. Taken together, our review shows that computational psychiatry offers powerful tools to advance our mechanistic insights into the cognitive anatomy of psychotic experiences.

Uncertainty alters the balance between incremental learning and episodic memory

A key question in decision-making is how humans arbitrate between competing learning and memory systems to maximize reward. We address this question by probing the balance between the effects, on choice, of incremental trial-and-error learning versus episodic memories of individual events. Although a rich literature has studied incremental learning in isolation, the role of episodic memory in decision-making has only recently drawn focus, and little research disentangles their separate contributions. We hypothesized that the brain arbitrates rationally between these two systems, relying on each in circumstances to which it is most suited, as indicated by uncertainty. We tested this hypothesis by directly contrasting contributions of episodic and incremental influence to decisions, while manipulating the relative uncertainty of incremental learning using a well-established manipulation of reward volatility. Across two large, independent samples of young adults, participants traded these influences off rationally, depending more on episodic information when incremental summaries were more uncertain. These results support the proposal that the brain optimizes the balance between different forms of learning and memory according to their relative uncertainties and elucidate the circumstances under which episodic memory informs decisions.

Computational models of adaptive behavior and prefrontal cortex

The real world is uncertain, and while ever changing, it constantly presents itself in terms of new sets of behavioral options. To attain the flexibility required to tackle these challenges successfully, most mammalian brains are equipped with certain computational abilities that rely on the prefrontal cortex (PFC). By examining learning in terms of internal models associating stimuli, actions, and outcomes, we argue here that adaptive behavior relies on specific interactions between multiple systems including: (1) selective models learning stimulus-action associations through rewards; (2) predictive models learning stimulus- and/or action-outcome associations through statistical inferences anticipating behavioral outcomes; and (3) contextual models learning external cues associated with latent states of the environment. Critically, the PFC combines these internal models by forming task sets to drive behavior and, moreover, constantly evaluates the reliability of actor task sets in predicting external contingencies to switch between task sets or create new ones. We review different models of adaptive behavior to demonstrate how their components map onto this unifying framework and specific PFC regions. Finally, we discuss how our framework may help to better understand the neural computations and the cognitive architecture of PFC regions guiding adaptive behavior.

How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective

Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.

A model for learning based on the joint estimation of stochasticity and volatility

Previous research has stressed the importance of uncertainty for controlling the speed of learning, and how such control depends on the learner inferring the noise properties of the environment, especially volatility: the speed of change. However, learning rates are jointly determined by the comparison between volatility and a second factor, moment-to-moment stochasticity. Yet much previous research has focused on simplified cases corresponding to estimation of either factor alone. Here, we introduce a learning model, in which both factors are learned simultaneously from experience, and use the model to simulate human and animal data across many seemingly disparate neuroscientific and behavioral phenomena. By considering the full problem of joint estimation, we highlight a set of previously unappreciated issues, arising from the mutual interdependence of inference about volatility and stochasticity. This interdependence complicates and enriches the interpretation of previous results, such as pathological learning in individuals with anxiety and following amygdala damage.

An empirical evaluation of active inference in multi-armed bandits

A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

Electrophysiological Signatures of Hierarchical Learning

Human perception and learning is thought to rely on a hierarchical generative model that is continuously updated via precision-weighted prediction errors (pwPEs). However, the neural basis of such cognitive process and how it unfolds during decision-making remain poorly understood. To investigate this question, we combined a hierarchical Bayesian model (i.e., Hierarchical Gaussian Filter [HGF]) with electroencephalography (EEG), while participants performed a probabilistic reversal learning task in alternatingly stable and volatile environments. Behaviorally, the HGF fitted significantly better than two control, nonhierarchical, models. Neurally, low-level and high-level pwPEs were independently encoded by the P300 component. Low-level pwPEs were reflected in the theta (4-8 Hz) frequency band, but high-level pwPEs were not. Furthermore, the expressions of high-level pwPEs were stronger for participants with better HGF fit. These results indicate that the brain employs hierarchical learning and encodes both low- and high-level learning signals separately and adaptively.

A continuum hypothesis of psychotomimetic rapid antidepressants

Ketamine, classical psychedelics and sleep deprivation are associated with rapid effects on depression. Interestingly, these interventions also have common psychotomimetic actions, mirroring aspects of psychosis such as an altered sense of self, perceptual distortions and distorted thinking. This raises the question whether these interventions might be acute antidepressants through the same mechanisms that underlie some of their psychotomimetic effects. That is, perhaps some symptoms of depression can be understood as occupying the opposite end of a spectrum where elements of psychosis can be found on the other side. This review aims at reviewing the evidence underlying a proposed continuum hypothesis of psychotomimetic rapid antidepressants, suggesting that a range of psychotomimetic interventions are also acute antidepressants as well as trying to explain these common features in a hierarchical predictive coding framework, where we hypothesise that these interventions share a common mechanism by increasing the flexibility of prior expectations. Neurobiological mechanisms at play and the role of different neuromodulatory systems affected by these interventions and their role in controlling the precision of prior expectations and new sensory evidence will be reviewed. The proposed hypothesis will also be discussed in relation to other existing theories of antidepressants. We also suggest a number of novel experiments to test the hypothesis and highlight research areas that could provide further insights, in the hope to better understand the acute antidepressant properties of these interventions.

Paranoia as a deficit in non-social belief updating

Paranoia is the belief that harm is intended by others. It may arise from selective pressures to infer and avoid social threats, particularly in ambiguous or changing circumstances. We propose that uncertainty may be sufficient to elicit learning differences in paranoid individuals, without social threat. We used reversal learning behavior and computational modeling to estimate belief updating across individuals with and without mental illness, online participants, and rats chronically exposed to methamphetamine, an elicitor of paranoia in humans. Paranoia is associated with a stronger prior on volatility, accompanied by elevated sensitivity to perceived changes in the task environment. Methamphetamine exposure in rats recapitulates this impaired uncertainty-driven belief updating and rigid anticipation of a volatile environment. Our work provides evidence of fundamental, domain-general learning differences in paranoid individuals. This paradigm enables further assessment of the interplay between uncertainty and belief-updating across individuals and species.

Everyone has had fleeting concerns that others might be against them at some point in their lives. Sometimes these concerns can escalate into paranoia and become debilitating. Paranoia is a common symptom in serious mental illnesses like schizophrenia. It can cause extreme distress and is linked with an increased risk of violence towards oneself or others. Understanding what happens in the brains of people experiencing paranoia might lead to better ways to treat or manage it.

Some experts argue that paranoia is caused by errors in the way people assess social situations. An alternative idea is that paranoia stems from the way the brain forms and updates beliefs about the world. Now, Reed et al. show that both people with paranoia and rats exposed to a paranoia-inducing substance expect the world will change frequently, change their minds often, and have a harder time learning in response to changing circumstances.

In the experiments, human volunteers with and without psychiatric disorders played a game where the best choices change. Then, the participants completed a survey to assess their level of paranoia. People with higher levels of paranoia predicted more changes would occur and made less predictable choices. In a second set of experiments, rats were put in a cage with three holes where they sometimes received sugar rewards. Some of the rats received methamphetamine, a drug that causes paranoia in humans. Rats given the drug also expected the location of the sugar reward would change often. The drugged animals had harder time learning and adapting to changing circumstances.

The experiments suggest that brain processes found in both rats, which are less social than humans, and humans contribute to paranoia. This suggests paranoia may make it harder to update beliefs. This may help scientists understand what causes paranoia and develop therapies or drugs that can reduce paranoia. This information may also help scientists understand why during societal crises like wars or natural disasters humans are prone to believing conspiracies. This is particularly important now as the world grapples with climate change and a global pandemic. Reed et al. note paranoia may impede the coordination of collaborative solutions to these challenging situations.