Hierarchical prediction errors in midbrain and basal forebrain during sensory learning

Aberrant computational mechanisms of social learning and decision-making in schizophrenia and borderline personality disorder

Psychiatric disorders are ubiquitously characterized by debilitating social impairments. These difficulties are thought to emerge from aberrant social inference. In order to elucidate the underlying computational mechanisms, patients diagnosed with major depressive disorder (N = 29), schizophrenia (N = 31), and borderline personality disorder (N = 31) as well as healthy controls (N = 34) performed a probabilistic reward learning task in which participants could learn from social and non-social information. Patients with schizophrenia and borderline personality disorder performed more poorly on the task than healthy controls and patients with major depressive disorder. Broken down by domain, borderline personality disorder patients performed better in the social compared to the non-social domain. In contrast, controls and major depressive disorder patients showed the opposite pattern and schizophrenia patients showed no difference between domains. In effect, borderline personality disorder patients gave up a possible overall performance advantage by concentrating their learning in the social at the expense of the non-social domain. We used computational modeling to assess learning and decision-making parameters estimated for each participant from their behavior. This enabled additional insights into the underlying learning and decision-making mechanisms. Patients with borderline personality disorder showed slower learning from social and non-social information and an exaggerated sensitivity to changes in environmental volatility, both in the non-social and the social domain, but more so in the latter. Regarding decision-making the modeling revealed that compared to controls and major depression patients, patients with borderline personality disorder and schizophrenia showed a stronger reliance on social relative to non-social information when making choices. Depressed patients did not differ significantly from controls in this respect. Overall, our results are consistent with the notion of a general interpersonal hypersensitivity in borderline personality disorder and schizophrenia based on a shared computational mechanism characterized by an over-reliance on beliefs about others in making decisions and by an exaggerated need to make sense of others during learning specifically in borderline personality disorder.

People suffering from psychiatric disorders frequently experience difficulties in social interaction, such as an impaired ability to use social signals to build representations of others and use these to guide behavior. Compuational models of learning and decision-making enable the characterization of individual patterns in learning and decision-making mechanisms that may be disorder-specific or disorder-general. We employed this approach to investigate the behavior of healthy participants and patients diagnosed with depression, schizophrenia, and borderline personality disorder while they performed a probabilistic reward learning task which included a social component. Patients with schizophrenia and borderline personality disorder performed more poorly on the task than controls and depressed patients. In addition, patients with borderline personality disorder concentrated their learning efforts more on the social compared to the non-social information. Computational modeling additionally revealed that borderline personality disorder patients showed a reduced flexibility in the weighting of newly obtained social and non-social information when learning about their predictive value. Instead, we found exaggerated learning of the volatility of social and non-social information. Additionally, we found a pattern shared between patients with borderline personality disorder and schizophrenia who both showed an over-reliance on predictions about social information during decision-making. Our modeling therefore provides a computational account of the exaggerated need to make sense of and rely on one’s interpretation of others’ behavior, which is prominent in both disorders.

What is optimal in optimal inference?

Inferring hidden structure from noisy observations is a problem addressed by Bayesian statistical learning, which aims to identify optimal models of the process that generated the observations given assumptions that constrain the space of potential solutions. Animals and machines face similar "model-selection" problems to infer latent properties and predict future states of the world. Here we review recent attempts to explain how intelligent agents address these challenges and how their solutions relate to Bayesian principles. We focus on how constraints on available information and resources affect inference and propose a general framework that uses benefit(accuracy) and accuracy(cost) curves to assess optimality under these constraints.

Neural arbitration between social and individual learning systems

Decision making requires integrating knowledge gathered from personal experiences with advice from others. The neural underpinnings of the process of arbitrating between information sources has not been fully elucidated. In this study, we formalized arbitration as the relative precision of predictions, afforded by each learning system, using hierarchical Bayesian modeling. In a probabilistic learning task, participants predicted the outcome of a lottery using recommendations from a more informed advisor and/or self-sampled outcomes. Decision confidence, as measured by the number of points participants wagered on their predictions, varied with our definition of arbitration as a ratio of precisions. Functional neuroimaging demonstrated that arbitration signals were independent of decision confidence and involved modality-specific brain regions. Arbitrating in favor of self-gathered information activated the dorsolateral prefrontal cortex and the midbrain, whereas arbitrating in favor of social information engaged the ventromedial prefrontal cortex and the amygdala. These findings indicate that relative precision captures arbitration between social and individual learning systems at both behavioral and neural levels.

Unreliable feedback deteriorates information processing in primary visual cortex

It is well-established that increased sensory uncertainty impairs perceptual decision-making and leads to degraded neural stimulus representations. Recently, we also showed that providing unreliable feedback to choices leads to changes in perceptual decision-making similar to those of increased stimulus noise: A deterioration in objective task performance, a decrease in subjective confidence and a lower reliance on sensory information for perceptual inference. To investigate the neural basis of such feedback-based changes in perceptual decision-making, in the present study, two groups of healthy human participants (n = 15 each) performed a challenging visual orientation discrimination task while undergoing functional magnetic resonance imaging (fMRI). Critically, one group received reliable feedback regarding their task performance in an intervention phase, whereas the other group correspondingly received unreliable feedback - thereby keeping stimulus information constant. The effects of feedback reliability on performance and stimulus representation in the primary visual cortex (V1) were studied by comparing the pre- and post-intervention test phases between the groups. Compared to participants who received reliable feedback, those receiving unreliable feedback showed a decline in task performance that was paralleled by reduced distinctness of fMRI response patterns in V1. These results show that environmental uncertainty can affect perceptual inference at the earliest cortical processing stages.

A simple model for learning in volatile environments

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Thinking, believing, and hallucinating self in schizophrenia

In this Personal View, we discuss the history and concept of self-disturbance in relation to the pathophysiology and subjective experience of schizophrenia in terms of three approaches: the perceptual anomalies approach of the early Heidelberg School of Psychiatry, the ipseity model, and the predictive coding framework. Despite the importance of these approaches, there has been a notable absence of efforts to compare them and consider how they might be integrated. This Personal View compares the three approaches and offers suggestions as to how they might work together, which represents a novel position. We view self-disturbances as transformations of self that form the inseparable background against which psychotic symptoms emerge. Integrating computational psychiatric approaches with those used by phenomenologists in the first two listed approaches, we argue that delusions and hallucinations are inferences produced under extraordinary conditions and are both statistically and experientially as real for patients as other mental events. Such inferences still approximate Bayes-optimality, given the personal, neurobiological, and environmental circumstances, and might be the only ones available to minimise prediction error. The added contribution we hope to make focuses on how the dialogue between neuroscience and phenomenology might improve clinical practice. We hope this Personal View will act as a timely primer and bridging point for the different approaches of computational psychiatry and phenomenological psychopathology for interested clinicians.

Ketamine Affects Prediction Errors about Statistical Regularities: A Computational Single-Trial Analysis of the Mismatch Negativity

The auditory mismatch negativity (MMN) is significantly reduced in schizophrenia. Notably, a similar MMN reduction can be achieved with NMDA receptor (NMDAR) antagonists. Both phenomena have been interpreted as reflecting an impairment of predictive coding or, more generally, the "Bayesian brain" notion that the brain continuously updates a hierarchical model to infer the causes of its sensory inputs. Specifically, neurobiological interpretations of predictive coding view perceptual inference as an NMDAR-dependent process of minimizing hierarchical precision-weighted prediction errors (PEs), and disturbances of this putative process play a key role in hierarchical Bayesian theories of schizophrenia. Here, we provide empirical evidence for this theory, demonstrating the existence of multiple, hierarchically related PEs in a "roving MMN" paradigm. We applied a hierarchical Bayesian model to single-trial EEG data from healthy human volunteers of either sex who received the NMDAR antagonist S-ketamine in a placebo-controlled, double-blind, within-subject fashion. Using an unrestricted analysis of the entire time-sensor space, our trial-by-trial analysis indicated that low-level PEs (about stimulus transitions) are expressed early (102-207 ms poststimulus), while high-level PEs (about transition probability) are reflected by later components (152-199 and 215-277 ms) of single-trial responses. Furthermore, we find that ketamine significantly diminished the expression of high-level PE responses, implying that NMDAR antagonism disrupts the inference on abstract statistical regularities. Our findings suggest that NMDAR dysfunction impairs hierarchical Bayesian inference about the world's statistical structure. Beyond the relevance of this finding for schizophrenia, our results illustrate the potential of computational single-trial analyses for assessing potential pathophysiological mechanisms.

Brain dynamics for confidence-weighted learning

Learning in a changing, uncertain environment is a difficult problem. A popular solution is to predict future observations and then use surprising outcomes to update those predictions. However, humans also have a sense of confidence that characterizes the precision of their predictions. Bayesian models use a confidence-weighting principle to regulate learning: for a given surprise, the update is smaller when the confidence about the prediction was higher. Prior behavioral evidence indicates that human learning adheres to this confidence-weighting principle. Here, we explored the human brain dynamics sub-tending the confidence-weighting of learning using magneto-encephalography (MEG). During our volatile probability learning task, subjects’ confidence reports conformed with Bayesian inference. MEG revealed several stimulus-evoked brain responses whose amplitude reflected surprise, and some of them were further shaped by confidence: surprise amplified the stimulus-evoked response whereas confidence dampened it. Confidence about predictions also modulated several aspects of the brain state: pupil-linked arousal and beta-range (15–30 Hz) oscillations. The brain state in turn modulated specific stimulus-evoked surprise responses following the confidence-weighting principle. Our results thus indicate that there exist, in the human brain, signals reflecting surprise that are dampened by confidence in a way that is appropriate for learning according to Bayesian inference. They also suggest a mechanism for confidence-weighted learning: confidence about predictions would modulate intrinsic properties of the brain state to amplify or dampen surprise responses evoked by discrepant observations.

Learning in a changing and uncertain world is difficult. In this context, facing a discrepancy between my current belief and new observations may reflect random fluctuations (e.g. my commute train is unexpectedly late, but it happens sometimes), if so, I should ignore this discrepancy and not change erratically my belief. However, this discrepancy could also denote a profound change (e.g. the train company changed and is less reliable), in this case, I should promptly revise my current belief. Human learning is adaptive: we change how much we learn from new observations, in particular, we promote flexibility when facing profound changes. A mathematical analysis of the problem shows that we should increase flexibility when the confidence about our current belief is low, which occurs when a change is suspected. Here, I show that human learners entertain rational confidence levels during the learning of changing probabilities. This confidence modulates intrinsic properties of the brain state (oscillatory activity and neuromodulation) which in turn amplifies or reduces, depending on whether confidence is low or high, the neural responses to discrepant observations. This confidence-weighting mechanism could underpin adaptive learning.

Impaired context-sensitive adjustment of behaviour in Parkinson's disease patients tested on and off medication: An fMRI study

The brain's sensitivity to and accentuation of unpredicted over predicted sensory signals plays a fundamental role in learning. According to recent theoretical models of the predictive coding framework, dopamine is responsible for balancing the interplay between bottom-up input and top-down predictions by controlling the precision of surprise signals that guide learning. Using functional MRI, we investigated whether patients with Parkinson's disease (PD) show impaired learning from prediction errors requiring either adaptation or stabilisation of current predictions. Moreover, we were interested in whether deficits in learning over a specific time scale would be accompanied by altered surprise responses in dopamine-related brain structures. To this end, twenty-one PD patients tested on and off dopaminergic medication and twenty-one healthy controls performed a digit prediction paradigm. During the task, violations of sequence-based predictions either signalled the need to update or to stabilise the current prediction and, thus, to react to them or ignore them, respectively. To investigate contextual adaptation to prediction errors, the probability (or its inverse, surprise) of the violations fluctuated across the experiment. When the probability of prediction errors over a specific time scale increased, healthy controls but not PD patients off medication became more flexible, i.e., error rates at violations requiring a motor response decreased in controls but increased in patients. On the neural level, this learning deficit in patients was accompanied by reduced signalling in the substantia nigra and the caudate nucleus. In contrast, differences between the groups regarding the probabilistic modulation of behaviour and neural responses were much less pronounced at prediction errors requiring only stabilisation but no adaptation. Interestingly, dopaminergic medication could neither improve learning from prediction errors nor restore the physiological, neurotypical pattern. Our findings point to a pivotal role of dysfunctions of the substantia nigra and caudate nucleus in deficits in learning from flexibility-demanding prediction errors in PD. Moreover, the data witness poor effects of dopaminergic medication on learning in PD.

Rethinking post-traumatic stress disorder - A predictive processing perspective

Predictive processing has become a popular framework in neuroscience and computational psychiatry, where it has provided a new understanding of various mental disorders. Here, we apply the predictive processing account to post-traumatic stress disorder (PTSD). We argue that the experience of a traumatic event in Bayesian terms can be understood as a perceptual hypothesis that is subsequently given a very high a-priori likelihood due to its (life-) threatening significance; thus, this hypothesis is re-selected although it does not fit the actual sensory input. Based on this account, we re-conceptualise the symptom clusters of PTSD through the lens of a predictive processing model. We particularly focus on re-experiencing symptoms as the hallmark symptoms of PTSD, and discuss the occurrence of flashbacks in terms of perceptual and interoceptive inference. This account provides not only a new understanding of the clinical profile of PTSD, but also a unifying framework for the corresponding pathologies at the neurobiological level. Finally, we derive directions for future research and discuss implications for psychological and pharmacological interventions.

Modulations of Insular Projections by Prior Belief Mediate the Precision of Prediction Error during Tactile Learning

Awareness for surprising sensory events is shaped by prior belief inferred from past experience. Here, we combined hierarchical Bayesian modeling with fMRI on an associative learning task in 28 male human participants to characterize the effect of the prior belief of tactile events on connections mediating the outcome of perceptual decisions. Activity in anterior insular cortex (AIC), premotor cortex (PMd), and inferior parietal lobule (IPL) were modulated by prior belief on unexpected targets compared with expected targets. On expected targets, prior belief decreased the connection strength from AIC to IPL, whereas it increased the connection strength from AIC to PMd when targets were unexpected. Individual differences in the modulatory strength of prior belief on insular projections correlated with the precision that increases the influence of prediction errors on belief updating. These results suggest complementary effects of prior belief on insular-frontoparietal projections mediating the precision of prediction during probabilistic tactile learning.SIGNIFICANCE STATEMENT In a probabilistic environment, the prior belief of sensory events can be inferred from past experiences. How this prior belief modulates effective brain connectivity for updating expectations for future decision-making remains unexplored. Combining hierarchical Bayesian modeling with fMRI, we show that during tactile associative learning, prior expectations modulate connections originating in the anterior insula cortex and targeting salience-related and attention-related frontoparietal areas (i.e., parietal and premotor cortex). These connections seem to be involved in updating evidence based on the precision of ascending inputs to guide future decision-making.

Confidence in Decision-Making during Probabilistic Tactile Learning Related to Distinct Thalamo–Prefrontal Pathways

The flexibility in adjusting the decision strategy from trial to trial is a prerequisite for learning in a probabilistic environment. Corresponding neural underpinnings remain largely unexplored. In the present study, 28 male humans were engaged in an associative learning task, in which they had to learn the changing probabilistic strengths of tactile sample stimuli. Combining functional magnetic resonance imaging with computational modeling, we show that an unchanged decision strategy over successively presented trials related to weakened functional connectivity between ventralmedial prefrontal cortex (vmPFC) and left secondary somatosensory cortex. The weaker the connection strength, the faster participants indicated their choice. If the decision strategy remained unchanged, participant’s decision confidence (i.e., prior belief) was related to functional connectivity between vmPFC and right pulvinar. While adjusting the decision strategy, we instead found confidence-related connections between left orbitofrontal cortex and left thalamic mediodorsal nucleus. The stronger the participant’s prior belief, the weaker the connection strengths. Together, these findings suggest that distinct thalamo–prefrontal pathways encode the confidence in keeping or changing the decision strategy during probabilistic learning. Low confidence in the decision strategy demands more thalamo–prefrontal processing resources, which is in-line with the theoretical accounts of the free-energy principle.

Atypical processing of uncertainty in individuals at risk for psychosis

•

Humans at psychosis clinical high risk (CHR) over-estimate environmental volatility.

•

Low-level prediction error (PE) signals evoke increased frontal activity in CHR.

•

Volatility-related PEs are associated with reduced frontal activity in CHR.

•

Frontal cortical activation to low-level PEs reflects impaired clinical functioning.

•

Atypical PE learning signal representations may promote delusion formation in CHR.

Current theories of psychosis highlight the role of abnormal learning signals, i.e., prediction errors (PEs) and uncertainty, in the formation of delusional beliefs. We employed computational analyses of behaviour and functional magnetic resonance imaging (fMRI) to examine whether such abnormalities are evident in clinical high risk (CHR) individuals.

Non-medicated CHR individuals (n = 13) and control participants (n = 13) performed a probabilistic learning paradigm during fMRI data acquisition. We used a hierarchical Bayesian model to infer subject-specific computations from behaviour – with a focus on PEs and uncertainty (or its inverse, precision) at different levels, including environmental ‘volatility’ – and used these computational quantities for analyses of fMRI data.

Computational modelling of CHR individuals’ behaviour indicated volatility estimates converged to significantly higher levels than in controls. Model-based fMRI demonstrated increased activity in prefrontal and insular regions of CHR individuals in response to precision-weighted low-level outcome PEs, while activations of prefrontal, orbitofrontal and anterior insula cortex by higher-level PEs (that serve to update volatility estimates) were reduced. Additionally, prefrontal cortical activity in response to outcome PEs in CHR was negatively associated with clinical measures of global functioning.

Our results suggest a multi-faceted learning abnormality in CHR individuals under conditions of environmental uncertainty, comprising higher levels of volatility estimates combined with reduced cortical activation, and abnormally high activations in prefrontal and insular areas by precision-weighted outcome PEs. This atypical representation of high- and low-level learning signals might reflect a predisposition to delusion formation.

An integrative framework for perceptual disturbances in psychosis

Perceptual disturbances in psychosis, such as auditory verbal hallucinations, are associated with increased baseline activity in the associative auditory cortex and increased dopamine transmission in the associative striatum. Perceptual disturbances are also associated with perceptual biases that suggest increased reliance on prior expectations. We review theoretical models of perceptual inference and key supporting physiological evidence, as well as the anatomy of associative cortico-striatal loops that may be relevant to auditory perceptual inference. Integrating recent findings, we outline a working framework that bridges neurobiology and the phenomenology of perceptual disturbances via theoretical models of perceptual inference.

Resting-state Functional Connectivity of the Right Temporoparietal Junction Relates to Belief Updating and Reorienting during Spatial Attention

Although multiple studies characterized the resting-state functional connectivity (rsFC) of the right temporoparietal junction (rTPJ), little is known about the link between rTPJ rsFC and cognitive functions. Given a putative involvement of rTPJ in both reorienting of attention and the updating of probabilistic beliefs, this study characterized the relationship between rsFC of rTPJ with dorsal and ventral attention systems and these two cognitive processes. Twenty-three healthy young participants performed a modified location-cueing paradigm with true and false prior information about the percentage of cue validity to assess belief updating and attentional reorienting. Resting-state fMRI was recorded before and after the task. Seed-based correlation analysis was employed, and correlations of each behavioral parameter with rsFC before the task, as well as with changes in rsFC after the task, were assessed in an ROI-based approach. Weaker rsFC between rTPJ and right intraparietal sulcus before the task was associated with relatively faster updating of the belief that the cue will be valid after false prior information. Moreover, relatively faster belief updating, as well as faster reorienting, were related to an increase in the interhemispheric rsFC between rTPJ and left TPJ after the task. These findings are in line with task-based connectivity studies on related attentional functions and extend results from stroke patients demonstrating the importance of interhemispheric parietal interactions for behavioral performance. The present results not only highlight the essential role of parietal rsFC for attentional functions but also suggest that cognitive processing during a task changes connectivity patterns in a performance-dependent manner.

A Computational Theory of Mindfulness Based Cognitive Therapy from the "Bayesian Brain" Perspective

Mindfulness Based Cognitive Therapy (MBCT) was developed to combine methods from cognitive behavioral therapy and meditative techniques, with the specific goal of preventing relapse in recurrent depression. While supported by empirical evidence from multiple clinical trials, the cognitive mechanisms behind the effectiveness of MBCT are not well understood in computational (information processing) or biological terms. This article introduces a testable theory about the computational mechanisms behind MBCT that is grounded in "Bayesian brain" concepts of perception from cognitive neuroscience, such as predictive coding. These concepts regard the brain as embodying a model of its environment (including the external world and the body) which predicts future sensory inputs and is updated by prediction errors, depending on how precise these error signals are. This article offers a concrete proposal how core concepts of MBCT-(i) the being mode (accepting whatever sensations arise, without judging or changing them), (ii) decentering (experiencing thoughts and percepts simply as events in the mind that arise and pass), and (iii) cognitive reactivity (changes in mood reactivate negative beliefs)-could be understood in terms of perceptual and metacognitive processes that draw on specific computational mechanisms of the "Bayesian brain." Importantly, the proposed theory can be tested experimentally, using a combination of behavioral paradigms, computational modelling, and neuroimaging. The novel theoretical perspective on MBCT described in this paper may offer opportunities for finessing the conceptual and practical aspects of MBCT.

Beyond Reward Prediction Errors: Human Striatum Updates Rule Values During Learning

Humans naturally group the world into coherent categories defined by membership rules. Rules can be learned implicitly by building stimulus-response associations using reinforcement learning or by using explicit reasoning. We tested if the striatum, in which activation reliably scales with reward prediction error, would track prediction errors in a task that required explicit rule generation. Using functional magnetic resonance imaging during a categorization task, we show that striatal responses to feedback scale with a "surprise" signal derived from a Bayesian rule-learning model and are inconsistent with RL prediction error. We also find that striatum and caudal inferior frontal sulcus (cIFS) are involved in updating the likelihood of discriminative rules. We conclude that the striatum, in cooperation with the cIFS, is involved in updating the values assigned to categorization rules when people learn using explicit reasoning.

Neural Response Patterns During Pavlovian-to-Instrumental Transfer Predict Alcohol Relapse and Young Adult Drinking

BACKGROUND

Pavlovian-to-instrumental transfer (PIT) describes the influence of conditioned stimuli on instrumental behaviors and is discussed as a key process underlying substance abuse. Here, we tested whether neural responses during alcohol-related PIT predict future relapse in alcohol-dependent patients and future drinking behavior in adolescents.

METHODS

Recently detoxified alcohol-dependent patients (n = 52) and young adults without dependence (n = 136) underwent functional magnetic resonance imaging during an alcohol-related PIT paradigm, and their drinking behavior was assessed in a 12-month follow-up. To predict future drinking behavior from PIT activation patterns, we used a multivoxel classification scheme based on linear support vector machines.

RESULTS

When training and testing the classification scheme in patients, PIT activation patterns predicted future relapse with 71.2% accuracy. Feature selection revealed that classification was exclusively based on activation patterns in medial prefrontal cortex. To probe the generalizability of this functional magnetic resonance imaging-based prediction of future drinking behavior, we applied the support vector machine classifier that had been trained on patients to PIT functional magnetic resonance imaging data from adolescents. An analysis of cross-classification predictions revealed that those young social drinkers who were classified as abstainers showed a greater reduction in alcohol consumption at 12-month follow-up than those classified as relapsers (Δ = -24.4 ± 6.0 g vs. -5.7 ± 3.6 g; p = .019).

CONCLUSIONS

These results suggest that neural responses during PIT could constitute a generalized prognostic marker for future drinking behavior in established alcohol use disorder and in at-risk states.

Dopamine neuron ensembles signal the content of sensory prediction errors

Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in the activity of individual neurons or population averages. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

Minimizing Precision-Weighted Sensory Prediction Errors via Memory Formation and Switching in Motor Adaptation

Humans predict the sensory consequences of motor commands by learning internal models of the body and of environment perturbations. When facing a sensory prediction error, should we attribute this error to a change in our body, and update the body internal model, or to a change in the environment? In the latter case, should we update an existing perturbation model or create a new model? Here, we propose that a decision-making process compares the models' prediction errors, weighted by their precisions, to select and update either the body model or an existing perturbation model. When no model can predict a perturbation, a new perturbation model is created and selected. When a model is selected, both the prediction's mean estimate and uncertainty are updated to minimize future prediction errors and to increase the precision of the predictions. Results from computer simulations, which we verified in an arm visuomotor adaptation experiment with subjects of both sexes, account for short aftereffects and large savings after adaptation to large, but not small, perturbations. Results also clarify previous data in the absence of errors (error-clamp): motor memories show an initial lack of decay after a large perturbation, but gradual decay after a small perturbation. Finally, qualitative individual differences in adaptation were explained by subjects selecting and updating either the body model or a perturbation model. Our results suggest that motor adaptation belongs to a general class of learning according to which memories are created when no existing memories can predict sensory data accurately and precisely.SIGNIFICANCE STATEMENT When movements are followed by unexpected outcomes, such as following the introduction of a visuomotor or a force field perturbation, or the sudden removal of such perturbations, it is unclear whether the CNS updates existing memories or creates new memories. Here, we propose a novel model of adaptation, and investigate, via computer simulations and behavioral experiments, how the amplitude and schedule of the perturbation, as well as the characteristics of the learner, lead to the selection and update of existing memories or the creation of new memories. Our results provide insights into a number of puzzling and contradictory motor adaptation data, as well as into qualitative individual differences in adaptation.

Towards a Unifying Cognitive, Neurophysiological, and Computational Neuroscience Account of Schizophrenia

Psychotic experiences may be understood as altered information processing due to aberrant neural computations. A prominent example of such neural computations is the computation of prediction errors (PEs), which signal the difference between expected and experienced events. Among other areas showing PE coding, hippocampal-prefrontal-striatal neurocircuits play a prominent role in information processing. Dysregulation of dopaminergic signaling, often secondary to psychosocial stress, is thought to interfere with the processing of biologically important events (such as reward prediction errors) and result in the aberrant attribution of salience to irrelevant sensory stimuli and internal representations. Bayesian hierarchical predictive coding offers a promising framework for the identification of dysfunctional neurocomputational processes and the development of a mechanistic understanding of psychotic experience. According to this framework, mismatches between prior beliefs encoded at higher levels of the cortical hierarchy and lower-level (sensory) information can also be thought of as PEs, with important consequences for belief updating. Low levels of precision in the representation of prior beliefs relative to sensory data, as well as dysfunctional interactions between prior beliefs and sensory data in an ever-changing environment, have been suggested as a general mechanism underlying psychotic experiences. Translating the promise of the Bayesian hierarchical predictive coding into patient benefit will come from integrating this framework with existing knowledge of the etiology and pathophysiology of psychosis, especially regarding hippocampal-prefrontal-striatal network function and neural mechanisms of information processing and belief updating.

Statistical context dictates the relationship between feedback-related EEG signals and learning

Learning should be adjusted according to the surprise associated with observed outcomes but calibrated according to statistical context. For example, when occasional changepoints are expected, surprising outcomes should be weighted heavily to speed learning. In contrast, when uninformative outliers are expected to occur occasionally, surprising outcomes should be less influential. Here we dissociate surprising outcomes from the degree to which they demand learning using a predictive inference task and computational modeling. We show that the P300, a stimulus-locked electrophysiological response previously associated with adjustments in learning behavior, does so conditionally on the source of surprise. Larger P300 signals predicted greater learning in a changing context, but less learning in a context where surprise was indicative of a one-off outlier (oddball). Our results suggest that the P300 provides a surprise signal that is interpreted by downstream learning processes differentially according to statistical context in order to appropriately calibrate learning across complex environments.

Tracking the Time Course of Bayesian Inference With Event-Related Potentials:A Study Using the Central Cue Posner Paradigm

In this study, we asked whether the event-related potentials associated to cue and target stimuli of a Central Cue Posner Paradigm (CCPP) may encode key parameters of Bayesian inference – prior expectation and surprise – on a trial-by-trial basis. Thirty-two EEG channel were recorded in a sample of 19 young adult subjects while performing a CCPP, in which a cue indicated (validly or invalidly) the position of an incoming auditory target. Three different types of blocks with validities of 50%, 64%, and 88%, respectively, were presented. Estimates of prior expectation and surprise were obtained on a trial-by-trial basis from participants’ responses, using a computational model implementing Bayesian learning. These two values were correlated on a trial-by-trial basis with the EEG values in all the electrodes and time bins. Therefore, a Spearman correlation metrics of the relationship between Bayesian parameters and the EEG was obtained. We report that the surprise parameter was able to classify the different validity blocks. Furthermore, the prior expectation parameter showed a significant correlation with the EEG in the cue-target period, in which the Contingent Negative Variation develops. Finally, in the post-target period the surprise parameter showed a significant correlation in the latencies and electrodes in which different event-related potentials are induced. Our results suggest that Bayesian parameters are coded in the EEG signals; and namely, the CNV would be related to prior expectation, while the post-target components P2a, P2, P3a, P3b, and SW would be related to surprise. This study thus provides novel support to the idea that human electrophysiological neural activity may implement a (Bayesian) predictive processing scheme.

Social noise interferes with learning in a volatile environment

To learn through feedback, feedback should be reliable. However, if feedback is blurred by irrelevant social information, learning in a volatile environment, which requires fast learning and adaptation, might be disturbed. In this study, we investigated how feedback with social noise interferes with learning in a volatile environment by designing a probabilistic associative learning task in which the association probability changes dynamically, and the outcome was randomly blurred by an emotional face with incongruent valence. Learning in this situation was modelled by HGF-S such that emotionally incongruent feedback induces perceptual uncertainty called social noise. The Bayesian model comparison showed that the HGF-S model explains the subjects' behaviour well, and the simulation showed that social noise interrupts both learning the association probability and the volatility. Furthermore, the learning interruption influenced the subsequent decision. Finally, we found that the individual difference in how the same emotionally incongruent feedback induces social noise in varying degrees was related to the differences in event-related desynchronization induced by happy and sad faces in the right anterior insula, which encodes the degree of emotional feeling. These results advance our understanding of how feedback with emotional interference affects learning.

Feature-specific prediction errors for visual mismatch

Predictive coding (PC) theory posits that our brain employs a predictive model of the environment to infer the causes of its sensory inputs. A fundamental but untested prediction of this theory is that the same stimulus should elicit distinct precision weighted prediction errors (pwPEs) when different (feature-specific) predictions are violated, even in the absence of attention. Here, we tested this hypothesis using functional magnetic resonance imaging (fMRI) and a multi-feature roving visual mismatch paradigm where rare changes in either color (red, green), or emotional expression (happy, fearful) of faces elicited pwPE responses in human participants. Using a computational model of learning and inference, we simulated pwPE and prediction trajectories of a Bayes-optimal observer and used these to analyze changes in blood oxygen level dependent (BOLD) responses to changes in color and emotional expression of faces while participants engaged in a distractor task. Controlling for visual attention by eye-tracking, we found pwPE responses to unexpected color changes in the fusiform gyrus. Conversely, unexpected changes of facial emotions elicited pwPE responses in cortico-thalamo-cerebellar structures associated with emotion and theory of mind processing. Predictions pertaining to emotions activated fusiform, occipital and temporal areas. Our results are consistent with a general role of PC across perception, from low-level to complex and socially relevant object features, and suggest that monitoring of the social environment occurs continuously and automatically, even in the absence of attention.

Contextual influence on confidence judgments in human reinforcement learning

The ability to correctly estimate the probability of one’s choices being correct is fundamental to optimally re-evaluate previous choices or to arbitrate between different decision strategies. Experimental evidence nonetheless suggests that this metacognitive process—confidence judgment- is susceptible to numerous biases. Here, we investigate the effect of outcome valence (gains or losses) on confidence while participants learned stimulus-outcome associations by trial-and-error. In two experiments, participants were more confident in their choices when learning to seek gains compared to avoiding losses, despite equal difficulty and performance between those two contexts. Computational modelling revealed that this bias is driven by the context-value, a dynamically updated estimate of the average expected-value of choice options, necessary to explain equal performance in the gain and loss domain. The biasing effect of context-value on confidence, revealed here for the first time in a reinforcement-learning context, is therefore domain-general, with likely important functional consequences. We show that one such consequence emerges in volatile environments, where the (in)flexibility of individuals’ learning strategies differs when outcomes are framed as gains or losses. Despite apparent similar behavior- profound asymmetries might therefore exist between learning to avoid losses and learning to seek gains.

In order to arbitrate between different decision strategies, as well as to inform future choices, a decision maker needs to estimate the probability of her choices being correct as precisely as possible. Surprisingly, this metacognitive operation, known as confidence judgment, has not been systematically investigated in the context of simple instrumental-learning tasks. Here, we assessed how confident individuals are in their choices when learning stimulus-outcome associations by trial-and-errors to maximize gains or to minimize losses. In two experiments, we show that individuals are more confident in their choices when learning to seek gains compared to avoiding losses, despite equal difficulty and performance between those two contexts. To simultaneously account for this pattern of choices and confidence judgments, we propose that individuals learn context-values, which approximate the average expected-value of choice options. We finally show that, in volatile environments, the biasing effect of context-value on confidence induces difference in learning flexibility when outcomes are framed as gains or losses.

Sensory prediction errors in the human midbrain signal identity violations independent of perceptual distance

The firing of dopaminergic midbrain neurons is thought to reflect prediction errors (PE) that depend on the difference between the value of expected and received rewards. However, recent work has demonstrated that unexpected changes in value-neutral outcome features, such as identity, can evoke similar responses. It remains unclear whether the magnitude of these identity PEs scales with the perceptual dissimilarity of expected and received rewards, or whether they are independent of perceptual similarity. We used a Pavlovian transreinforcer reversal task to elicit identity PEs for value-matched food odor rewards, drawn from two perceptual categories (sweet, savory). Replicating previous findings, identity PEs were correlated with fMRI activity in midbrain, OFC, piriform cortex, and amygdala. However, the magnitude of identity PE responses was independent of the perceptual distance between expected and received outcomes, suggesting that identity comparisons underlying sensory PEs may occur in an abstract state space independent of straightforward sensory percepts.

Sustained effects of corrupted feedback on perceptual inference

Feedback is central to most forms of learning, and its reliability is therefore critical. Here, we investigated the effects of corrupted, and hence unreliable, feedback on perceptual inference. Within the framework of Bayesian inference, we hypothesised that corrupting feedback in a demanding perceptual task would compromise sensory information processing and bias inference towards prior information if available. These hypotheses were examined by a simulation and in two behavioural experiments with visual detection (experiment 1) and discrimination (experiment 2) tasks. Both experiments consisted of two sessions comprising intervention runs with either corrupted or uncorrupted (correct) feedback, and pre- and post-intervention tests to assess the effects of feedback. In the tests alone, additional prior beliefs were induced through predictive auditory cues to assess sustained effects of feedback on the balance between sensory evidence and prior beliefs. Both experiments and the simulation showed the hypothesised decrease in performance and increased reliance on prior beliefs after corrupted but not uncorrupted feedback. Exploratory analyses indicated reduced confidence regarding perceptual decisions during delivery of corrupted feedback. Our results suggest that corrupted feedback on perceptual decisions leads to sustained changes in perceptual inference, characterised by a shift from sensory likelihood to prior beliefs when those are accessible.

Learning and forgetting using reinforced Bayesian change detection

Agents living in volatile environments must be able to detect changes in contingencies while refraining to adapt to unexpected events that are caused by noise. In Reinforcement Learning (RL) frameworks, this requires learning rates that adapt to past reliability of the model. The observation that behavioural flexibility in animals tends to decrease following prolonged training in stable environment provides experimental evidence for such adaptive learning rates. However, in classical RL models, learning rate is either fixed or scheduled and can thus not adapt dynamically to environmental changes. Here, we propose a new Bayesian learning model, using variational inference, that achieves adaptive change detection by the use of Stabilized Forgetting, updating its current belief based on a mixture of fixed, initial priors and previous posterior beliefs. The weight given to these two sources is optimized alongside the other parameters, allowing the model to adapt dynamically to changes in environmental volatility and to unexpected observations. This approach is used to implement the "critic" of an actor-critic RL model, while the actor samples the resulting value distributions to choose which action to undertake. We show that our model can emulate different adaptation strategies to contingency changes, depending on its prior assumptions of environmental stability, and that model parameters can be fit to real data with high accuracy. The model also exhibits trade-offs between flexibility and computational costs that mirror those observed in real data. Overall, the proposed method provides a general framework to study learning flexibility and decision making in RL contexts.

Annual Research Review: Developmental computational psychiatry

Most psychiatric disorders emerge during childhood and adolescence. This is also a period that coincides with the brain undergoing substantial growth and reorganisation. However, it remains unclear how a heightened vulnerability to psychiatric disorder relates to this brain maturation. Here, we propose 'developmental computational psychiatry' as a framework for linking brain maturation to cognitive development. We argue that through modelling some of the brain's fundamental cognitive computations, and relating them to brain development, we can bridge the gap between brain and cognitive development. This in turn can lead to a richer understanding of the ontogeny of psychiatric disorders. We illustrate this perspective with examples from reinforcement learning and dopamine function. Specifically, we show how computational modelling deepens an understanding of how cognitive processes, such as reward learning, effort learning, and social learning might go awry in psychiatric disorders. Finally, we sketch the promises and limitations of a developmental computational psychiatry.

Confidence resets reveal hierarchical adaptive learning in humans

Hierarchical processing is pervasive in the brain, but its computational significance for learning under uncertainty is disputed. On the one hand, hierarchical models provide an optimal framework and are becoming increasingly popular to study cognition. On the other hand, non-hierarchical (flat) models remain influential and can learn efficiently, even in uncertain and changing environments. Here, we show that previously proposed hallmarks of hierarchical learning, which relied on reports of learned quantities or choices in simple experiments, are insufficient to categorically distinguish hierarchical from flat models. Instead, we present a novel test which leverages a more complex task, whose hierarchical structure allows generalization between different statistics tracked in parallel. We use reports of confidence to quantitatively and qualitatively arbitrate between the two accounts of learning. Our results support the hierarchical learning framework, and demonstrate how confidence can be a useful metric in learning theory.

Neural correlates of instrumental responding in the context of alcohol-related cues index disorder severity and relapse risk

The influence of Pavlovian conditioned stimuli on ongoing behavior may contribute to explaining how alcohol cues stimulate drug seeking and intake. Using a Pavlovian-instrumental transfer task, we investigated the effects of alcohol-related cues on approach behavior (i.e., instrumental response behavior) and its neural correlates, and related both to the relapse after detoxification in alcohol-dependent patients. Thirty-one recently detoxified alcohol-dependent patients and 24 healthy controls underwent instrumental training, where approach or non-approach towards initially neutral stimuli was reinforced by monetary incentives. Approach behavior was tested during extinction with either alcohol-related or neutral stimuli (as Pavlovian cues) presented in the background during functional magnetic resonance imaging (fMRI). Patients were subsequently followed up for 6 months. We observed that alcohol-related background stimuli inhibited the approach behavior in detoxified alcohol-dependent patients (t = - 3.86, p < .001), but not in healthy controls (t = - 0.92, p = .36). This behavioral inhibition was associated with neural activation in the nucleus accumbens (NAcc) (t₍₃₀₎ = 2.06, p < .05). Interestingly, both the effects were only present in subsequent abstainers, but not relapsers and in those with mild but not severe dependence. Our data show that alcohol-related cues can acquire inhibitory behavioral features typical of aversive stimuli despite being accompanied by a stronger NAcc activation, suggesting salience attribution. The fact that these findings are restricted to abstinence and milder illness suggests that they may be potential resilience factors.Clinical trial: LeAD study, http://www.lead-studie.de , NCT01679145.

Age-related changes in Bayesian belief updating during attentional deployment and motor intention

Predicting upcoming events using past observations is a crucial component of an efficient allocation of attentional resources. Therefore, the deployment of attention is sensitive to different types of cues predicting upcoming events. Here we investigated probabilistic inference abilities in spatial and feature-based attentional, as well as in motor-intentional subsystems, focusing specifically on the age-related changes in these abilities. In two behavioral experiments, younger and older adults (20 younger and 20 older adults for each experiment) performed three versions of a cueing paradigm, where spatial, feature, or motor cues predicted the location, color, or motor response of a target stimulus. The percentage of cue validity (i.e., the probability of the cue being valid) changed over time, thereby creating a volatile environment. A Bayesian hierarchical model was used to estimate trial-wise beliefs concerning the cue validity from reaction times and to derive a subject-specific belief updating parameter ω in each task version. We also manipulated task difficulty: participants performed an easier version of the task in Experiment 1 and a more difficult version in Experiment 2. Results from Experiment 1 suggested a preserved ability of older adults to use the three different cues to generate predictions. However, the increased task demands of Experiment 2 uncovered a difference in belief updating between the two age groups, indicating moderate evidence for a reduction of the ability to update predictions with motor intention cues in older adults. These results point at a distinction of attentional and motor-intentional subsystems, with age-related differences tackling especially the motor-intentional subsystem.

Modality-specific effects of aversive expectancy in the anterior insula and medial prefrontal cortex

Expectations modulate the subjective experience of pain by increasing sensitivity to nociceptive inputs, an effect mediated by brain regions such as the insula. However, it is still unknown whether the neural structures underlying pain expectancy hold sensory-specific information or, alternatively, code for modality-independent features (eg, unpleasantness), potentially common with other negative experiences. We used functional magnetic resonance imaging to investigate neural activity underlying the expectation of different, but comparably unpleasant, pain and disgust. We presented participants with visual cues predicting either a painful heat or disgusting odor, and assessed how they affected the subsequent subjective experience of stimuli from the same (within-modality) or opposite (cross-modal) modality. We found a reliable influence of expectancy on the subjective experience of stimuli whose modality matched that of the previous cue. At the brain level, this effect was mediated by the intermediate dysgranular section of the insula, whereas it was suppressed by more anterior agranular portions of the same region. Instead, no expectancy modulation was observed when the modality of the cue differed from that of the subsequent stimulus. Our data suggest that the insular cortex encodes prospective aversive events in terms of their modality-specific features, and whether they match with subsequent stimulations.

Disrupted reinforcement learning during post-error slowing in ADHD

ADHD is associated with altered dopamine regulated reinforcement learning on prediction errors. Despite evidence of categorically altered error processing in ADHD, neuroimaging advances have largely investigated models of normal reinforcement learning in greater detail. Further, although reinforcement leaning critically relies on ventral striatum exerting error magnitude related thresholding influences on substantia nigra (SN) and dorsal striatum, these thresholding influences have never been identified with neuroimaging. To identify such thresholding influences, we propose that error magnitude related activities must first be separated from opposite activities in overlapping neural regions during error detection. Here we separate error detection from magnitude related adjustment (post-error slowing) during inhibition errors in the stop signal task in typically developing (TD) and ADHD adolescents using fMRI. In TD, we predicted that: 1) deactivation of dorsal striatum on error detection interrupts ongoing processing, and should be proportional to right frontoparietal response phase activity that has been observed in the SST; 2) deactivation of ventral striatum on post-error slowing exerts thresholding influences on, and should be proportional to activity in dorsal striatum. In ADHD, we predicted that ventral striatum would instead correlate with heightened amygdala responses to errors. We found deactivation of dorsal striatum on error detection correlated with response-phase activity in both groups. In TD, post-error slowing deactivation of ventral striatum correlated with activation of dorsal striatum. In ADHD, ventral striatum correlated with heightened amygdala activity. Further, heightened activities in locus coeruleus (norepinephrine), raphe nucleus (serotonin) and medial septal nuclei (acetylcholine), which all compete for control of DA, and are altered in ADHD, exhibited altered correlations with SN. All correlations in TD were replicated in healthy adults. Results in TD are consistent with dopamine regulated reinforcement learning on post-error slowing. In ADHD, results are consistent with heightened activities in the amygdala and non-dopaminergic neurotransmitter nuclei preventing reinforcement learning.

Computational mechanisms of curiosity and goal-directed exploration

Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. 'Hidden state' exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, 'model parameter' exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of 'Bayes-optimal' behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making.

Predicting change: Approximate inference under explicit representation of temporal structure in changing environments

In our daily lives timing of our actions plays an essential role when we navigate the complex everyday environment. It is an open question though how the representations of the temporal structure of the world influence our behavior. Here we propose a probabilistic model with an explicit representation of state durations which may provide novel insights in how the brain predicts upcoming changes. We illustrate several properties of the behavioral model using a standard reversal learning design and compare its task performance to standard reinforcement learning models. Furthermore, using experimental data, we demonstrate how the model can be applied to identify participants' beliefs about the latent temporal task structure. We found that roughly one quarter of participants seem to have learned the latent temporal structure and used it to anticipate changes, whereas the remaining participants' behavior did not show signs of anticipatory responses, suggesting a lack of precise temporal expectations. We expect that the introduced behavioral model will allow, in future studies, for a systematic investigation of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.

The Associations of Emotion Coping Appraisal With Both the Cue-Outcome Contingency and Perceived Verbal Abuse Exposure

Previous studies have reported an association between verbal abuse in early childhood and structural and functional alterations in the young adult brain, supporting the existence of critical periods in human brain development. In addition, exposure to verbal abuse in early childhood is strongly correlated with lifetime psychiatric illness. Resilience is defined as the ability to avoid the negative psychological, biological, and social consequences of stress that impair psychological and physical homeostasis and is used to cope with these psychiatric diseases. We attempted to explain the mediatable present function of resilience and its associations with several psychiatric disorders, with verbal abuse exposure in early childhood and with the present value of the readily measurable and conceptually connected generative Bayesian model parameter. Thirty-six subjects performed a cross-modal associative learning task requiring them to learn the predictive strength of auditory cues and predict a subsequent visual stimulus. The probability of the association changed across each trial block. Subjects' responses were modeled as a hierarchical Bayesian belief-updating process using a hierarchical Gaussian filter (HGF) with three levels, a Sutton K1 model, and a Rescorla-Wagner model. Subjects completed the Korean version of the Verbal Abuse Questionnaire (VAQ) for segmented periods (aged 0 to 6, 7 to 12, and 13 to 18 years), and their positive self-appraisal was estimated using the Resilience Appraisal Scale (RAS). Random-effects Bayesian model selection identified HGF as the best model. Of the VAQ values for specific periods, only preschool VAQ scores were negatively correlated with RAS scores. The tonic volatility parameter, ω₂, of HGF showed a negative relationship with RAS emotion coping scores. The linear regression model explained 18.3% of the variance of emotion coping appraisal with ω₂ and preschool VAQ scores. Based on the results obtained from young adults, decrease in emotion coping appraisal can be explained by an increase in the number of experiences of verbal abuse in early childhood and the increased tendency to update beliefs about the cue-outcome associative probability in a volatile environment.

Emotionally Aversive Cues Suppress Neural Systems Underlying Optimal Learning in Socially Anxious Individuals

Learning and decision-making are modulated by socio-emotional processing and such modulation is implicated in clinically relevant personality traits of social anxiety. The present study elucidates the computational and neural mechanisms by which emotionally aversive cues disrupt learning in socially anxious human individuals. Healthy volunteers with low or high trait social anxiety performed a reversal learning task requiring learning actions in response to angry or happy face cues. Choice data were best captured by a computational model in which learning rate was adjusted according to the history of surprises. High trait socially anxious individuals used a less-dynamic strategy for adjusting their learning rate in trials started with angry face cues and unlike the low social anxiety group, their dorsal anterior cingulate cortex (dACC) activity did not covary with the learning rate. Our results demonstrate that trait social anxiety is accompanied by disruption of optimal learning and dACC activity in threatening situations.SIGNIFICANCE STATEMENT Social anxiety is known to influence a broad range of cognitive functions. This study tests whether and how social anxiety affects human value-based learning as a function of uncertainty in the learning environment. The findings indicate that, in a threatening context evoked by an angry face, socially anxious individuals fail to benefit from a stable learning environment with highly predictable stimulus-response-outcome associations. Under those circumstances, socially anxious individuals failed to use their dorsal anterior cingulate cortex, a region known to adjust learning rate to environmental uncertainty. These findings open the way to modify neurobiological mechanisms of maladaptive learning in anxiety and depressive disorders.

Selective maintenance of value information helps resolve the exploration/exploitation dilemma

In natural environments with many options of uncertain value, one faces a difficult tradeoff between exploiting familiar, valuable options or searching for better alternatives. Reinforcement learning models of this exploration/exploitation dilemma typically modulate the rate of exploratory choices or preferentially sample uncertain options. The extent to which such models capture human behavior remains unclear, in part because they do not consider the constraints on remembering what is learned. Using reinforcement-based timing as a motivating example, we show that selectively maintaining high-value actions compresses the amount of information to be tracked in learning, as quantified by Shannon's entropy. In turn, the information content of the value representation controls the balance between exploration (high entropy) and exploitation (low entropy). Selectively maintaining preferred action values while allowing others to decay renders the choices increasingly exploitative across learning episodes. To adjudicate among alternative maintenance and sampling strategies, we developed a new reinforcement learning model, StrategiC ExPloration/ExPloitation of Temporal Instrumental Contingencies (SCEPTIC). In computational studies, a resource-rational selective maintenance approach was as successful as more resource-intensive strategies. Furthermore, human behavior was consistent with selective maintenance; information compression was most pronounced in subjects with superior performance and non-verbal intelligence, and in learnable vs. unlearnable contingencies. Cognitively demanding uncertainty-directed exploration recovered a more accurate representation in simulations with no foraging advantage and was strongly unsupported in our human study.

The Predictive Coding Account of Psychosis

Fueled by developments in computational neuroscience, there has been increasing interest in the underlying neurocomputational mechanisms of psychosis. One successful approach involves predictive coding and Bayesian inference. Here, inferences regarding the current state of the world are made by combining prior beliefs with incoming sensory signals. Mismatches between prior beliefs and incoming signals constitute prediction errors that drive new learning. Psychosis has been suggested to result from a decreased precision in the encoding of prior beliefs relative to the sensory data, thereby garnering maladaptive inferences. Here, we review the current evidence for aberrant predictive coding and discuss challenges for this canonical predictive coding account of psychosis. For example, hallucinations and delusions may relate to distinct alterations in predictive coding, despite their common co-occurrence. More broadly, some studies implicate weakened prior beliefs in psychosis, and others find stronger priors. These challenges might be answered with a more nuanced view of predictive coding. Different priors may be specified for different sensory modalities and their integration, and deficits in each modality need not be uniform. Furthermore, hierarchical organization may be critical. Altered processes at lower levels of a hierarchy need not be linearly related to processes at higher levels (and vice versa). Finally, canonical theories do not highlight active inference-the process through which the effects of our actions on our sensations are anticipated and minimized. It is possible that conflicting findings might be reconciled by considering these complexities, portending a framework for psychosis more equipped to deal with its many manifestations.

Attractor-like Dynamics in Belief Updating in Schizophrenia

Subjects with a diagnosis of schizophrenia (Scz) overweight unexpected evidence in probabilistic inference: such evidence becomes "aberrantly salient." A neurobiological explanation for this effect is that diminished synaptic gain (e.g., hypofunction of cortical NMDARs) in Scz destabilizes quasi-stable neuronal network states (or "attractors"). This attractor instability account predicts that (1) Scz would overweight unexpected evidence but underweight consistent evidence, (2) belief updating would be more vulnerable to stochastic fluctuations in neural activity, and (3) these effects would correlate. Hierarchical Bayesian belief updating models were tested in two independent datasets (n = 80 male and n = 167 female) comprising human subjects with Scz, and both clinical and nonclinical controls (some tested when unwell and on recovery) performing the "probability estimates" version of the beads task (a probabilistic inference task). Models with a standard learning rate, or including a parameter increasing updating to "disconfirmatory evidence," or a parameter encoding belief instability were formally compared. The "belief instability" model (based on the principles of attractor dynamics) had most evidence in all groups in both datasets. Two of four parameters differed between Scz and nonclinical controls in each dataset: belief instability and response stochasticity. These parameters correlated in both datasets. Furthermore, the clinical controls showed similar parameter distributions to Scz when unwell, but were no different from controls once recovered. These findings are consistent with the hypothesis that attractor network instability contributes to belief updating abnormalities in Scz, and suggest that similar changes may exist during acute illness in other psychiatric conditions.SIGNIFICANCE STATEMENT Subjects with a diagnosis of schizophrenia (Scz) make large adjustments to their beliefs following unexpected evidence, but also smaller adjustments than controls following consistent evidence. This has previously been construed as a bias toward "disconfirmatory" information, but a more mechanistic explanation may be that in Scz, neural firing patterns ("attractor states") are less stable and hence easily altered in response to both new evidence and stochastic neural firing. We model belief updating in Scz and controls in two independent datasets using a hierarchical Bayesian model, and show that all subjects are best fit by a model containing a belief instability parameter. Both this and a response stochasticity parameter are consistently altered in Scz, as the unstable attractor hypothesis predicts.

Forget-me-some: General versus special purpose models in a hierarchical probabilistic task

Humans build models of their environments and act according to what they have learnt. In simple experimental environments, such model-based behaviour is often well accounted for as if subjects are ideal Bayesian observers. However, more complex probabilistic tasks require more sophisticated forms of inference that are sufficiently computationally and statistically taxing as to demand approximation. Here, we study properties of two approximation schemes in the context of a serial reaction time task in which stimuli were generated from a hierarchical Markov chain. One, pre-existing, scheme was a generically powerful variational method for hierarchical inference which has recently become popular as an account of psychological and neural data across a wide swathe of probabilistic tasks. A second, novel, scheme was more specifically tailored to the task at hand. We show that the latter model fit significantly better than the former. This suggests that our subjects were sensitive to many of the particular constraints of a complex behavioural task. Further, the tailored model provided a different perspective on the effects of cholinergic manipulations in the task. Neither model fit the behaviour on more complex contingencies that competently. These results illustrate the benefits and challenges that come with the general and special purpose modelling approaches and raise important questions of how they can advance our current understanding of learning mechanisms in the brain.