Balancing New against Old Information: The Role of Puzzlement Surprise in Learning

Reward perseveration is shaped by GABA A -mediated dopamine pauses

Extinction learning is an essential form of cognitive flexibility, which enables obsolete reward associations to be discarded. Its downregulation can lead to perseveration, a symptom seen in several neuropsychiatric disorders. This balance is regulated by dopamine from VTA DA (ventral tegmental area dopamine) neurons, which in turn are largely controlled by GABA (gamma amino-butyric acid) synapses. However, the causal relationship of these circuit elements to extinction and perseveration remain incompletely understood. Here, we employ an innovative drug-targeting technology, DART (drug acutely restricted by tethering), to selectively block GABA A receptors on VTA DA neurons as mice engage in Pavlovian learning. DART eliminated GABA A -mediated pauses-brief decrements in VTA DA activity canonically thought to drive extinction learning. However, contrary to the hypothesis that blocking VTA DA pauses should eliminate extinction learning, we observed the opposite-accelerated extinction learning. Specifically, DART eliminated the naturally occurring perseveration seen in half of control mice. We saw no impact on Pavlovian conditioning, nor on other aspects of VTA DA neural firing. These findings challenge canonical theories, recasting GABA A -mediated VTA DA pauses from presumed facilitators of extinction to drivers of perseveration. More broadly, this study showcases the merits of targeted synaptic pharmacology, while hinting at circuit interventions for pathological perseveration.

Fast adaptation to rule switching using neuronal surprise

In humans and animals, surprise is a physiological reaction to an unexpected event, but how surprise can be linked to plausible models of neuronal activity is an open problem. We propose a self-supervised spiking neural network model where a surprise signal is extracted from an increase in neural activity after an imbalance of excitation and inhibition. The surprise signal modulates synaptic plasticity via a three-factor learning rule which increases plasticity at moments of surprise. The surprise signal remains small when transitions between sensory events follow a previously learned rule but increases immediately after rule switching. In a spiking network with several modules, previously learned rules are protected against overwriting, as long as the number of modules is larger than the total number of rules-making a step towards solving the stability-plasticity dilemma in neuroscience. Our model relates the subjective notion of surprise to specific predictions on the circuit level.

Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

A characterization of the neural representation of confidence during probabilistic learning

Learning in a stochastic and changing environment is a difficult task. Models of learning typically postulate that observations that deviate from the learned predictions are surprising and used to update those predictions. Bayesian accounts further posit the existence of a confidence-weighting mechanism: learning should be modulated by the confidence level that accompanies those predictions. However, the neural bases of this confidence are much less known than the ones of surprise. Here, we used a dynamic probability learning task and high-field MRI to identify putative cortical regions involved in the representation of confidence about predictions during human learning. We devised a stringent test based on the conjunction of four criteria. We localized several regions in parietal and frontal cortices whose activity is sensitive to the confidence of an ideal observer, specifically so with respect to potential confounds (surprise and predictability), and in a way that is invariant to which item is predicted. We also tested for functionality in two ways. First, we localized regions whose activity patterns at the subject level showed an effect of both confidence and surprise in qualitative agreement with the confidence-weighting principle. Second, we found neural representations of ideal confidence that also accounted for subjective confidence. Taken together, those results identify a set of cortical regions potentially implicated in the confidence-weighting of learning.

An intrinsically motivated learning algorithm based on Bayesian surprise for cognitive radar in autonomous vehicles

Introduction

This paper proposes a Bayesian surprise learning algorithm that internally motivates the cognitive radar to estimate a target's state (i.e., velocity, distance) from noisy measurements and make decisions to reduce the estimation error gradually. The work exhibits how the sensor learns from experiences, anticipates future responses, and adjusts its waveform parameters to achieve informative measurements based on the Bayesian surprise.

Methods

For a simple vehicle-following scenario where the radar measurements are generated from linear Gaussian state-space models, the article adopts the Kalman filter to carry out state estimation. According to the information within the filter's estimate, the sensor intrinsically assigns a surprise-based reward value to the immediate past action and updates the value-to-go function. Through a series of hypothetical steps, the cognitive radar considers the impact of future transmissions for a prescribed set of waveforms–available from the sensor profile library–to improve the estimation process.

Results and discussion

Numerous experiments investigate the performance of the proposed design for various surprise-based reward expressions. The robustness of the proposed method is compared to the state-of-the-art for practical and risky driving situations. Results show that the reward functions inspired by estimation credibility measures outperform their competitors when one-step planning is considered. Simulation results also indicate that multiple-step planning does not necessarily lead to lower error, particularly when the environment changes abruptly.

On the Application of Developmental Cognitive Neuroscience in Educational Environments

The paper overviews components of neurologic processing efficiencies to develop innovative methodologies and thinking to school-based applications and changes in educational leadership based on sound findings in the cognitive neurosciences applied to schools and learners. Systems science can allow us to better manage classroom-based learning and instruction on the basis of relatively easily evaluated efficiencies or inefficiencies and optimization instead of simply examining achievement. "Medicalizing" the learning process with concepts such as "learning disability" or employing grading methods such as pass-fail does little to aid in understanding the processes that learners employ to acquire, integrate, remember, and apply information learned. The paper endeavors to overview and provided reference to tools that can be employed that allow a better focus on nervous system-based strategic approaches to classroom learning.

Saccadic eye movement metrics reflect surprise and mental model updating

Two experiments investigated what eye movements can reveal about how we process surprising information and how we update mental models in dynamic and unstructured environments. Participants made saccades to visual targets presented one at a time, radially, around an invisible perimeter. Target locations were normally distributed and shifted at an unannounced point during the task. Trials following the shift were considered surprising and unexpected. These unexpected and surprising events prompted the need to update. Slower saccadic latencies were observed for surprising/unexpected events, perhaps indicative of the need to reorient attention to the unexpected target location. Longer dwell times were observed for events that signaled a change in the distribution. These data show that eye movement metrics provide a reliable indicator of mental model updating when contingencies change even in the absence of explicit change signals.

Spatiotemporal Signatures of Surprise Captured by Magnetoencephalography

Surprise and social influence are linked through several neuropsychological mechanisms. By garnering attention, causing arousal, and motivating engagement, surprise provides a context for effective or durable social influence. Attention to a surprising event motivates the formation of an explanation or updating of models, while high arousal experiences due to surprise promote memory formation. They both encourage engagement with the surprising event through efforts aimed at understanding the situation. By affecting the behavior of the individual or a social group via setting an attractive engagement context, surprise plays an important role in shaping personal and social change. Surprise is an outcome of the brain’s function in constantly anticipating the future of sensory inputs based on past experiences. When new sensory data is different from the brain’s predictions shaped by recent trends, distinct neural signals are generated to report this surprise. As a quantitative approach to modeling the generation of brain surprise, input stimuli containing surprising elements are employed in experiments such as oddball tasks during which brain activity is recorded. Although surprise has been well characterized in many studies, an information-theoretical model to describe and predict the surprise level of an external stimulus in the recorded MEG data has not been reported to date, and setting forth such a model is the main objective of this paper. Through mining trial-by-trial MEG data in an oddball task according to theoretical definitions of surprise, the proposed surprise decoding model employs the entire epoch of the brain response to a stimulus to measure surprise and assesses which collection of temporal/spatial components in the recorded data can provide optimal power for describing the brain’s surprise. We considered three different theoretical formulations for surprise assuming the brain acts as an ideal observer that calculates transition probabilities to estimate the generative distribution of the input. We found that middle temporal components and the right and left fronto-central regions offer the strongest power for decoding surprise. Our findings provide a practical and rigorous method for measuring the brain’s surprise, which can be employed in conjunction with behavioral data to evaluate the interactive and social effects of surprising events.

A Bayesian Surprise Approach in Designing Cognitive Radar for Autonomous Driving

This article proposes the Bayesian surprise as the main methodology that drives the cognitive radar to estimate a target’s future state (i.e., velocity, distance) from noisy measurements and execute a decision to minimize the estimation error over time. The research aims to demonstrate whether the cognitive radar as an autonomous system can modify its internal model (i.e., waveform parameters) to gain consecutive informative measurements based on the Bayesian surprise. By assuming that the radar measurements are constructed from linear Gaussian state-space models, the paper applies Kalman filtering to perform state estimation for a simple vehicle-following scenario. According to the filter’s estimate, the sensor measures the contribution of prospective waveforms—which are available from the sensor profile library—to state estimation and selects the one that maximizes the expectation of Bayesian surprise. Numerous experiments examine the estimation performance of the proposed cognitive radar for single-target tracking in practical highway and urban driving environments. The robustness of the proposed method is compared to the state-of-the-art for various error measures. Results indicate that the Bayesian surprise outperforms its competitors with respect to the mean square relative error when one-step and multiple-step planning is considered.

Serotonin neurons modulate learning rate through uncertainty

Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.

Preferred music listening is associated with perceptual learning enhancement at the expense of self-focused attention

Can preferred music listening improve following attentional and learning performances? Here we suggest that this may be the case. In Experiment 1, following preferred and non-preferred musical-piece listening, we recorded electrophysiological responses to an auditory roving-paradigm. We computed the mismatch negativity (MMN - the difference between responses to novel and repeated stimulation), as an index of perceptual learning, and we measured the correlation between trial-by-trial EEG responses and the fluctuations in Bayesian Surprise, as a quantification of the neural attunement with stimulus informational value. Furthermore, during music listening, we recorded oscillatory cortical activity. MMN and trial-by-trial correlation with Bayesian surprise were significantly larger after subjectively preferred versus non-preferred music, indicating the enhancement of perceptual learning. The analysis on oscillatory activity during music listening showed a selective alpha power increased in response to preferred music, an effect often related to cognitive enhancements. In Experiment 2, we explored whether this learning improvement was realized at the expense of self-focused attention. Therefore, after preferred versus non-preferred music listening, we collected Heart-Beat Detection (HBD) accuracy, as a measure of the attentional focus toward the self. HBD was significantly lowered following preferred music listening. Overall, our results suggest the presence of a specific neural mechanism that, in response to aesthetically pleasing stimuli, and through the modulation of alpha oscillatory activity, redirects neural resources away from the self and toward the environment. This attentional up-weighting of external stimuli might be fruitfully exploited in a wide area of human learning activities, including education, neurorehabilitation and therapy.

Computational models of adaptive behavior and prefrontal cortex

The real world is uncertain, and while ever changing, it constantly presents itself in terms of new sets of behavioral options. To attain the flexibility required to tackle these challenges successfully, most mammalian brains are equipped with certain computational abilities that rely on the prefrontal cortex (PFC). By examining learning in terms of internal models associating stimuli, actions, and outcomes, we argue here that adaptive behavior relies on specific interactions between multiple systems including: (1) selective models learning stimulus-action associations through rewards; (2) predictive models learning stimulus- and/or action-outcome associations through statistical inferences anticipating behavioral outcomes; and (3) contextual models learning external cues associated with latent states of the environment. Critically, the PFC combines these internal models by forming task sets to drive behavior and, moreover, constantly evaluates the reliability of actor task sets in predicting external contingencies to switch between task sets or create new ones. We review different models of adaptive behavior to demonstrate how their components map onto this unifying framework and specific PFC regions. Finally, we discuss how our framework may help to better understand the neural computations and the cognitive architecture of PFC regions guiding adaptive behavior.

Confidence-Controlled Hebbian Learning Efficiently Extracts Category Membership From Stimuli Encoded in View of a Categorization Task

In experiments on perceptual decision making, individuals learn a categorization task through trial-and-error protocols. We explore the capacity of a decision-making attractor network to learn a categorization task through reward-based, Hebbian-type modifications of the weights incoming from the stimulus encoding layer. For the latter, we assume a standard layer of a large number of stimulus-specific neurons. Within the general framework of Hebbian learning, we have hypothesized that the learning rate is modulated by the reward at each trial. Surprisingly, we find that when the coding layer has been optimized in view of the categorization task, such reward-modulated Hebbian learning (RMHL) fails to extract efficiently the category membership. In previous work, we showed that the attractor neural networks' nonlinear dynamics accounts for behavioral confidence in sequences of decision trials. Taking advantage of these findings, we propose that learning is controlled by confidence, as computed from the neural activity of the decision-making attractor network. Here we show that this confidence-controlled, reward-based Hebbian learning efficiently extracts categorical information from the optimized coding layer. The proposed learning rule is local and, in contrast to RMHL, does not require storing the average rewards obtained on previous trials. In addition, we find that the confidence-controlled learning rule achieves near-optimal performance. In accordance with this result, we show that the learning rule approximates a gradient descent method on a maximizing reward cost function.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

Neural surprise in somatosensory Bayesian learning

Tracking statistical regularities of the environment is important for shaping human behavior and perception. Evidence suggests that the brain learns environmental dependencies using Bayesian principles. However, much remains unknown about the employed algorithms, for somesthesis in particular. Here, we describe the cortical dynamics of the somatosensory learning system to investigate both the form of the generative model as well as its neural surprise signatures. Specifically, we recorded EEG data from 40 participants subjected to a somatosensory roving-stimulus paradigm and performed single-trial modeling across peri-stimulus time in both sensor and source space. Our Bayesian model selection procedure indicates that evoked potentials are best described by a non-hierarchical learning model that tracks transitions between observations using leaky integration. From around 70ms post-stimulus onset, secondary somatosensory cortices are found to represent confidence-corrected surprise as a measure of model inadequacy. Indications of Bayesian surprise encoding, reflecting model updating, are found in primary somatosensory cortex from around 140ms. This dissociation is compatible with the idea that early surprise signals may control subsequent model update rates. In sum, our findings support the hypothesis that early somatosensory processing reflects Bayesian perceptual learning and contribute to an understanding of its underlying mechanisms.

Our environment features statistical regularities, such as a drop of rain predicting imminent rainfall. Despite the importance for behavior and survival, much remains unknown about how these dependencies are learned, particularly for somatosensation. As surprise signalling about novel observations indicates a mismatch between one’s beliefs and the world, it has been hypothesized that surprise computation plays an important role in perceptual learning. By analyzing EEG data from human participants receiving sequences of tactile stimulation, we compare different formulations of surprise and investigate the employed underlying learning model. Our results indicate that the brain estimates transitions between observations. Furthermore, we identified different signatures of surprise computation and thereby provide a dissociation of the neural correlates of belief inadequacy and belief updating. Specifically, early surprise responses from around 70ms were found to signal the need for changes to the model, with encoding of its subsequent updating occurring from around 140ms. These results provide insights into how somatosensory surprise signals may contribute to the learning of environmental statistics.

Neural Computations of Threat

A host of learning, memory, and decision-making processes form the individual's response to threat and may be disrupted in anxiety and post-trauma psychopathology. Here we review the neural computations of threat, from the first encounter with a dangerous situation, through learning, storing, and updating cues that predict it, to making decisions about the optimal course of action. The overview highlights the interconnected nature of these processes and their reliance on shared neural and computational mechanisms. We propose an integrative approach to the study of threat-related processes, in which specific computations are studied across the various stages of threat experience rather than in isolation. This approach can generate new insights about the evolution, diagnosis, and treatment of threat-related psychopathology.

Learning in Volatile Environments With the Bayes Factor Surprise

Surprise-based learning allows agents to rapidly adapt to nonstationary stochastic environments characterized by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between forgetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, which we call the Bayes Factor Surprise, that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms, the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprise-based algorithms, one in the family of particle filters, one in the family of variational learning, and one in the family of message passing, that have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from the Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological indicators that dissociate the Bayes Factor Surprise from the Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior and to reinforcement learning in nonstationary environments.

Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing

Surprise signals a discrepancy between past and current beliefs. It is theorized to be linked to affective experiences, the creation of particularly resilient memories, and segmentation of the flow of experience into discrete perceived events. However, the ability to precisely measure naturalistic surprise has remained elusive. We used advanced basketball analytics to derive a quantitative measure of surprise and characterized its behavioral, physiological, and neural correlates in human subjects observing basketball games. We found that surprise was associated with segmentation of ongoing experiences, as reflected by subjectively perceived event boundaries and shifts in neocortical patterns underlying belief states. Interestingly, these effects differed by whether surprising moments contradicted or bolstered current predominant beliefs. Surprise also positively correlated with pupil dilation, activation in subcortical regions associated with dopamine, game enjoyment, and long-term memory. These investigations support key predictions from event segmentation theory and extend theoretical conceptualizations of surprise to real-world contexts.

Imprecise neural computations as a source of adaptive behaviour in volatile environments

In everyday life, humans face environments that feature uncertain and volatile or changing situations. Efficient adaptive behaviour must take into account uncertainty and volatility. Previous models of adaptive behaviour involve inferences about volatility that rely on complex and often intractable computations. Because such computations are presumably implausible biologically, it is unclear how humans develop efficient adaptive behaviours in such environments. Here, we demonstrate a counterintuitive result: simple, low-level inferences confined to uncertainty can produce near-optimal adaptive behaviour, regardless of the environmental volatility, assuming imprecisions in computation that conform to the psychophysical Weber law. We further show empirically that this Weber-imprecision model explains human behaviour in volatile environments better than optimal adaptive models that rely on high-level inferences about volatility, even when considering biologically plausible approximations of such models, as well as non-inferential models like adaptive reinforcement learning.

Risk prediction error signaling: A two-component response?

Organisms use rewards to navigate and adapt to (uncertain) environments. Error-based learning about rewards is supported by the dopaminergic system, which is thought to signal reward prediction errors to make adjustments to past predictions. More recently, the phasic dopamine response was suggested to have two components: the first rapid component is thought to signal the detection of a potentially rewarding stimulus; the second, slightly later component characterizes the stimulus by its reward prediction error. Error-based learning signals have also been found for risk. However, whether the neural generators of these signals employ a two-component coding scheme like the dopaminergic system is unknown. Here, using human high density EEG, we ask whether risk learning, or more generally speaking surprise-based learning under uncertainty, is similarly comprised of two temporally dissociable components. Using a simple card game, we show that the risk prediction error is reflected in the amplitude of the P3b component. This P3b modulation is preceded by an earlier component, that is modulated by the stimulus salience. Source analyses are compatible with the idea that both the early salience signal and the later risk prediction error signal are generated in insular, frontal, and temporal cortex. The identified sources are parts of the risk processing network that receives input from noradrenergic cells in the locus coeruleus. Finally, the P3b amplitude modulation is mirrored by an analogous modulation of pupil size, which is consistent with the idea that both the P3b and pupil size indirectly reflect locus coeruleus activity.

Pupil diameter encodes the idiosyncratic, cognitive complexity of belief updating

Pupils tend to dilate in response to surprising events, but it is not known whether these responses are primarily stimulus driven or instead reflect a more nuanced relationship between pupil-linked arousal systems and cognitive expectations. Using an auditory adaptive decision-making task, we show that evoked pupil diameter is more parsimoniously described as signaling violations of learned, top-down expectations than changes in low-level stimulus properties. We further show that both baseline and evoked pupil diameter is modulated by the degree to which individual subjects use these violations to update their subsequent expectations, as reflected in the complexity of their updating strategy. Together these results demonstrate a central role for idiosyncratic cognitive processing in how arousal systems respond to new inputs and, via our complexity-based analyses, offer a potential framework for understanding these effects in terms of both inference processes aimed to reduce belief uncertainty and more traditional notions of mental effort.

Novelty Manipulations, Memory Performance, and Predictive Coding: the Role of Unexpectedness

Novelty is central to the study of memory, but the wide range of experimental manipulations aimed to reveal its effects on learning produced inconsistent results. The novelty/encoding hypothesis suggests that novel information undergoes enhanced encoding and thus leads to benefits in memory, especially in recognition performance; however, recent studies cast doubts on this assumption. On the other hand, data from animal studies provided evidence on the robust effects of novelty manipulations on the neurophysiological correlates of memory processes. Conceptualizations and operationalizations of novelty are remarkably variable and were categorized into different subtypes, such as stimulus, context, associative or spatial novelty. Here, we summarize previous findings about the effects of novelty on memory and suggest that predictive coding theories provide a framework that could shed light on the differential influence of novelty manipulations on memory performance. In line with predictive coding theories, we emphasize the role of unexpectedness as a crucial property mediating the behavioral and neural effects of novelty manipulations.

Adaptive learning under expected and unexpected uncertainty

The outcome of a decision is often uncertain, and outcomes can vary over repeated decisions. Whether decision outcomes should substantially affect behaviour and learning depends on whether they are representative of a typically experienced range of outcomes or signal a change in the reward environment. Successful learning and decision-making therefore require the ability to estimate expected uncertainty (related to the variability of outcomes) and unexpected uncertainty (related to the variability of the environment). Understanding the bases and effects of these two types of uncertainty and the interactions between them - at the computational and the neural level - is crucial for understanding adaptive learning. Here, we examine computational models and experimental findings to distil computational principles and neural mechanisms for adaptive learning under uncertainty.

Information Theoretic Characterization of Uncertainty Distinguishes Surprise From Accuracy Signals in the Brain

Uncertainty presents a problem for both human and machine decision-making. While utility maximization has traditionally been viewed as the motive force behind choice behavior, it has been theorized that uncertainty minimization may supersede reward motivation. Beyond reward, decisions are guided by belief, i.e., confidence-weighted expectations. Evidence challenging a belief evokes surprise, which signals a deviation from expectation (stimulus-bound surprise) but also provides an information gain. To support the theory that uncertainty minimization is an essential drive for the brain, we probe the neural trace of uncertainty-related decision variables, namely confidence, surprise, and information gain, in a discrete decision with a deterministic outcome. Confidence and surprise were elicited with a gambling task administered in a functional magnetic resonance imaging experiment, where agents start with a uniform probability distribution, transition to a non-uniform probabilistic state, and end in a fully certain state. After controlling for reward expectation, we find confidence, taken as the negative entropy of a trial, correlates with a response in the hippocampus and temporal lobe. Stimulus-bound surprise, taken as Shannon information, correlates with responses in the insula and striatum. In addition, we also find a neural response to a measure of information gain captured by a confidence error, a quantity we dub accuracy. BOLD responses to accuracy were found in the cerebellum and precuneus, after controlling for reward prediction errors and stimulus-bound surprise at the same time point. Our results suggest that, even absent an overt need for learning, the human brain expends energy on information gain and uncertainty minimization.

A Curiosity-Based Learning Method for Spiking Neural Networks

Spiking Neural Networks (SNNs) have shown favorable performance recently. Nonetheless, the time-consuming computation on neuron level and complex optimization limit their real-time application. Curiosity has shown great performance in brain learning, which helps biological brains grasp new knowledge efficiently and actively. Inspired by this leaning mechanism, we propose a curiosity-based SNN (CBSNN) model, which contains four main learning processes. Firstly, the network is trained with biologically plausible plasticity principles to get the novelty estimations of all samples in only one epoch; secondly, the CBSNN begins to repeatedly learn the samples whose novelty estimations exceed the novelty threshold and dynamically update the novelty estimations of samples according to the learning results in five epochs; thirdly, in order to avoid the overfitting of the novel samples and forgetting of the learned samples, CBSNN retrains all samples in one epoch; finally, step two and step three are periodically taken until network convergence. Compared with the state-of-the-art Voltage-driven Plasticity-centric SNN (VPSNN) under standard architecture, our model achieves a higher accuracy of 98.55% with only 54.95% of its computation cost on the MNIST hand-written digit recognition dataset. Similar conclusion can also be found out in other datasets, i.e., Iris, NETtalk, Fashion-MNIST, and CIFAR-10, respectively. More experiments and analysis further prove that such curiosity-based learning theory is helpful in improving the efficiency of SNNs. As far as we know, this is the first practical combination of the curiosity mechanism and SNN, and these improvements will make the realistic application of SNNs possible on more specific tasks within the von Neumann framework.

A Tale of Two Positivities and the N400: Distinct Neural Signatures Are Evoked by Confirmed and Violated Predictions at Different Levels of Representation

It has been proposed that hierarchical prediction is a fundamental computational principle underlying neurocognitive processing. Here, we ask whether the brain engages distinct neurocognitive mechanisms in response to inputs that fulfill versus violate strong predictions at different levels of representation during language comprehension. Participants read three-sentence scenarios in which the third sentence constrained for a broad event structure, for example, {Agent caution animate-Patient}. High constraint contexts additionally constrained for a specific event/lexical item, for example, a two-sentence context about a beach, lifeguards, and sharks constrained for the event, {Lifeguards cautioned Swimmers}, and the specific lexical item swimmers. Low constraint contexts did not constrain for any specific event/lexical item. We measured ERPs on critical nouns that fulfilled and/or violated each of these constraints. We found clear, dissociable effects to fulfilled semantic predictions (a reduced N400), to event/lexical prediction violations (an increased late frontal positivity), and to event structure/animacy prediction violations (an increased late posterior positivity/P600). We argue that the late frontal positivity reflects a large change in activity associated with successfully updating the comprehender's current situation model with new unpredicted information. We suggest that the late posterior positivity/P600 is triggered when the comprehender detects a conflict between the input and her model of the communicator and communicative environment. This leads to an initial failure to incorporate the unpredicted input into the situation model, which may be followed by second-pass attempts to make sense of the discourse through reanalysis, repair, or reinterpretation. Together, these findings provide strong evidence that confirmed and violated predictions at different levels of representation manifest as distinct spatiotemporal neural signatures.

A Tale of Two Positivities and the N400: Distinct Neural Signatures Are Evoked by Confirmed and Violated Predictions at Different Levels of Representation

It has been proposed that hierarchical prediction is a fundamental computational principle underlying neurocognitive processing. Here, we ask whether the brain engages distinct neurocognitive mechanisms in response to inputs that fulfill versus violate strong predictions at different levels of representation during language comprehension. Participants read three-sentence scenarios in which the third sentence constrained for a broad event structure, for example, {Agent caution animate-Patient}. High constraint contexts additionally constrained for a specific event/lexical item, for example, a two-sentence context about a beach, lifeguards, and sharks constrained for the event, {Lifeguards cautioned Swimmers}, and the specific lexical item swimmers. Low constraint contexts did not constrain for any specific event/lexical item. We measured ERPs on critical nouns that fulfilled and/or violated each of these constraints. We found clear, dissociable effects to fulfilled semantic predictions (a reduced N400), to event/lexical prediction violations (an increased late frontal positivity), and to event structure/animacy prediction violations (an increased late posterior positivity/P600). We argue that the late frontal positivity reflects a large change in activity associated with successfully updating the comprehender's current situation model with new unpredicted information. We suggest that the late posterior positivity/P600 is triggered when the comprehender detects a conflict between the input and her model of the communicator and communicative environment. This leads to an initial failure to incorporate the unpredicted input into the situation model, which may be followed by second-pass attempts to make sense of the discourse through reanalysis, repair, or reinterpretation. Together, these findings provide strong evidence that confirmed and violated predictions at different levels of representation manifest as distinct spatiotemporal neural signatures.

Trial-by-trial surprise-decoding model for visual and auditory binary oddball tasks

Having to survive in a continuously changing environment has driven the human brain to actively predict the future state of its surroundings. Oddball tasks are specific types of experiments in which this nature of the human brain is studied. Detailed mathematical models have been constructed to explain the brain's perception in these tasks. These models consider a subject as an ideal observer who abstracts a hypothesis from the previous stimuli, and estimates its hyper-parameters - in order to make the next prediction. The corresponding prediction error is assumed to manifest the subjective surprise of the brain. While the approach of earlier works to this problem has been to suggest an encoding model, we investigated the reverse model: if the stimuli's surprise is assumed as the cause of the observer's surprise, it must be possible to decode the surprise of each stimulus, for every single subject, given only their neural responses, i.e. to tell how unexpected a specific stimulus has been for them. Employing machine learning tools, we developed a surprise decoding model for binary oddball tasks. We constructed our model using the ideal observer proposed by Meyniel et al. in 2016, and applied it to three datasets, one with visual, one with auditory, and one with both visual and auditory stimuli. We demonstrated that our decoding model performs very well for both of the sensory modalities with or without the presence of the subject's motor response.

Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.