When theory and biology differ: The relationship between reward prediction errors and expectancy

The neural correlates of continuous feedback processing

Feedback processing is commonly studied by analyzing the brain's response to discrete rather than continuous events. Such studies have led to the hypothesis that rapid phasic midbrain dopaminergic activity tracks reward prediction errors (RPEs), the effects of which are measurable at the scalp via electroencephalography (EEG). Although studies using continuous feedback are sparse, recent animal work suggests that moment-to-moment changes in reward are tracked by slowly ramping midbrain dopaminergic activity. Some have argued that these ramping signals index state values rather than RPEs. Our goal here was to develop an EEG measure of continuous feedback processing in humans, then test whether its behavior could be accounted for by the RPE hypothesis. Participants completed a stimulus-response learning task in which a continuous reward cue gradually increased or decreased over time. A regression-based unmixing approach revealed EEG activity with a topography and time course consistent with the stimulus-preceding negativity (SPN), a scalp potential previously linked to reward anticipation and tonic dopamine release. Importantly, this reward-related activity depended on outcome expectancy: as predicted by the RPE hypothesis, activity for expected reward cues was reduced compared to unexpected reward cues. These results demonstrate the possibility of using human scalp-recorded potentials to track continuous feedback processing, and test candidate hypotheses of this activity.

Oscillatory brain activity links experience to expectancy during associative learning

Associating a novel situation with a specific outcome involves a cascade of cognitive processes, including selecting relevant stimuli, forming predictions regarding expected outcomes, and updating memorized predictions based on experience. The present manuscript uses computational modeling and machine learning to test the hypothesis that alpha-band (8-12 Hz) oscillations are involved in the updating of expectations based on experience. Participants learned that a visual cue predicted an aversive loud noise with a probability of 50%. The Rescorla-Wagner model of associative learning explained trial-wise changes in self-reported noise expectancy as well as alpha power changes. Experience in the past trial and self-reported expectancy for the subsequent trial were accurately decoded based on the topographical distribution of alpha power at specific latencies. Decodable information during initial association formation and contingency report recurred when viewing the conditioned cue. Findings support the idea that alpha oscillations have multiple, temporally specific, roles in the formation of associations between cues and outcomes.

Champ versus Chump: Viewing an Opponent’s Face Engages Attention but Not Reward Systems

When we play competitive games, the opponents that we face act as predictors of the outcome of the game. For instance, if you are an average chess player and you face a Grandmaster, you anticipate a loss. Framed in a reinforcement learning perspective, our opponents can be thought of as predictors of rewards and punishments. The present study investigates whether facing an opponent would be processed as a reward or punishment depending on the level of difficulty the opponent poses. Participants played Rock, Paper, Scissors against three computer opponents while electroencephalographic (EEG) data was recorded. In a key manipulation, one opponent (HARD) was programmed to win most often, another (EASY) was made to lose most often, and the third (AVERAGE) had equiprobable outcomes of wins, losses, and ties. Through practice, participants learned to anticipate the relative challenge of a game based on the opponent they were facing that round. An analysis of our EEG data revealed that winning outcomes elicited a reward positivity relative to losing outcomes. Interestingly, our analysis of the predictive cues (i.e., the opponents’ faces) demonstrated that attentional engagement (P3a) was contextually sensitive to anticipated game difficulty. As such, our results for the predictive cue are contrary to what one might expect for a reinforcement model associated with predicted reward, but rather demonstrate that the neural response to the predictive cue was encoding the level of engagement with the opponent as opposed to value relative to the anticipated outcome.

Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG

Learning signals during reinforcement learning and cognitive control rely on valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by surprise magnitude. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). We combine behavioral modeling and single-trial EEG regression to disentangle sequential PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions characterized a spatio-tempo-spectral cascade from early valenced RPE value to non-valenced RPE magnitude, followed by outcome probability indexed by a late frontal positivity. Separating negative and positive outcomes revealed the valenced RPE value effect is an artifact of overlap between two non-valenced RPE magnitude responses: frontal theta feedback-related negativity on losses and posterior delta reward positivity on wins. These results reconcile longstanding debates on the sequence of components representing reward and salience PEs in the human EEG.

What happens when right means wrong? The impact of conflict arising from competing feedback responses

Humans often rely on feedback to learn. Indeed, in learning the difference between feedback and an expected outcome is computed to inform future actions. Further, recent work has found that reward and feedback have a unique role in modulating conflict processing and cognitive control. However, it is still not clear how conflict, especially concerning the processing and evaluation of feedback, impacts learning. To address this, we examined the effects of feedback competition on feedback evaluation in a reinforcement learning task. Specifically, we had participants play a simple two-choice gambling game while electroencephalographic (EEG) data were recorded. On half of the experiment blocks, we reversed the meaning of performance feedback for each trial from its prepotent meaning to induce response conflict akin to the Stroop effect (e.g., '✓' meant incorrect). Behaviourally, we found that participants' accuracy was reduced as a result of incongruent feedback. Paralleling this, an analysis of our EEG revealed that incongruent feedback resulted in a reduction in amplitude of the reward positivity and the P300, components of the human event-related brain potential implicated in reward processing. Our results demonstrate the negative impact of conflict on feedback evaluation and the impact of this on subsequent performance.

The ERP, frequency, and time-frequency correlates of feedback processing: Insights from a large sample study

Human learning, at least in part, appears to be dependent on the evaluation of how outcomes of our actions align with our expectations. Over the past 23 years, electroencephalography (EEG) has been used to probe the neural signatures of feedback processing. Seminal work demonstrated a difference in the human event-related potential (ERP) dependent on whether people were processing correct or incorrect feedback. Since then, these feedback evoked ERPs have been associated with reinforcement learning and conflict monitoring, tied to subsequent behavioral adaptations, and shown to be sensitive to a wide range of factors (e.g., Parkinson's disease). Recently, research has turned to frequency decomposition techniques to examine how changes in the EEG power spectra are related to underlying learning mechanisms. Although the literature on the neural correlates of feedback processing is vast, there are still methodological discrepancies and differences in results across studies. Here, we provide reference results and an investigation of methodological considerations for the ERP (reward positivity) and frequency (delta and theta power) correlates of feedback evaluation with a large sample size. Specifically, participants (n = 500) performed a two-armed bandit task while we recorded EEG. Our findings provide key information about the data characteristics and relationships that exist between the neural signatures of feedback evaluation. Additionally, we conclude with selected methodological recommendations for standardization of future research. All data and scripts are freely provided to facilitate open science.

Neural and Attentional Correlates of Intrinsic Motivation Resulting from Social Performance Expectancy

Some models of motivation distinguish between intrinsic and extrinsic motivation. While past work has examined the neural and cognitive correlates of extrinsic motivation, research on intrinsic motivation has relied primarily on behavioral measures of performance and learning. In particular, no past work has examined the neural and cognitive correlates of social performance expectancy, which is linked to intrinsic motivation. The current study manipulated expectancy of difficult (vs. easy) trials on a cued flanker task and assessed attentional scope and performance. EEG was used to examine motor-action preparation as measured by suppression of beta band activity over the motor cortex and feedback processing as measured by the Reward Positivity (RewP). Results revealed expectancy of difficult (vs. easy) trials narrowed attentional scope, reduced beta activity over the motor cortex, and enhanced RewP amplitudes to win feedback. These findings suggest that enhancing intrinsic motivation through expectancies of positive social comparison engages similar neural and cognitive correlates as extrinsic motivators high in motivational intensity.

Processing of performance errors predicts memory formation: Enhanced feedback-related negativities for corrected versus repeated errors in an associative learning paradigm

Learning from errors or negative feedback is crucial for adaptive behavior. FMRI studies have demonstrated enhanced anterior cingulate cortex activity for errors that were later corrected versus repeated errors even when a substantial delay between the error and the opportunity to correct was introduced. We aimed at identifying the electrophysiological correlates of these processes by investigating the feedback‐related negativity (FRN) and stimulus‐locked P3. Participants had to learn and recall the location of 2‐digit targets over consecutive rounds. Feedback was provided in two steps, first a color change indicated a correct or incorrect response (feedback phase) followed by presentation of the correct digit information (re‐encoding phase). Behaviorally, participants improved performance from the first to the third round. FRN amplitudes time‐locked to feedback were enhanced for corrected compared to repeated errors. The P3 in response to re‐encoding did not differ between the two error types. The finding that FRN amplitudes positively predicted memory performance is consistent with the idea that the FRN reflects prediction errors and the need for enhanced cognitive control. Interestingly, this happens early during feedback processing and not at a later time point when re‐encoding of correct information takes place. The prediction error signal reflected in the FRN is usually elicited by performance errors, but may thus also play a role in preparing/optimizing the system for memory formation. This supports the existence of a close link between action control and memory processes even when there is a substantial delay between error feedback and the opportunity to correct the error.