Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior

Altered cingulo-striatal function underlies reward drive deficits in schizophrenia

Amotivation in schizophrenia is assumed to involve dysfunctional dopaminergic signaling of reward prediction or anticipation. It is unclear, however, whether the translation of neural representation of reward value to behavioral drive is affected in schizophrenia. In order to examine how abnormal neural processing of response valuation and initiation affects incentive motivation in schizophrenia, we conducted functional MRI using a deterministic reinforcement learning task with variable intervals of contingency reversals in 20 clinically stable patients with schizophrenia and 20 healthy controls. Behaviorally, the advantage of positive over negative reinforcer in reinforcement-related responsiveness was not observed in patients. Patients showed altered response valuation and initiation-related striatal activity and deficient rostro-ventral anterior cingulate cortex activation during reward approach initiation. Among these neural abnormalities, rostro-ventral anterior cingulate cortex activation was correlated with positive reinforcement-related responsiveness in controls and social anhedonia and social amotivation subdomain scores in patients. Our findings indicate that the central role of the anterior cingulate cortex is in translating action value into driving force of action, and underscore the role of the cingulo-striatal network in amotivation in schizophrenia.

Post-stroke balance rehabilitation under multi-level electrotherapy: a conceptual review

Stroke is caused when an artery carrying blood from heart to an area in the brain bursts or a clot obstructs the blood flow thereby preventing delivery of oxygen and nutrients. About half of the stroke survivors are left with some degree of disability. Innovative methodologies for restorative neurorehabilitation are urgently required to reduce long-term disability. The ability of the nervous system to respond to intrinsic or extrinsic stimuli by reorganizing its structure, function, and connections is called neuroplasticity. Neuroplasticity is involved in post-stroke functional disturbances, but also in rehabilitation. It has been shown that active cortical participation in a closed-loop brain machine interface (BMI) can induce neuroplasticity in cortical networks where the brain acts as a controller, e.g., during a visuomotor task. Here, the motor task can be assisted with neuromuscular electrical stimulation (NMES) where the BMI will act as a real-time decoder. However, the cortical control and induction of neuroplasticity in a closed-loop BMI is also dependent on the state of brain, e.g., visuospatial attention during visuomotor task performance. In fact, spatial neglect is a hidden disability that is a common complication of stroke and is associated with prolonged hospital stays, accidents, falls, safety problems, and chronic functional disability. This hypothesis and theory article presents a multi-level electrotherapy paradigm toward motor rehabilitation in virtual reality that postulates that while the brain acts as a controller in a closed-loop BMI to drive NMES, the state of brain can be can be altered toward improvement of visuomotor task performance with non-invasive brain stimulation (NIBS). This leads to a multi-level electrotherapy paradigm where a virtual reality-based adaptive response technology is proposed for post-stroke balance rehabilitation. In this article, we present a conceptual review of the related experimental findings.

A neural signature of fairness in altruism: a game of theta?

In the dictator game, a proposer can share a certain amount of money between himself or herself and a receiver, who has no opportunity of influencing the offer. Rational choice theory predicts that dictators keep all money for themselves. But people often are offering money to receivers, despite their opportunity to maximize their own profit and therefore showing altruistic behavior. In this study, we investigated the influence of the altruism of the dictator, the anonymity of the decision and the income of the receiver on the offer made by a dictator. Additionally, we were interested in the influence of midfrontal theta activity prior to the offer, indicating the upcoming decision. The height of the offer made by the dictator was dependent on all variables investigated: Altruism of the dictator led to higher offers and income of the receiver led to higher offer the poorer the receivers are. The anonymity of the decision had two effects, depending on the altruism of the dictator, with higher offers for highly altruistic dictators, when they were not observed, and lower offers for less altruistic dictators in this condition. Finally, midfrontal theta activity predicts upcoming fair offers, maybe indicating altruistic motivation or empathy on physiological basis.

The role of GABAB receptors in human reinforcement learning

Behavioral evidence from human studies suggests that the γ-aminobutyric acid type B receptor (GABAB receptor) agonist baclofen modulates reinforcement learning and reduces craving in patients with addiction spectrum disorders. However, in contrast to the well established role of dopamine in reinforcement learning, the mechanisms by which the GABAB receptor influences reinforcement learning in humans remain completely unknown. To further elucidate this issue, a cross-over, double-blind, placebo-controlled study was performed in healthy human subjects (N=15) to test the effects of baclofen (20 and 50mg p.o.) on probabilistic reinforcement learning. Outcomes were the feedback-induced P2 component of the event-related potential, the feedback-related negativity, and the P300 component of the event-related potential. Baclofen produced a reduction of P2 amplitude over the course of the experiment, but did not modulate the feedback-related negativity. Furthermore, there was a trend towards increased learning after baclofen administration relative to placebo over the course of the experiment. The present results extend previous theories of reinforcement learning, which focus on the importance of mesolimbic dopamine signaling, and indicate that stimulation of cortical GABAB receptors in a fronto-parietal network leads to better attentional allocation in reinforcement learning. This observation is a first step in our understanding of how baclofen may improve reinforcement learning in healthy subjects. Further studies with bigger sample sizes are needed to corroborate this conclusion and furthermore, test this effect in patients with addiction spectrum disorder.

From Feedback- to Response-based Performance Monitoring in Active and Observational Learning

Humans can adapt their behavior by learning from the consequences of their own actions or by observing others. Gradual active learning of action–outcome contingencies is accompanied by a shift from feedback- to response-based performance monitoring. This shift is reflected by complementary learning-related changes of two ACC-driven ERP components, the feedback-related negativity (FRN) and the error-related negativity (ERN), which have both been suggested to signal events “worse than expected,” that is, a negative prediction error. Although recent research has identified comparable components for observed behavior and outcomes (observational ERN and FRN), it is as yet unknown, whether these components are similarly modulated by prediction errors and thus also reflect behavioral adaptation. In this study, two groups of 15 participants learned action–outcome contingencies either actively or by observation. In active learners, FRN amplitude for negative feedback decreased and ERN amplitude in response to erroneous actions increased with learning, whereas observational ERN and FRN in observational learners did not exhibit learning-related changes. Learning performance, assessed in test trials without feedback, was comparable between groups, as was the ERN following actively performed errors during test trials. In summary, the results show that action–outcome associations can be learned similarly well actively and by observation. The mechanisms involved appear to differ, with the FRN in active learning reflecting the integration of information about own actions and the accompanying outcomes.

Frontal glutamate and reward processing in adolescence and adulthood

The fronto-limbic network interaction, driven by glutamatergic and dopaminergic neurotransmission, represents a core mechanism of motivated behavior and personality traits. Reward seeking behavior undergoes tremendous changes in adolescence paralleled by neurobiological changes of this network including the prefrontal cortex, striatum and amygdala. Since fronto-limbic dysfunctions also underlie major psychiatric diseases beginning in adolescence, this investigation focuses on network characteristics separating adolescents from adults. To investigate differences in network interactions, the brain reward system activity (slot machine task) together with frontal glutamate concentration (anterior cingulate cortex, ACC) was measured in 28 adolescents and 26 adults employing functional magnetic resonance imaging and magnetic resonance spectroscopy, respectively. An inverse coupling of glutamate concentrations in the ACC and activation of the ventral striatum was observed in adolescents. Further, amygdala response in adolescents was negatively correlated with the personality trait impulsivity. For adults, no significant associations of network components or correlations with impulsivity were found. The inverse association between frontal glutamate concentration and striatal activation in adolescents is in line with the triadic model of motivated behavior stressing the important role of frontal top-down inhibition on limbic structures. Our data identified glutamate as the mediating neurotransmitter of this inhibitory process and demonstrates the relevance of glutamate on the reward system and related behavioral traits like impulsivity. This fronto-limbic coupling may represent a vulnerability factor for psychiatric disorders starting in adolescence but not in adulthood.

Practitioner review: Dysphoria and its regulation in child and adolescent depression

BACKGROUND

By emphasizing the importance of emotions, the 'affect revolution' in how human behavior is conceptualized has inspired a new generation of studies on dysphoric experience and its regulation in clinical depression, and novel efforts to characterize the precursors of affective disorders in juveniles at familial risk for depression.

METHOD

We review clinical, behavioral, and functional neuroimaging studies of dysphoric experience and its regulation in depressed children and adolescents, and in juvenile offspring of parents with histories of clinical depression. We discuss the implication of the literature in the context of maternal depression.

RESULTS

Findings confirm the high rate of clinically significant dysphoria in depressed children and adolescents and reveal notable affective lability in daily life as a function of context and activity. Findings also show that depressed youngsters have problems in attenuating dysphoria. Similarly, never-depressed offspring at familial risk for depression display problems in mood repair and impaired mood repair mechanisms. Brain neuroimaging findings indicate that, overall, depressed, and high-risk youngsters differ from never depressed controls in neural functioning (activation, connectivity) both at rest and in response to emotion triggers.

CONCLUSION

The evaluation of depressed youngsters should include questions about reactivity of dysphoric mood to the changing contexts of daily life and about how they manage (respond to) their own sadness and distress. The resultant information may help the clinician to restructure a young patient's day for the better and identify helpful mood repair responses. Evidence of impaired mood repair mechanisms in youngsters at high-risk for depression suggests the need for early intervention. But interventions must consider that many depressed and high-risk children have depressed mothers, who may be constrained in their ability to help offspring's emotion regulation efforts. To optimize treatment response of offspring, mothers of depressed children should therefore be routinely screened for depression and treated, as warranted.

Predictability influences whether outcomes are processed in terms of original or relative values

Previous studies have provided insights into the representations of original and relative values and the influence of predictability on decision making. However, whether the predictability of outcomes can influence the processing manner of outcomes (i.e. whether the outcomes are processed in terms of original or relative values) is still unknown. To investigate this issue, we had participants perform a monetary decision task which resulted in two outcomes with the same relative values but different original values in either a predictable or unpredictable condition, while recording event-related potentials (ERP). ERP results showed that the outcome processing in the unpredictable condition elicited more positive deflections in the time window of 300-500ms (P300) than did those in the predictable condition. More importantly, the outcome with high original value elicited a greater P300 component than did that with low original value in the unpredictable condition even though these two outcomes had the same relative values, while in the predictable condition no significant difference was observed between ERPs elicited by the two outcomes even though their original values were different. These results suggest that the outcomes might be processed in terms of relative values in the predictable condition but original values in the unpredictable condition.

Dissociating contributions of ACC and vmPFC in reward prediction, outcome, and choice

Acting in an uncertain environment requires estimating the probability and the value of potential outcomes. These computations are typically ascribed to various parts of the medial prefrontal cortex (mPFC), but the functional architecture of this region remains debated. The anterior cingulate cortex (ACC) encodes reward prediction and outcome (i.e. win vs lose, Silvetti, Seurinck, & Verguts, 2013. Cortex, 49(6), 1627-35. doi:10.1016/j.cortex.2012.05.008). An outcome-related value signal has also been reported in the ventromedial Prefrontal Cortex (vmPFC, Rangel & Hare, 2010. Current Opinion in Neurobiology, 20(2), 262-70. doi:10.1016/j.conb.2010.03.001). Whether a functional dissociation can be traced in these regions with respect to reward prediction and outcome has been suggested but not rigorously tested. Hence an fMRI study was designed to systematically examine the contribution of ACC and vmPFC to reward prediction and outcome. A striking dissociation was identified, with ACC coding for positive prediction errors and vmPFC responding to outcome, irrespective of probability. Moreover, ACC has been assigned a crucial role in the selection of intentional actions (decision-making) and computing the value associated to these actions (action-based value). Conversely, vmPFC seems to implement stimulus-based value processing (Rudebeck et al., 2008. Journal of Neuroscience, 28(51), 13775-85. doi:10.1523/JNEUROSCI.3541-08.2008; Rushworth, Behrens, Rudebeck, & Walton, 2007. Trends in Cognitive Sciences, 11(4), 168-76. doi:10.1016/j.tics.2007.01.004). Therefore, a decision-making factor (choice vs. no choice condition) was also implemented in the present paradigm to distinguish stimulus-based versus action-based value coding in the mPFC during both decision and outcome phase. We found that vmPFC was more activated during the outcome phase in the no-choice than in the choice condition, potentially confirming the role of this area in stimulus-based (more than action-based) value processing.

Perceptual properties of feedback stimuli influence the feedback-related negativity in the flanker gambling task

A negative deflection in the event-related potential is enhanced following error- and loss-related feedback in decision-making and simple gambling tasks. Researchers have assumed that the perceptual properties of the feedback stimuli are unimportant in explaining these effects. This assumption was tested in the present study through a flanker gambling task, in which the perceptual properties of the feedback were manipulated. Consistent with previous studies, loss elicited a larger feedback-related negativity (FRN) than gain feedback. However, this FRN reward effect was modulated by the perceptual properties of the feedback stimuli. When gain and loss feedback were perceptually similar to each other, the enhancement of the FRN following the loss feedback was smaller compared to when the gain and loss feedback were different from each other. In addition, incongruent feedback elicited a larger FRN than congruent feedback; this FRN congruency effect was larger following gain than loss feedback. These results suggested that perceptual properties of the feedback stimuli play a role in the elicitation of the FRN.

Anterior Cingulate Cortex Cells Identify Process-Specific Errors of Attentional Control Prior to Transient Prefrontal-Cingulate Inhibition

Errors indicate the need to adjust attention for improved future performance. Detecting errors is thus a fundamental step to adjust and control attention. These functions have been associated with the dorsal anterior cingulate cortex (dACC), predicting that dACC cells should track the specific processing states giving rise to errors in order to identify which processing aspects need readjustment. Here, we tested this prediction by recording cells in the dACC and lateral prefrontal cortex (latPFC) of macaques performing an attention task that dissociated 3 processing stages. We found that, across prefrontal subareas, the dACC contained the largest cell populations encoding errors indicating (1) failures of inhibitory control of the attentional focus, (2) failures to prevent bottom-up distraction, and (3) lapses when implementing a choice. Error-locked firing in the dACC showed the earliest latencies across the PFC, emerged earlier than reward omission signals, and involved a significant proportion of putative inhibitory interneurons. Moreover, early onset error-locked response enhancement in the dACC was followed by transient prefrontal-cingulate inhibition, possibly reflecting active disengagement from task processing. These results suggest a functional specialization of the dACC to track and identify the actual processes that give rise to erroneous task outcomes, emphasizing its role to control attentional performance.

The role of the midcingulate cortex in monitoring others' decisions

A plethora of research has implicated the cingulate cortex in the processing of social information (i.e., processing elicited by, about, and directed toward others) and reward-related information that guides decision-making. However, it is often overlooked that there is variability in the cytoarchitectonic properties and anatomical connections across the cingulate cortex, which is indicative of functional variability. Here we review evidence from lesion, single-unit recording and functional imaging studies. Taken together, these support the claim that the processing of information that has the greatest influence on social behavior can be localized to the gyral surface of the midcingulate cortex (MCC_g). We propose that the MCC_g is engaged when predicting and monitoring the outcomes of decisions during social interactions. In particular, the MCC_g processes statistical information that tracks the extent to which the outcomes of decisions meet goals when interacting with others. We provide a novel framework for the computational mechanisms that underpin such social information processing in the MCC_g. This framework provides testable hypotheses for the social deficits displayed in autism spectrum disorders and psychopathy.

How we learn to make decisions: rapid propagation of reinforcement learning prediction errors in humans

Our ability to make decisions is predicated upon our knowledge of the outcomes of the actions available to us. Reinforcement learning theory posits that actions followed by a reward or punishment acquire value through the computation of prediction errors-discrepancies between the predicted and the actual reward. A multitude of neuroimaging studies have demonstrated that rewards and punishments evoke neural responses that appear to reflect reinforcement learning prediction errors [e.g., Krigolson, O. E., Pierce, L. J., Holroyd, C. B., & Tanaka, J. W. Learning to become an expert: Reinforcement learning and the acquisition of perceptual expertise. Journal of Cognitive Neuroscience, 21, 1833-1840, 2009; Bayer, H. M., & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47, 129-141, 2005; O'Doherty, J. P. Reward representations and reward-related learning in the human brain: Insights from neuroimaging. Current Opinion in Neurobiology, 14, 769-776, 2004; Holroyd, C. B., & Coles, M. G. H. The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679-709, 2002]. Here, we used the brain ERP technique to demonstrate that not only do rewards elicit a neural response akin to a prediction error but also that this signal rapidly diminished and propagated to the time of choice presentation with learning. Specifically, in a simple, learnable gambling task, we show that novel rewards elicited a feedback error-related negativity that rapidly decreased in amplitude with learning. Furthermore, we demonstrate the existence of a reward positivity at choice presentation, a previously unreported ERP component that has a similar timing and topography as the feedback error-related negativity that increased in amplitude with learning. The pattern of results we observed mirrored the output of a computational model that we implemented to compute reward prediction errors and the changes in amplitude of these prediction errors at the time of choice presentation and reward delivery. Our results provide further support that the computations that underlie human learning and decision-making follow reinforcement learning principles.

Genetic risk for nicotine dependence in the cholinergic system and activation of the brain reward system in healthy adolescents

Genetic variation in a genomic region on chromosome 15q25.1, which encodes the alpha5, alpha3, and beta4 subunits of the cholinergic nicotinic receptor genes, confers risk to smoking and nicotine dependence (ND). Neural reward-related responses have previously been identified as important factors in the development of drug dependence involving ND. Applying an imaging genetics approach in two cohorts (N=487; N=478) of healthy non-smoking adolescents, we aimed to elucidate the impact of genome-wide significant smoking-associated variants in the CHRNA5-CHRNA3-CHRNB4 gene cluster on reward-related neural responses in central regions such as the striatum, orbitofrontal and anterior cingulate cortex (ACC), and personality traits related to addiction. In both samples, carriers of the rs578776 GG compared with AG/AA genotype showed a significantly lower neural response to reward outcomes in the right ventral and dorsal ACC but not the striatum or the orbitofrontal cortex. Rs578776 was unrelated to neural reward anticipation or reward magnitude. Significantly higher scores of anxiety sensitivity in GG compared with AG/AA carriers were found only in sample 1. Associations with other personality traits were not observed. Our findings suggest that the rs578776 risk variant influences susceptibility to ND by dampening the response of the ACC to reward feedback, without recruiting the striatum or orbitofrontal cortex during feedback or anticipation. Thus, it seems to have a major role in the processing of and behavioral adaptation to changing reward outcomes.

The day-after effect: long term, Hebbian-like restructuring of resting-state fMRI patterns induced by a single epoch of cortical activation

During rest, the cerebral cortex displays rich, coordinated patterns of spontaneous activity. The mechanism that shapes these patterns is largely unknown. Here we demonstrate that a Hebbian-like, sustained process plays a role in focusing these coherent patterns. Human subjects used an fMRI-based neurofeedback (NF) paradigm to intensely activate the dorsal anterior cingulate cortex for a single epoch (30 min). Resting-state correlations between all of the cortical voxels' BOLD time courses (functional connectivity) were mapped before, immediately after, and one day after the NF session. We found that the single epoch of cortical activation induced a lasting restructuring of the functional connections according to a Hebbian-like rule. Therefore, the change (increase and decrease) in functional connectivity strength of cortical voxels during rest reflected the level of their prior coactivation during the NF epoch. Interestingly, the effect was significantly enhanced 1 d after the NF activation epoch. The effect was evident in each subject individually, indicating its potential as a diagnostic window into the personal history of prior brain activations of both healthy and abnormal individuals.

Ventral striatum is related to within-subject learning performance

Learning from feedback involves a network of various cortical and subcortical regions. Although activation in this network has been shown to be especially strong in successful learners, it is currently unclear which of these regions are related to within-subject variation in learning performance. To this aim, 21 subjects performed a probabilistic feedback-learning task consisting of multiple independent Learning blocks and non-learning Control blocks, while functional magnetic resonance imaging data were acquired. In agreement with previous studies, activation in anterior, lateral and medial left prefrontal cortex, insula and superior and inferior parietal cortical regions were found when contrasting Learning and Control blocks. Furthermore, activation in the supplementary motor area, anterior cingulate cortex and bilateral striatum was associated specifically with the learning phase and not the application phase during Learning blocks. Finally, activation only in the ventral striatum was associated with within-subject learning performance across the Learning blocks. Taken together, these latter two results are argued to provide the answer to the main research question: ventral striatum activation is associated with within-subject variations in learning performance. The ventral striatum appears to play a vital role in learning by adjusting behavior based on feedback.

Agency matters! Social preferences in the three-person ultimatum game

In the present study EEG was recorded simultaneously while two participants were playing the three-person ultimatum game (UG). Both participants received different offers from changing proposers about how to split up a certain amount of money between the three players. One of the participants had no say, whereas the other, the responder, was able to harm the payoff of all other players. The aim of the study was to investigate how the outcomes of the respective other are evaluated by participants who were treated fairly or unfairly themselves and to what extent agency influences concerns for fairness. Analyses were focused on the medial frontal negativity (MFN) as an early index for subjective value assignment. Recipients with veto-power exhibited enhanced, more negative-going, MFN amplitudes following proposals that comprised a low share for both recipients, suggesting that responders favored offers with a fair amount to at least one of the two players. Though, the powerless players cared about the amount assigned to the responder, MFN amplitudes were larger following fair compared to unfair offers assigned to the responder. Similarly, concerns for fairness which determined the amplitude of the MFN, suggested that the powerless players exhibited negative and conversely the responders, positive social preferences.

Network Resets in Medial Prefrontal Cortex Mark the Onset of Behavioral Uncertainty

Regions within the prefrontal cortex are thought to process beliefs about the world, but little is known about the circuit dynamics underlying the formation and modification of these beliefs. Using a task that permits dissociation between the activity encoding an animal’s internal state and that encoding aspects of behavior, we found that transient increases in the volatility of activity in the rat medial prefrontal cortex accompany periods when an animal’s belief is modified after an environmental change. Activity across the majority of sampled neurons underwent marked, abrupt, and coordinated changes when prior belief was abandoned in favor of exploration of alternative strategies. These dynamics reflect network switches to a state of instability, which diminishes over the period of exploration as new stable representations are formed.

Reinforcement learning signals in the anterior cingulate cortex code for others' false beliefs

The ability to recognise that another's belief is false is a hallmark of our capacity to understand others' mental states. It has been suggested that the computational and neural mechanisms that underpin learning about others' mental states may be similar to those that underpin first-person Reinforcement Learning (RL). In RL, unexpected decision-making outcomes constitute prediction errors (PE), which are coded for by neurons in the Anterior Cingulate Cortex (ACC). Does the ACC signal the PEs (false beliefs) of others about the outcomes of their decisions? We scanned subjects using fMRI while they monitored a third-person's decisions and similar responses made by a computer. The outcomes of the trials were manipulated, such that the actual outcome was unexpectedly different from the predicted outcome on 1/3 of trials. We examined activity time-locked to privileged information which indicated the actual outcomes only to subjects. Activity in the gyral ACC was found when the outcomes of the third-person's decisions were unexpectedly positive. Activity in the sulcal ACC was found when the third-person's or computer's outcomes were unexpectedly positive. We suggest that a property of the ACC is that it codes PEs, with a portion of the gyral ACC specialised for processing the PEs of others.

Performance monitoring and the medial prefrontal cortex: a review of individual differences and context effects as a window on self-regulation

The medial prefrontal cortex (MPFC) is central to self-regulation and has been implicated in generating a cluster of event-related potential components, collectively referred to as medial frontal negativities (MFNs). These MFNs are elicited while individuals monitor behavioral and environmental consequences, and include the error-related negativity, Nogo N2, and the feedback-related negativity. A growing cognitive and affective neuroscience literature indicates that the activation of the anterior cingulate cortex (ACC) and surrounding medial prefrontal regions during performance monitoring is not only influenced by task context, but that these patterns of activity also vary as a function of individual differences (e.g., personality, temperament, clinical and non-clinical symptomatology, socio-political orientation, and genetic polymorphisms), as well as interactions between individual differences and task context. In this review we survey the neuroscience literature on the relations between performance monitoring, personality, task context, and brain functioning with a focus on the MPFC. We relate these issues to the role of affect in the paradigms used to elicit performance-monitoring neural responses and highlight some of the theoretical and clinical implications of this research. We conclude with a discussion of the complexity of these issues and how some of the basic assumptions required for their interpretation may be clarified with future research.

Do we care about the powerless third? An ERP study of the three-person ultimatum game

Recent years have provided increasing insights into the factors affecting economic decision-making. Little is known about how these factors influence decisions that also bear consequences for other people. We examined whether decisions that also affected a third, passive player modulate the behavioral and neural responses to monetary offers in a modified version of the three-person ultimatum game. We aimed to elucidate to what extent social preferences affect early neuronal processing when subjects were evaluating offers that were fair or unfair to themselves, to the third player, or to both. As an event-related potential (ERP) index for early evaluation processes in economic decision-making, we recorded the medial frontal negativity (MFN) component in response to such offers. Unfair offers were rejected more often than equitable ones, in particular when negatively affecting the subject. While the MFN amplitude was higher following unfair as compared to fair offers to the subject, MFN amplitude was not modulated by the shares assigned to the third, passive player. Furthermore, rejection rates and MFN amplitudes following fair offers were positively correlated, as subjects showing lower MFN amplitudes following fair offers tended to reject unfair offers more often—but only if those offers negatively affected their own payoff. Altogether, the rejection behavior suggests that humans mainly care about a powerless third when they are confronted with inequality as well. The correlation between rejection rates and the MFN amplitude supports the notion that this ERP component is also modulated by positive events and highlights how our expectations concerning other humans' behavior guide our own decisions. However, social preferences like inequality aversion and concern for the well-being of others are not reflected in this early neuronal response, but seem to result from later, deliberate and higher-order cognitive processes.

Distinct frontal networks are involved in adapting to internally and externally signaled errors

Errors trigger changes in behavior that help individuals adapt to new situations. The dorsal anterior cingulate cortex (dACC) is thought to be central to this response, but more lateral frontal regions are also activated by errors and may make distinct contributions. We investigated error processing by studying 2 distinct error types: commission and timing. Thirty-five subjects performed a version of the Simon Task designed to produce large number of errors. Commission errors were internally recognized and were not accompanied by explicit feedback. In contrast, timing errors were difficult to monitor internally and were explicitly signaled. Both types of error triggered changes in behavior consistent with increased cognitive control. As expected, robust activation within the dACC and bilateral anterior insulae (the Salience Network) was seen for commission errors. In contrast, timing errors were not associated with activation of this network but did activate a bilateral network that included the right ventral attentional system. Common activation for both error types occurred within the pars operculari and angular gyri. These results show that the dACC does not respond to all behaviorally salient errors. Instead, the error-processing system is multifaceted, and control can be triggered independently of the dACC when feedback is unexpected.

Beyond valence and magnitude: a flexible evaluative coding system in the brain

Outcome evaluation is a cognitive process that plays an important role in our daily lives. In most paradigms utilized in the field of experimental psychology, outcome valence and outcome magnitude are the two major features investigated. The classical "independent coding model" suggests that outcome valence and outcome magnitude are evaluated by separate neural mechanisms that may be mapped onto discrete event-related potential (ERP) components: feedback-related negativity (FRN) and the P3, respectively. To examine this model, we presented outcome valence and magnitude sequentially rather than simultaneously. The results reveal that when only outcome valence or magnitude is known, both the FRN and the P3 encode that outcome feature; when both aspects of outcome are known, the cognitive functions of the two components dissociate: the FRN responds to the information available in the current context, while the P3 pattern depends on outcome presentation sequence. The current study indicates that the human evaluative system, indexed in part by the FRN and the P3, is more flexible than previous theories suggested.

Frontal Oscillatory Dynamics Predict Feedback Learning and Action Adjustment

Frontal oscillatory dynamics in the theta (4–8 Hz) and beta (20–30 Hz) frequency bands have been implicated in cognitive control processes. Here we investigated the changes in coordinated activity within and between frontal brain areas during feedback-based response learning. In a time estimation task, participants learned to press a button after specific, randomly selected time intervals (300–2000 msec) using the feedback after each button press (correct, too fast, too slow). Consistent with previous findings, theta-band activity over medial frontal scalp sites (presumably reflecting medial frontal cortex activity) was stronger after negative feedback, whereas beta-band activity was stronger after positive feedback. Theta-band power predicted learning only after negative feedback, and beta-band power predicted learning after positive and negative feedback. Furthermore, negative feedback increased theta-band intersite phase synchrony (a millisecond resolution measure of functional connectivity) among right lateral prefrontal, medial frontal, and sensorimotor sites. These results demonstrate the importance of frontal theta- and beta-band oscillations and intersite communication in the realization of reinforcement learning.

Reward and punishment effects on error processing and conflict control

Recently, positive affect has been reported to reduce cognitive conflicts and adaptations related to conflict control. van Steenbergen et al. (2009) proposed that the aversive quality of conflicts drives short-term adaptations following a conflict. They reasoned that monetary gain and its positive emotional consequences might counteract the aversive quality of conflict and hence reduce subsequent adaptations. In two experiments, we combined Simon-type conflicts with monetary gains and losses in between trials and analyzed event-related brain potentials. In Experiment 1, gains and losses occurred randomly between trials as a lottery, whereas in Experiment 2 gains and losses were contingent upon performance, either rewarding the 25% fastest responses or penalizing the 25% slowest responses. In Experiment 1, conflict adaptation was completely unaffected by gains or losses; contrary to predictions, in Experiment 2, conflict adaptation in reward blocks was more pronounced after a gain. In Experiment 2 we also investigated the error-related negativity (ERN) – a brain signal proposed to be related to performance monitoring. The ERN and behavioral post-error slowing were enlarged in the context of reward; therefore, reward increases error adaptation, possibly by enhancing the subjective value of errors. In conclusion, affective modulations of conflict adaptations seem to be much more limited than previously asserted and adaptive mechanisms triggered by errors and conflicts dissociate.

Dissociation of response and feedback negativity in schizophrenia: electrophysiological and computational evidence for a deficit in the representation of value

Contrasting theories of schizophrenia propose that the disorder is characterized by a deficit in phasic changes in dopamine activity in response to ongoing events or, alternatively, by a weakness in the representation of the value of responses. Schizophrenia patients have reliably reduced brain activity following incorrect responses but other research suggests that they may have intact feedback-related potentials, indicating that the impairment may be specifically response-related. We used event-related brain potentials and computational modeling to examine this issue by comparing the neural response to outcomes with the neural response to behaviors that predict outcomes in patients with schizophrenia and psychiatrically healthy comparison subjects. We recorded feedback-related activity in a passive gambling task and a time estimation task and error-related activity in a flanker task. Patients' brain activity following an erroneous response was reduced compared to comparison subjects but feedback-related activity did not differ between groups. To test hypotheses about the possible causes of this pattern of results, we used computational modeling of the electrophysiological data to simulate the effects of an overall reduction in patients' sensitivity to feedback, selective insensitivity to positive or negative feedback, reduced learning rate, and a decreased representation of the value of the response given the stimulus on each trial. The results of the computational modeling suggest that schizophrenia patients exhibit weakened representation of response values, possibly due to failure of the basal ganglia to strongly associate stimuli with appropriate response alternatives.

Event-related potential correlates of performance-monitoring in a lateralized time-estimation task

Performance-monitoring as a key function of cognitive control covers a wide range of diverse processes to enable goal directed behavior and to avoid maladjustments. Several event-related brain potentials (ERP) are associated with performance-monitoring, but their conceptual background differs. For example, the feedback-related negativity (FRN) is associated with unexpected performance feedback and might serve as a teaching signal for adaptational processes, whereas the error-related negativity (ERN) is associated with error commission and subsequent behavioral adaptation. The N2 is visible in the EEG when the participant successfully inhibits a response following a cue and thereby adapts to a given stop-signal. Here, we present an innovative paradigm to concurrently study these different performance-monitoring-related ERPs. In 24 participants a tactile time-estimation task interspersed with infrequent stop-signal trials reliably elicited all three ERPs. Sensory input and motor output were completely lateralized, in order to estimate any hemispheric processing preferences for the different aspects of performance monitoring associated with these ERPs. In accordance with the literature our data suggest augmented inhibitory capabilities in the right hemisphere given that stop-trial performance was significantly better with left- as compared to right-hand stop-signals. In line with this, the N2 scalp distribution was generally shifted to the right in addition to an ipsilateral shift in relation to the response hand. Other than that, task lateralization affected neither behavior related to error and feedback processing nor ERN or FRN. Comparing the ERP topographies using the Global Map Dissimilarity index, a large topographic overlap was found between all considered components.With an evenly distributed set of trials and a split-half reliability for all ERP components ≥.85 the task is well suited to efficiently study N2, ERN, and FRN concurrently which might prove useful for group comparisons, especially in clinical populations.

Brain function during probabilistic learning in relation to IQ and level of education

Knowing how to adapt your behavior based on feedback lies at the core of successful learning. We investigated the relation between brain function, grey matter volume, educational level and IQ in a Dutch adolescent sample. In total 45 healthy volunteers between ages 13 and 16 were recruited from schools for pre-vocational and pre-university education. For each individual, IQ was estimated using two subtests from the WISC-III-R (similarities and block design). While in the magnetic resonance imaging (MRI) scanner, participants performed a probabilistic learning task. Behavioral comparisons showed that participants with higher IQ used a more adaptive learning strategy after receiving positive feedback. Analysis of neural activation revealed that higher IQ was associated with increased activation in DLPFC and dACC when receiving positive feedback, specifically for rules with low reward probability (i.e., unexpected positive feedback). Furthermore, VBM analyses revealed that IQ correlated positively with grey matter volume within these regions. These results provide support for IQ-related individual differences in the developmental time courses of neural circuitry supporting feedback-based learning. Current findings are interpreted in terms of a prolonged window of flexibility and opportunity for adolescents with higher IQ scores.

Modeling conflict and error in the medial frontal cortex

Despite intensive study, the role of the dorsal medial frontal cortex (dMFC) in error monitoring and conflict processing remains actively debated. The current experiment manipulated conflict type (stimulus conflict only or stimulus and response selection conflict) and utilized a novel modeling approach to isolate error and conflict variance during a multimodal numeric Stroop task. Specifically, hemodynamic response functions resulting from two statistical models that either included or isolated variance arising from relatively few error trials were directly contrasted. Twenty-four participants completed the task while undergoing event-related functional magnetic resonance imaging on a 1.5-Tesla scanner. Response times monotonically increased based on the presence of pure stimulus or stimulus and response selection conflict. Functional results indicated that dMFC activity was present during trials requiring response selection and inhibition of competing motor responses, but absent during trials involving pure stimulus conflict. A comparison of the different statistical models suggested that relatively few error trials contributed to a disproportionate amount of variance (i.e., activity) throughout the dMFC, but particularly within the rostral anterior cingulate gyrus (rACC). Finally, functional connectivity analyses indicated that an empirically derived seed in the dorsal ACC/pre-SMA exhibited strong connectivity (i.e., positive correlation) with prefrontal and inferior parietal cortex but was anti-correlated with the default-mode network. An empirically derived seed from the rACC exhibited the opposite pattern, suggesting that sub-regions of the dMFC exhibit different connectivity patterns with other large scale networks implicated in internal mentations such as daydreaming (default-mode) versus the execution of top-down attentional control (fronto-parietal).

Methylphenidate normalizes frontocingulate underactivation during error processing in attention-deficit/hyperactivity disorder

BACKGROUND

Children with attention-deficit/hyperactivity disorder (ADHD) have deficits in performance monitoring often improved with the indirect catecholamine agonist methylphenidate (MPH). We used functional magnetic resonance imaging to investigate the effects of single-dose MPH on activation of error processing brain areas in medication-naive boys with ADHD during a stop task that elicits 50% error rates.

METHODS

Twelve medication-naive boys with ADHD were scanned twice, under either a single clinical dose of MPH or placebo, in a randomized, double-blind design while they performed an individually adjusted tracking stop task, designed to elicit 50% failures. Brain activation was compared within patients under either drug condition. To test for potential normalization effects of MPH, brain activation in ADHD patients under either drug condition was compared with that of 13 healthy age-matched boys.

RESULTS

During failed inhibition, boys with ADHD under placebo relative to control subjects showed reduced brain activation in performance monitoring areas of dorsomedial and left ventrolateral prefrontal cortices, thalamus, cingulate, and parietal regions. MPH, relative to placebo, upregulated activation in these brain regions within patients and normalized all activation differences between patients and control subjects. During successful inhibition, MPH normalized reduced activation observed in patients under placebo compared with control subjects in parietotemporal and cerebellar regions.

CONCLUSIONS

MPH normalized brain dysfunction in medication-naive ADHD boys relative to control subjects in typical brain areas of performance monitoring, comprising left ventrolateral and dorsomedial frontal and parietal cortices. This could underlie the amelioration of MPH of attention and academic performance in ADHD.

Posterior cingulate cortex: adapting behavior to a changing world

When has the world changed enough to warrant a new approach? The answer depends on current needs, behavioral flexibility and prior knowledge about the environment. Formal approaches solve the problem by integrating the recent history of rewards, errors, uncertainty and context via Bayesian inference to detect changes in the world and alter behavioral policy. Neuronal activity in posterior cingulate cortex - a key node in the default network - is known to vary with learning, memory, reward and task engagement. We propose that these modulations reflect the underlying process of change detection and motivate subsequent shifts in behavior.

Feedback-related negativity codes prediction error but not behavioral adjustment during probabilistic reversal learning

We assessed electrophysiological activity over the medial frontal cortex (MFC) during outcome-based behavioral adjustment using a probabilistic reversal learning task. During recording, participants were presented two abstract visual patterns on each trial and had to select the stimulus rewarded on 80% of trials and to avoid the stimulus rewarded on 20% of trials. These contingencies were reversed frequently during the experiment. Previous EEG work has revealed feedback-locked electrophysiological responses over the MFC (feedback-related negativity; FRN), which correlate with the negative prediction error [Holroyd, C. B., & Coles, M. G. The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679-709, 2002] and which predict outcome-based adjustment of decision values [Cohen, M. X., & Ranganath, C. Reinforcement learning signals predict future decisions. Journal of Neuroscience, 27, 371-378, 2007]. Unlike previous paradigms, our paradigm enabled us to disentangle, on the one hand, mechanisms related to the reward prediction error, derived from reinforcement learning (RL) modeling, and on the other hand, mechanisms related to explicit rule-based adjustment of actual behavior. Our results demonstrate greater FRN amplitudes with greater RL model-derived prediction errors. Conversely expected negative outcomes that preceded rule-based behavioral reversal were not accompanied by an FRN. This pattern contrasted remarkably with that of the P3 amplitude, which was significantly greater for expected negative outcomes that preceded rule-based behavioral reversal than for unexpected negative outcomes that did not precede behavioral reversal. These data suggest that the FRN reflects prediction error and associated RL-based adjustment of decision values, whereas the P3 reflects adjustment of behavior on the basis of explicit rules.

Social status determines how we monitor and evaluate our performance

Since people with low status are more likely to experience social evaluative threat and are therefore more inclined to monitor for these threats and inhibit approach behaviour, we expected that low-status subjects would be more engaged in evaluating their own performance, compared with high-status subjects. We created a highly salient social hierarchy based on the performance of a simple time estimation task. Subjects could achieve high, middle or low status while performing this task simultaneously with other two players who were either higher or lower in status. Subjects received feedback on their own performance, as well as on the performance of the other two players simultaneously. Electroencephalography (EEG) was recorded from all three participants. The results showed that medial frontal negativity (an event-related potential reflecting performance evaluation) was significantly enhanced for low-status subjects. Implications for status-related differences in goal-directed behaviour are discussed.

Dissociated roles of the anterior cingulate cortex in reward and conflict processing as revealed by the feedback error-related negativity and N200

The reinforcement learning theory of the error-related negativity (ERN) holds that the impact of reward signals carried by the midbrain dopamine system modulates activity of the anterior cingulate cortex (ACC), alternatively disinhibiting and inhibiting the ACC following unpredicted error and reward events, respectively. According to a recent formulation of the theory, activity that is intrinsic to the ACC produces a component of the event-related brain potential (ERP) called the N200, and following unpredicted rewards, the N200 is suppressed by extrinsically applied positive dopamine reward signals, resulting in an ERP component called the feedback-ERN (fERN). Here we demonstrate that, despite extensive spatial and temporal overlap between the two ERP components, the functional processes indexed by the N200 (conflict) and the fERN (reward) are dissociable. These results point toward avenues for future investigation.

Disrupted reinforcement signaling in the orbitofrontal cortex and caudate in youths with conduct disorder or oppositional defiant disorder and a high level of psychopathic traits

OBJECTIVE

Dysfunction in the amygdala and orbitofrontal cortex has been reported in youths and adults with psychopathic traits. The specific nature of the functional irregularities within these structures remains poorly understood. The authors used a passive avoidance task to examine the responsiveness of these systems to early stimulus-reinforcement exposure, when prediction errors are greatest and learning maximized, and to reward in youths with psychopathic traits and comparison youths.

METHOD

While performing the passive avoidance learning task, 15 youths with conduct disorder or oppositional defiant disorder plus a high level of psychopathic traits and 15 healthy subjects completed a 3.0-T fMRI scan.

RESULTS

Relative to the comparison youths, the youths with a disruptive behavior disorder plus psychopathic traits showed less orbitofrontal responsiveness both to early stimulus-reinforcement exposure and to rewards, as well as less caudate response to early stimulus-reinforcement exposure. There were no group differences in amygdala responsiveness to these two task measures, but amygdala responsiveness throughout the task was lower in the youths with psychopathic traits.

CONCLUSIONS

Compromised sensitivity to early reinforcement information in the orbitofrontal cortex and caudate and to reward outcome information in the orbitofrontal cortex of youths with conduct disorder or oppositional defiant disorder plus psychopathic traits suggests that the integrated functioning of the amygdala, caudate, and orbitofrontal cortex may be disrupted. This provides a functional neural basis for why such youths are more likely to repeat disadvantageous decisions. New treatment possibilities are raised, as pharmacologic modulations of serotonin and dopamine can affect this form of learning.

Choice modulates the neural dynamics of prediction error processing during rewarded learning

Our ability to selectively engage with our environment enables us to guide our learning and to take advantage of its benefits. When facing multiple possible actions, our choices are a critical aspect of learning. In the case of learning from rewarding feedback, there has been substantial theoretical and empirical progress in elucidating the associated behavioral and neural processes, predominantly in terms of a reward prediction error, a measure of the discrepancy between actual versus expected reward. Nevertheless, the distinct influence of choice on prediction error processing and its neural dynamics remains relatively unexplored. In this study we used a novel paradigm to determine how choice influences prediction error processing and to examine whether there are correspondingly distinct neural dynamics. We recorded scalp electroencephalogram while healthy adults were administered a rewarded learning task in which choice trials were intermingled with control trials involving the same stimuli, motor responses, and probabilistic rewards. We used a temporal difference learning model of subjects' trial-by-trial choices to infer subjects' image valuations and corresponding prediction errors. As expected, choices were associated with lower overall prediction error magnitudes, most notably over the course of learning the stimulus-reward contingencies. Choices also induced a higher-amplitude relative positivity in the frontocentral event-related potential about 200 ms after reward signal onset that was negatively correlated with the differential effect of choice on the prediction error. Thus choice influences the neural dynamics associated with how reward signals are processed during learning. Behavioral, computational, and neurobiological models of rewarded learning should therefore accommodate a distinct influence for choice during rewarded learning.

Manipulation of feedback expectancy and valence induces negative and positive reward prediction error signals manifest in event-related brain potentials

The feedback-related negativity (FRN) has been hypothesized to be most sensitive to unexpected negative feedback. The present study investigated feedback expectancy and valence using a probabilistic gambling paradigm where subjects encountered expected or unexpected positive and negative feedback outcomes. In line with previous studies, FRN amplitude reflected a negative reward prediction error, but to a minor extent also a positive reward prediction error. Moreover, the P300 amplitude was largest after unexpected feedback, irrespective of valence. We propose to interpret the FRN in terms of a reinforcement learning signal which is detecting mismatch between internal and external representations indexed by the ACC to extract motivationally salient outcomes.