Dorsal anterior cingulate cortex integrates reinforcement history to guide voluntary behavior

Altered processing of contextual information during fear extinction in PTSD: an fMRI study

Medial prefrontal cortical areas have been hypothesized to underlie altered contextual processing in posttraumatic stress disorder (PTSD). We investigated brain signaling of contextual information in this disorder. Eighteen PTSD subjects and 16 healthy trauma-exposed subjects underwent a two-day fear conditioning and extinction paradigm. On day 1, within visual context A, a conditioned stimulus (CS) was followed 60% of the time by an electric shock (conditioning). The conditioned response was then extinguished (extinction learning) in context B. On day 2, recall of the extinction memory was tested in context B. Skin conductance response (SCR) and functional magnetic resonance imaging (fMRI) data were collected during context presentations. There were no SCR group differences in any context presentation. Concerning fMRI data, during late conditioning, when context A signaled danger, PTSD subjects showed dorsal anterior cingulate cortical (dACC) hyperactivation. During early extinction, when context B had not yet fully acquired signal value for safety, PTSD subjects still showed dACC hyperactivation. During late extinction, when context B had come to signal safety, they showed ventromedial prefrontal cortex (vmPFC) hypoactivation. During early extinction recall, when context B signaled safety, they showed both vmPFC hypoactivation and dACC hyperactivation. These findings suggest that PTSD subjects show alterations in the processing of contextual information related to danger and safety. This impairment is manifest even prior to a physiologically-measured, cue-elicited fear response, and characterized by hypoactivation in vmPFC and hyperactivation in dACC.

Neural processing of reward and loss in girls at risk for major depression

CONTEXT

Deficits in reward processing and their neural correlates have been associated with major depression. However, it is unclear if these deficits precede the onset of depression or are a consequence of this disorder.

OBJECTIVE

To determine whether anomalous neural processing of reward characterizes children at familial risk for depression in the absence of a personal history of diagnosable disorder.

DESIGN

Comparison of neural activity among children at low and high risk for depression as they process reward and loss.

SETTING

University functional magnetic resonance imaging facility.

PARTICIPANTS

Thirteen 10- to 14-year-old never-disordered daughters of mothers with recurrent depression ("high risk") and 13 age-matched never-disordered daughters with no family history of depression ("low risk"). Main Outcome Measure Neural activity, as measured using functional magnetic resonance imaging, in key reward and attention neural circuitry during anticipation and receipt of reward and loss.

RESULTS

While anticipating gains, high-risk participants showed less activation than did their low-risk counterparts in the putamen and left insula but showed greater activation in the right insula. When receiving punishment, high-risk participants showed greater activation in the dorsal anterior cingulate gyrus than did low-risk participants, who showed greater activation in the caudate and putamen.

CONCLUSIONS

Familial risk for depression affects neural mechanisms underlying the processing of reward and loss; young girls at risk for depression exhibit anomalies in the processing of reward and loss before the onset of depressive symptoms. Longitudinal studies are needed to examine whether these characteristics predict the subsequent onset of depression.

Growth hormone and selective attention: a review

INTRODUCTION

The relation between growth hormone (GH) secretion and general cognitive function has been established. General cognitive functioning depends on core functions including selective attention, which have not been addressed specifically in relation to GH. The present review addresses current insights about specific effects of growth hormone deficiency (GHD) on varieties of selective attention, as well as effects of GH suppletion.

MATERIALS AND METHODS

Studies investigating relationships between GH status and valid measures of selective or divided attention were reviewed.

RESULTS AND DISCUSSION

There are no indications that GHD is characterized by impaired attribute selection, interference control, or attentional switching. In contrast, a few studies point to a deficit in integrated processing of multiple dimensions, as well as speed of information processing. There is also weak evidence for beneficial effects of GH replacement in the opposite direction in these domains.

CONCLUSIONS

The function of integrated processing of multiple stimulus dimensions may be based on neural mechanisms in the anterior cingulate cortex and its extensive connections to the hippocampus, the latter being known to be rich in GH receptors.

Retest reliability of medial frontal negativities during performance monitoring

The error-related negativity (ERN) and feedback-related negativity (FRN) have been used as electrophysiological indices of performance monitoring produced in response to internally generated (errors) and externally generated (feedback) activations of the anterior cingulate cortex (ACC). No studies to date have systematically examined the measurement reliability of these components. In this article, we present the retest reliability of the ERN and FRN during response tasks designed to elicit errors or feedback responses on two occasions. Data from four experiments are presented in which participants performed tasks over various periods of time. Results indicate good retest reliability of the ERN and FRN amplitudes and source generation of these components. The present article provides important validation of the ERN and FRN as stable and trait-like electrophysiological reflections of performance monitoring and ACC functional integrity.

Better than expected or as bad as you thought? The neurocognitive development of probabilistic feedback processing

Learning from feedback lies at the foundation of adaptive behavior. Two prior neuroimaging studies have suggested that there are qualitative differences in how children and adults use feedback by demonstrating that dorsolateral prefrontal cortex (DLPFC) and parietal cortex were more active after negative feedback for adults, but after positive feedback for children. In the current study we used functional magnetic resonance imaging (fMRI) to test whether this difference is related to valence or informative value of the feedback by examining neural responses to negative and positive feedback while applying probabilistic rules. In total, 67 healthy volunteers between ages 8 and 22 participated in the study (8–11 years, n = 18; 13–16 years, n = 27; 18–22 years, n = 22). Behavioral comparisons showed that all participants were able to learn probabilistic rules equally well. DLPFC and dorsal anterior cingulate cortex were more active in younger children following positive feedback and in adults following negative feedback, but only when exploring alternative rules, not when applying the most advantageous rules. These findings suggest that developmental differences in neural responses to feedback are not related to valence per se, but that there is an age-related change in processing learning signals with different informative value.

Different methods to define utility functions yield similar results but engage different neural processes

Although the concept of utility is fundamental to many economic theories, up to now a generally accepted method determining a subject's utility function is not available. We investigated two methods that are used in economic sciences for describing utility functions by using response-locked event-related potentials in order to assess their neural underpinnings. For determining the certainty equivalent, we used a lottery game with probabilities to win p = 0.5, for identifying the subjects' utility functions directly a standard bisection task was applied. Although the lottery tasks' payoffs were only hypothetical, a pronounced negativity was observed resembling the error related negativity (ERN) previously described in action monitoring research, but this occurred only for choices far away from the indifference point between money and lottery. By contrast, the bisection task failed to evoke an remarkable ERN irrespective of the responses' correctness. Based on these findings we are reasoning that only decisions made in the lottery task achieved a level of subjective relevance that activates cognitive-emotional monitoring. In terms of economic sciences, our findings support the view that the bisection method is unaffected by any kind of probability valuation or other parameters related to risk and in combination with the lottery task can, therefore, be used to differentiate between payoff and probability valuation.

The importance of failure: feedback-related negativity predicts motor learning efficiency

Learning from past mistakes is of prominent importance for successful future behavior. In the present study, we tested whether reinforcement learning signals in the brain are predictive of adequate learning of a sequence of motor actions. We recorded event-related potentials (ERPs) while subjects engaged in a sequence learning task. The results showed that brain responses to feedback (the feedback-related negativity [FRN]) predicted whether subjects learned to avoid an erroneous response the next time this action had to be performed. Our findings add to a growing literature on feedback-based performance adjustment, by showing that FRN amplitudes may reflect the acquisition of motor skill and the consolidation of contingencies between stimuli or cues and their associated responses, providing evidence that learning efficiency and future performance can be predicted by the neural response to current feedback: FRN amplitude associated with a mistake is predictive of whether this mistake will be repeated, or learned from.

Receptor architecture of human cingulate cortex: evaluation of the four-region neurobiological model

The structural and functional organization of the human cingulate cortex is an ongoing focus; however, human imaging studies continue to use the century-old Brodmann concept of a two region cingulate cortex. Recently, a four-region neurobiological model was proposed based on structural, circuitry, and functional imaging observations. It encompasses the anterior cingulate, midcingulate, posterior cingulate, and retrosplenial cortices (ACC, MCC, PCC, and RSC, respectively). For the first time, this study performs multireceptor autoradiography of 15 neurotransmitter receptor ligands and multivariate statistics on human whole brain postmortem samples covering the entire cingulate cortex. We evaluated the validity of Brodmann's duality concept and of the four-region model using a hierarchical clustering analysis of receptor binding according to the degree of similarity of each area's receptor architecture. We could not find support for Brodmann's dual cingulate concept, because the anterior part of his area 24 has significantly higher AMPA, kainate, GABA(B), benzodiazepine, and M(3) but lower NMDA and GABA(A) binding site densities than the posterior part. The hierarchical clustering analysis distinguished ACC, MCC, PCC, and RSC as independent regions. The ACC has highest AMPA, kainate, alpha(2), 5-HT(1A), and D(1) but lowest GABA(A) densities. The MCC has lowest AMPA, kainate, alpha(2), and D(1) densities. Area 25 in ACC is similar in receptor-architecture to MCC, particularly the NMDA, GABA(A), GABA(B), and M(2) receptors. The PCC and RSC differ in the higher M(1) and alpha(1) but lower M(3) densities of PCC. Thus, multireceptor autoradiography supports the four-region neurobiological model of the cingulate cortex.

Single dose of a dopamine agonist impairs reinforcement learning in humans: evidence from event-related potentials and computational modeling of striatal-cortical function

Animal findings have highlighted the modulatory role of phasic dopamine (DA) signaling in incentive learning, particularly in the acquisition of reward-related behavior. In humans, these processes remain largely unknown. In a recent study, we demonstrated that a single low dose of a D2/D3 agonist (pramipexole)-assumed to activate DA autoreceptors and thus reduce phasic DA bursts-impaired reward learning in healthy subjects performing a probabilistic reward task. The purpose of this study was to extend these behavioral findings using event-related potentials and computational modeling. Compared with the placebo group, participants receiving pramipexole showed increased feedback-related negativity to probabilistic rewards and decreased activation in dorsal anterior cingulate regions previously implicated in integrating reinforcement history over time. Additionally, findings of blunted reward learning in participants receiving pramipexole were simulated by reduced presynaptic DA signaling in response to reward in a neural network model of striatal-cortical function. These preliminary findings offer important insights on the role of phasic DA signals on reinforcement learning in humans and provide initial evidence regarding the spatiotemporal dynamics of brain mechanisms underlying these processes.

Loss feedback negativity elicited by single- versus conjoined-feature stimuli

Event-related brain potential studies show that negative feedback in guessing tasks elicits a medial frontal negativity. Most theory and experimentation concerning this feedback-related negativity (FRN) has assumed that the FRN has little relationship to the perceptual characteristics of the feedback. This study challenges this assumption. We used a single visual feature or a conjunction of features to indicate reward feedback in a gambling task. In the single-feature condition, losses elicited a larger FRN than gains; in the conjoined-feature condition, that difference was not observed. The results are consistent with the proposal that the FRN is modulated by the deviation of feedback stimuli from a perceptual template. Future studies must not confound the perceptual properties and the valence of reward feedback.

Maternal responses to adolescent positive affect are associated with adolescents' reward neuroanatomy

The development of reward-based learning and decision-making, and the neural circuitry underlying these processes, appears to be influenced negatively by adverse child-rearing environments characterized by abuse and other forms of maltreatment. No research to-date has investigated whether normative variations in the child-rearing environment have effects on adolescent brain structure. We examined whether normative variations in maternal responses to adolescents' positive affective behavior were associated with morphometric measures of the adolescents' affective neural circuitry, namely the amygdala, orbitofrontal cortex (OFC), and anterior cingulate cortex (ACC). Healthy adolescents (N = 113) participated in laboratory-based interaction tasks with their mothers, and underwent high-resolution (3T) structural magnetic resonance imaging (MRI). The mother-adolescent interactions included a pleasant event-planning interaction (EPI) and a conflictual problem-solving interaction (PSI). Adolescents, whose mothers displayed more punishing responses to their positive affective behavior during both tasks, and only during the PSI, had larger left dorsal ACC and bilateral OFC volumes, respectively. In addition, boys whose mothers evidenced this pattern of behavior during the EPI had larger right amygdala volumes. These results suggest that normative variations in maternal responses to affective behavior are associated with the structural characteristics of adolescents' affective neural circuitry, which may have implications for the development of their social, cognitive and affective functioning.

When is an error not a prediction error? An electrophysiological investigation

A recent theory holds that the anterior cingulate cortex (ACC) uses reinforcement learning signals conveyed by the midbrain dopamine system to facilitate flexible action selection. According to this position, the impact of reward prediction error signals on ACC modulates the amplitude of a component of the event-related brain potential called the error-related negativity (ERN). The theory predicts that ERN amplitude is monotonically related to the expectedness of the event: It is larger for unexpected outcomes than for expected outcomes. However, a recent failure to confirm this prediction has called the theory into question. In the present article, we investigated this discrepancy in three trial-and-error learning experiments. All three experiments provided support for the theory, but the effect sizes were largest when an optimal response strategy could actually be learned. This observation suggests that ACC utilizes dopamine reward prediction error signals for adaptive decision making when the optimal behavior is, in fact, learnable.

Segregated and integrated coding of reward and punishment in the cingulate cortex

Reward and punishment have opposite affective value but are both processed by the cingulate cortex. However, it is unclear whether the positive and negative affective values of monetary reward and punishment are processed by separate or common subregions of the cingulate cortex. We performed a functional magnetic resonance imaging study using a free-choice task and compared cingulate activations for different levels of monetary gain and loss. Gain-specific activation (increasing activation for increasing gain, but no activation change in relation to loss) occurred mainly in the anterior part of the anterior cingulate and in the posterior cingulate cortex. Conversely, loss-specific activation (increasing activation for increasing loss, but no activation change in relation to gain) occurred between these areas, in the middle and posterior part of the anterior cingulate. Integrated coding of gain and loss (increasing activation throughout the full range, from biggest loss to biggest gain) occurred in the dorsal part of the anterior cingulate, at the border with the medial prefrontal cortex. Finally, unspecific activation increases to both gains and losses (increasing activation to increasing gains and increasing losses, possibly reflecting attention) occurred in dorsal and middle regions of the cingulate cortex. Together, these results suggest separate and common coding of monetary reward and punishment in distinct subregions of the cingulate cortex. Further meta-analysis suggested that the presently found reward- and punishment-specific areas overlapped with those processing positive and negative emotions, respectively.

Who is to blame? Neural correlates of causal attribution in social situations

In everyday life causal attribution is important in order to structure the complex world, provide explanations for events and to understand why our environment interacts with us in a particular way. This study used functional magnetic resonance imaging (fMRI) in 30 healthy subjects to separate the neural correlates of self vs. external responsibility for social events and explore the neural basis of self-serving attributions (internal attributions of positive events and external attributions of negative events). We presented short sentences describing positive and negative social events and asked participants to imagine the event, to decide the main cause and assign it to one of the categories (internal vs. external). FMRI data were analyzed using a 2 x 2 factorial design with the factors emotional valence and attribution. Internal compared to external attribution revealed activations along the right temporoparietal junction (TPJ). The reverse contrast showed a left lateralized network mainly involving the TPJ, the precuneus and the superior/medial frontal gyrus. These results confirmed the involvement of a fronto-temporoparietal network in differentiating self and external responsibility. Analysis of the self-serving bias yielded activation in the dorsal anterior cingulate and in the dorsal striatum, suggesting a rewarding value of these attributions.

Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-maze

In 2 human event-related brain potential (ERP) experiments, we examined the feedback error-related negativity (fERN), an ERP component associated with reward processing by the midbrain dopamine system, and the N170, an ERP component thought to be generated by the medial temporal lobe (MTL), to investigate the contributions of these neural systems toward learning to find rewards in a "virtual T-maze" environment. We found that feedback indicating the absence versus presence of a reward differentially modulated fERN amplitude, but only when the outcome was not predicted by an earlier stimulus. By contrast, when a cue predicted the reward outcome, then the predictive cue (and not the feedback) differentially modulated fERN amplitude. We further found that the spatial location of the feedback stimuli elicited a large N170 at electrode sites sensitive to right MTL activation and that the latency of this component was sensitive to the spatial location of the reward, occurring slightly earlier for rewards following a right versus left turn in the maze. Taken together, these results confirm a fundamental prediction of a dopamine theory of the fERN and suggest that the dopamine and MTL systems may interact in navigational learning tasks.

Neurocomputational mechanisms of reinforcement-guided learning in humans: a review

Adapting decision making according to dynamic and probabilistic changes in action-reward contingencies is critical for survival in a competitive and resource-limited world. Much research has focused on elucidating the neural systems and computations that underlie how the brain identifies whether the consequences of actions are relatively good or bad. In contrast, less empirical research has focused on the mechanisms by which reinforcements might be used to guide decision making. Here, I review recent studies in which an attempt to bridge this gap has been made by characterizing how humans use reward information to guide and optimize decision making. Regions that have been implicated in reinforcement processing, including the striatum, orbitofrontal cortex, and anterior cingulate, also seem to mediate how reinforcements are used to adjust subsequent decision making. This research provides insights into why the brain devotes resources to evaluating reinforcements and suggests a direction for future research, from studying the mechanisms of reinforcement processing to studying the mechanisms of reinforcement learning.

Individual differences in reinforcement learning: behavioral, electrophysiological, and neuroimaging correlates

During reinforcement learning, phasic modulations of activity in midbrain dopamine neurons are conveyed to the dorsal anterior cingulate cortex (dACC) and basal ganglia (BG) and serve to guide adaptive responding. While the animal literature supports a role for the dACC in integrating reward history over time, most human electrophysiological studies of dACC function have focused on responses to single positive and negative outcomes. The present electrophysiological study investigated the role of the dACC in probabilistic reward learning in healthy subjects using a task that required integration of reinforcement history over time. We recorded the feedback-related negativity (FRN) to reward feedback in subjects who developed a response bias toward a more frequently rewarded ("rich") stimulus ("learners") versus subjects who did not ("non-learners"). Compared to non-learners, learners showed more positive (i.e., smaller) FRNs and greater dACC activation upon receiving reward for correct identification of the rich stimulus. In addition, dACC activation and a bias to select the rich stimulus were positively correlated. The same participants also completed a monetary incentive delay (MID) task administered during functional magnetic resonance imaging. Compared to non-learners, learners displayed stronger BG responses to reward in the MID task. These findings raise the possibility that learners in the probabilistic reinforcement task were characterized by stronger dACC and BG responses to rewarding outcomes. Furthermore, these results highlight the importance of the dACC to probabilistic reward learning in humans.

The feedback correct-related positivity: sensitivity of the event-related brain potential to unexpected positive feedback

The N200 and the feedback error-related negativity (fERN) are two components of the event-related brain potential (ERP) that share similar scalp distributions, time courses, morphologies, and functional dependencies, which raises the question as to whether they are actually the same phenomenon. To investigate this issue, we recorded the ERP from participants engaged in two tasks that independently elicited the N200 and fERN. Our results indicate that they are, in fact, the same ERP component and further suggest that positive feedback elicits a positive-going deflection in the time range of the fERN. Taken together, these results indicate that negative feedback elicits a common N200 and that modulation of fERN amplitude results from the superposition on correct trials of a positive-going deflection that we term the feedback correct-related positivity.

Probing human and monkey anterior cingulate cortex in variable environments

Previous research has identified the anterior cingulate cortex (ACC) as an important node in the neural network underlying decision making in primates. Decision making can, however, be studied under a large variety of circumstances, ranging from the standard well-controlled lab situation to more natural, stochastic settings, in which multiple agents interact. Here, we illustrate how these different varieties of decision making studied can influence theories ofACC function in monkeys. Converging evidence from unit recordings and lesion studies now suggest that the ACC is important for interpreting outcome information according to the current task context to guide future action selection. We then apply this framework to the study of human ACC function and discuss its potential implications.