Dynamic changes in task preparation in a multi-task environment: The task transformation paradigm

A key element of human flexible behavior concerns the ability to continuously predict and prepare for sudden changes in tasks or actions. Here, we tested whether people can dynamically modulate task preparation processes and decision-making strategies when the identity of a to-be-performed task becomes uncertain. To this end, we developed a new paradigm where participants need to prepare for one of nine tasks on each trial. Crucially, in some blocks, the task being prepared could suddenly shift to a different task after a longer cue-target interval, by changing either the stimulus category or categorization rule that defined the initial task. We found that participants were able to dynamically modulate task preparation in the face of this task uncertainty. A second experiment shows that these changes in behavior were not simply a function of decreasing task expectancy, but rather of increasing switch expectancy. Finally, in the third and fourth experiment, we demonstrate that these dynamic modulations can be applied in a compositional manner, depending on whether either only the stimulus category or categorization rule would be expected to change.

Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

The controllosphere: The neural origin of cognitive effort

Why do some mental activities feel harder than others? The answer to this question is surprisingly controversial. Current theories propose that cognitive effort affords a computational benefit, such as instigating a switch from an activity with low reward value to a different activity with higher reward value. By contrast, in this article, I relate cognitive effort to the fact that brain neuroanatomy and neurophysiology render some neural states more energy-efficient than others. I introduce the concept of the "controllosphere," an energy-inefficient region of neural state space associated with high control, which surrounds the better known "intrinsic manifold," an energy-efficient subspace associated with low control. Integration of control-theoretic principles with classic neurocomputational models of cognitive control suggests that dorsolateral prefrontal cortex (DLPFC) implements a controller that can drive the system state into the controllosphere, anterior cingulate cortex (ACC) implements an observer that monitors changes of state of the controlled system, and cognitive effort reflects a mismatch between DLPFC and ACC energies for control and observation. On this account, cognitive effort scales with the energetic demands of the DLPFC control signal, especially when the consequences of the control are unobservable by ACC. Further, I propose that neural transitions through the controllosphere lead to a buildup of neural waste. Cognitive effort therefore prevents against neural damage by discouraging extended periods of high control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The successor representation subserves hierarchical abstraction for goal-directed behavior

Humans have the ability to craft abstract, temporally extended and hierarchically organized plans. For instance, when considering how to make spaghetti for dinner, we typically concern ourselves with useful "subgoals" in the task, such as cutting onions, boiling pasta, and cooking a sauce, rather than particulars such as how many cuts to make to the onion, or exactly which muscles to contract. A core question is how such decomposition of a more abstract task into logical subtasks happens in the first place. Previous research has shown that humans are sensitive to a form of higher-order statistical learning named "community structure". Community structure is a common feature of abstract tasks characterized by a logical ordering of subtasks. This structure can be captured by a model where humans learn predictions of upcoming events multiple steps into the future, discounting predictions of events further away in time. One such model is the "successor representation", which has been argued to be useful for hierarchical abstraction. As of yet, no study has convincingly shown that this hierarchical abstraction can be put to use for goal-directed behavior. Here, we investigate whether participants utilize learned community structure to craft hierarchically informed action plans for goal-directed behavior. Participants were asked to search for paintings in a virtual museum, where the paintings were grouped together in "wings" representing community structure in the museum. We find that participants' choices accord with the hierarchical structure of the museum and that their response times are best predicted by a successor representation. The degree to which the response times reflect the community structure of the museum correlates with several measures of performance, including the ability to craft temporally abstract action plans. These results suggest that successor representation learning subserves hierarchical abstractions relevant for goal-directed behavior.

Alpha Event-Related Desynchronization During Reward Processing in Schizophrenia

BACKGROUND

Alterations in the brain's reward system may underlie motivation and pleasure deficits in schizophrenia (SZ). Neuro-oscillatory desynchronization in the alpha band is thought to direct resource allocation away from the internal state, to prioritize processing salient environmental events, including reward feedback. We hypothesized reduced reward-related alpha event-related desynchronization (ERD) in SZ, consistent with less externally focused processing during reward feedback.

METHODS

Electroencephalography was recorded while participants with SZ (n = 54) and healthy control participants (n = 54) played a simple slot machine task. Total alpha band power (8-14 Hz), a measure of neural oscillation magnitude, was extracted via principal component analysis and compared between groups and reward outcomes. The clinical relevance of hypothesized alpha power alterations was examined by testing associations with negative symptoms within the SZ group and with trait rumination, dimensionally, across groups.

RESULTS

A group × reward outcome interaction (p = .018) was explained by healthy control participants showing significant posterior-occipital alpha power suppression to wins versus losses (p < .001), in contrast to participants with SZ (p > .1). Among participants with SZ, this alpha ERD was unrelated to negative symptoms (p > .1). Across all participants, less alpha ERD to reward outcomes covaried with greater trait rumination for both win (p = .005) and loss (p = .002) outcomes, with no group differences in slope.

CONCLUSIONS

These findings demonstrate alpha ERD alterations in SZ during reward outcome processing. Additionally, higher trait rumination was associated with less alpha ERD during reward feedback, suggesting that individual differences in rumination covary with external attention to reward processing, regardless of reward outcome valence or group membership.

Electrophysiological measures of conflict and reward processing are associated with decisions to engage in physical effort

Anterior cingulate cortex (ACC), a key brain region involved in cognitive control and decision making, is suggested to mediate effort- and value-based decision making, but the specific role of ACC in this process remains debated. Here we used frontal midline theta (FMT) and the reward positivity (RewP) to examine ACC function in a value-based decision making task requiring physical effort. We investigated whether (1) FMT power is sensitive to the difficulty of the decision or to selecting effortful actions, and (2) RewP is sensitive to the subjective value of reward outcomes as a function of effort investment. On each trial, participants chose to execute a low-effort or a high-effort behavior (that required squeezing a hand-dynamometer) to obtain smaller or larger rewards, respectively, while their brainwaves were recorded. We replicated prior findings that tonic FMT increased over the course of the hour-long task, which suggests increased application of control in the face of growing fatigue. RewP amplitude also increased following execution of high-effort compared to low-effort behavior, consistent with increased valuation of reward outcomes by ACC. Although neither phasic nor tonic FMT were associated with decision difficulty or effort selection per se, an exploratory analysis revealed that the interaction of phasic FMT and expected value of choice predicted effort choice. This interaction suggests that phasic FMT increases specifically under situations of decision difficulty when participants ultimately select a high-effort choice. These results point to a unique role for ACC in motivating and persisting at effortful behavior when decision conflict is high.

Task-level value affects trial-level reward processing

Despite disagreement about how anterior cingulate cortex (ACC) supports decision making, a recent hypothesis suggests that activity in this region is best understood in the context of a task or series of tasks. One important task-level variable is average reward because it is both a known driver of effortful behaviour and an important determiner of the tasks in which we choose to engage. Here we asked how average task value affects reward-related ACC activity. To answer this question, we measured a reward-related signal said to be generated in ACC called the reward positivity (RewP) while participants gambled in three tasks of differing average value. The RewP was reduced in the high-value task, an effect that was not explainable by either reward magnitude or outcome expectancy. This result suggests that ACC does not evaluate outcomes and cues in isolation, but in the context of the value of the current task.

Interbrain synchrony: on wavy ground

In recent years the study of dynamic, between-brain coupling mechanisms has taken social neuroscience by storm. In particular, interbrain synchrony (IBS) is a putative neural mechanism said to promote social interactions by enabling the functional integration of multiple brains. In this article, I argue that this research is beset with three pervasive and interrelated problems. First, the field lacks a widely accepted definition of IBS. Second, IBS wants for theories that can guide the design and interpretation of experiments. Third, a potpourri of tasks and empirical methods permits undue flexibility when testing the hypothesis. These factors synergistically undermine IBS as a theoretical construct. I finish by recommending measures that can address these issues.

Pain feedback interferes with reward positivity production

The reinforcement learning (RL) theory of the reward positivity (RewP) proposes that RewP indexes a reward prediction error (RPE) signal processed in the anterior cingulate cortex (ACC). According to this theory, RewP is an event-related potential (ERP) that is more positive-going for feedback stimuli that predict better-than-expected outcomes (positive feedback) than for feedback stimuli that predict worse-than-expected outcomes (negative feedback). Despite strong evidence for this hypothesis, findings have been equivocal for tasks involving painful outcomes. We hypothesized that the RewP is modulated by high-level task goals such that outcomes that are congruent with the goals elicit positive RPEs even if their immediate consequences are negative. Accordingly, changes in high-level task goals should modulate RewP amplitude for tasks that involve seeking pain compared to tasks that involve avoiding pain. We recorded the electroencephalogram from participants who were instructed to navigate a virtual T-Maze to find reward-predictive feedback in a reward condition and pain-predictive feedback in a pain condition. We expected more positive-going ERPs to reward feedback in the reward condition and more positive-going ERPs to pain feedback in the pain condition. Despite behavioral results indicating that participants complied with task instructions, contrary to our predictions, we did not find a RewP to pain feedback. We suggest that pain feedback interfered with the effect of high-level task goals on RewP amplitude, which is indicative of conflict at different levels of task hierarchy.

Neural Representations of Task Context and Temporal Order During Action Sequence Execution

Routine action sequences can share a great deal of similarity in terms of their stimulus response mappings. As a consequence, their correct execution relies crucially on the ability to preserve contextual and temporal information. However, there are few empirical studies on the neural mechanism and the brain areas maintaining such information. To address this gap in the literature, we recently recorded the blood-oxygen level dependent (BOLD) response in a newly developed coffee-tea making task. The task involves the execution of four action sequences that each comprise six consecutive decision states, which allows for examining the maintenance of contextual and temporal information. Here, we report a reanalysis of this dataset using a data-driven approach, namely multivariate pattern analysis, that examines context-dependent neural activity across several predefined regions of interest. Results highlight involvement of the inferior-temporal gyrus and lateral prefrontal cortex in maintaining temporal and contextual information for the execution of hierarchically organized action sequences. Furthermore, temporal information seems to be more strongly encoded in areas over the left hemisphere.

Hypnotic suggestions of safety reduce neuronal signals of delay discounting

Waiting for delayed rewards is important to reach long-term goals, yet most people prefer immediate rewards. This tendency is called delay discounting. Evidence shows that people are more willing to wait for delayed rewards when they believe that the delayed reward is certain. We hypothesized that feeling safe makes delayed outcomes subjectively more certain, which should in turn reduce neuronal signals of delay discounting. We hypnotized 24 highly suggestible participants and gave them a suggestion to feel safe. We then used EEG to measure their brain responses to immediate and delayed rewards while they played a delayed gratification game. As compared to a control condition without hypnosis, participants that were suggested to feel safe under hypnosis reported feeling significantly safer. Further, their reward-related brain activity differentiated less between immediate and delayed rewards. We conclude that feeling safe makes delayed outcomes subjectively more certain and therefore reduces neuronal signals of delay discounting.

Reward processing electrophysiology in schizophrenia: Effects of age and illness phase

BACKGROUND

Reward processing abnormalities may underlie characteristic pleasure and motivational impairments in schizophrenia. Some neural measures of reward processing show age-related modulation, highlighting the importance of considering age effects on reward sensitivity. We compared event-related potentials (ERPs) reflecting reward anticipation (stimulus-preceding negativity, SPN) and evaluation (reward positivity, RewP; late positive potential, LPP) across individuals with schizophrenia (SZ) and healthy controls (HC), with an emphasis on examining the effects of chronological age, brain age (i.e., predicted age based on neurobiological measures), and illness phase.

METHODS

Subjects underwent EEG while completing a slot-machine task for which rewards were not dependent on performance accuracy, speed, or response preparation. Slot-machine task EEG responses were compared between 54 SZ and 54 HC individuals, ages 19 to 65. Reward-related ERPs were analyzed with respect to chronological age, categorically-defined illness phase (early; ESZ versus chronic schizophrenia; CSZ), and were used to model brain age relative to chronological age.

RESULTS

Illness phase-focused analyses indicated there were no group differences in average SPN or RewP amplitudes. However, a group × reward outcome interaction revealed that ESZ differed from HC in later outcome processing, reflected by greater LPP responses following loss versus reward (a reversal of the HC pattern). While brain age estimates did not differ among groups, depressive symptoms in SZ were associated with older brain age estimates while controlling for negative symptoms.

CONCLUSIONS

ESZ and CSZ did not differ from HC in reward anticipation or early outcome processing during a cognitively undemanding reward task, highlighting areas of preserved functioning. However, ESZ showed altered later reward outcome evaluation, pointing to selective reward deficits during the early illness phase of schizophrenia. Further, an association between ERP-derived brain age and depressive symptoms in SZ extends prior findings linking depression with reward-related ERP blunting. Taken together, both illness phase and age may impact reward processing among SZ, and brain aging may offer a promising, novel marker of reward dysfunction that warrants further study.

Smoking Decisions: Altered Reinforcement Learning Signals Induced by Nicotine State

INTRODUCTION

Alterations in dopamine signaling play a key role in reinforcement learning and nicotine addiction, but the relationship between these two processes has not been well characterized. We investigated this relationship in young adult smokers using a combination of behavioral and computational measures of reinforcement learning.

METHODS

We asked moderately dependent smokers to engage in a reinforcement learning task three times: smoking as usual, smoking abstinence, and cigarette consumption. Participants' trial-to-trial training choices were modeled using a reinforcement learning model that calculates separate learning rates associated with positive and negative prediction errors.

RESULTS

We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. Finally, when tested with novel pairs of stimuli, participants were relatively better at selecting the positive feedback predicting stimuli than avoiding the negative feedback predicting stimuli during the smoking as usual session, a pattern that reversed following cigarette consumption.

CONCLUSIONS

These findings provide a specific computational account of altered reinforcement learning induced by smoking state (abstinence and consumption) and may represent a unique target for treatment of nicotine addiction.

IMPLICATIONS

This study illustrates the potential of computational psychiatry for understanding reinforcement learning deficits associated with substance use disorders in general and nicotine addiction in particular. We found that learning from positive prediction error signals is reduced during smoking abstinence and enhanced following cigarette consumption. By contrast, learning from negative prediction error signals was enhanced during smoking abstinence and reduced following cigarette consumption. By highlighting important computational differences between three states of smoking, these findings hold out promise for integrating experimental, computational, and theoretical analyses of decision-making function together with research on addiction-related disorders.

Neural mechanisms of affective instability and cognitive control in substance use

OBJECTIVE

We explored the impact of affect on cognitive control as this relates to individual differences in affective instability and substance use. Toward this end, we examined how different dimensions of affective instability interact to predict substance misuse and the effect of this on two event-related potential components, the reward positivity and the late positive potential, which are said to reflect the neural mechanisms of reward and emotion processing, respectively.

METHODS

We recorded the ongoing electroencephalogram from undergraduate students as they navigated two T-maze tasks in search of rewards. One of the tasks included neutral, pleasant, and unpleasant pictures from the International Affective Picture System. Participants also completed several questionnaires pertaining to substance use and personality.

RESULTS

A principal components analysis revealed a factor related to affective instability, which we named reactivity. This factor significantly predicted increased substance use. Individuals reporting higher levels of affective reactivity also displayed a larger reward positivity following stimuli with emotional content.

CONCLUSION

The current study uncovered a group of high-risk substance users who were characterized by greater levels of affective reactivity and context-specific increased sensitivity to rewards.

SIGNIFICANCE

These results help to elucidate the complex factors underlying substance use and may facilitate the creation of individually-tailored treatment programs for those struggling with substance use disorders.

Brain mechanisms underlying apathy

The past few decades have seen growing interest in the neuropsychiatric syndrome of apathy, conceptualised as a loss of motivation manifesting as a reduction of goal-directed behaviour. Apathy occurs frequently, and with substantial impact on quality of life, in a broad range of neurological and psychiatric conditions. Apathy is also consistently associated with neuroimaging changes in specific medial frontal cortex and subcortical structures, suggesting that disruption of a common systems-level mechanism may underlie its development, irrespective of the condition that causes it. In parallel with this growing recognition of the clinical importance of apathy, significant advances have been made in understanding normal motivated behaviour in humans and animals. These developments have occurred at several different conceptual levels, from work linking neural structures and neuromodulatory systems to specific aspects of motivated behaviour, to higher order computational models that aim to unite these findings within frameworks for normal goal-directed behaviour. In this review we develop a conceptual framework for understanding pathological apathy based on this current understanding of normal motivated behaviour. We first introduce prominent theories of motivated behaviour-which often involves sequences of actions towards a goal that needs to be maintained across time. Next, we outline the behavioural effects of disrupting these processes in animal models, highlighting the specific effects of these manipulations on different components of motivated behaviour. Finally, we relate these findings to clinical apathy, demonstrating the homologies between this basic neuroscience work and emerging behavioural and physiological evidence from patient studies of this syndrome.

Subgoal- and Goal-related Reward Prediction Errors in Medial Prefrontal Cortex

A longstanding view of the organization of human and animal behavior holds that behavior is hierarchically organized—in other words, directed toward achieving superordinate goals through the achievement of subordinate goals or subgoals. However, most research in neuroscience has focused on tasks without hierarchical structure. In past work, we have shown that negative reward prediction error (RPE) signals in medial prefrontal cortex (mPFC) can be linked not only to superordinate goals but also to subgoals. This suggests that mPFC tracks impediments in the progression toward subgoals. Using fMRI of human participants engaged in a hierarchical navigation task, here we found that mPFC also processes positive prediction errors at the level of subgoals, indicating that this brain region is sensitive to advances in subgoal completion. However, when subgoal RPEs were elicited alongside with goal-related RPEs, mPFC responses reflected only the goal-related RPEs. These findings suggest that information from different levels of hierarchy is processed selectively, depending on the task context.

Electrophysiological indices of anterior cingulate cortex function reveal changing levels of cognitive effort and reward valuation that sustain task performance

Successful execution of goal-directed behaviors often requires the deployment of cognitive control, which is thought to require cognitive effort. Recent theories have proposed that anterior cingulate cortex (ACC) regulates control levels by weighing the reward-related benefits of control against its effort-related costs. However, given that the sensations of cognitive effort and reward valuation are available only to introspection, this hypothesis is difficult to investigate empirically. We have proposed that two electrophysiological indices of ACC function, frontal midline theta and the reward positivity (RewP), provide objective measures of these functions. To explore this issue, we recorded the electroencephalogram (EEG) from participants engaged in an extended, cognitively-demanding task. Participants performed a time estimation task for 2 h in which they received reward and error feedback according to their task performance. We observed that the amplitude of the RewP, a feedback-locked component of the event related brain potential associated with reward processing, decreased with time-on-task. Conversely, frontal midline theta power, which consists of 4-8 Hz EEG oscillations associated with cognitive effort, increased with time-on-task. We also explored how these phenomena changed over time by conducting within-participant multi-level modeling analyses. Our results suggest that extended execution of a cognitively-demanding task is characterized by an early phase in which high control levels foster rapid improvements in task performance, and a later phase in which high control levels were necessary to maintain stable task performance, perhaps counteracting waning reward valuation.

Computational Models of Anterior Cingulate Cortex: At the Crossroads between Prediction and Effort

In the last two decades the anterior cingulate cortex (ACC) has become one of the most investigated areas of the brain. Extensive neuroimaging evidence suggests countless functions for this region, ranging from conflict and error coding, to social cognition, pain and effortful control. In response to this burgeoning amount of data, a proliferation of computational models has tried to characterize the neurocognitive architecture of ACC. Early seminal models provided a computational explanation for a relatively circumscribed set of empirical findings, mainly accounting for EEG and fMRI evidence. More recent models have focused on ACC's contribution to effortful control. In parallel to these developments, several proposals attempted to explain within a single computational framework a wider variety of empirical findings that span different cognitive processes and experimental modalities. Here we critically evaluate these modeling attempts, highlighting the continued need to reconcile the array of disparate ACC observations within a coherent, unifying framework.

I can't wait! Neural reward signals in impulsive individuals exaggerate the difference between immediate and future rewards

Waiting for rewards is difficult, and highly impulsive individuals with low self-control have an especially hard time with it. Here, we investigated whether neural responses to rewards in a delayed gratification task predict impulsivity and self-control. The EEG was recorded from participants engaged in a guessing game in which on each trial they could win either a large or small reward, paid either now or after 6 months. Ratings confirmed that participants preferred immediate, large rewards over small, delayed rewards. Electrophysiological reward signals reflecting the difference between immediate and future rewards predicted self-report measures of impulsivity and self-control. Further, these signals were highly reliable across two sessions over a 1-week interval, showing high temporal stability like stable personality traits. These results suggest that greater valuation of immediate rewards causes impulsive individuals to redirect control away from delayed rewards, indicating why it is so hard for them to wait.

Reward positivity: Reward prediction error or salience prediction error?

The reward positivity is a component of the human ERP elicited by feedback stimuli in trial-and-error learning and guessing tasks. A prominent theory holds that the reward positivity reflects a reward prediction error signal that is sensitive to outcome valence, being larger for unexpected positive events relative to unexpected negative events (Holroyd & Coles, 2002). Although the theory has found substantial empirical support, most of these studies have utilized either monetary or performance feedback to test the hypothesis. However, in apparent contradiction to the theory, a recent study found that unexpected physical punishments also elicit the reward positivity (Talmi, Atkinson, & El-Deredy, 2013). The authors of this report argued that the reward positivity reflects a salience prediction error rather than a reward prediction error. To investigate this finding further, in the present study participants navigated a virtual T maze and received feedback on each trial under two conditions. In a reward condition, the feedback indicated that they would either receive a monetary reward or not and in a punishment condition the feedback indicated that they would receive a small shock or not. We found that the feedback stimuli elicited a typical reward positivity in the reward condition and an apparently delayed reward positivity in the punishment condition. Importantly, this signal was more positive to the stimuli that predicted the omission of a possible punishment relative to stimuli that predicted a forthcoming punishment, which is inconsistent with the salience hypothesis.

Reward Sensitivity of ACC as an Intermediate Phenotype between DRD4-521T and Substance Misuse

The development and expression of the midbrain dopamine system is determined in part by genetic factors that vary across individuals such that dopamine-related genes are partly responsible for addiction vulnerability. However, a complete account of how dopamine-related genes predispose individuals to drug addiction remains to be developed. Adopting an intermediate phenotype approach, we investigated whether reward-related electrophysiological activity of ACC—a cortical region said to utilize dopamine reward signals to learn the value of extended, context-specific sequences of goal-directed behaviors—mediates the influence of multiple dopamine-related functional polymorphisms over substance use. We used structural equation modeling to examine whether two related electrophysiological phenomena associated with the control and reinforcement learning functions of ACC—theta power and the reward positivity—mediated the relationship between the degree of substance misuse and genetic polymorphisms that regulate dopamine processing in frontal cortex. Substance use data were collected from 812 undergraduate students. One hundred ninety-six returned on a subsequent day to participate in an electrophysiological experiment and to provide saliva samples for DNA analysis. We found that these electrophysiological signals mediated a relationship between the DRD4-521T dopamine receptor genotype and substance misuse. Our results provide a theoretical framework that bridges the gap between genes and behavior in drug addiction and illustrate how future interventions might be individually tailored for specific genetic and neurocognitive profiles.

Atypical valuation of monetary and cigarette rewards in substance dependent smokers

OBJECTIVE

Substance dependent (SD) relative to non-dependent (ND) individuals exhibit an attenuated reward positivity, an electrophysiological signal believed to index sensitivity of anterior cingulate cortex (ACC) to rewards. Here we asked whether this altered neural response reflects a specific devaluation of monetary rewards relative to drug-related rewards by ACC.

METHODS

We recorded the reward positivity from SD and ND individuals who currently smoke, following an overnight period of abstinence, while they engaged in two feedback tasks. In a money condition the feedback indicated either a monetary reward or no reward, and in a cigarette condition the feedback indicated either a drug-related reward or no reward.

RESULTS

Overall, cigarette relative to monetary rewards elicited a larger reward positivity. Further, for the subjects who engaged in the money condition first, the reward positivity was smaller for the SD compared to the ND participants, but for the subjects who engaged in the cigarette condition first, the reward positivity was larger for the SD compared to the ND participants.

CONCLUSIONS

Our results suggest that the initial category of feedback "primed" the response of the ACC to the alternative feedback type on subsequent trials, and that SD and ND individuals responded differently to this priming effect.

SIGNIFICANCE

We propose that for people who misuse addictive substances, the prospect of obtaining drug-related rewards engages the ACC to exert control over extended behaviors.

Reward feedback stimuli elicit high-beta EEG oscillations in human dorsolateral prefrontal cortex

Reward-related feedback stimuli have been observed to elicit a burst of power in the beta frequency range over frontal areas of the human scalp. Recent discussions have suggested possible neural sources for this activity but there is a paucity of empirical evidence on the question. Here we recorded EEG from participants while they navigated a virtual T-maze to find monetary rewards. Consistent with previous studies, we found that the reward feedback stimuli elicited an increase in beta power (20-30 Hz) over a right-frontal area of the scalp. Source analysis indicated that this signal was produced in the right dorsolateral prefrontal cortex (DLPFC). These findings align with previous observations of reward-related beta oscillations in the DLPFC in non-human primates. We speculate that increased power in the beta frequency range following reward receipt reflects the activation of task-related neural assemblies that encode the stimulus-response mapping in working memory.

Feedback-related negativity observed in rodent anterior cingulate cortex

The feedback-related negativity (FRN) refers to a difference in the human event-related potential (ERP) elicited by feedback indicating success versus failure: the difference appears negative when subtracting the success ERP from the failure ERP (Miltner et al., 1997). Although source localization techniques (e.g., BESA) suggest that the FRN is produced in the ACC, the inverse problem (that any given scalp distribution can be produced by an infinite number of possible dipole configurations) limits the certainty of this conclusion. The inverse problem can be circumvented by directly recording from the ACC in animal models. Although a non-human primate homologue of the FRN has been observed in the macaque monkey (e.g. Emeric et al., 2008), a homologue of the FRN has yet to be identified in rodents. We recorded local field potentials (LFPs) directly from the ACC in 6 rodents in a task based on the FRN paradigm. The animals were trained to poke their nose into a lighted port and received a feedback smell indicating whether or not a reward pellet would drop 1.5s later. We observed a FRN-like effect time-locked to the feedback scent whereby the LFP to feedback predicting no-reward was significantly more negative than the LFP to feedback predicting reward. This deflection began on average 130ms before behavioral changes in response to the feedback. Thus, we provide the first evidence of the existence of a rodent homologue of the FRN.

Task-specific effects of reward on task switching

Although cognitive control and reinforcement learning have been researched extensively over the last few decades, only recently have studies investigated their interrelationship. An important unanswered question concerns how the control system decides what task to execute and how vigorously to carry out the task once selected. Based on a recent theory of control formulated according to principles of hierarchical reinforcement learning, we asked whether rewards can affect top-down control over task performance at the level of task representation. Participants were rewarded for correctly performing only one of two tasks in a standard task-switching experiment. Reaction times and error rates were lower for the reinforced task compared to the non-reinforced task. Moreover, the switch cost in error rates for the non-reinforced task was significantly larger compared to the reinforced task, especially for trials in which the imperative stimulus afforded different responses for the two tasks, resulting in a "non-paradoxical" asymmetric switch cost. These findings suggest that reinforcement at the task level resulted in greater application of top-down control rather than in stronger stimulus-response pathways for the rewarded task.

Developmental changes in the reward positivity: an electrophysiological trajectory of reward processing

Children and adolescents learn to regulate their behavior by utilizing feedback from the environment but exactly how this ability develops remains unclear. To investigate this question, we recorded the event-related brain potential (ERP) from children (8-13 years), adolescents (14-17 years) and young adults (18-23 years) while they navigated a "virtual maze" in pursuit of monetary rewards. The amplitude of the reward positivity, an ERP component elicited by feedback stimuli, was evaluated for each age group. A current theory suggests the reward positivity is produced by the impact of reinforcement learning signals carried by the midbrain dopamine system on anterior cingulate cortex, which utilizes the signals to learn and execute extended behaviors. We found that the three groups produced a reward positivity of comparable size despite relatively longer ERP component latencies for the children, suggesting that the reward processing system reaches maturity early in development. We propose that early development of the midbrain dopamine system facilitates the development of extended goal-directed behaviors in anterior cingulate cortex.

The Impact of Deliberative Strategy Dissociates ERP Components Related to Conflict Processing vs. Reinforcement Learning

We applied the event-related brain potential (ERP) technique to investigate the involvement of two neuromodulatory systems in learning and decision making: The locus coeruleus-norepinephrine system (NE system) and the mesencephalic dopamine system (DA system). We have previously presented evidence that the N2, a negative deflection in the ERP elicited by task-relevant events that begins approximately 200 ms after onset of the eliciting stimulus and that is sensitive to low-probability events, is a manifestation of cortex-wide noradrenergic modulation recruited to facilitate the processing of unexpected stimuli. Further, we hold that the impact of DA reinforcement learning signals on the anterior cingulate cortex (ACC) produces a component of the ERP called the feedback-related negativity (FRN). The N2 and the FRN share a similar time range, a similar topography, and similar antecedent conditions. We varied factors related to the degree of cognitive deliberation across a series of experiments to dissociate these two ERP components. Across four experiments we varied the demand for a deliberative strategy, from passively watching feedback, to more complex/challenging decision tasks. Consistent with our predictions, the FRN was largest in the experiment involving active learning and smallest in the experiment involving passive learning whereas the N2 exhibited the opposite effect. Within each experiment, when subjects attended to color, the N2 was maximal at frontal-central sites, and when they attended to gender it was maximal over lateral-occipital areas, whereas the topology of the FRN was frontal-central in both task conditions. We conclude that both the DA system and the NE system act in concert when learning from rewards that vary in expectedness, but that the DA system is relatively more exercised when subjects are relatively more engaged by the learning task.

Motivation of extended behaviors by anterior cingulate cortex

Intense research interest over the past decade has yielded diverse and often discrepant theories about the function of anterior cingulate cortex (ACC). In particular, a dichotomy has emerged between neuropsychological theories suggesting a primary role for ACC in motivating or 'energizing' behavior, and neuroimaging-inspired theories emphasizing its contribution to cognitive control and reinforcement learning. To reconcile these views, we propose that ACC supports the selection and maintenance of 'options' - extended, context-specific sequences of behavior directed toward particular goals - that are learned through a process of hierarchical reinforcement learning. This theory accounts for ACC activity in relation to learning and control while simultaneously explaining the effects of ACC damage as disrupting the motivational context supporting the production of goal-directed action sequences.

Dissociation of response and feedback negativity in schizophrenia: electrophysiological and computational evidence for a deficit in the representation of value

Contrasting theories of schizophrenia propose that the disorder is characterized by a deficit in phasic changes in dopamine activity in response to ongoing events or, alternatively, by a weakness in the representation of the value of responses. Schizophrenia patients have reliably reduced brain activity following incorrect responses but other research suggests that they may have intact feedback-related potentials, indicating that the impairment may be specifically response-related. We used event-related brain potentials and computational modeling to examine this issue by comparing the neural response to outcomes with the neural response to behaviors that predict outcomes in patients with schizophrenia and psychiatrically healthy comparison subjects. We recorded feedback-related activity in a passive gambling task and a time estimation task and error-related activity in a flanker task. Patients' brain activity following an erroneous response was reduced compared to comparison subjects but feedback-related activity did not differ between groups. To test hypotheses about the possible causes of this pattern of results, we used computational modeling of the electrophysiological data to simulate the effects of an overall reduction in patients' sensitivity to feedback, selective insensitivity to positive or negative feedback, reduced learning rate, and a decreased representation of the value of the response given the stimulus on each trial. The results of the computational modeling suggest that schizophrenia patients exhibit weakened representation of response values, possibly due to failure of the basal ganglia to strongly associate stimuli with appropriate response alternatives.

Individual differences in substance dependence: at the intersection of brain, behaviour and cognition

Recent theories of drug dependence propose that the transition from occasional recreational substance use to harmful use and dependence results from the impact of disrupted midbrain dopamine signals for reinforcement learning on frontal brain areas that implement cognitive control and decision-making. We investigated this hypothesis in humans using electrophysiological and behavioral measures believed to assay the integrity of midbrain dopamine system and its neural targets. Our investigation revealed two groups of dependent individuals, one characterized by disrupted dopamine-dependent reward learning and the other by disrupted error learning associated with depression-proneness. These results highlight important neurobiological and behavioral differences between two classes of dependent users that can inform the development of individually tailored treatment programs.

Responsibility modulates neural mechanisms of outcome processing: an ERP study

The role of personal responsibility in decision-making and its influence on the outcome evaluation process have been investigated relatively rarely in cognitive neuroscience. The present event-related brain potential (ERP) study manipulated the subjective sense of responsibility by modifying outcome controllability in a gambling task. Participants reported a higher sense of responsibility and produced a larger fERN when they were told that the game was 'controllable' compared with when they were told that the game was 'uncontrollable.' In addition, fERN amplitude was correlated with individual self-reports of personal responsibility over the outcomes. These results indicate that self-attribution of responsibility associated with different degrees of controllability affects the outcome evaluation process and fERN amplitude.

Dissociated roles of the anterior cingulate cortex in reward and conflict processing as revealed by the feedback error-related negativity and N200

The reinforcement learning theory of the error-related negativity (ERN) holds that the impact of reward signals carried by the midbrain dopamine system modulates activity of the anterior cingulate cortex (ACC), alternatively disinhibiting and inhibiting the ACC following unpredicted error and reward events, respectively. According to a recent formulation of the theory, activity that is intrinsic to the ACC produces a component of the event-related brain potential (ERP) called the N200, and following unpredicted rewards, the N200 is suppressed by extrinsically applied positive dopamine reward signals, resulting in an ERP component called the feedback-ERN (fERN). Here we demonstrate that, despite extensive spatial and temporal overlap between the two ERP components, the functional processes indexed by the N200 (conflict) and the fERN (reward) are dissociable. These results point toward avenues for future investigation.

Hypersensitivity to reward in problem gamblers

BACKGROUND

Recent research has begun to examine the neurophysiologic basis of pathological gambling. However, direct evidence of a behavioral deficit and an accompanying neurofunctional deviation in a realistic gambling context such as Black Jack has not yet been reported.

METHODS

Electroencephalogram was recorded while 20 problem gamblers and 21 control participants played a computerized version of Black Jack. Participants were asked to decide at point scores between 11 and 21 whether they wanted to take another card ("hit") to arrive closer to 21 than the opponent (simulated by computer) or not to take another card ("sit") to avoid going over 21 ("bust").

RESULTS

At a critical point score of 16, problem gamblers decided more often to hit despite losses due to a bust on the preceding trial, whereas control participants decided more often to sit under these conditions. Furthermore, problem gamblers showed more reward-related positive amplitudes in the event-related brain potential than control participants after successful hit decisions at 16.

CONCLUSIONS

Here we provide experimental evidence for high-risk taking behavior in gamblers and its correlate in event-related brain potentials. Our results suggest that high-risk-taking behavior in problem gamblers is associated with an increased reward-related neural response to infrequent successes of this behavior.

Learning to become an expert: reinforcement learning and the acquisition of perceptual expertise

To elucidate the neural mechanisms underlying the development of perceptual expertise, we recorded ERPs while participants performed a categorization task. We found that as participants learned to discriminate computer generated "blob" stimuli, feedback modulated the amplitude of the error-related negativity (ERN)-an ERP component thought to reflect error evaluation within medial-frontal cortex. As participants improved at the categorization task, we also observed an increase in amplitude of an ERP component associated with object recognition (the N250). The increase in N250 amplitude preceded an increase in amplitude of an ERN component associated with internal error evaluation (the response ERN). Importantly, these electroencephalographic changes were not observed for participants who failed to improve on the categorization task. Our results suggest that the acquisition of perceptual expertise relies on interactions between the posterior perceptual system and the reinforcement learning system involving medial-frontal cortex.

Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-maze

In 2 human event-related brain potential (ERP) experiments, we examined the feedback error-related negativity (fERN), an ERP component associated with reward processing by the midbrain dopamine system, and the N170, an ERP component thought to be generated by the medial temporal lobe (MTL), to investigate the contributions of these neural systems toward learning to find rewards in a "virtual T-maze" environment. We found that feedback indicating the absence versus presence of a reward differentially modulated fERN amplitude, but only when the outcome was not predicted by an earlier stimulus. By contrast, when a cue predicted the reward outcome, then the predictive cue (and not the feedback) differentially modulated fERN amplitude. We further found that the spatial location of the feedback stimuli elicited a large N170 at electrode sites sensitive to right MTL activation and that the latency of this component was sensitive to the spatial location of the reward, occurring slightly earlier for rewards following a right versus left turn in the maze. Taken together, these results confirm a fundamental prediction of a dopamine theory of the fERN and suggest that the dopamine and MTL systems may interact in navigational learning tasks.

Electroencephalographic correlates of target and outcome errors

Different neural systems underlie the evaluation of different types of errors. Recent electroencephalographic evidence suggests that outcome errors -- errors indicating the failure to achieve a movement goal -- are evaluated within medial-frontal cortex (Krigolson and Holroyd 2006, 2007a, b). Conversely, evidence from a variety of manual aiming studies has demonstrated that target errors -- discrepancies between the actual and desired motor command brought about by an unexpected change in the movement environment -- are mediated within posterior parietal cortex (e.g., Desmurget et al. 1999, 2001; Diedrichsen et al. 2005). Here, event-related brain potentials (ERP) were recorded to assess medial-frontal and parietal ERP components associated with the evaluation of outcome and target errors during performance of a manual aiming task. In line with previous results (Krigolson and Holroyd 2007a), we found that target perturbations elicited an ERP component with a parietal scalp distribution, the P300. However, the timing of kinematic changes associated with accommodation of the target perturbations relative to the timing of the P300 suggests that the P300 component was not related to the online control of movement. Instead, we believe that the P300 evoked by target perturbations reflects the updating of an internal model of the movement environment. Our results also revealed that an error-related negativity, an ERP component typically associated with the evaluation of speeded response errors and error feedback, was elicited when participants missed the movement target. Importantly, this result suggests that a reinforcement learning system within medial-frontal cortex may play a role in improving subsequent motor output.

The feedback correct-related positivity: sensitivity of the event-related brain potential to unexpected positive feedback

The N200 and the feedback error-related negativity (fERN) are two components of the event-related brain potential (ERP) that share similar scalp distributions, time courses, morphologies, and functional dependencies, which raises the question as to whether they are actually the same phenomenon. To investigate this issue, we recorded the ERP from participants engaged in two tasks that independently elicited the N200 and fERN. Our results indicate that they are, in fact, the same ERP component and further suggest that positive feedback elicits a positive-going deflection in the time range of the fERN. Taken together, these results indicate that negative feedback elicits a common N200 and that modulation of fERN amplitude results from the superposition on correct trials of a positive-going deflection that we term the feedback correct-related positivity.