Neurocomputational models of basal ganglia function in learning, memory and choice

Functional brain imaging of appetite

Obesity is a neurobehavioral disorder that results from a combination of overeating and insufficient physical activity. Finely tuned mechanisms exist to match food intake to caloric expenditure. However, faced with abundant inexpensive and calorie-dense foods, many humans (and perhaps most) have a tendency to consume beyond their caloric needs. The brain controls food intake by sensing internal energy-balance signals and external cues of food availability, and by controlling feeding behavior; it is therefore at the centre of the obesity problem. This article reviews the recent use of functional brain imaging in humans to study the neural control of appetite, and how the neural systems involved may cause vulnerability to overeating in the obesogenic environment.

Effects of Monetary Incentives on Task Switching

In two experiments, effects of incentives on task switching were investigated. Incentives were provided as a monetary bonus. In both experiments, the availability of a bonus varied on a trial-to-trial basis. The main difference between the experiments relates to the association of incentives to individual tasks. In Experiment 1, the association of incentives to individual tasks was fixed. Under these conditions, the effect of incentives was largely due to reward expectancy. Switch costs were reduced to statistical insignificance. This was true even with the task that was not associated with a bonus. In Experiment 2, there was a variable association of incentives to individual tasks. Under these conditions, the reward expectancy effect was bound to conditions with a well-established bonus-task association. In conditions in which the bonus-task association was not established in advance, enhanced performance of the bonus task was accompanied by performance decrements with the task that was not associated with a bonus. Reward expectancy affected mainly the general level of performance. The outcome of this study may also inform recently suggested neurobiological accounts about the temporal dynamics of reward processing.

Negative symptoms and the failure to represent the expected reward value of actions: behavioral and computational modeling evidence

CONTEXT

Negative symptoms are a core feature of schizophrenia, but their pathogenesis remains unclear. Negative symptoms are defined by the absence of normal function. However, there must be a productive mechanism that leads to this absence.

OBJECTIVE

To test a reinforcement learning account suggesting that negative symptoms result from a failure in the representation of the expected value of rewards coupled with preserved loss-avoidance learning.

DESIGN

Participants performed a probabilistic reinforcement learning paradigm involving stimulus pairs in which choices resulted in reward or in loss avoidance. Following training, participants indicated their valuation of the stimuli in a transfer test phase. Computational modeling was used to distinguish between alternative accounts of the data.

SETTING

A tertiary care research outpatient clinic.

PATIENTS

In total, 47 clinically stable patients with a diagnosis of schizophrenia or schizoaffective disorder and 28 healthy volunteers participated in the study. Patients were divided into a high-negative symptom group and a low-negative symptom group.

MAIN OUTCOME MEASURES

The number of choices leading to reward or loss avoidance, as well as performance in the transfer test phase. Quantitative fits from 3 different models were examined.

RESULTS

Patients in the high-negative symptom group demonstrated impaired learning from rewards but intact loss-avoidance learning and failed to distinguish rewarding stimuli from loss-avoiding stimuli in the transfer test phase. Model fits revealed that patients in the high-negative symptom group were better characterized by an "actor-critic" model, learning stimulus-response associations, whereas control subjects and patients in the low-negative symptom group incorporated expected value of their actions ("Q learning") into the selection process.

CONCLUSIONS

Negative symptoms in schizophrenia are associated with a specific reinforcement learning abnormality: patients with high-negative symptoms do not represent the expected value of rewards when making decisions but learn to avoid punishments through the use of prediction errors. This computational framework offers the potential to understand negative symptoms at a mechanistic level.

Neuropsychopharmacology and neurogenetic aspects of executive functioning: should reward gene polymorphisms constitute a diagnostic tool to identify individuals at risk for impaired judgment?

Executive functions are processes that act in harmony to control behaviors necessary for maintaining focus and achieving outcomes. Executive dysfunction in neuropsychiatric disorders is attributed to structural or functional pathology of brain networks involving prefrontal cortex (PFC) and its connections with other brain regions. The PFC receives innervations from different neurons associated with a number of neurotransmitters, especially dopamine (DA). Here we review findings on the contribution of PFC DA to higher-order cognitive and emotional behaviors. We suggest that examination of multifactorial interactions of an individual's genetic history, along with environmental risk factors, can assist in the characterization of executive functioning for that individual. Based upon the results of genetic studies, we also propose genetic mapping as a probable diagnostic tool serving as a therapeutic adjunct for augmenting executive functioning capabilities. We conclude that preservation of the neurological underpinnings of executive functions requires the integrity of complex neural systems including the influence of specific genes and associated polymorphisms to provide adequate neurotransmission.

Dissociable reward and timing signals in human midbrain and ventral striatum

Reward prediction error (RPE) signals are central to current models of reward-learning. Temporal difference (TD) learning models posit that these signals should be modulated by predictions, not only of magnitude but also timing of reward. Here we show that BOLD activity in the VTA conforms to such TD predictions: responses to unexpected rewards are modulated by a temporal hazard function and activity between a predictive stimulus and reward is depressed in proportion to predicted reward. By contrast, BOLD activity in ventral striatum (VS) does not reflect a TD RPE, but instead encodes a signal on the variable relevant for behavior, here timing but not magnitude of reward. The results have important implications for dopaminergic models of cortico-striatal learning and suggest a modification of the conventional view that VS BOLD necessarily reflects inputs from dopaminergic VTA neurons signaling an RPE.

Double dissociation of stimulus-value and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage

Adaptive decision making involves selecting the most valuable option, typically by taking an action. Such choices require value comparisons, but there is debate about whether these comparisons occur at the level of stimuli (goods-based) value, action-based value, or both. One view is that value processes occur in series, with stimulus value informing action value. However, lesion work in nonhuman primates suggests that these two kinds of choice are dissociable. Here, we examined action-value and stimulus-value learning in humans with focal frontal lobe damage. Orbitofrontal damage disrupted the ability to sustain the correct choice of stimulus, but not of action, after positive feedback, while damage centered on dorsal anterior cingulate cortex led to the opposite deficit. These findings argue that there are distinct, domain-specific mechanisms by which outcome value is applied to guide subsequent decisions, depending on whether the choice is between stimuli or between actions.

Dissociation of response and feedback negativity in schizophrenia: electrophysiological and computational evidence for a deficit in the representation of value

Contrasting theories of schizophrenia propose that the disorder is characterized by a deficit in phasic changes in dopamine activity in response to ongoing events or, alternatively, by a weakness in the representation of the value of responses. Schizophrenia patients have reliably reduced brain activity following incorrect responses but other research suggests that they may have intact feedback-related potentials, indicating that the impairment may be specifically response-related. We used event-related brain potentials and computational modeling to examine this issue by comparing the neural response to outcomes with the neural response to behaviors that predict outcomes in patients with schizophrenia and psychiatrically healthy comparison subjects. We recorded feedback-related activity in a passive gambling task and a time estimation task and error-related activity in a flanker task. Patients' brain activity following an erroneous response was reduced compared to comparison subjects but feedback-related activity did not differ between groups. To test hypotheses about the possible causes of this pattern of results, we used computational modeling of the electrophysiological data to simulate the effects of an overall reduction in patients' sensitivity to feedback, selective insensitivity to positive or negative feedback, reduced learning rate, and a decreased representation of the value of the response given the stimulus on each trial. The results of the computational modeling suggest that schizophrenia patients exhibit weakened representation of response values, possibly due to failure of the basal ganglia to strongly associate stimuli with appropriate response alternatives.

Modulation of striatal projection systems by dopamine

The basal ganglia are a chain of subcortical nuclei that facilitate action selection. Two striatal projection systems--so-called direct and indirect pathways--form the functional backbone of the basal ganglia circuit. Twenty years ago, investigators proposed that the striatum's ability to use dopamine (DA) rise and fall to control action selection was due to the segregation of D(1) and D(2) DA receptors in direct- and indirect-pathway spiny projection neurons. Although this hypothesis sparked a debate, the evidence that has accumulated since then clearly supports this model. Recent advances in the means of marking neural circuits with optical or molecular reporters have revealed a clear-cut dichotomy between these two cell types at the molecular, anatomical, and physiological levels. The contrast provided by these studies has provided new insights into how the striatum responds to fluctuations in DA signaling and how diseases that alter this signaling change striatal function.

Molecular substrates of action control in cortico-striatal circuits

The purpose of this review is to describe the molecular mechanisms in the striatum that mediate reward-based learning and action control during instrumental conditioning. Experiments assessing the neural bases of instrumental conditioning have uncovered functional circuits in the striatum, including dorsal and ventral striatal sub-regions, involved in action-outcome learning, stimulus-response learning, and the motivational control of action by reward-associated cues. Integration of dopamine (DA) and glutamate neurotransmission within these striatal sub-regions is hypothesized to enable learning and action control through its role in shaping synaptic plasticity and cellular excitability. The extracellular signal regulated kinase (ERK) appears to be particularly important for reward-based learning and action control due to its sensitivity to combined DA and glutamate receptor activation and its involvement in a range of cellular functions. ERK activation in striatal neurons is proposed to have a dual role in both the learning and performance factors that contribute to instrumental conditioning through its regulation of plasticity-related transcription factors and its modulation of intrinsic cellular excitability. Furthermore, perturbation of ERK activation by drugs of abuse may give rise to behavioral disorders such as addiction.

A critical review of habit learning and the Basal Ganglia

The current paper briefly outlines the historical development of the concept of habit learning and discusses its relationship to the basal ganglia. Habit learning has been studied in many different fields of neuroscience using different species, tasks, and methodologies, and as a result it has taken on a wide range of definitions from these various perspectives. We identify five common but not universal, definitional features of habit learning: that it is inflexible, slow or incremental, unconscious, automatic, and insensitive to reinforcer devaluation. We critically evaluate for each of these how it has been defined, its utility for research in both humans and non-human animals, and the evidence that it serves as an accurate description of basal ganglia function. In conclusion, we propose a multi-faceted approach to habit learning and its relationship to the basal ganglia, emphasizing the need for formal definitions that will provide directions for future research.

Individual differences in substance dependence: at the intersection of brain, behaviour and cognition

Recent theories of drug dependence propose that the transition from occasional recreational substance use to harmful use and dependence results from the impact of disrupted midbrain dopamine signals for reinforcement learning on frontal brain areas that implement cognitive control and decision-making. We investigated this hypothesis in humans using electrophysiological and behavioral measures believed to assay the integrity of midbrain dopamine system and its neural targets. Our investigation revealed two groups of dependent individuals, one characterized by disrupted dopamine-dependent reward learning and the other by disrupted error learning associated with depression-proneness. These results highlight important neurobiological and behavioral differences between two classes of dependent users that can inform the development of individually tailored treatment programs.

Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI

The frontal lobes may be organized hierarchically such that more rostral frontal regions modulate cognitive control operations in caudal regions. In our companion paper (Frank MJ, Badre D. 2011. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits I: computational analysis. 22:509-526), we provide novel neural circuit and algorithmic models of hierarchical cognitive control in cortico-striatal circuits. Here, we test key model predictions using functional magnetic resonance imaging (fMRI). Our neural circuit model proposes that contextual representations in rostral frontal cortex influence the striatal gating of contextual representations in caudal frontal cortex. Reinforcement learning operates at each level, such that the system adaptively learns to gate higher order contextual information into rostral regions. Our algorithmic Bayesian "mixture of experts" model captures the key computations of this neural model and provides trial-by-trial estimates of the learner's latent hypothesis states. In the present paper, we used these quantitative estimates to reanalyze fMRI data from a hierarchical reinforcement learning task reported in Badre D, Kayser AS, D'Esposito M. 2010. Frontal cortex and the discovery of abstract action rules. Neuron. 66:315--326. Results validate key predictions of the models and provide evidence for an individual cortico-striatal circuit for reinforcement learning of hierarchical structure at a specific level of policy abstraction. These findings are initially consistent with the proposal that hierarchical control in frontal cortex may emerge from interactions among nested cortico-striatal circuits at different levels of abstraction.

Dopaminergic Balance between Reward Maximization and Policy Complexity

Previous reinforcement-learning models of the basal ganglia network have highlighted the role of dopamine in encoding the mismatch between prediction and reality. Far less attention has been paid to the computational goals and algorithms of the main-axis (actor). Here, we construct a top-down model of the basal ganglia with emphasis on the role of dopamine as both a reinforcement learning signal and as a pseudo-temperature signal controlling the general level of basal ganglia excitability and motor vigilance of the acting agent. We argue that the basal ganglia endow the thalamic-cortical networks with the optimal dynamic tradeoff between two constraints: minimizing the policy complexity (cost) and maximizing the expected future reward (gain). We show that this multi-dimensional optimization processes results in an experience-modulated version of the softmax behavioral policy. Thus, as in classical softmax behavioral policies, probability of actions are selected according to their estimated values and the pseudo-temperature, but in addition also vary according to the frequency of previous choices of these actions. We conclude that the computational goal of the basal ganglia is not to maximize cumulative (positive and negative) reward. Rather, the basal ganglia aim at optimization of independent gain and cost functions. Unlike previously suggested single-variable maximization processes, this multi-dimensional optimization process leads naturally to a softmax-like behavioral policy. We suggest that beyond its role in the modulation of the efficacy of the cortico-striatal synapses, dopamine directly affects striatal excitability and thus provides a pseudo-temperature signal that modulates the tradeoff between gain and cost. The resulting experience and dopamine modulated softmax policy can then serve as a theoretical framework to account for the broad range of behaviors and clinical states governed by the basal ganglia and dopamine systems.

Altered probabilistic learning and response biases in schizophrenia: behavioral evidence and neurocomputational modeling

OBJECTIVE

Patients with schizophrenia (SZ) show reinforcement learning impairments related to both the gradual/procedural acquisition of reward contingencies, and the ability to use trial-to-trial feedback to make rapid behavioral adjustments.

METHOD

We used neurocomputational modeling to develop plausible mechanistic hypotheses explaining reinforcement learning impairments in individuals with SZ. We tested the model with a novel Go/NoGo learning task in which subjects had to learn to respond or withhold responses when presented with different stimuli associated with different probabilities of gains or losses in points. We analyzed data from 34 patients and 23 matched controls, characterizing positive- and negative-feedback-driven learning in both a training phase and a test phase.

RESULTS

Consistent with simulations from a computational model of aberrant dopamine input to the basal ganglia patients, patients with SZ showed an overall increased rate of responding in the training phase, together with reduced response-time acceleration to frequently rewarded stimuli across training blocks, and a reduced relative preference for frequently rewarded training stimuli in the test phase. Patients did not differ from controls on measures of procedural negative-feedback-driven learning, although patients with SZ exhibited deficits in trial-to-trial adjustments to negative feedback, with these measures correlating with negative symptom severity.

CONCLUSIONS

These findings support the hypothesis that patients with SZ have a deficit in procedural "Go" learning, linked to abnormalities in DA transmission at D1-type receptors, despite a "Go bias" (increased response rate), potentially related to excessive tonic dopamine. Deficits in trial-to-trial reinforcement learning were limited to a subset of patients with SZ with severe negative symptoms, putatively stemming from prefrontal cortical dysfunction.

Computational models of motivated action selection in corticostriatal circuits

Computational models of the basal ganglia have matured and received increasing attention over the last decade. This article reviews some of the theoretical advances offered by these models, focusing on motor and cognitive action selection, learning, and the interaction between multiple corticostriatal circuits in selection and learning.

Adaptation, expertise, and giftedness: towards an understanding of cortical, subcortical, and cerebellar network contributions

Current cortico-centric models of cognition lack a cohesive neuroanatomic framework that sufficiently considers overlapping levels of function, from "pathological" through "normal" to "gifted" or exceptional ability. While most cognitive theories presume an evolutionary context, few actively consider the process of adaptation, including concepts of neurodevelopment. Further, the frequent co-occurrence of "gifted" and "pathological" function is difficult to explain from a cortico-centric point of view. This comprehensive review paper proposes a framework that includes the brain's vertical organization and considers "giftedness" from an evolutionary and neurodevelopmental vantage point. We begin by discussing the current cortico-centric model of cognition and its relationship to intelligence. We then review an integrated, dual-tiered model of cognition that better explains the process of adaptation by simultaneously allowing for both stimulus-based processing and higher-order cognitive control. We consider the role of the basal ganglia within this model, particularly in relation to reward circuitry and instrumental learning. We review the important role of white matter tracts in relation to speed of adaptation and development of behavioral mastery. We examine the cerebellum's critical role in behavioral refinement and in cognitive and behavioral automation, particularly in relation to expertise and giftedness. We conclude this integrated model of brain function by considering the savant syndrome, which we believe is best understood within the context of a dual-tiered model of cognition that allows for automaticity in adaptation as well as higher-order executive control.

Levodopa and the feedback process on set-shifting in Parkinson's disease

OBJECTIVE

To study the interaction between levodopa and the feedback process on set-shifting in Parkinson's disease (PD).

METHODS

Functional magnetic resonance imaging (fMRI) studies were performed on 13 PD subjects and 17 age-matched healthy controls while they performed a modified card-sorting task. Experimental time periods were defined based on the types of feedback provided. PD subjects underwent the fMRI experiment twice, once during "off" medication (PDoff) and again after levodopa replacement (PDon).

RESULTS

Compared with normal subjects, the cognitive processing times were prolonged in PDoff but not in PDon subjects during learning through positive outcomes. The ability to set-shift through negative outcomes was not affected in PD subjects, even when "off" medication. Intergroup comparisons showed the lateral prefrontal cortex was deactivated in PDoff subjects during positive feedback learning, especially following internal feedback cues. The cortical activations were increased in the posterior brain regions in PDoff subjects following external feedback learning, especially when negative feedback cues were provided. Levodopa replacement did not completely restore the activation patterns in PD subjects to normal although activations in the corticostriatal loops were restored.

CONCLUSION

PD subjects showed differential ability to set-shift, depending on the dopamine status as well as the types of feedback cues provided. PD subjects had difficulty performing set-shift tasks through positive outcomes when "off" medication, and showed improvement after levodopa replacement. The ability to set-shift through negative feedback was not affected in PD subjects even when "off" medication, possibly due to compensatory changes outside the nigrostriatal dopaminergic pathway.

The arbitration-extension hypothesis: a hierarchical interpretation of the functional organization of the Basal Ganglia

Based on known anatomy and physiology, we present a hypothesis where the basal ganglia motor loop is hierarchically organized in two main subsystems: the arbitration system and the extension system. The arbitration system, comprised of the subthalamic nucleus, globus pallidus, and pedunculopontine nucleus, serves the role of selecting one out of several candidate actions as they are ascending from various brain stem motor regions and aggregated in the centromedian thalamus or descending from the extension system or from the cerebral cortex. This system is an action-input/action-output system whose winner-take-all mechanism finds the strongest response among several candidates to execute. This decision is communicated back to the brain stem by facilitating the desired action via cholinergic/glutamatergic projections and suppressing conflicting alternatives via GABAergic connections. The extension system, comprised of the striatum and, again, globus pallidus, can extend the repertoire of responses by learning to associate novel complex states to certain actions. This system is a state-input/action-output system, whose organization enables it to encode arbitrarily complex Boolean logic rules using striatal neurons that only fire given specific constellations of inputs (Boolean AND) and pallidal neurons that are silenced by any striatal input (Boolean OR). We demonstrate the capabilities of this hierarchical system by a computational model where a simulated generic “animal” interacts with an environment by selecting direction of movement based on combinations of sensory stimuli, some being appetitive, others aversive or neutral. While the arbitration system can autonomously handle conflicting actions proposed by brain stem motor nuclei, the extension system is required to execute learned actions not suggested by external motor centers. Being precise in the functional role of each component of the system, this hypothesis generates several readily testable predictions.

From reinforcement learning models to psychiatric and neurological disorders

Over the last decade and a half, reinforcement learning models have fostered an increasingly sophisticated understanding of the functions of dopamine and cortico-basal ganglia-thalamo-cortical (CBGTC) circuits. More recently, these models, and the insights that they afford, have started to be used to understand important aspects of several psychiatric and neurological disorders that involve disturbances of the dopaminergic system and CBGTC circuits. We review this approach and its existing and potential applications to Parkinson's disease, Tourette's syndrome, attention-deficit/hyperactivity disorder, addiction, schizophrenia and preclinical animal models used to screen new antipsychotic drugs. The approach's proven explanatory and predictive power bodes well for the continued growth of computational psychiatry and computational neurology.

Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia

BACKGROUND

Negative symptoms are core features of schizophrenia (SZ); however, the cognitive and neural basis for individual negative symptom domains remains unclear. Converging evidence suggests a role for striatal and prefrontal dopamine in reward learning and the exploration of actions that might produce outcomes that are better than the status quo. The current study examines whether deficits in reinforcement learning and uncertainty-driven exploration predict specific negative symptom domains.

METHODS

We administered a temporal decision-making task, which required trial-by-trial adjustment of reaction time to maximize reward receipt, to 51 patients with SZ and 39 age-matched healthy control subjects. Task conditions were designed such that expected value (probability × magnitude) increased, decreased, or remained constant with increasing response times. Computational analyses were applied to estimate the degree to which trial-by-trial responses are influenced by reinforcement history.

RESULTS

Individuals with SZ showed impaired Go learning but intact NoGo learning relative to control subjects. These effects were most pronounced in patients with higher levels of negative symptoms. Uncertainty-based exploration was substantially reduced in individuals with SZ and selectively correlated with clinical ratings of anhedonia.

CONCLUSIONS

Schizophrenia patients, particularly those with high negative symptoms, failed to speed reaction times to increase positive outcomes and showed reduced tendency to explore when alternative actions could lead to better outcomes than the status quo. Results are interpreted in the context of current computational, genetic, and pharmacological data supporting the roles of striatal and prefrontal dopamine in these processes.

Opponency revisited: competition and cooperation between dopamine and serotonin

Affective valence lies on a spectrum ranging from punishment to reward. The coding of such spectra in the brain almost always involves opponency between pairs of systems or structures. There is ample evidence for the role of dopamine in the appetitive half of this spectrum, but little agreement about the existence, nature, or role of putative aversive opponents such as serotonin. In this review, we consider the structure of opponency in terms of previous biases about the nature of the decision problems that animals face, the conflicts that may thus arise between Pavlovian and instrumental responses, and an additional spectrum joining invigoration to inhibition. We use this analysis to shed light on aspects of the role of serotonin and its interactions with dopamine.

Neurogenetics and pharmacology of learning, motivation, and cognition

Many of the individual differences in cognition, motivation, and learning-and the disruption of these processes in neurological conditions-are influenced by genetic factors. We provide an integrative synthesis across human and animal studies, focusing on a recent spate of evidence implicating a role for genes controlling dopaminergic function in frontostriatal circuitry, including COMT, DARPP-32, DAT1, DRD2, and DRD4. These genetic effects are interpreted within theoretical frameworks developed in the context of the broader cognitive and computational neuroscience literature, constrained by data from pharmacological, neuroimaging, electrophysiological, and patient studies. In this framework, genes modulate the efficacy of particular neural computations, and effects of genetic variation are revealed by assays designed to be maximally sensitive to these computations. We discuss the merits and caveats of this approach and outline a number of novel candidate genes of interest for future study.

EEG oscillatory patterns are associated with error prediction during music performance and are altered in musician's dystonia

Skilled performance requires the ability to monitor ongoing behavior, detect errors in advance and modify the performance accordingly. The acquisition of fast predictive mechanisms might be possible due to the extensive training characterizing expertise performance. Recent EEG studies on piano performance reported a negative event-related potential (ERP) triggered in the ACC 70 ms before performance errors (pitch errors due to incorrect keypress). This ERP component, termed pre-error related negativity (pre-ERN), was assumed to reflect processes of error detection in advance. However, some questions remained to be addressed: (i) Does the electrophysiological marker prior to errors reflect an error signal itself or is it related instead to the implementation of control mechanisms? (ii) Does the posterior frontomedial cortex (pFMC, including ACC) interact with other brain regions to implement control adjustments following motor prediction of an upcoming error? (iii) Can we gain insight into the electrophysiological correlates of error prediction and control by assessing the local neuronal synchronization and phase interaction among neuronal populations? (iv) Finally, are error detection and control mechanisms defective in pianists with musician's dystonia (MD), a focal task-specific dystonia resulting from dysfunction of the basal ganglia-thalamic-frontal circuits? Consequently, we investigated the EEG oscillatory and phase synchronization correlates of error detection and control during piano performances in healthy pianists and in a group of pianists with MD. In healthy pianists, the main outcomes were increased pre-error theta and beta band oscillations over the pFMC and 13-15 Hz phase synchronization, between the pFMC and the right lateral prefrontal cortex, which predicted corrective mechanisms. In MD patients, the pattern of phase synchronization appeared in a different frequency band (6-8 Hz) and correlated with the severity of the disorder. The present findings shed new light on the neural mechanisms, which might implement motor prediction by means of forward control processes, as they function in healthy pianists and in their altered form in patients with MD.

Inputs to the dorsal striatum of the mouse reflect the parallel circuit architecture of the forebrain

The basal ganglia play a critical role in the regulation of voluntary action in vertebrates. Our understanding of the function of the basal ganglia relies heavily upon anatomical information, but continued progress will require an understanding of the specific functional roles played by diverse cell types and their connectivity. An increasing number of mouse lines allow extensive identification, characterization, and manipulation of specified cell types in the basal ganglia. Despite the promise of genetically modified mice for elucidating the functional roles of diverse cell types, there is relatively little anatomical data obtained directly in the mouse. Here we have characterized the retrograde labeling obtained from a series of tracer injections throughout the dorsal striatum of adult mice. We found systematic variations in input along both the medial–lateral and anterior–posterior neuraxes in close agreement with canonical features of basal ganglia anatomy in the rat. In addition to the canonical features we have provided experimental support for the importance of non-canonical inputs to the striatum from the raphe nuclei and the amygdala. To look for organization at a finer scale we have analyzed the correlation structure of labeling intensity across our entire dataset. Using this analysis we found substantial local heterogeneity within the large-scale order. From this analysis we conclude that individual striatal sites receive varied combinations of cortical and thalamic input from multiple functional areas, consistent with some earlier studies in the rat that have suggested the presence of a combinatorial map.

Contributions of ERK signaling in the striatum to instrumental learning and performance

The striatum is critical for learning and decision making; however, the molecular mechanisms that govern striatum function are not fully understood. The extracellular signal regulated kinase (ERK) cascade is an important signaling pathway that underlies synaptic plasticity, cellular excitability, learning and arousal. This review focuses on the role of ERK signaling in striatum function. ERK is activated in the striatum by coordinated dopamine and glutamate receptor signaling, where it underlies corticostriatal synaptic plasticity and influences striatal cell excitability. ERK activation in the dorsal striatum is necessary for action-outcome learning and performance of goal-directed actions. In the ventral striatum, ERK is necessary for the motivating effects of reward-associated stimuli on instrumental performance. Dysregulation of ERK signaling in the striatum by repeated drug exposure contributes to the development of addictive behavior. These results highlight the importance of ERK signaling in the striatum as a critical substrate for learning and as a regulator of ongoing behavior. Furthermore, they suggest that ERK may be a suitable target for therapeutics to treat disorders of learning and decision making that arise from compromised striatum function.

Focus on the positive: computational simulations implicate asymmetrical reward prediction error signals in childhood attention-deficit/hyperactivity disorder

A number of hypotheses have suggested that the principal neurological dysfunction responsible for the behavioural symptoms associated with Attention-Deficit/Hyperactive Disorder (ADHD) is likely rooted in abnormal phasic signals coded by the firing rate of midbrain dopamine neurons. We present a formal investigation of the impact atypical phasic dopamine signals have on behaviour by applying a TD(λ) reinforcement learning model to simulations of operant conditioning tasks that have been argued to quantify the hyperactive, inattentive and impulsive behaviour associated with ADHD. The results presented here suggest that asymmetrically effective dopamine signals encoded by a punctate increase or decrease in dopamine levels provide the best account for the behaviour of children with ADHD as well as an animal model of ADHD, the spontaneously hypertensive rat (SHR). The biological sources of this asymmetry are considered, as are other computational models of ADHD.

Dissociable responses to punishment in distinct striatal regions during reversal learning

Adaptive behavior depends on the ability to flexibly alter our choices in response to changes in reward and punishment contingencies. One brain region frequently implicated in such behavior is the striatum. However, this region is functionally diverse and there are a number of apparent inconsistencies across previous studies. For instance, how can significant BOLD responses in the ventral striatum during punishment-based reversal learning be reconciled with the frequently demonstrated role of the ventral striatum in reward processing? Here we attempt to address this question by separately examining BOLD responses during reversal learning driven by reward and during reversal learning driven by punishment. We demonstrate simultaneous valence-specific and valence-nonspecific signals in the striatum, with the posterior dorsal striatum responding only to unexpected reward, and the anterior ventral striatum responding to both unexpected punishment as well as unexpected reward. These data help to reconcile conflicting findings from previous studies by showing that distinct regions of the striatum exhibit dissociable responses to punishment during reversal learning.

Markers of basal ganglia dysfunction and conversion to psychosis: neurocognitive deficits and dyskinesias in the prodromal period

BACKGROUND

Movement abnormalities and cognitive deficits may represent external markers of an underlying neural process linked with the early etiology of psychosis. As basal ganglia function plays a governing role in both movement and cognitive processes, an understanding of the relationship between these phenomena stands to inform etiologic conceptualizations of vulnerability and psychotic disorders.

METHODS

In this investigation, trained raters coded movement abnormalities in videotapes from structured interviews of adolescents and young adults with a prodromal risk syndrome (n = 90). The participants were administered a neuropsychological battery including measures of verbal comprehension, perceptual organization, immediate/delayed auditory memory, and an estimate of full-scale intelligence quotient. Diagnostic status was followed for a 2-year period utilizing structured clinical interviews, during which time 24 high-risk participants (26.66%) converted to an Axis I psychotic disorder.

RESULTS

Elevated dyskinetic movements in the upper-body region were correlated with deficits in domains of verbal comprehension, perceptual organization, and both immediate and delayed auditory memory. Further, discriminant function analyses indicated that baseline movement abnormalities and neurocognitive deficits significantly classified those high-risk participants who would eventually convert to a psychotic disorder (72.3%).

CONCLUSIONS

Results support a common cortico-striato-pallido-thalamic circuit irregularity, underlying both movement abnormalities and cognitive deficits in individuals at high risk for psychosis. Models incorporating external markers of progressive basal ganglia dysfunction may enhance detection and preventive intervention for those high-risk individuals most in need of treatment.

General cognitive principles for learning structure in time and space

How are hierarchically structured sequences of objects, events or actions learned from experience and represented in the brain? When several streams of regularities present themselves, which will be learned and which ignored? Can statistical regularities take effect on their own, or are additional factors such as behavioral outcomes expected to influence statistical learning? Answers to these questions are starting to emerge through a convergence of findings from naturalistic observations, behavioral experiments, neurobiological studies, and computational analyses and simulations. We propose that a small set of principles are at work in every situation that involves learning of structure from patterns of experience and outline a general framework that accounts for such learning.

Functional disconnection of the substantia nigra pars compacta from the pedunculopontine nucleus impairs learning of a conditioned avoidance task

The pedunculopontine tegmental nucleus (PPTg) targets nuclei in the basal ganglia, including the substantia nigra pars compacta (SNc), in which neuronal loss occurs in Parkinson's disease, a condition in which patients show cognitive as well as motor disturbances. Partial loss and functional abnormalities of neurons in the PPTg are also associated with Parkinson's disease. We hypothesized that the interaction of PPTg and SNc might be important for cognitive impairments and so investigated whether disrupting the connections between the PPTg and SNc impaired learning of a conditioned avoidance response (CAR) by male Wistar rats. The following groups were tested: PPTg unilateral; SNc unilateral; PPTg-SNc ipsilateral (ipsilateral lesions in PPTg and SNc); PPTg-SNc contralateral (contralateral lesions in PPTg and SNc); sham lesions (of each type). SNc lesions were made with 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine HCl (MPTP, 0.6micromol); PPTg lesions with ibotenate (24nmol). After recovery, all rats underwent 50-trial sessions of 2-way active avoidance conditioning for 3 consecutive days. Rats with unilateral lesions in PPTg or SNc learnt this, however rats with contralateral (but not ipsilateral) combined lesions in both structures presented no sign of learning. This effect was not likely to be due to sensorimotor impairment because lesions did not affect reaction time to the tone or footshock during conditioning. However, an increased number of non-responses were observed in the rats with contralateral lesions. The results support the hypothesis that a functional interaction between PPTg and SNc is needed for CAR learning and performance.

Frontal cortex and the discovery of abstract action rules

Although we often encounter circumstances with which we have no prior experience, we rapidly learn how to behave in these novel situations. Such adaptive behavior relies on abstract behavioral rules that are generalizable, rather than concrete rules mapping specific cues to specific responses. Although the frontal cortex is known to support concrete rule learning, less well understood are the neural mechanisms supporting the acquisition of abstract rules. Here, we use a reinforcement learning paradigm to demonstrate that more anterior regions along the rostro-caudal axis of frontal cortex support rule learning at higher levels of abstraction. Moreover, these results indicate that when humans confront new rule learning problems, this rostro-caudal division of labor supports the search for relationships between context and action at multiple levels of abstraction simultaneously.

Firing properties and functional connectivity of substantia nigra pars compacta neurones recorded with a multi-electrode array in vitro

Dopamine (DA) neurones of the substantia nigra pars compacta (SNc) are involved in a wide variety of functions, including motor control and reward-based learning. In order to gain new insights into the firing properties of neuronal ensembles in the SNc, we recorded extracellular single units from spontaneously active neurones, using a multi-electrode array (MEA) device in midbrain slices. The majority of neurones (50.21%) had a low firing frequency (1-3 Hz) and a stable pacemaker-like pattern, while others (44.84%) were irregular, but still firing at a low rate. The remaining population (4.95%) comprised neurones with a regular higher firing rate (5-10 Hz). High rate neurones, on the whole, were insensitive to DA (30 mum), while low rate neurones were mostly inhibited by DA, although responding either with a prominent or a weak inhibition. However, we recorded low rate regular neurones that were insensitive to DA, or irregular low rate neurones excited by DA. Interestingly, we found pairs of active neurones (12.10 +/- 3.14%) with a significant proportion of spikes occurring synchronously. Moreover, the crosscorrelation probability in each pair tended to increase in response to DA. In conclusion, MEA recordings in midbrain slices reveal a much more complex picture than previously reported with regard to the firing pattern and DA sensitivity of spontaneously active SNc neurones. Moreover, the study opens new prospectives for the in vitro investigation of functional connectivity in the midbrain dopaminergic system, thus proposing new targets for the pharmacological treatment of DA-dependent neurological disorders.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

Neurocomputational models of motor and cognitive deficits in Parkinson's disease

We review the contributions of biologically constrained computational models to our understanding of motor and cognitive deficits in Parkinson's disease (PD). The loss of dopaminergic neurons innervating the striatum in PD, and the well-established role of dopamine (DA) in reinforcement learning (RL), enable neural network models of the basal ganglia (BG) to derive concrete and testable predictions. We focus in this review on one simple underlying principle - the notion that reduced DA increases activity and causes long-term potentiation in the indirect pathway of the BG. We show how this theory can provide a unified account of diverse and seemingly unrelated phenomena in PD including progressive motor degeneration as well as cognitive deficits in RL, decision making and working memory. DA replacement therapy and deep brain stimulation can alleviate some aspects of these impairments, but can actually introduce negative effects such as motor dyskinesias and cognitive impulsivity. We discuss these treatment effects in terms of modulation of specific mechanisms within the computational framework. In addition, we review neurocomputational interpretations of increased impulsivity in the face of response conflict in patients with deep-brain-stimulation.

The role of dopamine in modulating the structure and function of striatal circuits

Dopamine (DA) is a key regulator of action selection and associative learning. The striatum has long been thought to be a major locus of DA action in this process. Although all striatal cell types express G protein-coupled receptors for DA, the effects of DA on principal medium spiny neurons (MSNs) understandably have received the most attention. In the two principal classes of MSN, DA receptor expression diverges, with striatonigral MSNs robustly expressing D(1) receptors and striatopallidal MSNs expressing D(2) receptors. In the last couple of years, our understanding of how these receptors and the intracellular signalling cascades that they couple to modulate dendritic physiology and synaptic plasticity has rapidly expanded, fuelled in large measure by the development of new optical and genetic tools. These tools also have enabled a rapid expansion of our understanding of the striatal adaptations in models of Parkinson's disease. This chapter highlights some of the major advances in these areas.

Dopamine precursor depletion improves punishment prediction during reversal learning in healthy females but not males

INTRODUCTION

The neurotransmitter dopamine has frequently been implicated in reward processing but is also, increasingly, implicated in punishment processing. We have previously shown that both patients with Parkinson's disease and healthy individuals with low dopamine (DA) synthesis are better at reversal learning based on punishment than reward. Here, we extend these prior findings by examining the effects of artificially reducing DA synthesis in healthy individuals performing this previously employed task.

METHODS

In a double-blind, placebo-controlled crossover design, we applied the acute tyrosine and phenylalanine depletion (ATPD) procedure to reduce global DA synthesis in 15 female and 14 male subjects. Each subject performed the reward- and punishment-based reversal-learning paradigm.

RESULTS

There was a significant three-way interaction between ATPD, the valence of the outcome signalling reversal and the gender of the participants. Examination of punishment and reward-based reversals separately revealed that this was driven by a significant improvement in punishment processing in female but not male subjects following DA depletion.

CONCLUSIONS

Reducing DA synthesis in healthy individuals shifted sensitivity of performance from reward to punishment processing. Gender differences in DA synthesis might underlie the selectivity of this effect to female subjects. Such gender biases may go some way towards explaining the gender biases in certain psychiatric disorders such as depression and Parkinson's disease.

Dopaminergic drugs modulate learning rates and perseveration in Parkinson's patients in a dynamic foraging task

Making appropriate choices often requires the ability to learn the value of available options from experience. Parkinson's disease is characterized by a loss of dopamine neurons in the substantia nigra, neurons hypothesized to play a role in reinforcement learning. Although previous studies have shown that Parkinson's patients are impaired in tasks involving learning from feedback, they have not directly tested the widely held hypothesis that dopamine neuron activity specifically encodes the reward prediction error signal used in reinforcement learning models. To test a key prediction of this hypothesis, we fit choice behavior from a dynamic foraging task with reinforcement learning models and show that treatment with dopaminergic drugs alters choice behavior in a manner consistent with the theory. More specifically, we found that dopaminergic drugs selectively modulate learning from positive outcomes. We observed no effect of dopaminergic drugs on learning from negative outcomes. We also found a novel dopamine-dependent effect on decision making that is not accounted for by reinforcement learning models: perseveration in choice, independent of reward history, increases with Parkinson's disease and decreases with dopamine therapy.

Dopamine and synaptic plasticity in dorsal striatal circuits controlling action selection

The striatum is thought to play a central role in learning how to choose acts that lead to reward and avoid punishment. Dopamine-dependent modification of striatal synapses in the action selection circuitry has long been thought to be a key step toward this type of learning. The development of new genetic and optical tools has pushed this field forward in the last couple of years, demanding a re-evaluation of models of how experience controls dopamine-dependent synaptic plasticity and how disease states like Parkinson's disease affect the striatal circuitry.

Robust pacemaking in substantia nigra dopaminergic neurons

Dopaminergic neurons of the substantia nigra pars compacta are autonomous pacemakers. This activity is responsible for the sustained release of dopamine necessary for the proper functioning of target structures, such as the striatum. Somatodendritic L-type Ca2+ channels have long been viewed as important, if not necessary, for this activity. The studies reported here challenge this viewpoint. Using a combination of optical and electrophysiological approaches in brain slices, it was found that antagonism of L-type Ca2+ channel effectively stopped dendritic Ca2+ oscillations but left autonomous pacemaking unchanged. Moreover, damping intracellular Ca2+ oscillations with exogenous buffer had little effect on pacemaking rate. Although not necessary for pacemaking, L-type channels helped support pacemaking when challenged with cationic channel blockers. Simulations suggested that the insensitivity to antagonism of L-type channels reflected the multichannel nature of the pacemaking process. The robustness of pacemaking underscores its biological importance and provides a framework for understanding how therapeutics targeting L-type Ca2+ channels might protect dopaminergic neurons in Parkinson's disease without compromising their function.

Genetic contributions to avoidance-based decisions: striatal D2 receptor polymorphisms

Individuals differ in their tendencies to seek positive decision outcomes or to avoid negative ones. At the neurobiological level, our model suggests that phasic changes in dopamine support learning to reinforce good decisions via striatal D1 receptors, and to avoid maladaptive choices via striatal D2 receptors. Accordingly, in a previous study individual differences in positive and negative learning were strongly modulated by two genetic polymorphisms factors related to striatal D1 and D2 function, respectively. Nevertheless, whereas the role for dopamine in positive learning is relatively well accepted, that in learning to avoid negative outcomes is more controversial. Here we further explore D2-receptor-related genetic contributions to probabilistic avoidance in humans, in light of recent data showing that particular DRD2 polymorphisms are associated with functional modulation of receptor expression [Zhang Y, Bertolino A, Fazio L, Blasi G, Rampino A, Romano R, Lee M-LT, Xiao T, Papp A, Wang D, Sadée W (2007) Polymorphisms in human dopamine d2 receptor gene affect gene expression, splicing, and neuronal activity during working memory. Proc Natl Acad Sci U S A 104(51):20552-20557]. We find that a promoter polymorphism rs12364283 associated with transcription and D2 receptor density was strongly and selectively predictive of avoidance-based decisions. Two further polymorphisms (rs2283265 and rs1076560) associated with relatively reduced presynaptic relative to postsynaptic D2 receptor expression were predictive of relative impairments in negative compared to positive decisions. These previously undocumented effects of DRD2 polymorphisms were largely independent of those we reported previously for the C957T polymorphism (rs6277) associated with striatal D2 density. In contrast, effects of the commonly studied Taq1A polymorphism on reinforcement-based decisions were due to indirect association with C957T. Taken together these findings suggest multiple D2-dependent genetic mechanisms contributing to avoidance. We discuss these effects in the context of neurocomputational models of reinforcement leaning in the basal ganglia.