Shaping of motor responses by incentive values through the basal ganglia

A whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots

Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information. In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.

Rat Paraventricular Neurons Encode Predictive and Incentive Information of Reward Cues

The paraventricular nucleus of the thalamus (PVT) has been implicated in cue-induced motivated behaviors. Although reward-associated cues (conditioned stimuli, CSs) contain different types of information including predictive information of future reward delivery and incentive (motivational) value of the reward, it remains unknown whether PVT neurons represent predictive and incentive information of CSs. It is suggested that neural activity just after the onset of CSs (early activity) and that just before reward delivery (late activity) might more strongly represent predictive and incentive information, respectively. In this study, rats were trained to lick a tube, which was protruded close to their mouth just after a CS, to obtain a reward (sucrose or water) (cue-induced licking task). Auditory and visual CSs were used: each elemental cue (CS) predicted reward or non-reward outcome, while simultaneous presentation of the two elemental cues (configural cues) predicted the opposite reward outcome. We recorded PVT neurons in the cue-induced licking task, and report that half of the CS-responsive PVT neurons responded selectively to the CSs predicting reward outcome regardless of physical property of the cues (CS+-selective). In addition, the early activity of the CS+-selective neurons discriminated reward/non-reward association (predictive information) and was less sensitive to reward value and motivation reflected by lick latency (incentive information), while the late activity of the CS+-selective neurons was correlated with reward value and motivation rather than reward/non-reward association. Early and late population activity of the CS+-selective neurons also represented predictive and incentive information of the CSs, respectively. On the other hand, activity of more than half of the PVT neurons was correlated with individual licking during licking to acquire reward. Taken together, the results suggest that the PVT neurons engage in different neural processes involved in cue-induced motivated behaviors: CS encoding to determine reward availability and form motivation for reward-seeking behavior, and hedonic mouth movements during reward consumption.

Towards Automaticity in Reinforcement Learning: A Model-Based Functional Magnetic Resonance Imaging Study

Introduction

Previous studies showed that over the course of learning many neurons in the medial prefrontal cortex adapt their firing rate towards the options with highest predicted value reward but it was showed that during later learning trials the brain switches to a more automatic processing mode governed by the basal ganglia. Based on this evidence, we hypothesized that during the early learning trials the predicted values of chosen options will be coded by a goal directed system in the medial frontal cortex but during the late trials the predicted values will be coded by the habitual learning system in the dorsal striatum.

Methods

In this study, using a 3 Tesla functional magnetic resonance imaging scanner (fMRI), blood oxygen level dependent signal (BOLD) data was collected whilst participants (N=12) performed a reinforcement learning task. The task consisted of instrumental conditioning trials wherein each trial a participant choose one of the two available options in order to win or avoid losing money. In addition to that, depending on the experimental condition, participants received either monetary reward (gain money), monetary penalty (lose money) or neural outcome.

Results

Using model-based analysis for functional magnetic resonance imaging (fMRI) event related designs; region of interest (ROI) analysis was performed to nucleus accumbens, medial frontal cortex, caudate nucleus, putamen and globus pallidus internal and external segments. In order to compare the difference in brain activity for early (goal directed) versus late learning (habitual, automatic) trials, separate ROI analyses were performed for each anatomical sub-region. For the reward condition, we found significant activity in the medial frontal cortex (p<0.05) only for early learning trials but activity is shifted to bilateral putamen (p<0.05) during later trials. However, for the loss condition no significant activity was found for early trials except globus pallidus internal segment showed a significant activity (p<0.05) for later trials.

Conclusion

We found that during reinforcement learning activation in the brain shifted from the medial frontal regions to dorsal regions of the striatum. These findings suggest that there are two separable (early goal directed and late habitual) learning systems in the brain.

A Computational Model of Dual Competition between the Basal Ganglia and the Cortex

We propose a model that includes interactions between the cortex, the basal ganglia (BG), and the thalamus based on a dual competition. We hypothesize that the striatum, the subthalamic nucleus (STN), the internal globus pallidus (GPi), the thalamus, and the cortex are involved in closed feedback loops through the hyperdirect and direct pathways. These loops support a competition process that results in the ability of BG to make a cognitive decision followed by a motor one. Considering lateral cortical interactions, another competition takes place inside the cortex allowing the latter to make a cognitive and a motor decision. We show how this dual competition endows the model with two regimes. One is driven by reinforcement learning and the other by Hebbian learning. The final decision is made according to a combination of these two mechanisms with a gradual transfer from the former to the latter. We confirmed these theoretical results on primates (Macaca mulatta) using a novel paradigm predicted by the model.

Insufficient task-outcome association promotes task procrastination through a decrease of hippocampal-striatal interaction

Theories on procrastination propose that associating tasks with higher valued incentive outcomes results in less task procrastination. However, it remains unknown how representation of incentive outcomes and task-outcome association are mediated by the human brain. Using event-related functional magnetic resonance imaging, we scanned human participants while they were thinking about both tasks and the incentive outcomes each task can yield in an unconstrained way. Results showed that tasks that are more likely to be procrastinated are associated with less value in incentive outcomes. Interestingly, procrastination was more likely if it was more difficult for participants to associate a task with its valued incentives when thinking about the task (i.e., the decreased task-outcome association). On the neural level, higher value of rewarding outcomes was correlated with increased putamen activations, which further negatively predicted task procrastination. On the other hand, when participants were associating tasks with the incentive outcomes, the decreasing task-outcome association corresponded to decreasing activation in putamen, and a decreasing hippocampus-putamen coupling which further mediated the effect of the insufficient task-outcome association on procrastination. In particular, the current findings show that procrastination is more likely when people are less able to associate tasks with highly valued incentives, which is accompanied by reduced hippocampal-striatal interactions during task construction.

A natural history of skills

The dorsal pallium (a.k.a. cortex in mammals) makes a loop circuit with the basal ganglia and the thalamus known to control and adapt behavior but the who's who of the functional roles of these structures is still debated. Influenced by the Triune brain theory that was proposed in the early sixties, many current theories propose a hierarchical organization on the top of which stands the cortex to which the subcortical structures are subordinated. In particular, habits formation has been proposed to reflect a switch from conscious on-line control of behavior by the cortex, to a fully automated subcortical control. In this review, we propose to revalue the function of the network in light of the current experimental evidence concerning the anatomy and physiology of the basal ganglia-cortical circuits in vertebrates. We briefly review the current theories and show that they could be encompassed in a broader framework of skill learning and performance. Then, after reminding the state of the art concerning the anatomical architecture of the network and the underlying dynamic processes, we summarize the evolution of the anatomical and physiological substrate of skill learning and performance among vertebrates. We then review experimental evidence supporting for the hypothesis that the development of automatized skills relies on the BG teaching cortical circuits and is actually a late feature linked with the development of a specialized cortex or pallium that evolved in parallel in different taxa. We finally propose a minimal computational framework where this hypothesis can be explicitly implemented and tested.

Striatal action-value neurons reconsidered

It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.

Dynamic Encoding of Effort and Reward throughout the Execution of Action by External Globus Pallidus Neurons in Monkeys

Humans and animals must evaluate the costs and expected benefits of their actions to make adaptive choices. Prior studies have demonstrated the involvement of the basal ganglia in this evaluation. However, little is known about the role of the external part of the globus pallidus (GPe), which is well positioned to integrate motor and reward-related information, in this process. To investigate this role, the activity of 126 neurons was recorded in the associative and limbic parts of the GPe of two monkeys performing a behavioral task in which different levels of force were required to obtain different amounts of liquid reward. The results first revealed that the activity of associative and limbic GPe neurons could be modulated not only by cognitive and limbic but also motor information at the same time, both during a single period or during different periods throughout the trial, mainly in an independent way. Moreover, as a population, GPe neurons encoded these types of information dynamically throughout the trial, when each piece of information was the most relevant for the achievement of the action. Taken together, these results suggest that GPe neurons could be dedicated to the parallel monitoring of task parameters essential to adjusting and maintaining goal-directed behavior.

Vigor of reaching movements: reward discounts the cost of effort

Making a movement may be thought of as an economic decision in which one spends effort to acquire reward. Time discounts reward, which predicts that the magnitude of reward should affect movement vigor: we should move faster, spending greater effort, when there is greater reward at stake. Indeed, saccade peak velocities are greater and reaction-times shorter when a target is paired with reward. In this study, we focused on human reaching and asked whether movement kinematics were affected by expectation of reward. Participants made out-and-back reaching movements to one of four quadrants of a 14-cm circle. During various periods of the experiment only one of the four quadrants was paired with reward, and the transition from reward to nonreward status of a quadrant occurred randomly. Our experiment design minimized dependence of reward on accuracy, granting the subjects wide latitude in self-selecting their movement speed, amplitude, and variability. When a quadrant was paired with reward, reaching movements had a shorter reaction time, higher peak velocity, and greater amplitude. Despite this greater vigor, movements toward the rewarded quadrant suffered from less variability: both reaction times and reach kinematics were less variable when there was expectation of reward. Importantly, the effect of reward on vigor was specific to the movement component that preceded the time of reward (outward reach), not the movement component that followed it (return reach). Our results suggest that expectation of reward not only increases vigor of human reaching but also decreases its variability. NEW & NOTEWORTHY Movements may be thought of as an economic transaction where the vigor of the movement represents the effort that the brain is willing to expend to acquire a rewarding state. We show that in reaching, reward discounts the cost of effort, producing movements with shorter reaction time, higher velocity, greater amplitude, and reduced reaction-time variability. These results complement earlier observations in saccades, suggesting a common principle of economics across modalities of motor control.

Reward-Predictive Neural Activities in Striatal Striosome Compartments

The striatum has been shown to play a critical role in reward prediction. It is composed of two neurochemically and anatomically distinct compartments known as the striosomes and the matrix. The striosomes comprise only about 15% of the striatum by volume and are distributed mosaically therein. Accordingly, it has been difficult to identify striosomal neurons in electrophysiological recordings and it has been unclear whether striosomal neurons, which project to midbrain dopaminergic neurons, engage in reward prediction. In this study, we utilized a mouse line (Sepw1-NP67) selectively expressing Cre in striosomal neurons, combined with endoscopic in vivo calcium imaging to selectively record activities of striosomal neurons during an odor-conditioning task. As mice learned the task, striosomal neurons in the dorsomedial striatum (DMS) showed predictive activities to odor cues that were associated with water rewards or aversive air puffs. These activities were proportional to the expected reward or air-puff intensity. Intriguingly, repeated recordings of the same striosomal neurons over a period of weeks revealed that predictive activities were learning-stage specific. That is, these activities disappeared after continuous training. Furthermore, presentations of rewards or air puffs activated some striosomal neurons. These findings suggest that the striosomes participate in reward prediction with learning stage-specific neural ensembles, and that they also send reward and aversive signals to dopaminergic neurons.

The Basal Ganglia Do Not Select Reach Targets but Control the Urgency of Commitment

Prominent theories of decision making suggest that the basal ganglia (BG) play a causal role in deliberation between action choices. An alternative hypothesis is that deliberation occurs in cortical regions, while the BG control the speed-accuracy trade-off (SAT) between committing to a choice versus continuing to deliberate. Here, we test these hypotheses by recording activity in the internal and external segments of the globus pallidus (GPi/GPe) while monkeys perform a task dissociating the process of deliberation, the moment of commitment, and adjustment of the SAT. Our data suggest that unlike premotor and motor cortical regions, pallidal output does not contribute to the process of deliberation but instead provides a time-varying signal that controls the SAT and reflects the growing urgency to commit to a choice. Once a target is selected by cortical regions, GP activity confirms commitment to the decision and invigorates the subsequent movement.

A Biologically Plausible Architecture of the Striatum to Solve Context-Dependent Reinforcement Learning Tasks

Basal ganglia circuit is an important subcortical system of the brain thought to be responsible for reward-based learning. Striatum, the largest nucleus of the basal ganglia, serves as an input port that maps cortical information. Microanatomical studies show that the striatum is a mosaic of specialized input-output structures called striosomes and regions of the surrounding matrix called the matrisomes. We have developed a computational model of the striatum using layered self-organizing maps to capture the center-surround structure seen experimentally and explain its functional significance. We believe that these structural components could build representations of state and action spaces in different environments. The striatum model is then integrated with other components of basal ganglia, making it capable of solving reinforcement learning tasks. We have proposed a biologically plausible mechanism of action-based learning where the striosome biases the matrisome activity toward a preferred action. Several studies indicate that the striatum is critical in solving context dependent problems. We build on this hypothesis and the proposed model exploits the modularity of the striatum to efficiently solve such tasks.

Roles of Multiple Globus Pallidus Territories of Monkeys and Humans in Motivation, Cognition and Action: An Anatomical, Physiological and Pathophysiological Review

The globus pallidus (GP) communicates with widespread cortical areas that support various functions, including motivation, cognition and action. Anatomical tract-tracing studies revealed that the anteroventral GP communicates with the medial prefrontal and orbitofrontal cortices, which are involved in motivational control; the anterodorsal GP communicates with the lateral prefrontal cortex, which is involved in cognitive control; and the posterior GP communicates with the frontal motor cortex, which is involved in action control. This organization suggests that distinct subdivisions within the GP play specific roles. Neurophysiological studies examining GP neurons in monkeys during behavior revealed that the types of information coding performed within these subdivisions differ greatly. The anteroventral GP is characterized by activities related to motivation, such as reward seeking and aversive avoidance; the anterodorsal GP is characterized by activity that reflects cognition, such as goal decision and action selection; and the posterior GP is characterized by activity associated with action preparation and execution. Pathophysiological studies have shown that GABA-related substances or GP lesions result in abnormal activity in the GP, which causes site-specific behavioral and motor symptoms. The present review article discusses the anatomical organization, physiology and pathophysiology of the three major GP territories in nonhuman primates and humans.

Decision making under uncertainty in a spiking neural network model of the basal ganglia

The mechanisms of decision-making and action selection are generally thought to be under the control of parallel cortico-subcortical loops connecting back to distinct areas of cortex through the basal ganglia and processing motor, cognitive and limbic modalities of decision-making. We have used these properties to develop and extend a connectionist model at a spiking neuron level based on a previous rate model approach. This model is demonstrated on decision-making tasks that have been studied in primates and the electrophysiology interpreted to show that the decision is made in two steps. To model this, we have used two parallel loops, each of which performs decision-making based on interactions between positive and negative feedback pathways. This model is able to perform two-level decision-making as in primates. We show here that, before learning, synaptic noise is sufficient to drive the decision-making process and that, after learning, the decision is based on the choice that has proven most likely to be rewarded. The model is then submitted to lesion tests, reversal learning and extinction protocols. We show that, under these conditions, it behaves in a consistent manner and provides predictions in accordance with observed experimental data.

Neostriatal Neuronal Activity Correlates Better with Movement Kinematics under Certain Rewards

This study investigated how the activity of neostriatal neurons is related to the kinematics of movement when monkeys performed visually and vibratory cued wrist extensions and flexions. Single-unit recordings of 142/236 neostriatal neurons showed pre-movement activity (PMA) in a reaction time task with unpredictable reward. Monkeys were pseudo-randomly (75%) rewarded for correct performance. A regression model was used to determine whether the correlation between neostriatal neuronal activity and the kinematic variables (position, velocity, and acceleration) of wrist movement changes as a function of reward contingency, sensory cues, and movement direction. The coefficients of determination (CoD) representing the proportion of the variance in neuronal activity explained by the regression model on a trial by trial basis, together with their temporal occurrences (time of best regression/correlation, ToC) were compared across sensory modality, movement direction, and reward contingency. The best relationship (correlation) between neuronal activity and movement kinematic variables, given by the average coefficient of determination (CoD), was: (a) greater during trials in which rewards were certain, called "A" trials, as compared with those in which reward was uncertain called ("R") trials, (b) greater during flexion (Flex) trials as compared with extension (Ext) trials, and (c) greater during visual (VIS) cued trials than during vibratory (VIB) cued trials, for the same type of trial and the same movement direction. These results are consistent with the hypothesis that predictability of reward for correct performance is accompanied by faster linkage between neostriatal PMA and the vigor of wrist movement kinematics. Furthermore, the results provide valuable insights for building an upper-limb neuroprosthesis.

Preliminary evidence for human globus pallidus pars interna neurons signaling reward and sensory stimuli

•

Non-motor responses of human GPi neurons are described.

•

Cells were identified that showed increased firing to reward-stimuli.

•

Visual-sensory responses unrelated to reward also observed.

The globus pallidus pars interna (GPi) is a component of the basal ganglia, a network of subcortical nuclei that process motor, associative, and limbic information. While non-human primate studies have suggested a role for the GPi in non-motor functions, there have been no single-unit studies of non-motor electrophysiological behavior of human GPi neurons. We therefore sought to extend these findings by collecting single-unit recordings from awake patients during functional stereotactic neurosurgery targeting the GPi for deep brain stimulation. To assess cellular responses to non-motor information, patients performed a reward task where virtual money could be won, lost, or neither, depending on their performance while cellular activity was monitored. Changes in the firing rates of isolated GPi neurons after the presentation of reward-related stimuli were compared between different reward contingencies (win, loss, null). We observed neurons that modulated their firing rate significantly to the presentation of reward-related stimuli. We furthermore found neurons that responded to visual-stimuli more broadly. This is the first single-unit evidence of human GPi neurons carrying non-motor information. These results are broadly consistent with previous findings in the animal literature and suggest non-motor information may be represented in the single-unit activity of human GPi neurons.

The Striatum and Subthalamic Nucleus as Independent and Collaborative Structures in Motor Control

The striatum and the subthalamic nucleus (STN) are two separate input structures into the basal ganglia (BG). Accordingly, research to date has primarily focused on the distinct roles of these structures in motor control and cognition, often through investigation of Parkinson’s disease (PD). Both structures are divided into sensorimotor, associative, and limbic subdivisions based on cortical connectivity. The more recent discovery of the STN as an input structure into the BG drives comparison of these two structures and their respective roles in cognition and motor control. This review compares the role of the striatum and STN in motor response inhibition and execution, competing motor programs, feedback based learning, and response planning. Through comparison, it is found that the striatum and STN have highly independent roles in motor control but also collaborate in order to execute desired actions. There is also the possibility that inhibition or activation of one of these structures indirectly contributes to the function of other connected anatomical structures. Both structures contribute to selective motor response inhibition, which forms the basis of many tasks, but the STN additionally contributes to global inhibition through the hyperdirect pathway. Research is warranted on the functional connectivity of the network for inhibition involving the rIFG, preSMA, striatum, and STN.

The globus pallidus pars interna in goal-oriented and routine behaviors: Resolving a long-standing paradox

BACKGROUND

There is an apparent contradiction between experimental data showing that the basal ganglia are involved in goal-oriented and routine behaviors and clinical observations. Lesion or disruption by deep brain stimulation of the globus pallidus interna has been used for various therapeutic purposes ranging from the improvement of dystonia to the treatment of Tourette's syndrome. None of these approaches has reported any severe impairment in goal-oriented or automatic movement.

METHOD

To solve this conundrum, we trained 2 monkeys to perform a variant of a 2-armed bandit-task (with different reward contingencies). In the latter we alternated blocks of trials with choices between familiar rewarded targets that elicit routine behavior and blocks with novel pairs of targets that require an intentional learning process.

RESULTS

Bilateral inactivation of the globus pallidus interna, by injection of muscimol, prevents animals from learning new contingencies while performance remains intact, although slower for the familiar stimuli. We replicate in silico these data by adding lateral competition and Hebbian learning in the cortical layer of the theoretical model of the cortex-basal ganglia loop that provided the framework of our experimental approach.

CONCLUSION

The basal ganglia play a critical role in the deliberative process that underlies learning but are not necessary for the expression of routine movements. Our approach predicts that after pallidotomy or during stimulation, patients should have difficulty with complex decision-making processes or learning new goal-oriented behaviors. © 2016 Movement Disorder Society.

Overstatement in happiness reporting with ordinal, bounded scale

There are various methods by which people can express subjective evaluations quantitatively. For example, happiness can be measured on a scale from 1 to 10, and has been suggested as a measure of economic policy. However, there is resistance to these types of measurement from economists, who often regard welfare to be a cardinal, unbounded quantity. It is unclear whether there are differences between subjective evaluation reported on ordinal, bounded scales and on cardinal, unbounded scales. To answer this question, we developed functional magnetic resonance imaging experimental tasks for reporting happiness from monetary gain and the perception of visual stimulus. Subjects tended to report higher values when they used ordinal scales instead of cardinal scales. There were differences in neural activation between ordinal and cardinal reporting scales. The posterior parietal area showed greater activation when subjects used an ordinal scale instead of a cardinal scale. Importantly, the striatum exhibited greater activation when asked to report happiness on an ordinal scale than when asked to report on a cardinal scale. The finding that ordinal (bounded) scales are associated with higher reported happiness and greater activation in the reward system shows that overstatement bias in happiness data must be considered.

Reward functions of the basal ganglia

Besides their fundamental movement function evidenced by Parkinsonian deficits, the basal ganglia are involved in processing closely linked non-motor, cognitive and reward information. This review describes the reward functions of three brain structures that are major components of the basal ganglia or are closely associated with the basal ganglia, namely midbrain dopamine neurons, pedunculopontine nucleus, and striatum (caudate nucleus, putamen, nucleus accumbens). Rewards are involved in learning (positive reinforcement), approach behavior, economic choices and positive emotions. The response of dopamine neurons to rewards consists of an early detection component and a subsequent reward component that reflects a prediction error in economic utility, but is unrelated to movement. Dopamine activations to non-rewarded or aversive stimuli reflect physical impact, but not punishment. Neurons in pedunculopontine nucleus project their axons to dopamine neurons and process sensory stimuli, movements and rewards and reward-predicting stimuli without coding outright reward prediction errors. Neurons in striatum, besides their pronounced movement relationships, process rewards irrespective of sensory and motor aspects, integrate reward information into movement activity, code the reward value of individual actions, change their reward-related activity during learning, and code own reward in social situations depending on whose action produces the reward. These data demonstrate a variety of well-characterized reward processes in specific basal ganglia nuclei consistent with an important function in non-motor aspects of motivated behavior.

Pallidal Deep Brain Stimulation Improves Higher Control of the Oculomotor System in Parkinson's Disease

The frontal cortex and basal ganglia form a set of parallel but mostly segregated circuits called cortico-basal ganglia loops. The oculomotor loop controls eye movements and can direct relatively simple movements, such as reflexive prosaccades, without external help but needs input from "higher" loops for more complex behaviors. The antisaccade task requires the dorsolateral prefrontal cortex, which is part of the prefrontal loop. Information flows from prefrontal to oculomotor circuits in the striatum, and directional errors in this task can be considered a measure of failure of prefrontal control over the oculomotor loop. The antisaccadic error rate (AER) is increased in Parkinson's disease (PD). Deep brain stimulation (DBS) of the subthalamic nucleus (STN) has no effect on the AER, but a previous case suggested that DBS of the globus pallidus interna (GPi) might. Our aim was to compare the effects of STN DBS and GPi DBS on the AER. We tested eye movements in 14 human DBS patients and 10 controls. GPi DBS substantially reduced the AER, restoring lost higher control over oculomotor function. Interloop information flow involves striatal neurons that receive cortical input and project to pallidum. They are normally silent when quiescent, but in PD they fire randomly, creating noise that may account for the degradation in interloop control. The reduced AER with GPi DBS could be explained by retrograde stimulation of striatopallidal axons with consequent activation of inhibitory collaterals and reduction in background striatal firing rates. This study may help explain aspects of PD pathophysiology and the mechanism of action of GPi DBS. Significance statement: Parkinson's disease causes symptoms including stiffness, slowness of movement, and tremor. Electrical stimulation of specific areas deep in the brain can effectively treat these symptoms, but exactly how is not fully understood. Part of the cause of such symptoms may be impairments in the way information flows from one circuit within the brain to another, as a result of overactivity of certain nerve cells. By demonstrating that stimulation of an area called the globus pallidus interna partially reverses deficits in voluntary control of eye movements, this study shows that stimulation can improve information flow between circuits, probably by calming down the overactive cells.

Parallel Representation of Value-Based and Finite State-Based Strategies in the Ventral and Dorsal Striatum

Previous theoretical studies of animal and human behavioral learning have focused on the dichotomy of the value-based strategy using action value functions to predict rewards and the model-based strategy using internal models to predict environmental states. However, animals and humans often take simple procedural behaviors, such as the “win-stay, lose-switch” strategy without explicit prediction of rewards or states. Here we consider another strategy, the finite state-based strategy, in which a subject selects an action depending on its discrete internal state and updates the state depending on the action chosen and the reward outcome. By analyzing choice behavior of rats in a free-choice task, we found that the finite state-based strategy fitted their behavioral choices more accurately than value-based and model-based strategies did. When fitted models were run autonomously with the same task, only the finite state-based strategy could reproduce the key feature of choice sequences. Analyses of neural activity recorded from the dorsolateral striatum (DLS), the dorsomedial striatum (DMS), and the ventral striatum (VS) identified significant fractions of neurons in all three subareas for which activities were correlated with individual states of the finite state-based strategy. The signal of internal states at the time of choice was found in DMS, and for clusters of states was found in VS. In addition, action values and state values of the value-based strategy were encoded in DMS and VS, respectively. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

The neural mechanism of decision-making, a cognitive process to select one action among multiple possibilities, is a fundamental issue in neuroscience. Previous studies have revealed the roles of the cerebral cortex and the basal ganglia in decision-making, by assuming that subjects take a value-based reinforcement learning strategy, in which the expected reward for each action candidate is updated. However, animals and humans often use simple procedural strategies, such as “win-stay, lose-switch.” In this study, we consider a finite state-based strategy, in which a subject acts depending on its discrete internal state and updates the state based on reward feedback. We found that the finite state-based strategy could reproduce the choice behavior of rats in a binary choice task with higher accuracy than the value-based strategy. Interestingly, neuronal activity in the striatum, a crucial brain region for reward-based learning, encoded information regarding both strategies. These results suggest that both the value-based strategy and the finite state-based strategy are implemented in the striatum.

Characteristics of fast-spiking neurons in the striatum of behaving monkeys

Inhibitory interneurons are the fundamental constituents of neural circuits that organize network outputs. The striatum as part of the basal ganglia is involved in reward-directed behaviors. However, the role of the inhibitory interneurons in this process remains unclear, especially in behaving monkeys. We recorded the striatal single neuron activity while monkeys performed reward-directed hand or eye movements. Presumed parvalbumin-containing GABAergic interneurons (fast-spiking neurons, FSNs) were identified based on narrow spike shapes in three independent experiments, though they were a small population (4.2%, 42/997). We found that FSNs are characterized by high-frequency and less-bursty discharges, which are distinct from the basic firing properties of the presumed projection neurons (phasically active neurons, PANs). Besides, the encoded information regarding actions and outcomes was similar between FSNs and PANs in terms of proportion of neurons, but the discharge selectivity was higher in PANs than that of FSNs. The coding of actions and outcomes in FSNs and PANs was consistently observed under various behavioral contexts in distinct parts of the striatum (caudate nucleus, putamen, and anterior striatum). Our results suggest that FSNs may enhance the discharge selectivity of postsynaptic output neurons (PANs) in encoding crucial variables for a reward-directed behavior.

Neuroimaging studies of the striatum in cognition Part I: healthy individuals

The striatum has traditionally mainly been associated with playing a key role in the modulation of motor functions. Indeed, lesion studies in animals and studies of some neurological conditions in humans have brought further evidence to this idea. However, better methods of investigation have raised concerns about this notion, and it was proposed that the striatum could also be involved in different types of functions including cognitive ones. Although the notion was originally a matter of debate, it is now well-accepted that the caudate nucleus contributes to cognition, while the putamen could be involved in motor functions, and to some extent in cognitive functions as well. With the arrival of modern neuroimaging techniques in the early 1990, knowledge supporting the cognitive aspect of the striatum has greatly increased, and a substantial number of scientific papers were published studying the role of the striatum in healthy individuals. For the first time, it was possible to assess the contribution of specific areas of the brain during the execution of a cognitive task. Neuroanatomical studies have described functional loops involving the striatum and the prefrontal cortex suggesting a specific interaction between these two structures. This review examines the data up to date and provides strong evidence for a specific contribution of the fronto-striatal regions in different cognitive processes, such as set-shifting, self-initiated responses, rule learning, action-contingency, and planning. Finally, a new two-level functional model involving the prefrontal cortex and the dorsal striatum is proposed suggesting an essential role of the dorsal striatum in selecting between competing potential responses or actions, and in resolving a high level of ambiguity.

Response properties of neurons in the cat's putamen during auditory discrimination

The striatum integrates diverse convergent input and plays a critical role in the goal-directed behaviors. To date, the auditory functions of striatum are less studied. Recently, it was demonstrated that auditory cortico-striatal projections influence behavioral performance during a frequency discrimination task. To reveal the functions of striatal neurons in auditory discrimination, we recorded the single-unit spike activities in the putamen (dorsal striatum) of free-moving cats while performing a Go/No-go task to discriminate the sounds with different modulation rates (12.5 Hz vs. 50 Hz) or envelopes (damped vs. ramped). We found that the putamen neurons can be broadly divided into four groups according to their contributions to sound discrimination. First, 40% of neurons showed vigorous responses synchronized to the sound envelope, and could precisely discriminate different sounds. Second, 18% of neurons showed a high preference of ramped to damped sounds, but no preference for modulation rate. They could only discriminate the change of sound envelope. Third, 27% of neurons rapidly adapted to the sound stimuli, had no ability of sound discrimination. Fourth, 15% of neurons discriminated the sounds dependent on the reward-prediction. Comparing to passively listening condition, the activities of putamen neurons were significantly enhanced by the engagement of the auditory tasks, but not modulated by the cat's behavioral choice. The coexistence of multiple types of neurons suggests that the putamen is involved in the transformation from auditory representation to stimulus-reward association.

Task-rest modulation of basal ganglia connectivity in mild to moderate Parkinson's disease

Parkinson's disease (PD) is associated with abnormal synchronization in basal ganglia-thalamo-cortical loops. We tested whether early PD patients without demonstrable cognitive impairment exhibit abnormal modulation of functional connectivity at rest, while engaged in a task, or both. PD and healthy controls underwent two functional MRI scans: a resting-state scan and a Stroop Match-to-Sample task scan. Rest-task modulation of basal ganglia (BG) connectivity was tested using seed-to-voxel connectivity analysis with task and rest time series as conditions. Despite substantial overlap of BG-cortical connectivity patterns in both groups, connectivity differences between groups had clinical and behavioral correlates. During rest, stronger putamen-medial parietal and pallidum-occipital connectivity in PD than controls was associated with worse task performance and more severe PD symptoms suggesting that abnormalities in resting-state connectivity denote neural network dedifferentiation. During the executive task, PD patients showed weaker BG-cortical connectivity than controls, i.e., between caudate-supramarginal gyrus and pallidum-inferior prefrontal regions, that was related to more severe PD symptoms and worse task performance. Yet, task processing also evoked stronger striatal-cortical connectivity, specifically between caudate-prefrontal, caudate-precuneus, and putamen-motor/premotor regions in PD relative to controls, which was related to less severe PD symptoms and better performance on the Stroop task. Thus, stronger task-evoked striatal connectivity in PD demonstrated compensatory neural network enhancement to meet task demands and improve performance levels. fMRI-based network analysis revealed that despite resting-state BG network compromise in PD, BG connectivity to prefrontal, premotor, and precuneus regions can be adequately invoked during executive control demands enabling near normal task performance.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

Striatal and midbrain connectivity with the hippocampus selectively boosts memory for contextual novelty

The role of contextual expectation in processing familiar and novel stimuli was investigated in a series of experiments combining eye tracking, functional magnetic resonance imaging, and behavioral methods. An experimental paradigm emphasizing either familiarity or novelty detection at retrieval was used. The detection of unexpected familiar and novel stimuli, which were characterized by lower probability, engaged activity in midbrain and striatal structures. Specifically, detecting unexpected novel stimuli, relative to expected novel stimuli, produced greater activity in the substantia nigra/ventral tegmental area (SN/VTA), whereas the detection of unexpected familiar, relative to expected, familiar stimuli, elicited activity in the striatum/globus pallidus (GP). An effective connectivity analysis showed greater functional coupling between these two seed areas (GP and SN/VTA) and the hippocampus, for unexpected than for expected stimuli. Within this network of midbrain/striatal-hippocampal interactions two pathways are apparent; the direct SN-hippocampal pathway sensitive to unexpected novelty and the perirhinal-GP-hippocampal pathway sensitive to unexpected familiarity. In addition, increased eye fixations and pupil dilations also accompanied the detection of unexpected relative to expected familiar and novel stimuli, reflecting autonomic activity triggered by the functioning of these two pathways. Finally, subsequent memory for unexpected, relative to expected, familiar, and novel stimuli was characterized by enhanced recollection, but not familiarity, accuracy. Taken together, these findings suggest that a hippocampal-midbrain network, characterized by two distinct pathways, mediates encoding facilitation and most critically, that this facilitation is driven by contextual novelty, rather than by the absolute novelty of a stimulus. This contextually sensitive neural mechanism appears to elicit increased exploratory behavior, leading subsequently to greater recollection of the unexpected stimulus.

Movement-related discharge in the macaque globus pallidus during high-frequency stimulation of the subthalamic nucleus

Deep brain stimulation (DBS) of the subthalamic nucleus (STN-DBS) has largely replaced ablative therapies for Parkinson's disease. Because of the similar efficacies of the two treatments, it has been proposed that DBS acts by creating an "informational lesion," whereby pathologic neuronal firing patterns are replaced by low-entropy, stimulus-entrained firing patterns. The informational lesion hypothesis, in its current form, states that DBS blocks the transmission of all information from the basal ganglia, including both pathologic firing patterns and normal, task-related modulations in activity. We tested this prediction in two healthy rhesus macaques by recording single-unit spiking activity from the globus pallidus (232 neurons) while the animals completed choice reaction time reaching movements with and without STN-DBS. Despite strong effects of DBS on the activity of most pallidal cells, reach-related modulations in firing rate were equally prevalent in the DBS-on and DBS-off states. This remained true even when the analysis was restricted to cells affected significantly by DBS. In addition, the overall form and timing of perimovement modulations in firing rate were preserved between DBS-on and DBS-off states in the majority of neurons (66%). Active movement and DBS had largely additive effects on the firing rate of most neurons, indicating an orthogonal relationship in which both inputs contribute independently to the overall firing rate of pallidal neurons. These findings suggest that STN-DBS does not act as an indiscriminate informational lesion but rather as a filter that permits task-related modulations in activity while, presumably, eliminating the pathological firing associated with parkinsonism.

Distinct neural representation in the dorsolateral, dorsomedial, and ventral parts of the striatum during fixed- and free-choice tasks

The striatum is a major input site of the basal ganglia, which play an essential role in decision making. Previous studies have suggested that subareas of the striatum have distinct roles: the dorsolateral striatum (DLS) functions in habitual action, the dorsomedial striatum (DMS) in goal-directed actions, and the ventral striatum (VS) in motivation. To elucidate distinctive functions of subregions of the striatum in decision making, we systematically investigated information represented by phasically active neurons in DLS, DMS, and VS. Rats performed two types of choice tasks: fixed- and free-choice tasks. In both tasks, rats were required to perform nose poking to either the left or right hole after cue-tone presentation. A food pellet was delivered probabilistically depending on the presented cue and the selected action. The reward probability was fixed in fixed-choice task and varied in a block-wise manner in free-choice task. We found the following: (1) when rats began the tasks, a majority of VS neurons increased their firing rates and information regarding task type and state value was most strongly represented in VS; (2) during action selection, information of action and action values was most strongly represented in DMS; (3) action-command information (action representation before action selection) was stronger in the fixed-choice task than in the free-choice task in both DLS and DMS; and (4) action-command information was strongest in DLS, particularly when the same choice was repeated. We propose a hypothesis of hierarchical reinforcement learning in the basal ganglia to coherently explain these results.

Enhanced corticobulbar excitability in chronic smokers during visual exposure to cigarette smoking cues

BACKGROUND

Neuroimaging studies of chronic smokers report altered activity of several neural regions involved in the processing of rewarding outcomes. Neuroanatomical evidence suggests that these regions are directly connected to the tongue muscle through the corticobulbar pathways. Accordingly, we examined whether corticobulbar excitability might be considered a somatic marker for nicotine craving.

METHODS

We compared motor-evoked potential (MEP) amplitudes recorded from the tongue and the extensor carpi radialis (control muscle) of chronic smokers under drug withdrawal and intake conditions as well as a nonsmoker group. All participants were tested during passive exposure to pictures showing a smoking cue or a meaningless stimulus. In the intake condition, chronic smokers were asked to smoke a real cigarette (CSn: group 1) or a placebo (CSp: group 2).

RESULTS

Results show that MEP amplitudes recorded from the tongues of participants in the CSn and CSp groups under the withdrawal condition were selectively enhanced during exposure to a smoking cue. However, this effect on tongue MEP amplitudes disappeared in the intake condition for both the CSn and CSp groups.

LIMITATIONS

Limitations include the fact that the study was conducted in 2 different laboratories, the small sample size, the absence of data on chronic smoker craving strength and the different tastes of the real and placebo cigarettes.

CONCLUSION

These results suggest that, in chronic smokers, tongue muscle MEP amplitudes are sensitive to neural processes active under the physiological status of nicotine craving. This finding implicates a possible functional link between neural excitability of the corticobulbar pathway and the reward system in chronic smokers.

Representation of spatial- and object-specific behavioral goals in the dorsal globus pallidus of monkeys during reaching movement

The dorsal aspect of the globus pallidus (GP) communicates with the prefrontal cortex and higher-order motor areas, indicating that it plays a role in goal-directed behavior. We examined the involvement of dorsal GP neurons in behavioral goal monitoring and maintenance, essential components of executive function. We trained two macaque monkeys to choose a reach target based on relative target position in a spatial goal task or a target shape in an object-goal task. The monkeys were trained to continue to choose a certain behavioral goal when reward volume was constant and to switch the goals when the volume began to decrease. Because the judgment for the next goal was made in the absence of visual signals, the monkeys were required to monitor and maintain the chosen goals during the reaching movement. We obtained three major findings. (1) GP neurons reflected more of the relative spatial position than the shape of the reaching target during the spatial goal task. During the object-goal task, the shape of the reaching object was represented more than the relative position. (2) The selectivity of individual neurons for the relative position was enhanced during the spatial goal task, whereas the object-shape selectivity was enhanced during the object-goal task. (3) When the monkeys switched the goals, the selectivity for either the position or shape also switched. Together, these findings suggest that the dorsal GP is involved in behavioral goal monitoring and maintenance during execution of goal-oriented actions, presumably in collaboration with the prefrontal cortex.

Complex population response of dorsal putamen neurons predicts the ability to learn

Day-to-day variability in performance is a common experience. We investigated its neural correlate by studying learning behavior of monkeys in a two-alternative forced choice task, the two-armed bandit task. We found substantial session-to-session variability in the monkeys’ learning behavior. Recording the activity of single dorsal putamen neurons we uncovered a dual function of this structure. It has been previously shown that a population of neurons in the DLP exhibits firing activity sensitive to the reward value of chosen actions. Here, we identify putative medium spiny neurons in the dorsal putamen that are cue-selective and whose activity builds up with learning. Remarkably we show that session-to-session changes in the size of this population and in the intensity with which this population encodes cue-selectivity is correlated with session-to-session changes in the ability to learn the task. Moreover, at the population level, dorsal putamen activity in the very beginning of the session is correlated with the performance at the end of the session, thus predicting whether the monkey will have a "good" or "bad" learning day. These results provide important insights on the neural basis of inter-temporal performance variability.

Neural correlates of motor vigour and motor urgency during exercise

This article reviews the brain structures and neural circuitry underlying the motor system as it pertains to endurance exercise. Some obvious phenomena that occur during endurance racing events that need to be explained neurophysiologically are variable pacing strategies, the end spurt, motivation and the rating of perceived exertion. Understanding the above phenomena physiologically is problematic due to the sheer complexity of obtaining real-time brain measurements during exercise. In those rare instances where brain measurements have been made during exercise, the measurements have usually been limited to the sensory and motor cortices; or the exercise itself was limited to small muscle groups. Without discounting the crucial importance of the primary motor cortex in the execution of voluntary movement, it is surprising that very few exercise studies pay any attention to the complex and dynamic organization of motor action in relation to the subcortical nuclei, given that they are essential for the execution of normal movement patterns. In addition, the findings from laboratory-based exercise performance trials are hampered by the absence of objective measures of the motivational state of subjects. In this review we propose that some of the above phenomena may be explained by distinguishing between voluntary, vigorous and urgent motor behaviours during exercise, given that different CNS structures and neurotransmitters are involved in the execution of these different motor behaviours.

Different populations of subthalamic neurons encode cocaine vs. sucrose reward and predict future error

The search for treatment of cocaine addiction raises the challenge to find a way to diminish motivation for the drug without decreasing it for natural rewards. Subthalamic nucleus (STN) inactivation decreases motivation for cocaine while increasing motivation for food, suggesting that STN can dissociate different rewards. Here, we investigated how rat STN neurons respond to cues predicting cocaine or sucrose and to reward delivery while rats are performing a discriminative stimuli task. We show that different neuronal populations of STN neurons encode cocaine and sucrose. In addition, we show that STN activity at the cue onset predicts future error. When changing the reward predicted unexpectedly, STN neurons show capacities of adaptation, suggesting a role in reward-prediction error. Furthermore, some STN neurons show a response to executive error (i.e., "oops neurons") that is specific to the missed reward. These results position the STN as a nexus where natural rewards and drugs of abuse are coded differentially and can influence the performance. Therefore, STN can be viewed as a structure where action could be taken for the treatment of cocaine addiction.

Signal enhancement in the output stage of the basal ganglia by synaptic short-term plasticity in the direct, indirect, and hyperdirect pathways

Many of the synapses in the basal ganglia display short-term plasticity. Still, computational models have not yet been used to investigate how this affects signaling. Here we use a model of the basal ganglia network, constrained by available data, to quantitatively investigate how synaptic short-term plasticity affects the substantia nigra reticulata (SNr), the basal ganglia output nucleus. We find that SNr becomes particularly responsive to the characteristic burst-like activity seen in both direct and indirect pathway striatal medium spiny neurons (MSN). As expected by the standard model, direct pathway MSNs are responsible for decreasing the activity in SNr. In particular, our simulations indicate that bursting in only a few percent of the direct pathway MSNs is sufficient for completely inhibiting SNr neuron activity. The standard model also suggests that SNr activity in the indirect pathway is controlled by MSNs disinhibiting the subthalamic nucleus (STN) via the globus pallidus externa (GPe). Our model rather indicates that SNr activity is controlled by the direct GPe-SNr projections. This is partly because GPe strongly inhibits SNr but also due to depressing STN-SNr synapses. Furthermore, depressing GPe-SNr synapses allow the system to become sensitive to irregularly firing GPe subpopulations, as seen in dopamine depleted conditions, even when the GPe mean firing rate does not change. Similar to the direct pathway, simulations indicate that only a few percent of bursting indirect pathway MSNs can significantly increase the activity in SNr. Finally, the model predicts depressing STN-SNr synapses, since such an assumption explains experiments showing that a brief transient activation of the hyperdirect pathway generates a tri-phasic response in SNr, while a sustained STN activation has minor effects. This can be explained if STN-SNr synapses are depressing such that their effects are counteracted by the (known) depressing GPe-SNr inputs.

Interaction between cognitive and motor cortico-basal ganglia loops during decision making: a computational study

In a previous modeling study, Leblois et al. (2006) demonstrated an action selection mechanism in cortico-basal ganglia loops based on competition between the positive feedback, direct pathway through the striatum and the negative feedback, hyperdirect pathway through the subthalamic nucleus. The present study investigates how multiple level action selection could be performed by the basal ganglia. To do this, the model is extended in a manner consistent with known anatomy and electrophysiology in three main areas. First, two-level decision making has been incorporated, with a cognitive level selecting based on cue shape and a motor level selecting based on cue position. We show that the decision made at the cognitive level can be used to bias the decision at the motor level. We then demonstrate that, for accurate transmission of information between decision-making levels, low excitability of striatal projection neurons is necessary, a generally observed electrophysiological finding. Second, instead of providing a biasing signal between cue choices as an external input to the network, we show that the action selection process can be driven by reasonable levels of noise. Finally, we incorporate dopamine modulated learning at corticostriatal synapses. As learning progresses, the action selection becomes based on learned visual cue values and is not interfered with by the noise that was necessary before learning.

Neuronal activity correlated with checking behaviour in the subthalamic nucleus of patients with obsessive-compulsive disorder

Doubt, and its behavioural correlate, checking, is a normal phenomenon of human cognition that is dramatically exacerbated in obsessive-compulsive disorder. We recently showed that deep brain stimulation in the associative-limbic area of the subthalamic nucleus, a central core of the basal ganglia, improved obsessive-compulsive disorder. To understand the physiological bases of symptoms in such patients, we recorded the activity of individual neurons in the therapeutic target during surgery while subjects performed a cognitive task that gave them the possibility of unrestricted repetitive checking after they had made a choice. We postulated that the activity of neurons in this region could be influenced by doubt and checking behaviour. Among the 63/87 task-related neurons recorded in 10 patients, 60% responded to various combinations of instructions, delay, movement or feedback, thus highlighting their role in the integration of different types of information. In addition, task-related activity directed towards decision-making increased during trials with checking in comparison with those without checking. These results suggest that the associative-limbic subthalamic nucleus plays a role in doubt-related repetitive thoughts. Overall, our results not only provide new insight into the role of the subthalamic nucleus in human cognition but also support the fact that subthalamic nucleus modulation by deep brain stimulation reduced compulsive behaviour in patients with obsessive-compulsive disorder.

Uncertainty in action-value estimation affects both action choice and learning rate of the choice behaviors of rats

The estimation of reward outcomes for action candidates is essential for decision making. In this study, we examined whether and how the uncertainty in reward outcome estimation affects the action choice and learning rate. We designed a choice task in which rats selected either the left-poking or right-poking hole and received a reward of a food pellet stochastically. The reward probabilities of the left and right holes were chosen from six settings (high, 100% vs. 66%; mid, 66% vs. 33%; low, 33% vs. 0% for the left vs. right holes, and the opposites) in every 20-549 trials. We used Bayesian Q-learning models to estimate the time course of the probability distribution of action values and tested if they better explain the behaviors of rats than standard Q-learning models that estimate only the mean of action values. Model comparison by cross-validation revealed that a Bayesian Q-learning model with an asymmetric update for reward and non-reward outcomes fit the choice time course of the rats best. In the action-choice equation of the Bayesian Q-learning model, the estimated coefficient for the variance of action value was positive, meaning that rats were uncertainty seeking. Further analysis of the Bayesian Q-learning model suggested that the uncertainty facilitated the effective learning rate. These results suggest that the rats consider uncertainty in action-value estimation and that they have an uncertainty-seeking action policy and uncertainty-dependent modulation of the effective learning rate.

Action-specific value signals in reward-related regions of the human brain

Estimating the value of potential actions is crucial for learning and adaptive behavior. We know little about how the human brain represents action-specific value outside of motor areas. This is, in part, due to a difficulty in detecting the neural correlates of value using conventional (region of interest) functional magnetic resonance imaging (fMRI) analyses, due to a potential distributed representation of value. We address this limitation by applying a recently developed multivariate decoding method to high-resolution fMRI data in subjects performing an instrumental learning task. We found evidence for action-specific value signals in circumscribed regions, specifically ventromedial prefrontal cortex, putamen, thalamus, and insula cortex. In contrast, action-independent value signals were more widely represented across a large set of brain areas. Using multivariate Bayesian model comparison, we formally tested whether value-specific responses are spatially distributed or coherent. We found strong evidence that both action-specific and action-independent value signals are represented in a distributed fashion. Our results suggest that a surprisingly large number of classical reward-related areas contain distributed representations of action-specific values, representations that are likely to mediate between reward and adaptive behavior.

The primate ventral pallidum encodes expected reward value and regulates motor action

Motor actions are facilitated when expected reward value is high. It is hypothesized that there are neurons that encode expected reward values to modulate impending actions and potentially represent motivation signals. Here, we present evidence suggesting that the ventral pallidum (VP) may participate in this process. We recorded single neuronal activity in the monkey VP using a saccade task with a direction-dependent reward bias. Depending on the amount of the expected reward, VP neurons increased or decreased their activity tonically until the reward was delivered, for both ipsiversive and contraversive saccades. Changes in expected reward values were also associated with changes in saccade performance (latency and velocity). Furthermore, bilateral muscimol-induced inactivation of the VP abolished the reward-dependent changes in saccade latencies. These data suggest that the VP provides expected reward value signals that are used to facilitate or inhibit motor actions.

Coding of the long-term value of multiple future rewards in the primate striatum

Decisions maximizing benefits involve a tradeoff between the quantity of a reward and the cost of elapsed time until an animal receives it. The estimation of long-term reward values is critical to attain the most desirable outcomes over a certain period of time. Reinforcement learning theories have established algorithms to estimate the long-term reward values of multiple future rewards in which the values of future rewards are discounted as a function of how many steps of choices are necessary to achieve them. Here, we report that presumed striatal projection neurons represent the long-term values of multiple future rewards estimated by a standard reinforcement learning model while monkeys are engaged in a series of trial-and-error choices and adaptive decisions for multiple rewards. We found that the magnitude of activity of a subset of neurons was positively correlated with the long-term reward values, and that of another subset of neurons was negatively correlated throughout the entire decision-making process in individual trials: from the start of the task trial, estimation of the values and their comparison among alternatives, choice execution, and evaluation of the received rewards. An idiosyncratic finding was that neurons showing negative correlations represented reward values in the near future (high discounting), while neurons showing positive correlations represented reward values not only in the near future, but also in the far future (low discounting). These findings provide a new insight that long-term value signals are embedded in two subsets of striatal neurons as high and low discounting of multiple future rewards.

Reward prediction error signaling in posterior dorsomedial striatum is action specific

Neural correlates of reward prediction errors (RPEs) have been found in dorsal striatum. Such signals may be important for updating associative action representations within striatum. In order that the appropriate representations can be updated, it might be important for the RPE signal to be specific for the action that led to that error. However, RPEs signaled by midbrain dopamine neurons, which project heavily to striatum, are not action-specific. Here we tested whether RPE-like activity in dorsal striatum is action-specific; we recorded single-unit activity in posterior dorsomedial and dorsolateral striatum as rats performed a task in which the reward predictions associated with two different actions were repeatedly violated, thereby eliciting RPEs. We separately analyzed fast firing neurons (FFNs) and phasically firing neurons (total n = 1076). Only among FFNs recorded in posterior dorsomedial striatum did we find a population with RPE-like characteristics (19 of all 196 FFNs, 10%). This population showed a phasic increase in activity during unexpected rewards, a phasic decrease in activity during unexpected omission of rewards, and a phasic increase in activity during cues when they predicted high-value reward. However, unlike a classical RPE signal, this signal was linked to the action that elicited the prediction error, in that neurons tended to signal RPEs only after their anti-preferred action. This action-specific RPE-like signal could provide a mechanism for updating specific associative action representations in posterior dorsomedial striatum.

Scaling of movement is related to pallidal γ oscillations in patients with dystonia

Neuronal synchronization in the gamma (γ) band is considered important for information processing through functional integration of neuronal assemblies across different brain areas. Movement-related γ synchronization occurs in the human basal ganglia where it is centered at ~70 Hz and more pronounced contralateral to the moved hand. However, its functional significance in motor performance is not yet well understood. Here, we assessed whether event-related γ synchronization (ERS) recorded from the globus pallidus internus in patients undergoing deep brain stimulation for medically intractable primary focal and segmental dystonia might code specific motor parameters. Pallidal local field potentials were recorded in 22 patients during performance of a choice-reaction-time task. Movement amplitude of the forearm pronation-supination movements was parametrically modulated with an angular degree of 30°, 60°, and 90°. Only patients with limbs not affected by dystonia were tested. A broad contralateral γ band (35-105 Hz) ERS occurred at movement onset with a maximum reached at peak velocity of the movement. The pallidal oscillatory γ activity correlated with movement parameters: the larger and faster the movement, the stronger was the synchronization in the γ band. In contrast, the event-related decrease in beta band activity was similar for all movements. Gamma band activity did not change with movement direction and did not occur during passive movements. The stepwise increase of γ activity with movement size and velocity suggests a role of neuronal synchronization in this frequency range in basal ganglia control of the scaling of ongoing movements.

Neural mechanisms underlying motivation of mental versus physical effort

Mental and physical efforts, such as paying attention and lifting weights, have been shown to involve different brain systems. These cognitive and motor systems, respectively, include cortical networks (prefronto-parietal and precentral regions) as well as subregions of the dorsal basal ganglia (caudate and putamen). Both systems appeared sensitive to incentive motivation: their activity increases when we work for higher rewards. Another brain system, including the ventral prefrontal cortex and the ventral basal ganglia, has been implicated in encoding expected rewards. How this motivational system drives the cognitive and motor systems remains poorly understood. More specifically, it is unclear whether cognitive and motor systems can be driven by a common motivational center or if they are driven by distinct, dedicated motivational modules. To address this issue, we used functional MRI to scan healthy participants while performing a task in which incentive motivation, cognitive, and motor demands were varied independently. We reasoned that a common motivational node should (1) represent the reward expected from effort exertion, (2) correlate with the performance attained, and (3) switch effective connectivity between cognitive and motor regions depending on task demand. The ventral striatum fulfilled all three criteria and therefore qualified as a common motivational node capable of driving both cognitive and motor regions of the dorsal striatum. Thus, we suggest that the interaction between a common motivational system and the different task-specific systems underpinning behavioral performance might occur within the basal ganglia.

Incentive motivation refers to the process in the brain by which we translate the expectation of a potential reward into the effort required to do an action, as for instance when the expected paycheck brings the employee to work. Different types of effort can be implemented in everyday life, some being more cognitive, like paying attention, and others more motor-involved, like lifting weights. Reward, cognitive, and motor representations are known to rely on distinct regions of the frontal cortex and basal ganglia. However, how expected rewards motivate these different types of efforts remains poorly understood. Here, we addressed this question by developing a functional neuroimaging approach where we independently varied a monetary reward as well as the cognitive and motor demand of the task. Our results suggest that the expectation of a reward is encoded in the ventral striatum, which can then drive either the motor or cognitive part of the dorsal striatum, depending on the task, in order to boost behavioral performance. We conclude that intra-striatal effective connectivity may explain how both motor and cognitive efforts can be driven by a single motivational module.

Early life stress affects cerebral glucose metabolism in adult rhesus monkeys (Macaca mulatta)

Early life stress (ELS) is a risk factor for anxiety, mood disorders and alterations in stress responses. Less is known about the long-term neurobiological impact of ELS. We used [(18)F]-fluorodeoxyglucose Positron Emission Tomography (FDG-PET) to assess neural responses to a moderate stress test in adult monkeys that experienced ELS as infants. Both groups of monkeys showed hypothalamic-pituitary-adrenal (HPA) axis stress-induced activations and cardiac arousal in response to the stressor. A whole brain analysis detected significantly greater regional cerebral glucose metabolism (rCGM) in superior temporal sulcus, putamen, thalamus, and inferotemporal cortex of ELS animals compared to controls. Region of interest (ROI) analyses performed in areas identified as vulnerable to ELS showed greater activity in the orbitofrontal cortex of ELS compared to control monkeys, but greater hippocampal activity in the control compared to ELS monkeys. Together, these results suggest hyperactivity in emotional and sensory processing regions of adult monkeys with ELS, and greater activity in stress-regulatory areas in the controls. Despite these neural responses, no group differences were detected in neuroendocrine, autonomic or behavioral responses, except for a trend towards increased stillness in the ELS monkeys. Together, these data suggest hypervigilance in the ELS monkeys in the absence of immediate danger.

Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates

Recent work has given rise to the view that reward-based decision making is governed by two key controllers: a habit system, which stores stimulus-response associations shaped by past reward, and a goal-oriented system that selects actions based on their anticipated outcomes. The current literature provides a rich body of computational theory addressing habit formation, centering on temporal-difference learning mechanisms. Less progress has been made toward formalizing the processes involved in goal-directed decision making. We draw on recent work in cognitive neuroscience, animal conditioning, cognitive and developmental psychology, and machine learning to outline a new theory of goal-directed decision making. Our basic proposal is that the brain, within an identifiable network of cortical and subcortical structures, implements a probabilistic generative model of reward, and that goal-directed decision making is effected through Bayesian inversion of this model. We present a set of simulations implementing the account, which address benchmark behavioral and neuroscientific findings, and give rise to a set of testable predictions. We also discuss the relationship between the proposed framework and other models of decision making, including recent models of perceptual choice, to which our theory bears a direct connection.