Reward prediction error signaling in posterior dorsomedial striatum is action specific

Exploring value learning in pigeons: the role of dual pathways in the basal ganglia and synaptic plasticity

Understanding value learning in animals is a key focus in cognitive neuroscience. Current models used in research are often simple, and while more complex models have been proposed, it remains unclear which assumptions align with actual value-learning strategies of animals. This study investigated the computational mechanisms behind value learning in pigeons using a free-choice task. Three models were constructed based on different assumptions about the role of the basal ganglia's dual pathways and synaptic plasticity in value computation, followed by model comparison and neural correlation analysis. Among the three models tested, the dual-pathway reinforcement learning model with Hebbian rules most closely matched the pigeons' behavior. Furthermore, the striatal gamma band connectivity showed the highest correlation with the values estimated by this model. Additionally, enhanced beta band connectivity in the nidopallium caudolaterale supported value learning. This study provides valuable insights into reinforcement learning mechanisms in non-human animals.

Striatal function scrutinized through the PAN-TAN-FSI triumvirate

Understanding the information encoded by distinct components of the neuronal circuitry in the striatum represents an avenue for elucidating the role of this subcortical region in adaptive behavior and its dysfunction in pathological conditions. In behaving animals, conventional single neuron recordings generally differentiated between three main electrophysiologically identified neuron subtypes in the striatum, referred to as phasically active neurons (PANs), tonically active neurons (TANs), and fast-spiking interneurons (FSIs), assumed to correspond to GABAergic spiny projection neurons, cholinergic interneurons, and parvalbumin-containing GABAergic interneurons, respectively. Considerable research has been devoted to exploring the behavior-related activities of neurons classified electrophysiologically into PANs, TANs, and FSIs in animals engaged in task performance, mostly monkeys. Although precise neuron identification remains a major challenge, such electrophysiological studies have provided insights into the functional properties of presumed distinct striatal neuronal populations. In this review, we will focus on current ideas about the functions subserved by these neuron subtypes, emphasizing their link to specific aspects of behaviors. We will also underline the issues that are yet to be resolved regarding the classification of striatal neurons into distinct subgroups which emphasize the importance of considering the potential overlap among electrophysiological characteristics and the molecular diversity of neuron types in the striatum.

Coding Dynamics of the Striatal Networks During Learning

The rat dorsomedial (DMS) and dorsolateral striatum (DLS), equivalent to caudate nucleus and putamen in primates, are required for goal-directed and habit behaviour, respectively. However, it is still unclear whether and how this functional dichotomy emerges in the course of learning. In this study, we investigated this issue by recording DMS and DLS single neuron activity in rats performing a continuous spatial alternation task, from the acquisition to optimized performance. We first applied a classical analytical approach to identify task-related activity based on the modifications of single neuron firing rate in relation to specific task events or maze trajectories. We then used an innovative approach based on Hawkes process to reconstruct a directed connectivity graph of simultaneously recorded neurons, that was used to decode animal behavior. This approach enabled us to better unravel the role of DMS and DLS neural networks across learning stages. We showed that DMS and DLS display different task-related activity throughout learning stages, and the proportion of coding neurons over time decreases in the DMS and increases in the DLS. Despite these major differences, the decoding power of both networks increases during learning. These results suggest that DMS and DLS neural networks gradually reorganize in different ways in order to progressively increase their control over the behavioral performance.

Different Effects of Peer Sex on Operant Responding for Social Interaction and Striatal Dopamine Activity

When rats are given discrete choices between social interactions with a peer and opioid or psychostimulant drugs, they choose social interaction, even after extensive drug self-administration experience. Studies show that like drug and nondrug food reinforcers, social interaction is an operant reinforcer and induces dopamine release. However, these studies were conducted with same-sex peers. We examined if peer sex influences operant social interaction and the role of estrous cycle and striatal dopamine in same- versus opposite-sex social interaction. We trained male and female rats (n = 13 responders/12 peers) to lever-press (fixed-ratio 1 [FR1] schedule) for 15 s access to a same- or opposite-sex peer for 16 d (8 d/sex) while tracking females' estrous cycle. Next, we transfected GRAB-DA2m and implanted optic fibers into nucleus accumbens (NAc) core and dorsomedial striatum (DMS). We then retrained the rats for 15 s social interaction (FR1 schedule) for 16 d (8 d/sex) and recorded striatal dopamine during operant responding for a peer for 8 d (4 d/sex). Finally, we assessed economic demand by manipulating FR requirements for a peer (10 d/sex). In male, but not female rats, operant responding was higher for the opposite-sex peer. Female's estrous cycle fluctuations had no effect on operant social interaction. Striatal dopamine signals for operant social interaction were dependent on the peer's sex and striatal region (NAc core vs DMS). Results indicate that estrous cycle fluctuations did not influence operant social interaction and that NAc core and DMS dopamine activity reflect sex-dependent features of volitional social interaction.

Beta Oscillations in Monkey Striatum Encode Reward Prediction Error Signals

Reward prediction error (RPE) signals are crucial for reinforcement learning and decision-making as they quantify the mismatch between predicted and obtained rewards. RPE signals are encoded in the neural activity of multiple brain areas, such as midbrain dopaminergic neurons, prefrontal cortex, and striatum. However, it remains unclear how these signals are expressed through anatomically and functionally distinct subregions of the striatum. In the current study, we examined to which extent RPE signals are represented across different striatal regions. To do so, we recorded local field potentials (LFPs) in sensorimotor, associative, and limbic striatal territories of two male rhesus monkeys performing a free-choice probabilistic learning task. The trial-by-trial evolution of RPE during task performance was estimated using a reinforcement learning model fitted on monkeys' choice behavior. Overall, we found that changes in beta band oscillations (15-35 Hz), after the outcome of the animal's choice, are consistent with RPE encoding. Moreover, we provide evidence that the signals related to RPE are more strongly represented in the ventral (limbic) than dorsal (sensorimotor and associative) part of the striatum. To conclude, our results suggest a relationship between striatal beta oscillations and the evaluation of outcomes based on RPE signals and highlight a major contribution of the ventral striatum to the updating of learning processes.SIGNIFICANCE STATEMENT Reward prediction error (RPE) signals are crucial for reinforcement learning and decision-making as they quantify the mismatch between predicted and obtained rewards. Current models suggest that RPE signals are encoded in the neural activity of multiple brain areas, including the midbrain dopaminergic neurons, prefrontal cortex and striatum. However, it remains elusive whether RPEs recruit anatomically and functionally distinct subregions of the striatum. Our study provides evidence that RPE-related modulations in local field potential (LFP) power are dominant in the striatum. In particular, they are stronger in the rostro-ventral rather than the caudo-dorsal striatum. Our findings contribute to a better understanding of the role of striatal territories in reward-based learning and may be relevant for neuropsychiatric and neurologic diseases that affect striatal circuits.

Neural signals implicated in the processing of appetitive and aversive events in social and non-social contexts

In 2014, we participated in a special issue of Frontiers examining the neural processing of appetitive and aversive events. Specifically, we reviewed brain areas that contribute to the encoding of prediction errors and value versus salience, attention and motivation. Further, we described how we disambiguated these cognitive processes and their neural substrates by using paradigms that incorporate both appetitive and aversive stimuli. We described a circuit in which the orbitofrontal cortex (OFC) signals expected value and the basolateral amygdala (BLA) encodes the salience and valence of both appetitive and aversive events. This information is integrated by the nucleus accumbens (NAc) and dopaminergic (DA) signaling in order to generate prediction and prediction error signals, which guide decision-making and learning via the dorsal striatum (DS). Lastly, the anterior cingulate cortex (ACC) is monitoring actions and outcomes, and signals the need to engage attentional control in order to optimize behavioral output. Here, we expand upon this framework, and review our recent work in which within-task manipulations of both appetitive and aversive stimuli allow us to uncover the neural processes that contribute to the detection of outcomes delivered to a conspecific and behaviors in social contexts. Specifically, we discuss the involvement of single-unit firing in the ACC and DA signals in the NAc during the processing of appetitive and aversive events in both social and non-social contexts.

Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors

Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning, signaling discrepancies between actual versus expected reward predicted by the cues.^1-5 While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. The vast majority of previous studies examining dopamine responses primarily have used discrete reward-predictive stimuli,^1-15 whether Pavlovian conditioned stimuli for which no action is required to earn reward or explicit discriminative stimuli that essentially instruct an animal how and when to respond for reward. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single and sequential instrumental actions, in behavioral contexts with minimal overt changes in the animal's external environment. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal's own action, relative to non-contingent passive stimulation. This dopamine suppression generalized to food rewards was specific to the reinforced action, was temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. These findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.

Interoceptive active inference and self-representation in social anxiety disorder (SAD): exploring the neurocognitive traits of the SAD self

This article provides an interoceptive active inference (IAI) account of social anxiety disorder (SAD). Through a neurocognitive framework, we argue that the cognitive and behavioural profile of SAD is best conceived of as a form of maladaptive IAI produced by a negatively biased self-model that cannot reconcile inconsistent tendencies to approach and avoid social interaction. Anticipated future social interactions produce interoceptive prediction error (bodily states of arousal). These interoceptive states are transcribed and experienced as states of distress due to the influence of inconsistent and unstable self-models across a hierarchy of interrelated systems involved in emotional, interoceptive and affective processing. We highlight the role of the insula cortex, in concert with the striatum, amygdala and dorsal anterior cingulate in the generation and reduction of interoceptive prediction errors as well as the resolution of social approach-avoidance conflict. The novelty of our account is a shift in explanatory priority from the representation of the social world in SAD to the representation of the SAD self. In particular, we show how a high-level conceptual self-model of social vulnerability and inadequacy fails to minimize prediction errors produced by a basic drive for social affiliation combined with strong avoidant tendencies. The result is a cascade of interoceptive prediction errors whose attempted minimization through action (i.e. active inference) yields the symptom profile of SAD. We conclude this article by proposing testable hypotheses to further investigate the neurocognitive traits of the SAD self with respect to IAI.

Activity of fast-spiking interneurons in the monkey striatum during reaching movements guided by external cues or by a free choice

Parvalbumin-containing GABAergic interneurons in the striatum, electrophysiologically identified as fast-spiking interneurons (FSIs), exert inhibitory control over striatal output to drive appropriate behavior. While a number of studies have emphasized their importance in motor control, it is unknown how these putative interneurons adapt their functional properties to different modes of movement selection. Here, we tested whether FSIs are sensitive to externally versus internally selected movements by recording their activity while two male rhesus monkeys performed reaching movements to visual targets. Two variants were used: an external condition, in which movements were instructed via external cues, and an internal condition, in which movements were guided by an internal representation of the target location. These conditions allowed to contrast the FSI activity associated with either externally cued or internally driven movement selection. After extensive training, reaching performance was only marginally affected by the type of movement, albeit with some differences between the monkeys. Over two-thirds of the FSIs were modulated around movement onset, regardless of the condition, and consisting mostly of increased activity. We found that a subset of FSIs showed stronger activation related to the initiation of movements in the external condition than in the internal condition, suggesting a dependence on movement selection mode. Moreover, this difference in the strength of FSI activation was predominant in the motor striatum. These data indicate that changes in FSI activity carry information that is scaled by constraints on action selection reflecting the involvement of local striatal inhibitory circuits in adaptation of behavior according to task demands.

Differential encoding of action selection by orbitofrontal and striatal population dynamics

Survival relies on the ability to flexibly choose between different actions according to varying environmental circumstances. Many lines of evidence indicate that action selection involves signaling in corticostriatal circuits, including the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS). While choice-specific responses have been found in individual neurons from both areas, it is unclear whether populations of OFC or DMS neurons are better at encoding an animal's choice. To address this, we trained head-fixed mice to perform an auditory guided two-alternative choice task, which required moving a joystick forward or backward. We then used silicon microprobes to simultaneously measure the spiking activity of OFC and DMS ensembles, allowing us to directly compare population dynamics between these areas within the same animals. Consistent with previous literature, both areas contained neurons that were selective for specific stimulus-action associations. However, analysis of concurrently recorded ensemble activity revealed that the animal's trial-by-trial behavior could be decoded more accurately from DMS dynamics. These results reveal substantial regional differences in encoding action selection, suggesting that DMS neural dynamics are more specialized than OFC at representing an animal's choice of action.NEW & NOTEWORTHY While previous literature shows that both orbitofrontal cortex (OFC) and dorsomedial striatum (DMS) represent information relevant to selecting specific actions, few studies have directly compared neural signals between these areas. Here we compared OFC and DMS dynamics in mice performing a two-alternative choice task. We found that the animal's choice could be decoded more accurately from DMS population activity. This work provides among the first evidence that OFC and DMS differentially represent information about an animal's selected action.

Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making

When choosing between options, whether menu items or career paths, we can evaluate how rewarding each one will be, or how congruent it is with our current choice goal (e.g., to point out the best option or the worst one.). Past decision-making research interpreted findings through the former lens, but in these experiments the most rewarding option was always most congruent with the task goal (choosing the best option). It is therefore unclear to what extent expected reward vs. goal congruency can account for choice value findings. To deconfound these two variables, we performed three behavioral studies and an fMRI study in which the task goal varied between identifying the best vs. the worst option. Contrary to prevailing accounts, we find that goal congruency dominates choice behavior and neural activity. We separately identify dissociable signals of expected reward. Our findings call for a reinterpretation of previous research on value-based choice.

Decision-making research has confounded the reward value of options with their goal-congruency, as the task goal was always to pick the most rewarding option. Here, authors separately asked participants to select the least rewarding of a set of options, revealing a dominant role for goal congruency.

Distributed Encoding of Reinforcement in Rat Cortico-Striatal-Limbic Networks

Decision-making in the mammalian brain typically involves multiple brain structures within the midbrain, thalamus, striatum, limbic system, and cortex. Although task specific contributions of each brain region have been identified, neurons responding to reinforcement have been found throughout these structures. We sought to determine if any brain area, or cluster of areas, are the source of information, and if the fidelity of information varies among the areas. We recorded simultaneous field potentials (FPs) in rats from seven brain regions as they completed a binary choice task. The FPs of a 0.5 s window following reinforcement were given as input to a classifier that attempted to predict whether or not the rat received reward on each trial. The classifier correctly categorized reward on 77% of trials. Any region-specific signal could be omitted without lowering accuracy. Frequencies above 40 Hz and signals recorded later than 0.25 s following reinforcement were necessary to achieve this accuracy. Further, the classifier was able to predict reinforcement outcome above chance levels when using FPs from any single recorded brain region. Some combinations of structures, however, were more predictive than others. Analysis of FPs prior to reward revealed most regions reflected the prior probability of reward. Lastly, analyses of information flow suggested reinforcement information does not originate within a single structure of the network, within the resolution afforded by FP recordings. These data suggest reward delivery information is rapidly distributed non-uniformly across the network, and there is no canonical flow of information about reward events in the recorded structures.

Success and failure of controlling the real-time functional magnetic resonance imaging neurofeedback signal are reflected in the striatum

INTRODUCTION

Over the last decades, neurofeedback has been applied in variety of research contexts and therapeutic interventions. Despite this extensive use, its neural mechanisms are still under debate. Several scientific advances have suggested that different networks become jointly active during neurofeedback, including regions generally involved in self-regulation, regions related to the specific mental task driving the neurofeedback and regions generally involved in feedback learning (Sitaram et al., 2017, Nature Reviews Neuroscience, 18, 86).

METHODS

To investigate the neural mechanisms specific to neurofeedback but independent from general effects of self-regulation, we compared brain activation as measured with functional magnetic resonance imaging (fMRI) across different mental tasks involving gradual self-regulation with and without providing neurofeedback. Ten participants freely chose one self-regulation task and underwent two training sessions during fMRI scanning, one with and one without receiving neurofeedback. During neurofeedback sessions, feedback signals were provided in real-time based on activity in task-related, individually defined target regions. In both sessions, participants aimed at reaching and holding low, medium, or high brain-activation levels in the target region.

RESULTS

During gradual self-regulation with neurofeedback, a network of cortical control regions as well as regions implicated in reward and feedback processing were activated. Self-regulation with feedback was accompanied by stronger activation within the striatum across different mental tasks. Additional time-resolved single-trial analysis revealed that neurofeedback performance was positively correlated with a delayed brain response in the striatum that reflected the accuracy of self-regulation.

CONCLUSION

Overall, these findings support that neurofeedback contributes to self-regulation through task-general regions involved in feedback and reward processing.

Prefrontal Corticotropin-Releasing Factor (CRF) Neurons Act Locally to Modulate Frontostriatal Cognition and Circuit Function

The PFC and extended frontostriatal circuitry support higher cognitive processes that guide goal-directed behavior. PFC-dependent cognitive dysfunction is a core feature of multiple psychiatric disorders. Unfortunately, a major limiting factor in the development of treatments for PFC cognitive dysfunction is our limited understanding of the neural mechanisms underlying PFC-dependent cognition. We recently demonstrated that activation of corticotropin-releasing factor (CRF) receptors in the caudal dorsomedial PFC (dmPFC) impairs higher cognitive function, as measured in a working memory task. Currently, there remains much unknown about CRF-dependent regulation of cognition, including the source of CRF for cognition-modulating receptors and the output pathways modulated by these receptors. To address these issues, the current studies used a viral vector-based approach to chemogenetically activate or inhibit PFC CRF neurons in working memory-tested male rats. Chemogenetic activation of caudal, but not rostral, dmPFC CRF neurons potently impaired working memory, whereas inhibition of these neurons improved working memory. Importantly, the cognition-impairing actions of PFC CRF neurons were dependent on local CRF receptors coupled to protein kinase A. Additional electrophysiological recordings demonstrated that chemogenetic activation of caudal dmPFC CRF neurons elicits a robust degradation of task-related coding properties of dmPFC pyramidal neurons and, to a lesser extent, medium spiny neurons in the dorsomedial striatum. Collectively, these results demonstrate that local CRF release within the caudal dmPFC impairs frontostriatal cognitive and circuit function and suggest that CRF may represent a potential target for treating frontostriatal cognitive dysfunction.SIGNIFICANCE STATEMENT The dorsomedial PFC and its striatal targets play a critical role in higher cognitive function. PFC-dependent cognitive dysfunction is associated with many psychiatric disorders. Although it has long-been known that corticotropin-releasing factor (CRF) neurons are prominent within the PFC, their role in cognition has remained unclear. Using a novel chemogenetic viral vector system, the present studies demonstrate that PFC CRF neurons impair working memory via activation of local PKA-coupled CRF receptors, an action associated with robust degradation in task-related frontostriatal neuronal coding. Conversely, suppression of constitutive PFC CRF activity improved working memory. Collectively, these studies provide novel insight into the neurobiology of cognition and suggest that CRF may represent a novel target for the treatment of cognitive dysfunction.

Two-photon imaging in mice shows striosomes and matrix have overlapping but differential reinforcement-related responses

Striosomes were discovered several decades ago as neurochemically identified zones in the striatum, yet technical hurdles have hampered the study of the functions of these striatal compartments. Here we used 2-photon calcium imaging in neuronal birthdate-labeled Mash1-CreER;Ai14 mice to image simultaneously the activity of striosomal and matrix neurons as mice performed an auditory conditioning task. With this method, we identified circumscribed zones of tdTomato-labeled neuropil that correspond to striosomes as verified immunohistochemically. Neurons in both striosomes and matrix responded to reward-predicting cues and were active during or after consummatory licking. However, we found quantitative differences in response strength: striosomal neurons fired more to reward-predicting cues and encoded more information about expected outcome as mice learned the task, whereas matrix neurons were more strongly modulated by recent reward history. These findings open the possibility of harnessing in vivo imaging to determine the contributions of striosomes and matrix to striatal circuit function.

Functional brain networks involved in decision-making under certain and uncertain conditions

PURPOSE

The aim of this study was to describe imaging markers of decision-making under uncertain conditions in normal individuals, in order to provide baseline activity to compare to impaired decision-making in pathological states.

METHODS

In this cross-sectional study, 19 healthy subjects ages 18-35 completed a novel decision-making card-matching task using a Phillips T3 Scanner and a 32-channel head coil. Functional data were collected in six functional runs. In one condition of the task, the participant was certain of the rule to apply to match the cards; in the other condition, the participant was uncertain. We performed cluster-based comparison of the two conditions using FSL fMRI Expert Analysis Tool and network-based analysis using MATLAB.

RESULTS

The uncertain > certain comparison yielded three clusters-a midline cluster that extended through the midbrain, the thalamus, bilateral prefrontal cortex, the striatum, and bilateral parietal/occipital clusters. The certain > uncertain comparison yielded bilateral clusters in the insula, parietal and temporal lobe, as well as a medial frontal cluster. A larger, more connected functional network was found in the uncertain condition.

CONCLUSION

The involvement of the insula, parietal cortex, temporal cortex, ventromedial prefrontal cortex, and orbitofrontal cortex of the certain condition reinforces the notion that certainty is inherently rewarding. For the uncertain condition, the involvement of the prefrontal cortex, parietal cortex, striatum, thalamus, amygdala, and hippocampal involvement was expected, as these are areas involved in resolving uncertainty and rule updating. The involvement of occipital cortical involvement and midbrain involvement may be attributed to increased visual attention and increased motor control.

Background activity and visual responsiveness of caudate nucleus neurons in halothane anesthetized and in awake, behaving cats

This study focuses on the important question whether brain activity recorded from anesthetized, paralyzed animals is comparable to that recorded from awake, behaving ones. We compared neuronal activity recorded from the caudate nucleus (CN) of two halothane-anesthetized, paralyzed and two awake, behaving cats. In both models, extracellular recordings were made from the CN during static and dynamic visual stimulation. The anesthesia was maintained during the recordings by a gaseous mixture of air and halothane (1.0%). The behaving animals were trained to perform a visual fixation task. Based on their electrophysiological properties, the recorded CN neurons were separated into three different classes: phasically active (PANs), high firing (HFNs), and tonically active (TANs) neurons. Halothane anesthesia significantly decreased the background activity of the CN neurons in all three classes. The anesthesia had the most remarkable suppressive effect on PANs, where the background activity was consistently under 1 spike/s. The analysis of these responses was almost impossible due to the extremely low activity. The evoked responses during both static and dynamic visual stimulation were obvious in the behaving cats. On the other hand, only weak visual responses were found in some neurons of halothane anesthetized cats. These results show that halothane gas anesthesia has a marked suppressive effect on the feline CN. We suggest that for the purposes of the visual and related multisensory/sensorimotor electrophysiological exploration of the CN, behaving animal models are preferable over anesthetized ones.

Changes in activity of fast-spiking interneurons of the monkey striatum during reaching at a visual target

Recent works highlight the importance of local inhibitory interneurons in regulating the function of the striatum. In particular, fast-spiking interneurons (FSIs), which likely correspond to a subgroup of GABAergic interneurons, have been involved in the control of movement by exerting strong inhibition on striatal output pathways. However, little is known about the exact contribution of these presumed interneurons in movement preparation, initiation, and execution. We recorded the activity of FSIs in the striatum of monkeys as they performed reaching movements to a visual target under two task conditions: one in which the movement target was presented at unsignaled left or right locations, and another in which advance information about target location was available, thus allowing monkeys to react faster. Modulations of FSI activity around the initiation of movement (53% of 55 neurons) consisted mostly of increases reaching maximal firing immediately before or, less frequently, after movement onset. Another subset of FSIs showed decreases in activity during movement execution. Rarely did movement-related changes in FSI firing depend on response direction and movement speed. Modulations of FSI activity occurring relatively early in relation to movement initiation were more influenced by the preparation for movement, compared with those occurring later. Conversely, FSI activity remained unaffected, as monkeys were preparing a movement toward a specific location and instead moved to the opposite direction when the trigger occurred. These results provide evidence that changes in activity of presumed GABAergic interneurons of the primate striatum could make distinct contributions to processes involved in movement generation.

NEW & NOTEWORTHY

We explored the functional contributions of striatal fast-spiking interneurons (FSIs), presumed GABAergic interneurons, to distinct steps of movement generation in monkeys performing a reaching task. The activity of individual FSIs was modulated before and during the movement, consisting mostly of increased in firing rates. Changes in activity also occurred during movement preparation. We interpret this variety of modulation types at different moments of task performance as reflecting differential FSI control over distinct phases of movement.

Reward functions of the basal ganglia

Besides their fundamental movement function evidenced by Parkinsonian deficits, the basal ganglia are involved in processing closely linked non-motor, cognitive and reward information. This review describes the reward functions of three brain structures that are major components of the basal ganglia or are closely associated with the basal ganglia, namely midbrain dopamine neurons, pedunculopontine nucleus, and striatum (caudate nucleus, putamen, nucleus accumbens). Rewards are involved in learning (positive reinforcement), approach behavior, economic choices and positive emotions. The response of dopamine neurons to rewards consists of an early detection component and a subsequent reward component that reflects a prediction error in economic utility, but is unrelated to movement. Dopamine activations to non-rewarded or aversive stimuli reflect physical impact, but not punishment. Neurons in pedunculopontine nucleus project their axons to dopamine neurons and process sensory stimuli, movements and rewards and reward-predicting stimuli without coding outright reward prediction errors. Neurons in striatum, besides their pronounced movement relationships, process rewards irrespective of sensory and motor aspects, integrate reward information into movement activity, code the reward value of individual actions, change their reward-related activity during learning, and code own reward in social situations depending on whose action produces the reward. These data demonstrate a variety of well-characterized reward processes in specific basal ganglia nuclei consistent with an important function in non-motor aspects of motivated behavior.

Using fMRI to study reward processing in humans: past, present, and future

Functional magnetic resonance imaging (fMRI) is a noninvasive tool used to probe cognitive and affective processes. Although fMRI provides indirect measures of neural activity, the advent of fMRI has allowed for1) the corroboration of significant animal findings in the human brain, and2) the expansion of models to include more common human attributes that inform behavior. In this review, we briefly consider the neural basis of the blood oxygenation level dependent signal to set up a discussion of how fMRI studies have applied it in examining cognitive models in humans and the promise of using fMRI to advance such models. Specifically, we illustrate the contribution that fMRI has made to the study of reward processing, focusing on the role of the striatum in encoding reward-related learning signals that drive anticipatory and consummatory behaviors. For instance, we discuss how fMRI can be used to link neural signals (e.g., striatal responses to rewards) to individual differences in behavior and traits. While this functional segregation approach has been constructive to our understanding of reward-related functions, many fMRI studies have also benefitted from a functional integration approach that takes into account how interconnected regions (e.g., corticostriatal circuits) contribute to reward processing. We contend that future work using fMRI will profit from using a multimodal approach, such as combining fMRI with noninvasive brain stimulation tools (e.g., transcranial electrical stimulation), that can identify causal mechanisms underlying reward processing. Consequently, advancements in implementing fMRI will promise new translational opportunities to inform our understanding of psychopathologies.

Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences

Over a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naive, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate an integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates.

Activity of Caudate Nucleus Neurons in a Visual Fixation Paradigm in Behaving Cats

Beside its motor functions, the caudate nucleus (CN), the main input structure of the basal ganglia, is also sensitive to various sensory modalities. The goal of the present study was to investigate the effects of visual stimulation on the CN by using a behaving, head-restrained, eye movement-controlled feline model developed recently for this purpose. Extracellular multielectrode recordings were made from the CN of two cats in a visual fixation paradigm applying static and dynamic stimuli. The recorded neurons were classified in three groups according to their electrophysiological properties: phasically active (PAN), tonically active (TAN) and high-firing (HFN) neurons. The response characteristics were investigated according to this classification. The PAN and TAN neurons were sensitive primarily to static stimuli, while the HFN neurons responded primarily to changes in the visual environment i.e. to optic flow and the offset of the stimuli. The HFNs were the most sensitive to visual stimulation; their responses were stronger than those of the PANs and TANs. The majority of the recorded units were insensitive to the direction of the optic flow, regardless of group, but a small number of direction-sensitive neurons were also found. Our results demonstrate that both the static and the dynamic components of the visual information are represented in the CN. Furthermore, these results provide the first piece of evidence on optic flow processing in the CN, which, in more general terms, indicates the possible role of this structure in dynamic visual information processing.

Neuroimaging studies of the striatum in cognition Part I: healthy individuals

The striatum has traditionally mainly been associated with playing a key role in the modulation of motor functions. Indeed, lesion studies in animals and studies of some neurological conditions in humans have brought further evidence to this idea. However, better methods of investigation have raised concerns about this notion, and it was proposed that the striatum could also be involved in different types of functions including cognitive ones. Although the notion was originally a matter of debate, it is now well-accepted that the caudate nucleus contributes to cognition, while the putamen could be involved in motor functions, and to some extent in cognitive functions as well. With the arrival of modern neuroimaging techniques in the early 1990, knowledge supporting the cognitive aspect of the striatum has greatly increased, and a substantial number of scientific papers were published studying the role of the striatum in healthy individuals. For the first time, it was possible to assess the contribution of specific areas of the brain during the execution of a cognitive task. Neuroanatomical studies have described functional loops involving the striatum and the prefrontal cortex suggesting a specific interaction between these two structures. This review examines the data up to date and provides strong evidence for a specific contribution of the fronto-striatal regions in different cognitive processes, such as set-shifting, self-initiated responses, rule learning, action-contingency, and planning. Finally, a new two-level functional model involving the prefrontal cortex and the dorsal striatum is proposed suggesting an essential role of the dorsal striatum in selecting between competing potential responses or actions, and in resolving a high level of ambiguity.

Neural correlates of rules and conflict in medial prefrontal cortex during decision and feedback epochs

The ability to properly adjust behavioral responses to cues in a changing environment is crucial for survival. Activity in the medial Prefrontal Cortex (mPFC) is thought to both represent rules to guide behavior as well as detect and resolve conflicts between rules in changing contingencies. However, while lesion and pharmacological studies have supported a crucial role for mPFC in this type of set-shifting, an understanding of how mPFC represents current rules or detects and resolves conflict between different rules is unclear. Here, we directly address the role of rat mPFC in shifting rule based behavioral strategies using a novel behavioral task designed to tease apart neural signatures of rules, conflict and direction. We demonstrate that activity of single neurons in rat mPFC represent distinct rules. Further, we show increased firing on high conflict trials in a separate population of mPFC neurons. Reduced firing in both populations of neurons was associated with poor performance. Moreover, activity in both populations increased and decreased firing during the outcome epoch when reward was and was not delivered on correct and incorrect trials, respectively. In addition, outcome firing was modulated by the current rule and the degree of conflict associated with the previous decision. These results promote a greater understanding of the role that mPFC plays in switching between rules, signaling both rule and conflict to promote improved behavioral performance.

Coexistence of glutamatergic spine synapses and shaft synapses in substantia nigra dopamine neurons

Dopamine neurons of the substantia nigra have long been believed to have multiple aspiny dendrites which receive many glutamatergic synaptic inputs from several regions of the brain. But, here, using high-resolution two-photon confocal microscopy in the mouse brain slices, we found a substantial number of common dendritic spines in the nigral dopamine neurons including thin, mushroom, and stubby types of spines. However, the number of dendritic spines of the dopamine neurons was approximately five times lower than that of CA1 pyramidal neurons. Immunostaining and morphological analysis revealed that glutamatergic shaft synapses were present two times more than spine synapses. Using local two-photon glutamate uncaging techniques, we confirmed that shaft synapses and spine synapses had both AMPA and NMDA receptors, but the AMPA/NMDA current ratios differed. The evoked postsynaptic potentials of spine synapses showed lower amplitudes but longer half-widths than those of shaft synapses. Therefore, we provide the first evidence that the midbrain dopamine neurons have two morphologically and functionally distinct types of glutamatergic synapses, spine synapses and shaft synapses, on the same dendrite. This peculiar organization could be a new basis for unraveling many physiological and pathological functions of the midbrain dopamine neurons.

Dorsolateral striatum is critical for the expression of surprise-induced enhancements in cue associability

The dorsolateral striatum (DLS) is frequently implicated in sensory-motor integration, including the performance of sensory orienting responses (ORs) and learned stimulus-response habits. Our laboratory previously identified a role for the DLS in rats' performance of conditioned ORs to Pavlovian cues for food delivery. Here, we considered whether DLS is also critical to another aspect of attention in associative learning, the surprise-induced enhancement of cue associability. A large behavioral literature shows that a cue present when an expected event is omitted enters into new associations more rapidly when that cue is subsequently paired with food. Research from our laboratory has shown that both cue associability enhancements and conditioned ORs depend on the function of a circuit that includes the amygdala central nucleus and the substantia nigra pars compacta. In three experiments, we explored the involvement of DLS in surprise-induced associability enhancements, using a three-stage serial prediction task that permitted separation of DLS function in registering surprise (prediction error) and enhancing cue associability, and in using that increased associability to learn more rapidly about that cue later. The results showed that DLS is critical to the expression, but not the establishment, of the enhanced cue associability normally produced by surprise in this task. They extend the role of DLS and the amygdalo-nigro-striatal circuit underlying learned orienting to more subtle aspects of attention in associative learning, but are consistent with the general notion that DLS is more important in the expression of previously acquired tendencies than in their acquisition.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task

Goal-directed and habit-based behaviors are driven by multiple but dissociable decision making systems involving several different brain areas, including the hippocampus and dorsal striatum. On repetitive tasks, behavior transitions from goal directed to habit based with experience. Hippocampus has been implicated in initial learning and dorsal striatum in automating behavior, but recent studies suggest that subregions within the dorsal striatum have distinct roles in mediating habit-based and goal-directed behavior. We compared neural activity in the CA1 region of hippocampus with anterior dorsolateral and posterior dorsomedial striatum in rats on a spatial choice task, in which subjects experienced reward delivery changes that forced them to adjust their behavioral strategy. Our results confirm the importance of the hippocampus in evaluating predictive steps during goal-directed behavior, while separate circuits in the basal ganglia integrated relevant information during automation of actions and recognized when new behaviors were needed to continue obtaining rewards.

Goal-directed, habitual and Pavlovian prosocial behavior

Although prosocial behaviors have been widely studied across disciplines, the mechanisms underlying them are not fully understood. Evidence from psychology, biology and economics suggests that prosocial behaviors can be driven by a variety of seemingly opposing factors: altruism or egoism, intuition or deliberation, inborn instincts or learned dispositions, and utility derived from actions or their outcomes. Here we propose a framework inspired by research on reinforcement learning and decision making that links these processes and explains characteristics of prosocial behaviors in different contexts. More specifically, we suggest that prosocial behaviors inherit features of up to three decision-making systems employed to choose between self- and other- regarding acts: a goal-directed system that selects actions based on their predicted consequences, a habitual system that selects actions based on their reinforcement history, and a Pavlovian system that emits reflexive responses based on evolutionarily prescribed priors. This framework, initially described in the field of cognitive neuroscience and machine learning, provides insight into the potential neural circuits and computations shaping prosocial behaviors. Furthermore, it identifies specific conditions in which each of these three systems should dominate and promote other- or self- regarding behavior.

Integrative hippocampal and decision-making neurocircuitry during goal-relevant predictions and encoding

It has become clear that the hippocampus plays a critical role in the identification of new contexts and for the detection of changes in familiar contexts. The hippocampus accomplishes these goals through a continual process of comparing predicted features of a context or situation to those actually experienced. A mismatch between expected and experienced context expectations is thought to lead to the generation of a context prediction error (Mizumori, 2013) that functionally alerts connected brain areas to alter subsequent decision making and response selection. Little is understood about how hippocampal context analyses impact downstream decision processes. This issue is evaluated here first by comparing the nature of the information represented in hippocampus and decision-related midbrain-striatal structures, while rats perform a hippocampal-dependent spatial memory task in which rewards of different value are found at different locations. In contrast to place-specific and egocentric neural representations, neural representations of goal information are broadly distributed in hippocampal and decision neural circuitry, but they appear in different forms for different brain structures. It is suggested that further researching on how goal information processing occurs in hippocampus and decision neural circuitry may reveal insights into the nature of the interaction between memory and decision systems. The second part of this review describes neural pathways by which hippocampal context information might arrive within the decision circuit. The third section presents a hypothesis that the nature of the interactions between hippocampal and midbrain-striatal circuitry is regulated by the prefrontal cortex.

Responses of rat medial prefrontal cortical neurons to Pavlovian conditioned stimuli and to delivery of appetitive reward

In the present experiments, medial prefrontal cortical (mPFC) neurons were extracellularly recorded by means of tetrode electrodes to examine their possible role in the prediction of appetitive reward. Two different cue tones (CS) and sucrose solution or water reward (US) were associated in a Pavlovian conditioning paradigm. In order to test behavioral correlate of the CS-US association, the head acceleration before the first lick of licking cluster was measured. Neuronal activity changes in the mPFC were analyzed (i) during the CS presentations; (ii) before the first lick of licking clusters; (iii) during consummation; and (iv) we also examined whether consummation was represented in neurons responding to the CSs. There was a difference between the head accelerations to the different USs during early or late occurring first approaches, but there was no such a difference during intercluster approaches. A significant proportion of neurons changed their firing rate during the CS presentation, before the first lick of licking cluster or during licking of the reward. Both, excitatory and inhibitory responses were observed. A subpopulation of neurons responding to the CSs also responded during reward consumption. Differential population activities of excitatory neurons were recorded in response to the different CSs, CS evoked approach behaviors and consumption of different rewards. Neuronal responses also discriminated among the CSs and trials with or without consummation. These results provided evidence for the involvement of mPFC neurons in the prediction, representation and organization of conditioned behavioral actions, such as approaches to rewards and consummation.

From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making

The striatum is critical for reward-guided and habitual behavior. Anatomical and interference studies suggest a functional heterogeneity within striatum. Medial regions, such as nucleus accumbens core and dorsal medial striatum play roles in goal-directed behavior, while dorsal lateral striatum is critical for control of habitual action. Subdivisions of striatum are topographically connected with different cortical and subcortical structures forming channels that carry information related to limbic, associative, and sensorimotor functions. Here, we describe data showing that as one progresses from ventral-medial to dorsal-lateral striatum, there is a shift from more prominent value encoding to activity more closely related to associative and motor aspects of decision-making. In addition, we will describe data suggesting that striatal circuits work in parallel to control behavior and that regions within striatum can compensate for each other when functions are disrupted.

Striatal firing rate reflects head movement velocity

Although the basal ganglia have long been implicated in the initiation of actions, their contribution to movement remains a matter of dispute. Using wireless multi-electrode recording and motion tracking, we examined the relationship between single-unit activity in the sensorimotor striatum and movement kinematics. We recorded single-unit activity from medium spiny projection neurons and fast-spiking interneurons while monitoring the movements of mice using motion tracking. In Experiment 1, we trained mice to generate movements reliably by water-depriving them and giving them periodic cued sucrose rewards. We found high correlations between single-unit activity and movement velocity in particular directions. This correlation was found in both putative medium spiny projection neurons and fast-spiking interneurons. In Experiment 2, to rule out the possibility that the observed correlations were due to reward expectancy, we repeated the same procedure but added trials in which sucrose delivery was replaced by an aversive air puff stimulus. The air puff generated avoidance movements that were clearly different from movements on rewarded trials, but the same neurons that showed velocity correlation on reward trials exhibited a similar correlation on air puff trials. These experiments show for the first time that the firing rate of striatal neurons reflects movement velocity for different types of movements, whether to seek rewards or to avoid harm.

Homeostatic regulation of memory systems and adaptive decisions

While it is clear that many brain areas process mnemonic information, understanding how their interactions result in continuously adaptive behaviors has been a challenge. A homeostatic-regulated prediction model of memory is presented that considers the existence of a single memory system that is based on a multilevel coordinated and integrated network (from cells to neural systems) that determines the extent to which events and outcomes occur as predicted. The “multiple memory systems of the brain” have in common output that signals errors in the prediction of events and/or their outcomes, although these signals differ in terms of what the error signal represents (e.g., hippocampus: context prediction errors vs. midbrain/striatum: reward prediction errors). The prefrontal cortex likely plays a pivotal role in the coordination of prediction analysis within and across prediction brain areas. By virtue of its widespread control and influence, and intrinsic working memory mechanisms. Thus, the prefrontal cortex supports the flexible processing needed to generate adaptive behaviors and predict future outcomes. It is proposed that prefrontal cortex continually and automatically produces adaptive responses according to homeostatic regulatory principles: prefrontal cortex may serve as a controller that is intrinsically driven to maintain in prediction areas an experience-dependent firing rate set point that ensures adaptive temporally and spatially resolved neural responses to future prediction errors. This same drive by prefrontal cortex may also restore set point firing rates after deviations (i.e. prediction errors) are detected. In this way, prefrontal cortex contributes to reducing uncertainty in prediction systems. An emergent outcome of this homeostatic view may be the flexible and adaptive control that prefrontal cortex is known to implement (i.e. working memory) in the most challenging of situations. Compromise to any of the prediction circuits should result in rigid and suboptimal decision making and memory as seen in addiction and neurological disease. © 2013 The Authors. Hippocampus Published by Wiley Periodicals, Inc.

Differential entrainment and learning-related dynamics of spike and local field potential activity in the sensorimotor and associative striatum

Parallel cortico-basal ganglia loops are thought to have distinct but interacting functions in motor learning and habit formation. In rats, the striatal projection neuron populations (MSNs) in the dorsolateral and dorsomedial striatum, respectively corresponding to sensorimotor and associative regions of the striatum, exhibit contrasting dynamics as rats acquire T-maze tasks (Thorn et al., 2010). Here, we asked whether these patterns could be related to the activity of local interneuron populations in the striatum and to the local field potential activity recorded simultaneously in the corresponding regions. We found that dorsolateral and dorsomedial striatal fast-spiking interneurons exhibited task-specific and training-related dynamics consistent with those of corresponding MSN populations. Moreover, both MSNs and interneuron populations in both regions became entrained to theta-band (5-12 Hz) frequencies during task acquisition. However, the predominant entrainment frequencies were different for the sensorimotor and associative zones. Dorsolateral striatal neurons became entrained mid-task to oscillations centered ∼ 5 Hz, whereas simultaneously recorded neurons in the dorsomedial region became entrained to higher frequency (∼ 10 Hz) rhythms. These region-specific patterns of entrainment evolved dynamically with the development of region-specific patterns of interneuron and MSN activity, indicating that, with learning, these two striatal regions can develop different frequency-modulated circuit activities in parallel. We suggest that such differential entrainment of sensorimotor and associative neuronal populations, acquired through learning, could be critical for coordinating information flow throughout each trans-striatal network while simultaneously enabling nearby components of the separate networks to operate independently.

A new window to understanding individual differences in reward sensitivity from attentional networks

Existing evidence suggests that the presence of reward cues modifies the activity in attentional networks, however, the nature of these influences remains poorly understood. Here, we performed independent component analysis (ICA) in two fMRI datasets corresponding to two incentive delay tasks, which compared the response to reward (money and erotic pictures) and neutral cues, and yielded activations in the ventral striatum using a general linear model approach. Across both experiments, ICA revealed that both the right frontoparietal network and default mode network time courses were positively and negatively modulated by reward cues, respectively. Moreover, this dual neural response pattern was enhanced in individuals with strong reward sensitivity. Therefore, ICA may be a complementary tool to investigate the relevant role of attentional networks on reward processing, and to investigate reward sensitivity in normal and pathological populations.

Functional reorganization of neural networks during repeated exposure to the traumatic memory in posttraumatic stress disorder: an exploratory fMRI study

BACKGROUND

Repeated exposure to the traumatic memory (RETM) is a common component of treatments for posttraumatic stress disorder (PTSD). This treatment is based on a fear extinction model; however, the degree to which this treatment actually engages and modifies neural networks mediating fear extinction is unknown. Therefore, the purpose of the current exploratory study was to define the dynamic changes in neural processing networks while participants completed a novel adaptation of RETM.

METHOD

Participants were adult women (N = 16) with PTSD related to physical or sexual assault. Prior to scanning, participants provided written narratives of a traumatic event related to their PTSD as well as a neutral control event. RETM during fMRI consisted of 5 sequential presentations of the blocked narrative types, lasting three minutes each. Self-reported anxiety was assessed after each presentation.

RESULTS

Relative to changes in functional connectivity during the neutral control script, RETM was associated with strengthened functional connectivity of the right amygdala with the right hippocampus and right anterior insular cortex, left amygdala with the right insular cortex, medial PFC with right anterior insula, left hippocampus with striatum and dorsal cingulate cortex, and right hippocampus with striatum and orbitofrontal cortex. Greater PTSD severity generally led to less changes in functional connectivity with the right insular cortex.

CONCLUSIONS

These results provide evidence that RETM engages and modifies functional connectivity pathways with neural regions implicated in fear extinction. The results also implicate the engagement of the right insular cortex and striatum during RETM and suggest their importance in human fear extinction to trauma memories. However, comorbidity in the sample and the lack of a control group limit inferences regarding RETM with PTSD populations specifically.

The role of the striatum in social behavior

Where and how does the brain code reward during social behavior? Almost all elements of the brain's reward circuit are modulated during social behavior. The striatum in particular is activated by rewards in social situations. However, its role in social behavior is still poorly understood. Here, we attempt to review its participation in social behaviors of different species ranging from voles to humans. Human fMRI experiments show that the striatum is reliably active in relation to others' rewards, to reward inequity and also while learning about social agents. Social contact and rearing conditions have long-lasting effects on behavior, striatal anatomy and physiology in rodents and primates. The striatum also plays a critical role in pair-bond formation and maintenance in monogamous voles. We review recent findings from single neuron recordings showing that the striatum contains cells that link own reward to self or others' actions. These signals might be used to solve the agency-credit assignment problem: the question of whose action was responsible for the reward. Activity in the striatum has been hypothesized to integrate actions with rewards. The picture that emerges from this review is that the striatum is a general-purpose subcortical region capable of integrating social information into coding of social action and reward.

Nucleus accumbens responses differentiate execution and restraint in reward-directed behavior

Our behavior is powerfully driven by environmental cues that signal the availability of rewarding stimuli. We frequently encounter stimuli-a bowl of candy or an alert from our smartphone-that trigger actions to obtain those rewards, even though there may be positive outcomes associated with not acting. The inability to restrain one's action in the presence of reward-associated cues is one type of impulsive behavior and a component of such maladaptive behaviors as overeating, gambling, and substance abuse. The nucleus accumbens (NAc) is ideally situated to integrate multiple cognitive and affective inputs to bias action via outputs through the basal ganglia. NAc neurons have been shown to respond to cues that predict reward availability, goal-directed behaviors aimed at obtaining them, and delivery of the reward itself. As these processes are typically associated, it is difficult to discern whether signals in the NAc are more closely related to processing reward-predictive aspects of goal-directed behavior or selection of behavioral response. To dissociate these possibilities, we recorded the activity of NAc neurons while rats performed a task in which two different cues both informed rats of reward availability but required them to either press a lever (Go) or withhold pressing (NoGo) to obtain the reward. Individual cue-responsive neurons showed either increases or decreases in activity at cue onset. Increases in activity were larger, and decreases smaller, when rats withheld lever pressing, whether correctly for NoGo trials or in error on Go trials. Thus NAc cue responses correlated with action, regardless of cue type or accuracy.

Reward-modulated motor information in identified striatum neurons

It is widely accepted that dorsal striatum neurons participate in either the direct pathway (expressing dopamine D1 receptors) or the indirect pathway (expressing D2 receptors), controlling voluntary movements in an antagonistically balancing manner. The D1- and D2-expressing neurons are activated and inactivated, respectively, by dopamine released from substantia nigra neurons encoding reward expectation. However, little is known about the functional representation of motor information and its reward modulation in individual striatal neurons constituting the two pathways. In this study, we juxtacellularly recorded the spike activity of single neurons in the dorsolateral striatum of rats performing voluntary forelimb movement in a reward-predictable condition. Some of these neurons were identified morphologically by a combination of juxtacellular visualization and in situ hybridization for D1 mRNA. We found that the striatal neurons exhibited distinct functional activations before and during the forelimb movement, regardless of the expression of D1 mRNA. They were often positively, but rarely negatively, modulated by expecting a reward for the correct motor response. The positive reward modulation was independent of behavioral differences in motor performance. In contrast, regular-spiking and fast-spiking neurons in any layers of the motor cortex displayed only minor and unbiased reward modulation of their functional activation in relation to the execution of forelimb movement. Our results suggest that the direct and indirect pathway neurons cooperatively rather than antagonistically contribute to spatiotemporal control of voluntary movements, and that motor information is subcortically integrated with reward information through dopaminergic and other signals in the skeletomotor loop of the basal ganglia.

The thalamostriatal pathway and cholinergic control of goal-directed action: interlacing new with existing learning in the striatum

The capacity for goal-directed action depends on encoding specific action-outcome associations, a learning process mediated by the posterior dorsomedial striatum (pDMS). In a changing environment, plasticity has to remain flexible, requiring interference between new and existing learning to be minimized, yet it is not known how new and existing learning are interlaced in this way. Here we investigated the role of the thalamostriatal pathway linking the parafascicular thalamus (Pf) with cholinergic interneurons (CINs) in the pDMS in this process. Removing the excitatory input from Pf to the CINs was found to reduce the firing rate and intrinsic activity of these neurons and produced an enduring deficit in goal-directed learning after changes in the action-outcome contingency. Disconnection of the Pf-pDMS pathway produced similar behavioral effects. These data suggest that CINs reduce interference between new and existing learning, consistent with claims that the thalamostriatal pathway exerts state control over learning-related plasticity.

Contextual behavior and neural circuits

Animals including humans engage in goal-directed behavior flexibly in response to items and their background, which is called contextual behavior in this review. Although the concept of context has long been studied, there are differences among researchers in defining and experimenting with the concept. The current review aims to provide a categorical framework within which not only the neural mechanisms of contextual information processing but also the contextual behavior can be studied in more concrete ways. For this purpose, we categorize contextual behavior into three subcategories as follows by considering the types of interactions among context, item, and response: contextual response selection, contextual item selection, and contextual item–response selection. Contextual response selection refers to the animal emitting different types of responses to the same item depending on the context in the background. Contextual item selection occurs when there are multiple items that need to be chosen in a contextual manner. Finally, when multiple items and multiple contexts are involved, contextual item–response selection takes place whereby the animal either chooses an item or inhibits such a response depending on item–context paired association. The literature suggests that the rhinal cortical regions and the hippocampal formation play key roles in mnemonically categorizing and recognizing contextual representations and the associated items. In addition, it appears that the fronto-striatal cortical loops in connection with the contextual information-processing areas critically control the flexible deployment of adaptive action sets and motor responses for maximizing goals. We suggest that contextual information processing should be investigated in experimental settings where contextual stimuli and resulting behaviors are clearly defined and measurable, considering the dynamic top-down and bottom-up interactions among the neural systems for contextual behavior.

Selective effects of dopamine depletion and L-DOPA therapy on learning-related firing dynamics of striatal neurons

Despite evidence that dopamine neurotransmission in the striatum is critical for learning as well as for movement control, little is yet known about how the learning-related dynamics of striatal activity are affected by dopamine depletion, a condition faced in Parkinson's disease. We made localized intrastriatal 6-hydroxydopamine lesions in rats and recorded within the dopamine-depleted sensorimotor striatal zone and its contralateral correspondent as the animals learned a conditional maze task. Rather than producing global, nonspecific elevations in firing rate across the task, the dopamine depletion altered striatal projection neuron activity and fast-spiking interneuron activity selectively, with sharply task-specific and cell type-specific effects, and often, with learning-stage selective effects as well. Striatal projection neurons with strong responses during the maze runs had especially elevated responsiveness during the maze runs. Projection neurons that, instead, fired most strongly before maze running showed elevated pre-start firing rates, but not during maze running, as learning progressed. The intrastriatal dopamine depletion severely affected the learning-related patterning of fast-spiking interneuron ensembles, especially during maze running and after extended training. Remarkably, L-DOPA treatment almost entirely reversed the depletion-induced elevations in pre-run firing of the projection neurons, and elevated their responses around start and end of maze runs. By contrast, L-DOPA failed to normalize fast-spiking interneuron activity. Thus the effects of striatal dopamine depletion and restoration on striatal activity are highly dependent not only on cell type, as previously shown, but also on the behavioral activity called for and the state of behavioral learning achieved.

Action-specific value signals in reward-related regions of the human brain

Estimating the value of potential actions is crucial for learning and adaptive behavior. We know little about how the human brain represents action-specific value outside of motor areas. This is, in part, due to a difficulty in detecting the neural correlates of value using conventional (region of interest) functional magnetic resonance imaging (fMRI) analyses, due to a potential distributed representation of value. We address this limitation by applying a recently developed multivariate decoding method to high-resolution fMRI data in subjects performing an instrumental learning task. We found evidence for action-specific value signals in circumscribed regions, specifically ventromedial prefrontal cortex, putamen, thalamus, and insula cortex. In contrast, action-independent value signals were more widely represented across a large set of brain areas. Using multivariate Bayesian model comparison, we formally tested whether value-specific responses are spatially distributed or coherent. We found strong evidence that both action-specific and action-independent value signals are represented in a distributed fashion. Our results suggest that a surprisingly large number of classical reward-related areas contain distributed representations of action-specific values, representations that are likely to mediate between reward and adaptive behavior.