Dual credit assignment processes underlie dopamine signals in a complex spatial environment

A neural mechanism for learning from delayed postingestive feedback

Animals learn the value of foods on the basis of their postingestive effects and thereby develop aversions to foods that are toxic1-10 and preferences to those that are nutritious11-13. However, it remains unclear how the brain is able to assign credit to flavours experienced during a meal with postingestive feedback signals that can arise after a substantial delay. Here we reveal an unexpected role for the postingestive reactivation of neural flavour representations in this temporal credit-assignment process. To begin, we leverage the fact that mice learn to associate novel14,15, but not familiar, flavours with delayed gastrointestinal malaise signals to investigate how the brain represents flavours that support aversive postingestive learning. Analyses of brain-wide activation patterns reveal that a network of amygdala regions is unique in being preferentially activated by novel flavours across every stage of learning (consumption, delayed malaise and memory retrieval). By combining high-density recordings in the amygdala with optogenetic stimulation of malaise-coding hindbrain neurons, we show that delayed malaise signals selectively reactivate flavour representations in the amygdala from a recent meal. The degree of malaise-driven reactivation of individual neurons predicts the strengthening of flavour responses upon memory retrieval, which in turn leads to stabilization of the population-level representation of the recently consumed flavour. By contrast, flavour representations in the amygdala degrade in the absence of unexpected postingestive consequences. Thus, we demonstrate that postingestive reactivation and plasticity of neural flavour representations may support learning from delayed feedback.

Reward expectation and receipt differentially modulate the spiking of accumbens D1+ and D2+ neurons

The nucleus accumbens (NAc) helps govern motivation to pursue reward. Two distinct sets of NAc projection neurons-expressing dopamine D1 vs. D2 receptors-are thought to promote and suppress motivated behaviors, respectively. However, support for this conceptual framework is limited: in particular, the spiking patterns of these distinct cell types during motivated behavior have been largely unknown. Using optogenetic tagging, we recorded the spiking of identified D1+ and D2+ neurons in the NAc core as unrestrained rats performed an operant task in which motivation to initiate work tracks recent reward rate. D1+ neurons preferentially increased firing as rats initiated trials and fired more when reward expectation was higher. By contrast, D2+ cells preferentially increased firing later in the trial, especially in response to reward delivery-a finding not anticipated from current theoretical models. Our results provide new evidence for the specific contribution of NAc D1+ cells to self-initiated approach behavior and will spur updated models of how D2+ cells contribute to learning.

A corticostriatal learning mechanism linking excess striatal dopamine and auditory hallucinations

Auditory hallucinations are linked to elevated striatal dopamine, but their underlying computational mechanisms have been obscured by regional heterogeneity in striatal dopamine signaling. To address this, we developed a normative circuit model in which corticostriatal plasticity in the ventral striatum is modulated by reward prediction errors to drive reinforcement learning while that in the sensory-dorsal striatum is modulated by sensory prediction errors derived from internal belief to drive self-supervised learning. We then validate the key predictions of this model using dopamine recordings across striatal regions in mice, as well as human behavior in a hybrid learning task. Finally, we find that changes in learning resulting from optogenetic stimulation of the sensory striatum in mice and individual variability in hallucination proneness in humans are best explained by selectively enhancing dopamine levels in the model sensory striatum. These findings identify plasticity mechanisms underlying biased learning of sensory expectations as a biologically plausible link between excess dopamine and hallucinations.

Ventral tegmental area dopamine neural activity switches simultaneously with rule representations in the prefrontal cortex and hippocampus

Multiple brain regions need to coordinate activity to support cognitive flexibility and behavioral adaptation. Neural activity in both the hippocampus (HPC) and prefrontal cortex (PFC) is known to represent spatial context and is sensitive to reward and rule alterations. Midbrain dopamine (DA) activity is key in reward seeking behavior and learning. There is abundant evidence that midbrain DA modulates HPC and PFC activity. However, it remains underexplored how these networks engage dynamically and coordinate temporally when animals must adjust their behavior according to changing reward contingencies. In particular, is there any relationship between DA reward prediction change during rule switching, and rule representation changes in PFC and CA1? We addressed these questions using simultaneous recording of neuronal population activity from the hippocampal area CA1, PFC and ventral tegmental area (VTA) in male TH-Cre rats performing two spatial working memory tasks with frequent rule switches in blocks of trials. CA1 and PFC ensembles showed rule-specific activity both during maze running and at reward locations, with PFC rule coding more consistent across animals compared to CA1. Optogenetically tagged VTA DA neuron firing activity responded to and predicted reward outcome. We found that the correct prediction in DA emerged gradually over trials after rule-switching in coordination with transitions in PFC and CA1 ensemble representations of the current rule after a rule switch, followed by behavioral adaptation to the correct rule sequence. Therefore, our study demonstrates a crucial temporal coordination between the rule representation in PFC/CA1, the dopamine reward signal and behavioral strategy.

Hunger modulates exploration through suppression of dopamine signaling in the tail of striatum

Caloric depletion leads to behavioral changes that help an animal find food and restore its homeostatic balance. Hunger increases exploration and risk-taking behavior, allowing an animal to forage for food despite risks; however, the neural circuitry underlying this change is unknown. Here, we characterize how hunger restructures an animal's spontaneous behavior as well as its directed exploration of a novel object. We show that hunger-induced changes in exploration are accompanied by and result from modulation of dopamine signaling in the tail of the striatum (TOS). Dopamine signaling in the TOS is modulated by internal hunger state through the activity of agouti-related peptide (AgRP) neurons, putative "hunger neurons" in the arcuate nucleus of the hypothalamus. These AgRP neurons are poly-synaptically connected to TOS-projecting dopaminergic neurons through the lateral hypothalamus, the central amygdala, and the periaqueductal grey. We thus delineate a hypothalamic-midbrain circuit that coordinates changes in exploration behavior in the hungry state.

A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors

Striatal acetylcholine and dopamine critically regulate movement, motivation, and reward-related learning. Pauses in cholinergic interneuron (CIN) firing are thought to coincide with dopamine pulses encoding reward prediction errors (RPE) to jointly enable synaptic plasticity. Here, we examine the firing of identified CINs during reward-guided decision-making in freely moving rats and compare this firing to dopamine release. Relationships between CINs, dopamine, and behavior varied strongly by subregion. In the dorsal-lateral striatum, a Go! cue evoked burst-pause CIN spiking, followed by a brief dopamine pulse that was unrelated to RPE. In the dorsal-medial striatum, this cue evoked only a CIN pause, that was curtailed by a movement-selective rebound in firing. Finally, in the ventral striatum, a reward cue evoked RPE-coding increases in both dopamine and CIN firing, without a consistent pause. Our results demonstrate a spatial and temporal dissociation between CIN pauses and dopamine RPE signals and will inform future models of striatal information processing under both normal and pathological conditions.

A neural mechanism for learning from delayed postingestive feedback

Animals learn the value of foods based on their postingestive effects and thereby develop aversions to foods that are toxic1-6 and preferences to those that are nutritious7-14. However, it remains unclear how the brain is able to assign credit to flavors experienced during a meal with postingestive feedback signals that can arise after a substantial delay. Here, we reveal an unexpected role for postingestive reactivation of neural flavor representations in this temporal credit assignment process. To begin, we leverage the fact that mice learn to associate novel15-18, but not familiar, flavors with delayed gastric malaise signals to investigate how the brain represents flavors that support aversive postingestive learning. Surveying cellular resolution brainwide activation patterns reveals that a network of amygdala regions is unique in being preferentially activated by novel flavors across every stage of the learning process: the initial meal, delayed malaise, and memory retrieval. By combining high-density recordings in the amygdala with optogenetic stimulation of genetically defined hindbrain malaise cells, we find that postingestive malaise signals potently and specifically reactivate amygdalar novel flavor representations from a recent meal. The degree of malaise-driven reactivation of individual neurons predicts strengthening of flavor responses upon memory retrieval, leading to stabilization of the population-level representation of the recently consumed flavor. In contrast, meals without postingestive consequences degrade neural flavor representations as flavors become familiar and safe. Thus, our findings demonstrate that interoceptive reactivation of amygdalar flavor representations provides a neural mechanism to resolve the temporal credit assignment problem inherent to postingestive learning.

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

Dopamine Release in the Nucleus Accumbens Core Encodes the General Excitatory Components of Learning

Dopamine release in the nucleus accumbens core (NAcC) is generally considered to be a proxy for phasic firing of the ventral tegmental area dopamine (VTADA) neurons. Thus, dopamine release in NAcC is hypothesized to reflect a unitary role in reward prediction error signaling. However, recent studies reveal more diverse roles of dopamine neurons, which support an emerging idea that dopamine regulates learning differently in distinct circuits. To understand whether the NAcC might regulate a unique component of learning, we recorded dopamine release in NAcC while male rats performed a backward conditioning task where a reward is followed by a neutral cue. We used this task because we can delineate different components of learning, which include sensory-specific inhibitory and general excitatory components. Furthermore, we have shown that VTADA neurons are necessary for both the specific and general components of backward associations. Here, we found that dopamine release in NAcC increased to the reward across learning while reducing to the cue that followed as it became more expected. This mirrors the dopamine prediction error signal seen during forward conditioning and cannot be accounted for temporal-difference reinforcement learning. Subsequent tests allowed us to dissociate these learning components and revealed that dopamine release in NAcC reflects the general excitatory component of backward associations, but not their sensory-specific component. These results emphasize the importance of examining distinct functions of different dopamine projections in reinforcement learning.

From innate to instructed: A new look at perceptual decision-making

Understanding how subjects perceive sensory stimuli in their environment and use this information to guide appropriate actions is a major challenge in neuroscience. To study perceptual decision-making in animals, researchers use tasks that either probe spontaneous responses to stimuli (often described as "naturalistic") or train animals to associate stimuli with experimenter-defined responses. Spontaneous decisions rely on animals' pre-existing knowledge, while trained tasks offer greater versatility, albeit often at the cost of extensive training. Here, we review emerging approaches to investigate perceptual decision-making using both spontaneous and trained behaviors, highlighting their strengths and limitations. Additionally, we propose how trained decision-making tasks could be improved to achieve faster learning and a more generalizable understanding of task rules.

Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Mesolimbic dopamine ramps reflect environmental timescales

Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.

Dopamine transients follow a striatal gradient of reward time horizons

Animals make predictions to guide their behavior and update those predictions through experience. Transient increases in dopamine (DA) are thought to be critical signals for updating predictions. However, it is unclear how this mechanism handles a wide range of behavioral timescales-from seconds or less (for example, if singing a song) to potentially hours or more (for example, if hunting for food). Here we report that DA transients in distinct rat striatal subregions convey prediction errors based on distinct time horizons. DA dynamics systematically accelerated from ventral to dorsomedial to dorsolateral striatum, in the tempo of spontaneous fluctuations, the temporal integration of prior rewards and the discounting of future rewards. This spectrum of timescales for evaluative computations can help achieve efficient learning and adaptive motivation for a broad range of behaviors.

Prioritizing replay when future goals are unknown

Although hippocampal place cells replay nonlocal trajectories, the computational function of these events remains controversial. One hypothesis, formalized in a prominent reinforcement learning account, holds that replay plans routes to current goals. However, recent puzzling data appear to contradict this perspective by showing that replayed destinations lag current goals. These results may support an alternative hypothesis that replay updates route information to build a "cognitive map." Yet no similar theory exists to formalize this view, and it is unclear how such a map is represented or what role replay plays in computing it. We address these gaps by introducing a theory of replay that learns a map of routes to candidate goals, before reward is available or when its location may change. Our work extends the planning account to capture a general map-building function for replay, reconciling it with data, and revealing an unexpected relationship between the seemingly distinct hypotheses.