Cue-Evoked Dopamine Promotes Conditioned Responding during Learning

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Computational models of intrinsic motivation for curiosity and creativity

We link Ivancovsky et al.'s novelty-seeking model (NSM) to computational models of intrinsically motivated behavior and learning. We argue that dissociating different forms of curiosity, creativity, and memory based on the involvement of distinct intrinsic motivations (e.g., surprise and novelty) is essential to empirically test the conceptual claims of the NSM.

Role of dopamine neurons in familiarity

Dopamine neurons signal the salience of environmental stimuli and influence learning, although it is less clear if these neurons also determine the salience of memories. Ventral tegmental area (VTA) dopamine neurons increase their firing in the presence of new objects and reduce it upon repeated, inconsequential exposures, marking the shift from novelty to familiarity. This study investigates how dopamine neuron activity during repeated familiar object exposure affects an animal's preference for new objects in a subsequent novel object recognition (NOR) test. We hypothesize that a single familiarization session will not sufficiently lower dopamine activity, such that the memory of a familiar object remains salient, leading to equal exploration of familiar and novel objects and weaker NOR discrimination. In contrast, multiple familiarization sessions likely suppress dopamine activity more effectively, reducing the salience of the familiar object and enhancing subsequent novelty discrimination. Our experiments in mice indicated that multiple familiarization sessions reduce VTA dopamine neuron activation, as measured by c-Fos expression, and enhance novelty discrimination compared with a single familiarization session. Dopamine neurons that show responsiveness to novelty were primarily located in the paranigral nucleus of the VTA and expressed vesicular glutamate transporter 2 transcripts, marking them as dopamine-glutamate neurons. Chemogenetic inhibition of dopamine neurons during a single session paralleled the effects of multiple sessions, improving NOR. These findings suggest that a critical role of dopamine neurons during the transition from novelty to familiarity is to modulate the salience of an object's memory.

Altered neuronal response to visual food stimuli in adolescents undergoing chronic sleep restriction

STUDY OBJECTIVES

Poor sleep in adolescents can increase the risk of obesity, possibly due to changes in dietary patterns. Prior neuroimaging evidence, mostly in adults, suggests that lacking sleep results in increased response to food cues in reward-processing brain regions. Needed is a clarification of the mechanisms by which food reward processing is altered by the kind of chronic sleep restriction (SR) typically experienced by adolescents. This study aimed to elucidate the impact of sleep duration on response to visual food stimuli in healthy adolescents using functional neuroimaging, hypothesizing increased reward processing response after SR compared to a well-rested condition.

METHODS

Thirty-nine healthy adolescents, 14-17 years old, completed a 3-week protocol: (1) sleep phase stabilization; (2) SR (~6.5 h nightly); and (3) healthy sleep (HS) duration (~9 h nightly). Participants underwent functional MRI while performing a visual food paradigm. Contrasts of food versus nonfood responses were compared within-subject between conditions of SR and HS.

RESULTS

Under SR, there was a greater response to food stimuli compared to HS in a voxel cluster including the left ventral tegmental area and substantia nigra. No change in food appeal rating due to the sleep manipulation was detected.

CONCLUSIONS

Outcomes of this study suggest that SR, as commonly experienced by healthy adolescents, results in the elevated dopaminergic drive of the reward network that may augment motivation to seek food in the context of individual food appeal and inhibitory profiles. Countermeasures that reduce food salience could include promoting consistent HS habits.

Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

A septal-ventral tegmental area circuit drives exploratory behavior

To survive, animals need to balance their exploratory drive with their need for safety. Subcortical circuits play an important role in initiating and modulating movement based on external demands and the internal state of the animal; however, how motivation and onset of locomotion are regulated remain largely unresolved. Here, we show that a glutamatergic pathway from the medial septum and diagonal band of Broca (MSDB) to the ventral tegmental area (VTA) controls exploratory locomotor behavior in mice. Using a self-supervised machine learning approach, we found an overrepresentation of exploratory actions, such as sniffing, whisking, and rearing, when this projection is optogenetically activated. Mechanistically, this role relies on glutamatergic MSDB projections that monosynaptically target a subset of both glutamatergic and dopaminergic VTA neurons. Taken together, we identified a glutamatergic basal forebrain to midbrain circuit that initiates locomotor activity and contributes to the expression of exploration-associated behavior.

Reduction in the activity of VTA/SNc dopaminergic neurons underlies aging-related decline in novelty seeking

Curiosity, or novelty seeking, is a fundamental mechanism motivating animals to explore and exploit environments to improve survival, and is also positively associated with cognitive, intrapersonal and interpersonal well-being in humans. However, curiosity declines as humans age, and the decline even positively predicts the extent of cognitive decline in Alzheimer's disease patients. Therefore, determining the underlying mechanism, which is currently unknown, is an urgent task for the present aging society that is growing at an unprecedented rate. This study finds that seeking behaviors for both social and inanimate novelties are compromised in aged mice, suggesting that the aging-related decline in curiosity and novelty-seeking is a biological process. This study further identifies an aging-related reduction in the activity (manifesting as a reduction in spontaneous firing) of dopaminergic neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc). Finally, this study establishes that this reduction in activity causally underlies the aging-related decline in novelty-seeking behaviors. This study potentially provides an interventional strategy for maintaining high curiosity in the aged population, i.e., compensating for the reduced activity of VTA/SNc dopaminergic neurons, enabling the aged population to cope more smoothly with the present growing aging society, physically, cognitively and socioeconomically.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

The influence of dopamine autoreceptors on temperament and addiction risk

As a major regulator of dopamine (DA), DA autoreceptors (DAARs) exert substantial influence over DA-mediated behaviors. This paper reviews the physiological and behavioral impact of DAARs. Individual differences in DAAR functioning influences temperamental traits such as novelty responsivity and impulsivity, both of which are associated with vulnerability to addictive behavior in animal models and a broad array of externalizing behaviors in humans. DAARs additionally impact the response to psychostimulants and other drugs of abuse. Human PET studies of D2-like receptors in the midbrain provide evidence for parallels to the animal literature. These data lead to the proposal that weak DAAR regulation is a risk factor for addiction and externalizing problems. The review highlights the potential to build translational models of the functional role of DAARs in behavior. It also draws attention to key limitations in the current literature that would need to be addressed to further advance a weak DAAR regulation model of addiction and externalizing risk.

[Effect of Sleep Deprivation on the Metabolism of Hippocampal Amino Acids and Monoamine Neurotransmitters in Mice and Their Behaviors]

Objective

To investigate the effect of sleep deprivation on the metabolism of the hippocampal region in mice.

Methods

The mice were randomly assigned to three groups, a control group, a 24-h sleep deprivation (SD) group, and a 48-h SD group. Each group had 10 mice. The sleep deprivation model was induced by the modified multiple platform method. The mice's anxiety-like behaviors were assessed with the open field test (OFT) and their depression-like behaviors were assessed with the sucrose preference test (SPT), the forced swimming test (FST), and tail suspension test (TST). High performance liquid chromatography (HPLC) was performed to determine the levels of 6 monoamine neurotransmitters, including 5-hydroxytryptamine (5-HT), norepinephrine (NE), dopamine (DA), gamma-aminobutyric acid (GABA), 5-dihydroxyphenylacetic acid (5-DOPAC), and homovanillic acid (HVA), and 4 amino acids, including glutamic acid (Glu), aspartic acid (Asp), serine (Ser), and taurine (Tau), in the hippocampal region. Immunofluorescence staining was performed to examine the expression of glial cells in the hippocampal region of the mice. The main indicators measured were the levels of monoamine neurotransmitters and amino acids.

Results

According to the results of the behavioral analysis, in comparison with the findings for the control group, the 24-h SD mice exhibited increased consumption of sucrose in SFT, significantly decreased total immobility time in FST and TST, and increased total distance covered in OFT, while the 48-h SD mice showed decreased consumption of sucrose in SFT, prolonged total immobility time in FST and TST, and decreased total distance covered in OFT. The results of the HPLC analysis of the monoamine neurotransmitter showed that 24-h SD mice had in their hippocampal region increased levels of DA (P<0.001) and NE (P<0.01) and decreased levels of GABA (P<0.05) in comparison with those of the control mice, while their 5-HT, 5-DOPAC, and HVA levels were not significantly different from those of the control mice. In comparison with those of the control mice, the 48-h SD mice had, in their hippocampal region, decreased levels of 5-HT and NE (all P<0.05), decreased DA (P<0.01), and increased level of GABA (P<0.01), while the levels of 5-DOPAC and HAV were not significantly different. The 48-h SD group showed a significant decrease in the levels of Tau and Glu in comparison with those of the 24-h SD group (all P<0.05). According to the results of immunofluorescence assay, there was no significant difference between the control group and the 24-h SD group in the cell count of glial fibrillary acidic protein (GFAP)-positive cells, while a decline in GFAP-positive cells in comparison with that of the control group was observed in the 48-h SD group.

Conclusion

SD of 24 hours may induce anxiety-like behavioral changes in mice by activating their hippocampal glial cells, upregulating the levels of 5-HT, DA, and NE, and increasing the levels of Glu and Tau in the hippocampal region. SD of 48 hours may induce depression-like behavioral changes in mice by inhibiting the activation of glial cells in the hippocampal region and regulating in the opposite direction the levels of the above-mentioned monoamine neurotransmitters and amino acids in the hippocampal region.

Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping

The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.

Acute inhibition of hunger-sensing AgRP neurons promotes context-specific learning in mice

OBJECTIVE

An environmental context, which reliably predicts food availability, can increase the appetitive food drive within the same environment context. However, hunger is required for the development of such a context-induced feeding (CIF) response, suggesting the neural circuits sensitive to hunger link an internal energy state with a particular environment context. Since Agouti related peptide (AgRP) neurons are activated by energy deficit, we hypothesised that AgRP neurons are both necessary and sufficient to drive CIF.

METHODS

To examine the role of AgRP neurons in the CIF process, we used fibre photometry with GCaMP7f, chemogenetic activation of AgRP neurons, as well as optogenetic control of AgRP neurons to facilitate acute temporal control not permitted with chemogenetics.

RESULTS

A CIF response at test was only observed when mice were fasted during context training and AgRP population activity at test showed an attenuated inhibitory response to food, suggesting increased food-seeking and/or decreased satiety signalling drives the increased feeding response at test. Intriguingly, chemogenetic activation of AgRP neurons during context training did not increase CIF, suggesting precise temporal firing properties may be required. Indeed, termination of AgRP neuronal photostimulation during context training (ON-OFF in context), in the presence or absence of food, increased CIF. Moreover, photoinhibition of AgRP neurons during context training in fasted mice was sufficient to drive a subsequent CIF in the absence of food.

CONCLUSIONS

Our results suggest that AgRP neurons regulate the acquisition of CIF when the acute inhibition of AgRP activity is temporally matched to context exposure. These results establish acute AgRP inhibition as a salient neural event underscoring the effect of hunger on associative learning.

Dopaminergic neurons dynamically update sensory values during olfactory maneuver

Dopaminergic neurons (DANs) drive associative learning to update the value of sensory cues, but their contribution to the assessment of sensory values outside the context of association remains largely unexplored. Here, we show in Drosophila that DANs in the mushroom body encode the innate value of odors and constantly update the current value by inducing plasticity during olfactory maneuver. Our connectome-based network model linking all the way from the olfactory neurons to DANs reproduces the characteristics of DAN responses, proposing a concrete circuit mechanism for computation. Downstream of DANs, odors alone induce value- and dopamine-dependent changes in the activity of mushroom body output neurons, which store the current value of odors. Consistent with this neural plasticity, specific sets of DANs bidirectionally modulate flies' steering in a virtual olfactory environment. Thus, the DAN circuit known for discrete, associative learning also continuously updates odor values in a nonassociative manner.

Role of Dopamine Neurons in Familiarity

Dopamine neurons signal the salience of environmental stimuli, influencing learning and motivation. However, research has not yet identified whether dopamine neurons also modulate the salience of memory content. Dopamine neuron activity in the ventral tegmental area (VTA) increases in response to novel objects and diminishes as objects become familiar through repeated presentations. We proposed that the declined rate of dopamine neuron activity during familiarization affects the salience of a familiar object's memory. This, in turn, influences the degree to which an animal distinguishes between familiar and novel objects in a subsequent novel object recognition (NOR) test. As such, a single familiarization session may not sufficiently reduce dopamine activity, allowing the memory of a familiar object to maintain its salience and potentially attenuating NOR. In contrast, multiple familiarization sessions could lead to more pronounced dopamine activity suppression, strengthening NOR. Our data in mice reveals that, compared to a single session, multiple sessions result in decreased VTA dopamine neuron activation, as indicated by c-Fos measurements, and enhanced novelty discrimination. Critically, when VTA dopamine neurons are chemogenetically inhibited during a single familiarization session, NOR improves, mirroring the effects of multiple familiarization sessions. In summary, our findings highlight the pivotal function of dopamine neurons in familiarity and suggest a role in modulating the salience of memory content.

Striatal Dopamine Signals and Reward Learning

We are constantly bombarded by sensory information and constantly making decisions on how to act. In order to optimally adapt behavior, we must judge which sequences of sensory inputs and actions lead to successful outcomes in specific circumstances. Neuronal circuits of the basal ganglia have been strongly implicated in action selection, as well as the learning and execution of goal-directed behaviors, with accumulating evidence supporting the hypothesis that midbrain dopamine neurons might encode a reward signal useful for learning. Here, we review evidence suggesting that midbrain dopaminergic neurons signal reward prediction error, driving synaptic plasticity in the striatum underlying learning. We focus on phasic increases in action potential firing of midbrain dopamine neurons in response to unexpected rewards. These dopamine neurons prominently innervate the dorsal and ventral striatum. In the striatum, the released dopamine binds to dopamine receptors, where it regulates the plasticity of glutamatergic synapses. The increase of striatal dopamine accompanying an unexpected reward activates dopamine type 1 receptors (D1Rs) initiating a signaling cascade that promotes long-term potentiation of recently active glutamatergic input onto striatonigral neurons. Sensorimotor-evoked glutamatergic input, which is active immediately before reward delivery will thus be strengthened onto neurons in the striatum expressing D1Rs. In turn, these neurons cause disinhibition of brainstem motor centers and disinhibition of the motor thalamus, thus promoting motor output to reinforce rewarded stimulus-action outcomes. Although many details of the hypothesis need further investigation, altogether, it seems likely that dopamine signals in the striatum might underlie important aspects of goal-directed reward-based learning.

Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.

Excitatory and Inhibitory Signaling in the Nucleus Accumbens Encode Different Aspects of a Pavlovian Cue in Sign Tracking and Goal Tracking Rats

When a Pavlovian cue is presented separately from its associated reward, some animals will acquire a sign tracking (ST) response - approach and/or interaction with the cue - while others will acquire a goal tracking response - approach to the site of reward. We have previously shown that cue-evoked excitations in the nucleus accumbens (NAc) encode the vigor of both behaviors; in contrast, reward-related responses diverge over the course of training, possibly reflecting neurochemical differences between sign tracker and goal tracker individuals. However, a substantial subset of neurons in the NAc exhibit inhibitory, rather than excitatory, cue-evoked responses, and the evolution of their signaling during Pavlovian conditioning remains unknown. Using single-neuron recordings in behaving rats, we show that NAc neurons with cue-evoked inhibitions have distinct coding properties from neurons with cue-evoked excitations. Cue-evoked inhibitions become more numerous over the course of training and, like excitations, may encode the vigor of sign tracking and goal tracking behavior. However, the responses of cue-inhibited neurons do not evolve differently between sign tracker and goal tracker individuals. Moreover, cue-evoked inhibitions, unlike excitations, are insensitive to extinction of the cue-reward relationship. Finally, we show that cue-evoked excitations are greatly diminished by reward devaluation, while inhibitory cue responses are virtually unaffected. Overall, these findings converge with existing evidence that cue-excited neurons in NAc, but not cue-inhibited neurons, are profoundly sensitive to the same behavior variations that are often associated with changes in dopamine release.

The respective activation and silencing of striatal direct and indirect pathway neurons support behavior encoding

The basal ganglia are known to control actions and modulate movements. Neuronal activity in the two efferent pathways of the dorsal striatum is critical for appropriate behavioral control. Previous evidence has led to divergent conclusions on the respective engagement of both pathways during actions. Using calcium imaging to evaluate how neurons in the direct and indirect pathways encode behaviors during self-paced spontaneous explorations in an open field, we observed that the two striatal pathways exhibit distinct tuning properties. Supervised learning algorithms revealed that direct pathway neurons encode behaviors through their activation, whereas indirect pathway neurons exhibit behavior-specific silencing. These properties remain stable for weeks. Our findings highlight a complementary encoding of behaviors with congruent activations in the direct pathway encoding multiple accessible behaviors in a given context, and in the indirect pathway encoding the suppression of competing behaviors. This model reconciles previous conflicting conclusions on motor encoding in the striatum.

Changes in dorsomedial striatum activity mediate expression of goal-directed vs. habit-like cue-induced cocaine seeking

A preclinical model of cue exposure therapy, cue extinction, reduces cue-induced cocaine seeking when drug seeking is goal-directed but not habitual. Goal-directed and habitual behaviors differentially rely on the dorsomedial striatum (DMS) and dorsolateral striatum (DLS), but the effects of cue extinction on dorsal striatal responses to cue-induced drug seeking are unknown. We used fiber photometry to examine how dorsal striatal intracellular calcium and extracellular dopamine activity differs between goal-directed and habitual cue-induced cocaine seeking and how it is impacted by cue extinction. Rats trained to self-administer cocaine paired with an audiovisual cue on schedules of reinforcement that promote goal-directed or habitual cocaine seeking had different patterns of dorsal striatal calcium and dopamine responses to cue-reinforced lever presses. Cue extinction reduced calcium and dopamine responses during subsequent drug seeking in the DMS, but not in the DLS. Therefore, cue extinction may reduce goal-directed behavior through its effects on the DMS, whereas habitual behavior and the DLS are unaffected.

Inhibition of dopamine neurons prevents incentive value encoding of a reward cue: With revelations from deep phenotyping

The survival of an organism is dependent on their ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the nucleus accumbens is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCut revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient towards and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.

Significance Statement

Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCut revealed that cue-directed behaviors do not emerge without VTA dopamine. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of VTA dopamine during cue presentation to encode the incentive value of reward cues.

Novelty selectively permits learning-associated plasticity in ventral tegmental-hippocampal-prefrontal circuitry

Modifying established behavior in novel situations is essential, and patients with neuropsychiatric disorders often lack this flexibility. Understanding how novelty affects behavioral flexibility therefore has therapeutic potential. Here, novelty differentially impacts connectivity within the ventral tegmental-hippocampal-medial prefrontal (VTA-HPC-mPFC) circuit, thereby enhancing the ability of mice to overcome established behavioral bias and adapt to new rules. Circuit connectivity was measured by local field potential (LFP) coherence. As mice exposed to novelty learned to overcome previously established spatial bias, the ventral HPC (vHPC) strengthens its coherence with the VTA and mPFC in theta frequency (4-8 Hz). Novelty or learning did not affect circuits involving the dorsal HPC (dHPC). Without novelty, however, mice continued following established spatial bias and connectivity strength remained stable in the VTA-HPC-mPFC circuit. Pharmacologically blocking dopamine D1-receptors (D1Rs) in the vHPC abolished the behavioral and physiological impacts of novelty. Thus, novelty promotes behavioral adaptation by permitting learning-associated plasticity in the vHPC-mPFC and VTA-vHPC circuit, a process mediated by D1Rs in the vHPC.

Locomotor and olfactory responses in dopamine neurons of the Drosophila superior-lateral brain

The Drosophila brain contains about 50 distinct morphological types of dopamine neurons.^1,2,3,4 Physiological studies of Drosophila dopamine neurons have been largely limited to one brain region, the mushroom body,^{5,6,7,8,9,10,11,12,13} where they are implicated in learning.^{14,15,16,17,18} By comparison, we know little about the physiology of other Drosophila dopamine neurons. Interestingly, a recent whole-brain imaging study found that dopamine neuron activity in several fly brain regions is correlated with locomotion.¹⁹ This is notable because many dopamine neurons in the rodent brain are also correlated with locomotion or other movements^{20,21,22,23,24,25,26,27,28,29,30}; however, most rodent studies have focused on learned and rewarded behaviors, and few have investigated dopamine neuron activity during spontaneous (self-timed) movements. In this study, we monitored dopamine neurons in the Drosophila brain during self-timed locomotor movements, focusing on several previously uncharacterized cell types that arborize in the superior-lateral brain, specifically the lateral horn and superior-lateral protocerebrum. We found that activity of all of these dopamine neurons correlated with spontaneous fluctuations in walking speed, with different cell types showing different speed correlations. Some dopamine neurons also responded to odors, but these responses were suppressed by repeated odor encounters. Finally, we found that the same identifiable dopamine neuron can encode different combinations of locomotion and odor in different individuals. If these dopamine neurons promote synaptic plasticity-like the dopamine neurons of the mushroom body-then, their tuning profiles would imply that plasticity depends on a flexible integration of sensory signals, motor signals, and recent experience.

Integrated world modeling theory expanded: Implications for the future of consciousness

Integrated world modeling theory (IWMT) is a synthetic theory of consciousness that uses the free energy principle and active inference (FEP-AI) framework to combine insights from integrated information theory (IIT) and global neuronal workspace theory (GNWT). Here, I first review philosophical principles and neural systems contributing to IWMT's integrative perspective. I then go on to describe predictive processing models of brains and their connections to machine learning architectures, with particular emphasis on autoencoders (perceptual and active inference), turbo-codes (establishment of shared latent spaces for multi-modal integration and inferential synergy), and graph neural networks (spatial and somatic modeling and control). Future directions for IIT and GNWT are considered by exploring ways in which modules and workspaces may be evaluated as both complexes of integrated information and arenas for iterated Bayesian model selection. Based on these considerations, I suggest novel ways in which integrated information might be estimated using concepts from probabilistic graphical models, flow networks, and game theory. Mechanistic and computational principles are also considered with respect to the ongoing debate between IIT and GNWT regarding the physical substrates of different kinds of conscious and unconscious phenomena. I further explore how these ideas might relate to the "Bayesian blur problem," or how it is that a seemingly discrete experience can be generated from probabilistic modeling, with some consideration of analogies from quantum mechanics as potentially revealing different varieties of inferential dynamics. I go on to describe potential means of addressing critiques of causal structure theories based on network unfolding, and the seeming absurdity of conscious expander graphs (without cybernetic symbol grounding). Finally, I discuss future directions for work centered on attentional selection and the evolutionary origins of consciousness as facilitated "unlimited associative learning." While not quite solving the Hard problem, this article expands on IWMT as a unifying model of consciousness and the potential future evolution of minds.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Salience memories formed by value, novelty and aversiveness jointly shape object responses in the prefrontal cortex and basal ganglia

Ecological fitness depends on maintaining object histories to guide future interactions. Recent evidence shows that value memory changes passive visual responses to objects in ventrolateral prefrontal cortex (vlPFC) and substantia nigra reticulata (SNr). However, it is not known whether this effect is limited to reward history and if not how cross-domain representations are organized within the same or different neural populations in this corticobasal circuitry. To address this issue, visual responses of the same neurons across appetitive, aversive and novelty domains were recorded in vlPFC and SNr. Results showed that changes in visual responses across domains happened in the same rather than separate populations and were related to salience rather than valence of objects. Furthermore, while SNr preferentially encoded outcome related salience memory, vlPFC encoded salience memory across all domains in a correlated fashion, consistent with its role as an information hub to guide behavior.

Phosphorylation Signals Downstream of Dopamine Receptors in Emotional Behaviors: Association with Preference and Avoidance

Dopamine regulates emotional behaviors, including rewarding and aversive behaviors, through the mesolimbic dopaminergic pathway, which projects dopamine neurons from the ventral tegmental area to the nucleus accumbens (NAc). Protein phosphorylation is critical for intracellular signaling pathways and physiological functions, which are regulated by neurotransmitters in the brain. Previous studies have demonstrated that dopamine stimulated the phosphorylation of intracellular substrates, such as receptors, ion channels, and transcription factors, to regulate neuronal excitability and synaptic plasticity through dopamine receptors. We also established a novel database called KANPHOS that provides information on phosphorylation signals downstream of monoamines identified by our kinase substrate screening methods, including dopamine, in addition to those reported in the literature. Recent advances in proteomics techniques have enabled us to clarify the mechanisms through which dopamine controls rewarding and aversive behaviors through signal pathways in the NAc. In this review, we discuss the intracellular phosphorylation signals regulated by dopamine in these two emotional behaviors.

Dopamine signaling in the nucleus accumbens core mediates latent inhibition

Studies investigating the neural mechanisms by which associations between cues and predicted outcomes control behavior often use associative learning frameworks to understand the neural control of behavior. These frameworks do not always account for the full range of effects that novelty can have on behavior and future associative learning. Here, in mice, we show that dopamine in the nucleus accumbens core is evoked by novel, neutral stimuli, and the trajectory of this response over time tracked habituation to these stimuli. Habituation to novel cues before associative learning reduced future associative learning, a phenomenon known as latent inhibition. Crucially, trial-by-trial dopamine response patterns tracked this phenomenon. Optogenetic manipulation of dopamine responses to the cue during the habituation period bidirectionally influenced future associative learning. Thus, dopamine signaling in the nucleus accumbens core has a causal role in novelty-based learning in a way that cannot be predicted based on purely associative factors.

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning

A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multiple trials. Here we demonstrate that such a gradual shift occurs both at the level of dopaminergic cellular activity and dopamine release in the ventral striatum in mice. Our results establish a long-sought link between dopaminergic activity and the TD learning algorithm, providing fundamental insights into how the brain associates cues and rewards that are separated in time.

Electrophysiological Evidence of Enhanced Processing of Novel Pornographic Images in Individuals With Tendencies Toward Problematic Internet Pornography Use

Novelty seeking is regarded as a core feature in substance use disorders. However, few studies thus far have investigated this feature in problematic Internet pornography use (PIPU). The main aim of the present study was to examine group differences in electrophysiological activity associated with novelty processing in participants with high tendencies toward PIPU vs. low tendencies using event-related potentials (ERPs). Twenty-seven participants with high tendencies toward PIPU and 25 with low tendencies toward PIPU completed a modified three-stimulus oddball task while electroencephalogram (EEG) was recorded. Participants were instructed to detect neutral target stimuli from distracting stimuli. The distracting stimuli contained a familiar sexual stimulus and a set of novel sexual stimuli. The novel-familiar difference waves were calculated to identify specific group difference in novelty effect. While both groups demonstrated a sustained novelty effect in the late positive potential (LPP) within the 500–800 ms time windows, the novelty effect was greater in the high tendencies toward PIPU group than in the low tendencies toward PIPU group. This result suggests that individuals with high tendencies toward PIPU allocate more attentional resources for novelty processing. Enhanced brain responding to novel sexual stimuli may facilitate pornographic consumption and play an essential role in the development and maintenance of PIPU.

Dopamine errors drive excitatory and inhibitory components of backward conditioning in an outcome-specific manner

For over two decades, phasic activity in midbrain dopamine neurons was considered synonymous with the prediction error in temporal-difference reinforcement learning.^1-4 Central to this proposal is the notion that reward-predictive stimuli become endowed with the scalar value of predicted rewards. When these cues are subsequently encountered, their predictive value is compared to the value of the actual reward received, allowing for the calculation of prediction errors.^5,6 Phasic firing of dopamine neurons was proposed to reflect this computation,^1,2 facilitating the backpropagation of value from the predicted reward to the reward-predictive stimulus, thus reducing future prediction errors. There are two critical assumptions of this proposal: (1) that dopamine errors can only facilitate learning about scalar value and not more complex features of predicted rewards, and (2) that the dopamine signal can only be involved in anticipatory cue-reward learning in which cues or actions precede rewards. Recent work^7-15 has challenged the first assumption, demonstrating that phasic dopamine signals across species are involved in learning about more complex features of the predicted outcomes, in a manner that transcends this value computation. Here, we tested the validity of the second assumption. Specifically, we examined whether phasic midbrain dopamine activity would be necessary for backward conditioning-when a neutral cue reliably follows a rewarding outcome.^16-20 Using a specific Pavlovian-to-instrumental transfer (PIT) procedure,^21-23 we show rats learn both excitatory and inhibitory components of a backward association, and that this association entails knowledge of the specific identity of the reward and cue. We demonstrate that brief optogenetic inhibition of VTA_DA neurons timed to the transition between the reward and cue reduces both of these components of backward conditioning. These findings suggest VTA_DA neurons are capable of facilitating associations between contiguously occurring events, regardless of the content of those events. We conclude that these data may be in line with suggestions that the VTA_DA error acts as a universal teaching signal. This may provide insight into why dopamine function has been implicated in myriad psychological disorders that are characterized by very distinct reinforcement-learning deficits.

Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning

Learning associations between cues and rewards require the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well trained animals. However, these value-related dopamine responses are absent during early training sessions when cues signal differences in the reward rate. These findings suggest cue-evoked dopamine release conveys differences between outcomes only after extensive training, though it is unclear whether this is unique to when cues signal differences in reward rate, or whether this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we used a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout learning. While female rats exhibited higher levels of conditioned responding, a faster latency to respond, and elevated post-reward head entries relative to male rats, there were no sex differences in the dopamine response to cues. Multiple training sessions were required before cue-evoked dopamine release signaled differences in reward size. Reward-evoked dopamine release scaled with reward size, though females displayed lower reward-evoked dopamine responses relative to males. Conditioned responding related to the decrease in the peak reward-evoked dopamine response and not to cue-evoked dopamine release. Collectively, these data illustrate sex differences in behavioral responding as well as in reward-evoked dopamine release during Pavlovian learning.

Hippocampal Disinhibition Reduces Contextual and Elemental Fear Conditioning While Sparing the Acquisition of Latent Inhibition

Hippocampal neural disinhibition, i.e., reduced GABAergic inhibition, is a key feature of schizophrenia pathophysiology. The hippocampus is an important part of the neural circuitry that controls fear conditioning and can also modulate prefrontal and striatal mechanisms, including dopamine signaling, which play a role in salience modulation. Consequently, hippocampal neural disinhibition may contribute to impairments in fear conditioning and salience modulation reported in schizophrenia. Therefore, we examined the effect of ventral hippocampus (VH) disinhibition in male rats on fear conditioning and salience modulation, as reflected by latent inhibition (LI), in a conditioned emotional response (CER) procedure. A flashing light was used as the conditioned stimulus (CS), and conditioned suppression was used to index conditioned fear. In experiment 1, VH disinhibition via infusion of the GABA-A receptor antagonist picrotoxin before CS pre-exposure and conditioning markedly reduced fear conditioning to both the CS and context; LI was evident in saline-infused controls but could not be detected in picrotoxin-infused rats because of the low level of fear conditioning to the CS. In experiment 2, VH picrotoxin infusions only before CS pre-exposure did not affect the acquisition of fear conditioning or LI. Together, these findings indicate that VH neural disinhibition disrupts contextual and elemental fear conditioning, without affecting the acquisition of LI. The disruption of fear conditioning resembles aversive conditioning deficits reported in schizophrenia and may reflect a disruption of neural processing both within the hippocampus and in projection sites of the hippocampus.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

Slowly evolving dopaminergic activity modulates the moment-to-moment probability of reward-related self-timed movements

Clues from human movement disorders have long suggested that the neurotransmitter dopamine plays a role in motor control, but how the endogenous dopaminergic system influences movement is unknown. Here we examined the relationship between dopaminergic signaling and the timing of reward-related movements in mice. Animals were trained to initiate licking after a self-timed interval following a start-timing cue; reward was delivered in response to movements initiated after a criterion time. The movement time was variable from trial-to-trial, as expected from previous studies. Surprisingly, dopaminergic signals ramped-up over seconds between the start-timing cue and the self-timed movement, with variable dynamics that predicted the movement/reward time on single trials. Steeply rising signals preceded early lick-initiation, whereas slowly rising signals preceded later initiation. Higher baseline signals also predicted earlier self-timed movements. Optogenetic activation of dopamine neurons during self-timing did not trigger immediate movements, but rather caused systematic early-shifting of movement initiation, whereas inhibition caused late-shifting, as if modulating the probability of movement. Consistent with this view, the dynamics of the endogenous dopaminergic signals quantitatively predicted the moment-by-moment probability of movement initiation on single trials. We propose that ramping dopaminergic signals, likely encoding dynamic reward expectation, can modulate the decision of when to move.

Dissociable contributions of phasic dopamine activity to reward and prediction

Sensory cues that precede reward acquire predictive (expected value) and incentive (drive reward-seeking action) properties. Mesolimbic dopamine neurons' responses to sensory cues correlate with both expected value and reward-seeking action. This has led to the proposal that phasic dopamine responses may be sufficient to inform value-based decisions, elicit actions, and/or induce motivational states; however, causal tests are incomplete. Here, we show that direct dopamine neuron stimulation, both calibrated to physiological and greater intensities, at the time of reward can be sufficient to induce and maintain reward seeking (reinforcing) although replacement of a cue with stimulation is insufficient to induce reward seeking or act as an informative cue. Stimulation of descending cortical inputs, one synapse upstream, are sufficient for reinforcement and cues to future reward. Thus, physiological activation of mesolimbic dopamine neurons can be sufficient for reinforcing properties of reward without being sufficient for the predictive and incentive properties of cues.

Dopamine release and its control over early Pavlovian learning differs between the NAc core and medial NAc shell

Dopamine neurons respond to cues to reflect the value of associated outcomes. These cue-evoked dopamine responses can encode the relative rate of reward in rats with extensive Pavlovian training. Specifically, a cue that always follows the previous reward by a short delay (high reward rate) evokes a larger dopamine response in the nucleus accumbens (NAc) core relative to a distinct cue that always follows the prior reward by a long delay (low reward rate). However, it was unclear if these reward rate dopamine signals are evident during early Pavlovian training sessions and across NAc subregions. To address this, we performed fast-scan cyclic voltammetry recordings of dopamine levels to track the pattern of cue- and reward-evoked dopamine signals in the NAc core and medial NAc shell. We identified regional differences in the progression of cue-evoked dopamine signals across training. However, the dopamine response to cues did not reflect the reward rate in either the NAc core or the medial NAc shell during early training sessions. Pharmacological experiments found that dopamine-sensitive conditioned responding emerged in the NAc core before the medial NAc shell. Together, these findings illustrate regional differences in NAc dopamine release and its control over behavior during early Pavlovian learning.

Neural Representation of Costs and Rewards in Decision Making

Decision making is crucial for animal survival because the choices they make based on their current situation could influence their future rewards and could have potential costs. This review summarises recent developments in decision making, discusses how rewards and costs could be encoded in the brain, and how different options are compared such that the most optimal one is chosen. The reward and cost are mainly encoded by the forebrain structures (e.g., anterior cingulate cortex, orbitofrontal cortex), and their value is updated through learning. The recent development on dopamine and the lateral habenula's role in reporting prediction errors and instructing learning will be emphasised. The importance of dopamine in powering the choice and accounting for the internal state will also be discussed. While the orbitofrontal cortex is the place where the state values are stored, the anterior cingulate cortex is more important when the environment is volatile. All of these structures compare different attributes of the task simultaneously, and the local competition of different neuronal networks allows for the selection of the most appropriate one. Therefore, the total value of the task is not encoded as a scalar quantity in the brain but, instead, as an emergent phenomenon, arising from the computation at different brain regions.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.

A New Paradigm for Training Hyperactive Dopamine Transporter Knockout Rats: Influence of Novel Stimuli on Object Recognition

Attention deficit hyperactivity disorder (ADHD) is believed to be connected with a high level of hyperactivity caused by alterations of the control of dopaminergic transmission in the brain. The strain of hyperdopaminergic dopamine transporter knockout (DAT-KO) rats represents an optimal model for investigating ADHD-related pathological mechanisms. The goal of this work was to study the influence of the overactivated dopamine system in the brain on a motor cognitive task fulfillment. The DAT-KO rats were trained to learn an object recognition task and store it in long-term memory. We found that DAT-KO rats can learn to move an object and retrieve food from the rewarded familiar objects and not to move the non-rewarded novel objects. However, we observed that the time of task performance and the distances traveled were significantly increased in DAT-KO rats in comparison with wild-type controls. Both groups of rats explored the novel objects longer than the familiar cubes. However, unlike controls, DAT-KO rats explored novel objects significantly longer and with fewer errors, since they preferred not to move the non-rewarded novel objects. After a 3 months' interval that followed the training period, they were able to retain the learned skills in memory and to efficiently retrieve them. The data obtained indicate that DAT-KO rats have a deficiency in learning the cognitive task, but their hyperactivity does not prevent the ability to learn a non-spatial cognitive task under the presentation of novel stimuli. The longer exploration of novel objects during training may ensure persistent learning of the task paradigm. These findings may serve as a base for developing new ADHD learning paradigms.

Rare rewards amplify dopamine responses

Dopamine prediction error responses are essential components of universal learning mechanisms. However, it is unknown whether individual dopamine neurons reflect the shape of reward distributions. Here, we used symmetrical distributions with differently weighted tails to investigate how the frequency of rewards and reward prediction errors influence dopamine signals. Rare rewards amplified dopamine responses, even when conventional prediction errors were identical, indicating a mechanism for learning the complexities of real-world incentives.

What is dopamine doing in model-based reinforcement learning?

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

Dopamine, Updated: Reward Prediction Error and Beyond

Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine's role in movement, motivation, and goal-directed planning.

Dopamine: The Neuromodulator of Long-Term Synaptic Plasticity, Reward and Movement Control

Dopamine (DA) is a key neurotransmitter involved in multiple physiological functions including motor control, modulation of affective and emotional states, reward mechanisms, reinforcement of behavior, and selected higher cognitive functions. Dysfunction in dopaminergic transmission is recognized as a core alteration in several devastating neurological and psychiatric disorders, including Parkinson's disease (PD), schizophrenia, bipolar disorder, attention deficit hyperactivity disorder (ADHD) and addiction. Here we will discuss the current insights on the role of DA in motor control and reward learning mechanisms and its involvement in the modulation of synaptic dynamics through different pathways. In particular, we will consider the role of DA as neuromodulator of two forms of synaptic plasticity, known as long-term potentiation (LTP) and long-term depression (LTD) in several cortical and subcortical areas. Finally, we will delineate how the effect of DA on dendritic spines places this molecule at the interface between the motor and the cognitive systems. Specifically, we will be focusing on PD, vascular dementia, and schizophrenia.

Tonic firing mode of midbrain dopamine neurons continuously tracks reward values changing moment-by-moment

Animal behavior is regulated based on the values of future rewards. The phasic activity of midbrain dopamine neurons signals these values. Because reward values often change over time, even on a subsecond-by-subsecond basis, appropriate behavioral regulation requires continuous value monitoring. However, the phasic dopamine activity, which is sporadic and has a short duration, likely fails continuous monitoring. Here, we demonstrate a tonic firing mode of dopamine neurons that effectively tracks changing reward values. We recorded dopamine neuron activity in monkeys during a Pavlovian procedure in which the value of a cued reward gradually increased or decreased. Dopamine neurons tonically increased and decreased their activity as the reward value changed. This tonic activity was evoked more strongly by non-burst spikes than burst spikes producing a conventional phasic activity. Our findings suggest that dopamine neurons change their firing mode to effectively signal reward values in a given situation.

Neurobiology of reward-related learning

A major goal in psychology is to understand how environmental stimuli associated with primary rewards come to function as conditioned stimuli, acquiring the capacity to elicit similar responses to those elicited by primary rewards. Our neurobiological model is predicated on the Hebbian idea that concurrent synaptic activity on the primary reward neural substrate-proposed to be ventral tegmental area (VTA) dopamine (DA) neurons-strengthens the synapses involved. We propose that VTA DA neurons receive both a strong unconditioned stimulus signal (acetylcholine stimulation of DA cells) from the primary reward capable of unconditionally activating DA cells and a weak stimulus signal (glutamate stimulation of DA cells) from the neutral stimulus. Through joint stimulation the weak signal is potentiated and capable of activating the VTA DA cells, eliciting a conditioned response. The learning occurs when this joint stimulation initiates intracellular second-messenger cascades resulting in enhanced glutamate-DA synapses. In this review we present evidence that led us to propose this model and the most recent evidence supporting it.

The Tail of the Striatum: From Anatomy to Connectivity and Function

The dorsal striatum, the largest subcortical structure of the basal ganglia, is critical in controlling motor, procedural, and reinforcement-based behaviors. Although in mammals the striatum extends widely along the rostro-caudal axis, current knowledge and derived theories about its anatomo-functional organization largely rely on results obtained from studies of its rostral sectors, leading to potentially oversimplified working models of the striatum as a whole. Recent findings indicate that the extreme caudal part of the striatum, also referred to as the tail of striatum (TS), represents an additional functional domain. Here, we provide an overview of past and recent studies revealing that the TS displays a heterogeneous cell-type-specific organization, and a unique input-output connectivity, which poises the TS as an integrator of sensory processing.

Midbrain circuits of novelty processing

Novelty triggers an increase in orienting behavior that is critical to evaluate the potential salience of unknown events. As novelty becomes familiar upon repeated encounters, this increase in response rapidly habituates as a form of behavioral adaptation underlying goal-directed behaviors. Many neurodevelopmental, psychiatric and neurodegenerative disorders are associated with abnormal responses to novelty and/or familiarity, although the neuronal circuits and cellular/molecular mechanisms underlying these natural behaviors in the healthy brain are largely unknown, as is the maladaptive processes that occur to induce impairment of novelty signaling in diseased brains. In rodents, the development of cutting-edge tools that allow for measurements of real time activity dynamics in selectively identified neuronal ensembles by gene expression signatures is beginning to provide advances in understanding the neural bases of the novelty response. Accumulating evidence indicate that midbrain circuits, the majority of which linked to dopamine transmission, promote exploratory assessments and guide approach/avoidance behaviors to different types of novelty via specific projection sites. The present review article focuses on midbrain circuit analysis relevant to novelty processing and habituation with familiarity.

The effect of feedback novelty on neural correlates of feedback processing

It has been suggested that stimulus novelty itself can be rewarding and recent evidence suggests that novelty processing and reward processing share common neural mechanisms. For feedback processing, this can be beneficial as well as detrimental: If novelty lends a rewarding characteristic to a stimulus, then this should particularly decrease the impact of negative feedback. The present study investigated whether such an effect of feedback novelty on feedback processing is reflected in electrophysiological markers of reinforcement learning (feedback-related negativity, FRN) and feedback processing (feedback-P300) in a simple decision-making task. In this task, participants had to chose between two stimuli in a learning trial followed by a novel or a familiar feedback stimulus. Learning from feedback allowed them to optimize their payoff in a later test trial. As expected, we found that the FRN effect, i.e. the difference between the FRN amplitudes after negative and positive feedback, was reduced for novel compared to familiar feedback stimuli. In addition, the amplitude of the feedback-P300 was decreased by feedback novelty, both for the anterior P3a and the posterior P3b. Together, these results indicate that feedback novelty can affect feedback processing as reflected by feedback-related brain activity.

Dopamine Signals Learn New Tricks

In this issue of Neuron, Morrens et al. (2020) show that stimulus-evoked dopamine responses are enhanced by novelty and increase the rate at which animals acquire conditioned responses. These results provide a candidate neural mechanism for latent inhibition and illustrate a new role of dopamine signals in learning.