Cue-Evoked Dopamine Promotes Conditioned Responding during Learning

Nucleus Accumbens Dopamine Encodes the Trace Period during Appetitive Pavlovian Conditioning

Pavlovian conditioning tasks have been used to identify the neural systems involved with learning cue-outcome relationships. In delay conditioning, the conditioned stimulus (CS) overlaps or co-terminates with the unconditioned stimulus (US). Prior studies demonstrate that dopamine in the nucleus accumbens (NAc) regulates behavioral responding during delay conditioning. Furthermore, the dopamine response to the CS reflects the relative value of the upcoming reward in these tasks. In contrast to delay conditioning, trace conditioning involves a "trace" period separating the end of the CS and the US delivery. While dopamine has been implicated in trace conditioning, no studies have examined how NAc dopamine responds to reward-related stimuli in these tasks. Here, we developed a within-subject trace conditioning task where distinct CSs signaled either a short trace period (5 s) or a long trace period (55 s) prior to food reward delivery. Male rats exhibited greater conditioned responding and a faster response latency to the Short Trace CS relative to the Long Trace CS. Voltammetry recordings in the NAc found that the CS-evoked dopamine response increased on Short Trace trials but decreased on Long Trace trials. Conversely, US-evoked dopamine responses were greater on Long Trace trials relative to Short Trace trials. The CS dopamine response correlated with the response latency and not with conditioned responding. Furthermore, the relationship between CS dopamine and latency was best explained by an exponential function. Our results collectively illustrate that the trace period is encoded by the bidirectional NAc dopamine response to the CS during pavlovian conditioning.

Nucleus accumbens dopamine encodes the trace period during appetitive Pavlovian conditioning

Pavlovian conditioning tasks have been used to identify the neural systems involved with learning cue-outcome relationships. In delay conditioning, the conditioned stimulus (CS) overlaps or co-terminates with the delivery of the unconditioned stimulus (US). Prior studies demonstrate that dopamine in the nucleus accumbens (NAc) regulates behavioral responding during delay conditioning. Furthermore, the dopamine response to the CS reflects the relative value of the upcoming reward in these tasks. In contrast to delay conditioning, trace conditioning involves a 'trace' period separating the end of the CS and the US delivery. While dopamine has been implicated in trace conditioning, no studies have examined how NAc dopamine responds to reward-related stimuli in these tasks. Here, we developed a within-subject trace conditioning task where distinct CSs signaled either a short trace period (5s) or a long trace period (55s) prior to food reward delivery. Male rats exhibited greater conditioned responding and a faster response latency to the Short Trace CS relative to the Long Trace CS. Voltammetry recordings in the NAc found that the CS-evoked dopamine response increased on Short Trace trials but decreased on Long Trace trials. Conversely, US-evoked dopamine responses were greater on Long Trace trials relative to Short Trace trials. The CS dopamine response correlated with the response latency and not with conditioned responding. Furthermore, the relationship between CS dopamine and latency was best explained by an exponential function. Our results collectively illustrate that the trace period is encoded by the bidirectional NAc dopamine response to the CS during Pavlovian conditioning.

Significance statement

Learning to associate a cue with given outcome is a fundamental process underlying reward seeking behavior. Striatal dopamine is important for associating cues with rewards during Pavlovian conditioning. However, it is unclear how the dopamine system responds to cues during trace conditioning when there is temporal gap between the cue and reward. Here, we performed voltammetry recordings of striatal dopamine levels in male rats during trace conditioning. We find that cue-evoked dopamine signals encode the trace period and is related to the response latency. While prior reports find dopamine neurons signal the relative reward value by increases in dopamine levels, we demonstrate that the dopamine response to reward-predictive cues can signal the reward value through bidirectional changes in dopamine transmission.

Astrocyte ensembles manipulated with AstroLight tune cue-motivated behavior

Astrocytes, dynamic cells crucial to brain function, have traditionally been overshadowed by the emphasis on neuronal activity in regulating behavior. Unlike neurons, which are organized into ensembles that encode different brain representations, astrocytes have long been considered a homogeneous population. This is partly because of the lack of tools available to map and manipulate specific subsets of astrocytes based on their functional activity, obscuring the extent of their specialization in circuits. Here, using AstroLight, a tool that translates astrocytic activity-mediated calcium signals into gene expression in a light-dependent manner, we have identified an astrocytic ensemble, a functionally specified subset of astrocytes that emerges upon activity during cue-motivated behaviors in the nucleus accumbens, an integrator hub in the reward system. Furthermore, through gain-of-function and loss-of-function manipulations, we demonstrate that this ensemble is essential for modulating cue-reward associations. These findings highlight the specialization of astrocytes into ensembles and their fine-tuning role in shaping salient behavior.

Review: Dopamine, Serotonin, and the Translational Neuroscience of Aggression in Autism Spectrum Disorder

Objective

Autism spectrum disorder (ASD) is a neurodevelopmental disorder with a 1% to 2% prevalence in children. In addition to social communication deficits and restricted or repetitive behavior, ASD is often characterized by a heightened propensity for aggression. In fact, aggressive behavior is the primary reason for hospitalization in children with ASD, and current treatment options, despite some efficacy, are often associated with prominent side effects. Despite such high clinical toll, the neurobiology of aggression in ASD remains poorly understood.

Method

The neural circuits linked to both ASD and aggression were reviewed, with the goal of identifying overlapping components to help guide future treatment development. In discussing the clinical phenotype of aggression in ASD, some of the triggers and risk factors were noted to differ from those that cause aggression in neurotypical children. Preclinical and clinical studies on the neurobiology of aggression and ASD were synthesized to combine evidence from genetics, neuroimaging, pharmacology, and circuit manipulations. Dopamine and serotonin, 2 neuromodulators that contribute to development and behavioral control, were specifically studied.

Results

The literature indicates that the intricate interplay of the dopamine and serotonin systems has a pivotal role in shaping behavior, including the expression of aggression.

Conclusion

Understanding the balance between dopamine as an accelerator and serotonin as a brake may provide insights into the mechanisms of aggression in children with ASD. Although much work remains to be done, new perspectives promise to bridge the gap between human and animal studies and pinpoint the neurobiology of aggression in ASD.

Diversity & Inclusion Statement

One or more of the authors of this paper self-identifies as a member of one or more historically underrepresented sexual and/or gender groups in science. We actively worked to promote sex and gender balance in our author group.

Exploring novelty to unpack the black-box of motivation

Murayama and Jach point out that we do not sufficiently understand the constructs and mental computations underlying higher-order motivated behaviors. Although this may be generally true, we would like to add and contribute to the discussion by outlining how interdisciplinary research on novelty-evoked exploration has advanced the study of learning and curiosity.

Opioid-environment interaction: Contrasting effects of morphine administered in a novel versus familiar environment on acute and repeated morphine induced behavioral effects and on acute morphine ERK activation in reward associated brain areas

We report that environmental context can have a major impact on morphine locomotor behavior and ERK effects. We manipulated environmental context in terms of an environmental novelty/ familiarity dimension and measured morphine behavioral effects in both acute and chronic morphine treatment protocols. Wistar rats (n=7 per group) were injected with morphine 10 mg/kg or vehicle (s.c.), and immediately placed into an arena for 5 min, and locomotor activity was measured after one or 5 days. The morphine treatments were initiated either when the environment was novel or began after the rats had been familiarized with the arena by being given 5 daily nondrug tests in the arena. The results showed that acute and chronic morphine effects were strongly modified by whether the environment was novel or familiar. Acute morphine administered in a novel environment increased ERK activity more substantially in several brain areas, particularly in reward-associated areas such as the VTA in comparison to when morphine was given in a familiar environment. Repeated morphine treatments initiated in a novel environment induced a strong locomotor sensitization, whereas repeated morphine treatments initiated in a familiar environment did not induce a locomotor stimulant effect but rather a drug discriminative stimulus dis-habituation effect. The marked differential effects of environmental novelty/familiarity and ongoing dopamine activity on acute and chronic morphine treatments may be of potential clinical relevance for opioid drug addiction.

Novelty triggers time-dependent theta oscillatory dynamics in cortical-hippocampal-midbrain circuitry

Rapid adaptation to novel environments is crucial for survival, and this ability is impaired in many neuropsychiatric disorders. Understanding neural adaptation to novelty exposure therefore has therapeutic implications. Here, I found that novelty induces time-dependent theta (4-12Hz) oscillatory dynamics in brain circuits including the medial prefrontal cortex (mPFC), ventral hippocampus (vHPC), and ventral tegmental area (VTA), but not dorsal hippocampus (dHPC), as mice adapt to a novel environment. Local field potential (LFP) recordings were performed while mice were freely behaving in a novel or a familiar arena for 10 min. Initially, mice exhibited increased exploratory behavior upon exposure to novelty, which gradually decreased to levels observed in mice exposed to the familiar arena. Over the same time course, the mPFC, vHPC, and VTA displayed progressively increasing theta power through novelty exposure. Additionally, theta coherence and theta phase synchrony measures demonstrated that novelty weakened the connectivity between these areas, which then gradually strengthened to the level observed in the familiar group. Conversely, mice exposed to the familiar arena showed steady and consistent behavior as well as theta dynamics in all areas. Treatment with a dopamine D1-receptor (D1R) antagonist in the vHPC disrupted neurophysiological adaptation to novelty specifically in the vHPC-mPFC and vHPC-VTA circuits, without affecting behavior. Thus, novelty induces distinct theta dynamics that are not readily dictated by behavior in the mPFC, vHPC, and VTA circuits, a process mediated by D1Rs in the vHPC. The observed time-dependent circuit dynamics in the key learning and memory circuit would provide new insights for treating neuropsychiatric disorders that often show impaired novelty processing.

Pre-existing visual responses in a projection-defined dopamine population explain individual learning trajectories

A key challenge of learning a new task is that the environment is high dimensional-there are many different sensory features and possible actions, with typically only a small reward-relevant subset. Although animals can learn to perform complex tasks that involve arbitrary associations between stimuli, actions, and rewards,1,2,3,4,5,6 a consistent and striking result across varied experimental paradigms is that in initially acquiring such tasks, large differences between individuals are apparent in the learning process.7,8,9,10,11,12 What neural mechanisms contribute to initial task acquisition, and why do some individuals learn a new task much more quickly than others? To address these questions, we recorded longitudinally from dopaminergic (DA) axon terminals in mice learning a visual decision-making task.7 Across striatum, DA responses tracked idiosyncratic and side-specific learning trajectories, consistent with widespread reward prediction error coding across DA terminals. However, even before any rewards were delivered, contralateral-side-specific visual responses were present in DA terminals, primarily in the dorsomedial striatum (DMS). These pre-existing responses predicted the extent of learning for contralateral stimuli. Moreover, activation of these terminals improved contralateral performance. Thus, the initial conditions of a projection-specific and feature-specific DA signal help explain individual learning trajectories. More broadly, this work suggests that functional heterogeneity across DA projections may serve to bias target regions toward learning about different subsets of task features, providing a potential mechanism to address the dimensionality of the initial task learning problem.

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

The nucleus accumbens in reward and aversion processing: insights and implications

The nucleus accumbens (NAc), a central component of the brain's reward circuitry, has been implicated in a wide range of behaviors and emotional states. Emerging evidence, primarily drawing from recent rodent studies, suggests that the function of the NAc in reward and aversion processing is multifaceted. Prolonged stress or drug use induces maladaptive neuronal function in the NAc circuitry, which results in pathological conditions. This review aims to provide comprehensive and up-to-date insights on the role of the NAc in motivated behavior regulation and highlights areas that demand further in-depth analysis. It synthesizes the latest findings on how distinct NAc neuronal populations and pathways contribute to the processing of opposite valences. The review examines how a range of neuromodulators, especially monoamines, influence the NAc's control over various motivational states. Furthermore, it delves into the complex underlying mechanisms of psychiatric disorders such as addiction and depression and evaluates prospective interventions to restore NAc functionality.

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Changes in dorsomedial striatum activity during expression of goal-directed vs. habit-like cue-induced cocaine seeking

A preclinical model of cue exposure therapy, cue extinction, reduces cue-induced cocaine seeking that is goal-directed but not habit-like. Goal-directed and habitual behaviors differentially rely on the dorsomedial striatum (DMS) and dorsolateral striatum (DLS), but the effects of cue extinction on dorsal striatal responses to cue-induced drug seeking are unknown. We used fiber photometry in rats trained to self-administer cocaine paired with an audiovisual cue to examine how dorsal striatal intracellular calcium and extracellular dopamine activity differs between goal-directed and habit-like cue-induced cocaine seeking and how it is impacted by cue extinction. After minimal fixed-ratio training, rats showed enhanced DMS and DLS calcium responses to cue-reinforced compared to unreinforced lever presses. After rats were trained on goal-promoting fixed ratio schedules or habit-promoting second-order schedules of reinforcement, different patterns of dorsal striatal calcium and dopamine responses to cue-reinforced lever presses emerged. Rats trained on habit-promoting second-order schedules showed reduced DMS calcium responses and enhanced DLS dopamine responses to cue-reinforced lever presses. Cue extinction reduced calcium responses during subsequent drug seeking in the DMS, but not in the DLS. Therefore, cue extinction may reduce goal-directed behavior through its effects on the DMS, whereas habit-like behavior and the DLS are unaffected.

Computational models of intrinsic motivation for curiosity and creativity

We link Ivancovsky et al.'s novelty-seeking model (NSM) to computational models of intrinsically motivated behavior and learning. We argue that dissociating different forms of curiosity, creativity, and memory based on the involvement of distinct intrinsic motivations (e.g., surprise and novelty) is essential to empirically test the conceptual claims of the NSM.

Role of dopamine neurons in familiarity

Dopamine neurons signal the salience of environmental stimuli and influence learning, although it is less clear if these neurons also determine the salience of memories. Ventral tegmental area (VTA) dopamine neurons increase their firing in the presence of new objects and reduce it upon repeated, inconsequential exposures, marking the shift from novelty to familiarity. This study investigates how dopamine neuron activity during repeated familiar object exposure affects an animal's preference for new objects in a subsequent novel object recognition (NOR) test. We hypothesize that a single familiarization session will not sufficiently lower dopamine activity, such that the memory of a familiar object remains salient, leading to equal exploration of familiar and novel objects and weaker NOR discrimination. In contrast, multiple familiarization sessions likely suppress dopamine activity more effectively, reducing the salience of the familiar object and enhancing subsequent novelty discrimination. Our experiments in mice indicated that multiple familiarization sessions reduce VTA dopamine neuron activation, as measured by c-Fos expression, and enhance novelty discrimination compared with a single familiarization session. Dopamine neurons that show responsiveness to novelty were primarily located in the paranigral nucleus of the VTA and expressed vesicular glutamate transporter 2 transcripts, marking them as dopamine-glutamate neurons. Chemogenetic inhibition of dopamine neurons during a single session paralleled the effects of multiple sessions, improving NOR. These findings suggest that a critical role of dopamine neurons during the transition from novelty to familiarity is to modulate the salience of an object's memory.

Altered neuronal response to visual food stimuli in adolescents undergoing chronic sleep restriction

STUDY OBJECTIVES

Poor sleep in adolescents can increase the risk of obesity, possibly due to changes in dietary patterns. Prior neuroimaging evidence, mostly in adults, suggests that lacking sleep results in increased response to food cues in reward-processing brain regions. Needed is a clarification of the mechanisms by which food reward processing is altered by the kind of chronic sleep restriction (SR) typically experienced by adolescents. This study aimed to elucidate the impact of sleep duration on response to visual food stimuli in healthy adolescents using functional neuroimaging, hypothesizing increased reward processing response after SR compared to a well-rested condition.

METHODS

Thirty-nine healthy adolescents, 14-17 years old, completed a 3-week protocol: (1) sleep phase stabilization; (2) SR (~6.5 h nightly); and (3) healthy sleep (HS) duration (~9 h nightly). Participants underwent functional MRI while performing a visual food paradigm. Contrasts of food versus nonfood responses were compared within-subject between conditions of SR and HS.

RESULTS

Under SR, there was a greater response to food stimuli compared to HS in a voxel cluster including the left ventral tegmental area and substantia nigra. No change in food appeal rating due to the sleep manipulation was detected.

CONCLUSIONS

Outcomes of this study suggest that SR, as commonly experienced by healthy adolescents, results in the elevated dopaminergic drive of the reward network that may augment motivation to seek food in the context of individual food appeal and inhibitory profiles. Countermeasures that reduce food salience could include promoting consistent HS habits.

Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action-reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

A septal-ventral tegmental area circuit drives exploratory behavior

To survive, animals need to balance their exploratory drive with their need for safety. Subcortical circuits play an important role in initiating and modulating movement based on external demands and the internal state of the animal; however, how motivation and onset of locomotion are regulated remain largely unresolved. Here, we show that a glutamatergic pathway from the medial septum and diagonal band of Broca (MSDB) to the ventral tegmental area (VTA) controls exploratory locomotor behavior in mice. Using a self-supervised machine learning approach, we found an overrepresentation of exploratory actions, such as sniffing, whisking, and rearing, when this projection is optogenetically activated. Mechanistically, this role relies on glutamatergic MSDB projections that monosynaptically target a subset of both glutamatergic and dopaminergic VTA neurons. Taken together, we identified a glutamatergic basal forebrain to midbrain circuit that initiates locomotor activity and contributes to the expression of exploration-associated behavior.

Reduction in the activity of VTA/SNc dopaminergic neurons underlies aging-related decline in novelty seeking

Curiosity, or novelty seeking, is a fundamental mechanism motivating animals to explore and exploit environments to improve survival, and is also positively associated with cognitive, intrapersonal and interpersonal well-being in humans. However, curiosity declines as humans age, and the decline even positively predicts the extent of cognitive decline in Alzheimer's disease patients. Therefore, determining the underlying mechanism, which is currently unknown, is an urgent task for the present aging society that is growing at an unprecedented rate. This study finds that seeking behaviors for both social and inanimate novelties are compromised in aged mice, suggesting that the aging-related decline in curiosity and novelty-seeking is a biological process. This study further identifies an aging-related reduction in the activity (manifesting as a reduction in spontaneous firing) of dopaminergic neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc). Finally, this study establishes that this reduction in activity causally underlies the aging-related decline in novelty-seeking behaviors. This study potentially provides an interventional strategy for maintaining high curiosity in the aged population, i.e., compensating for the reduced activity of VTA/SNc dopaminergic neurons, enabling the aged population to cope more smoothly with the present growing aging society, physically, cognitively and socioeconomically.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

The influence of dopamine autoreceptors on temperament and addiction risk

As a major regulator of dopamine (DA), DA autoreceptors (DAARs) exert substantial influence over DA-mediated behaviors. This paper reviews the physiological and behavioral impact of DAARs. Individual differences in DAAR functioning influences temperamental traits such as novelty responsivity and impulsivity, both of which are associated with vulnerability to addictive behavior in animal models and a broad array of externalizing behaviors in humans. DAARs additionally impact the response to psychostimulants and other drugs of abuse. Human PET studies of D2-like receptors in the midbrain provide evidence for parallels to the animal literature. These data lead to the proposal that weak DAAR regulation is a risk factor for addiction and externalizing problems. The review highlights the potential to build translational models of the functional role of DAARs in behavior. It also draws attention to key limitations in the current literature that would need to be addressed to further advance a weak DAAR regulation model of addiction and externalizing risk.

[Effect of Sleep Deprivation on the Metabolism of Hippocampal Amino Acids and Monoamine Neurotransmitters in Mice and Their Behaviors]

Objective

To investigate the effect of sleep deprivation on the metabolism of the hippocampal region in mice.

Methods

The mice were randomly assigned to three groups, a control group, a 24-h sleep deprivation (SD) group, and a 48-h SD group. Each group had 10 mice. The sleep deprivation model was induced by the modified multiple platform method. The mice's anxiety-like behaviors were assessed with the open field test (OFT) and their depression-like behaviors were assessed with the sucrose preference test (SPT), the forced swimming test (FST), and tail suspension test (TST). High performance liquid chromatography (HPLC) was performed to determine the levels of 6 monoamine neurotransmitters, including 5-hydroxytryptamine (5-HT), norepinephrine (NE), dopamine (DA), gamma-aminobutyric acid (GABA), 5-dihydroxyphenylacetic acid (5-DOPAC), and homovanillic acid (HVA), and 4 amino acids, including glutamic acid (Glu), aspartic acid (Asp), serine (Ser), and taurine (Tau), in the hippocampal region. Immunofluorescence staining was performed to examine the expression of glial cells in the hippocampal region of the mice. The main indicators measured were the levels of monoamine neurotransmitters and amino acids.

Results

According to the results of the behavioral analysis, in comparison with the findings for the control group, the 24-h SD mice exhibited increased consumption of sucrose in SFT, significantly decreased total immobility time in FST and TST, and increased total distance covered in OFT, while the 48-h SD mice showed decreased consumption of sucrose in SFT, prolonged total immobility time in FST and TST, and decreased total distance covered in OFT. The results of the HPLC analysis of the monoamine neurotransmitter showed that 24-h SD mice had in their hippocampal region increased levels of DA (P<0.001) and NE (P<0.01) and decreased levels of GABA (P<0.05) in comparison with those of the control mice, while their 5-HT, 5-DOPAC, and HVA levels were not significantly different from those of the control mice. In comparison with those of the control mice, the 48-h SD mice had, in their hippocampal region, decreased levels of 5-HT and NE (all P<0.05), decreased DA (P<0.01), and increased level of GABA (P<0.01), while the levels of 5-DOPAC and HAV were not significantly different. The 48-h SD group showed a significant decrease in the levels of Tau and Glu in comparison with those of the 24-h SD group (all P<0.05). According to the results of immunofluorescence assay, there was no significant difference between the control group and the 24-h SD group in the cell count of glial fibrillary acidic protein (GFAP)-positive cells, while a decline in GFAP-positive cells in comparison with that of the control group was observed in the 48-h SD group.

Conclusion

SD of 24 hours may induce anxiety-like behavioral changes in mice by activating their hippocampal glial cells, upregulating the levels of 5-HT, DA, and NE, and increasing the levels of Glu and Tau in the hippocampal region. SD of 48 hours may induce depression-like behavioral changes in mice by inhibiting the activation of glial cells in the hippocampal region and regulating in the opposite direction the levels of the above-mentioned monoamine neurotransmitters and amino acids in the hippocampal region.

Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping

The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.

Acute inhibition of hunger-sensing AgRP neurons promotes context-specific learning in mice

OBJECTIVE

An environmental context, which reliably predicts food availability, can increase the appetitive food drive within the same environment context. However, hunger is required for the development of such a context-induced feeding (CIF) response, suggesting the neural circuits sensitive to hunger link an internal energy state with a particular environment context. Since Agouti related peptide (AgRP) neurons are activated by energy deficit, we hypothesised that AgRP neurons are both necessary and sufficient to drive CIF.

METHODS

To examine the role of AgRP neurons in the CIF process, we used fibre photometry with GCaMP7f, chemogenetic activation of AgRP neurons, as well as optogenetic control of AgRP neurons to facilitate acute temporal control not permitted with chemogenetics.

RESULTS

A CIF response at test was only observed when mice were fasted during context training and AgRP population activity at test showed an attenuated inhibitory response to food, suggesting increased food-seeking and/or decreased satiety signalling drives the increased feeding response at test. Intriguingly, chemogenetic activation of AgRP neurons during context training did not increase CIF, suggesting precise temporal firing properties may be required. Indeed, termination of AgRP neuronal photostimulation during context training (ON-OFF in context), in the presence or absence of food, increased CIF. Moreover, photoinhibition of AgRP neurons during context training in fasted mice was sufficient to drive a subsequent CIF in the absence of food.

CONCLUSIONS

Our results suggest that AgRP neurons regulate the acquisition of CIF when the acute inhibition of AgRP activity is temporally matched to context exposure. These results establish acute AgRP inhibition as a salient neural event underscoring the effect of hunger on associative learning.

Dopaminergic neurons dynamically update sensory values during olfactory maneuver

Dopaminergic neurons (DANs) drive associative learning to update the value of sensory cues, but their contribution to the assessment of sensory values outside the context of association remains largely unexplored. Here, we show in Drosophila that DANs in the mushroom body encode the innate value of odors and constantly update the current value by inducing plasticity during olfactory maneuver. Our connectome-based network model linking all the way from the olfactory neurons to DANs reproduces the characteristics of DAN responses, proposing a concrete circuit mechanism for computation. Downstream of DANs, odors alone induce value- and dopamine-dependent changes in the activity of mushroom body output neurons, which store the current value of odors. Consistent with this neural plasticity, specific sets of DANs bidirectionally modulate flies' steering in a virtual olfactory environment. Thus, the DAN circuit known for discrete, associative learning also continuously updates odor values in a nonassociative manner.

Role of Dopamine Neurons in Familiarity

Dopamine neurons signal the salience of environmental stimuli, influencing learning and motivation. However, research has not yet identified whether dopamine neurons also modulate the salience of memory content. Dopamine neuron activity in the ventral tegmental area (VTA) increases in response to novel objects and diminishes as objects become familiar through repeated presentations. We proposed that the declined rate of dopamine neuron activity during familiarization affects the salience of a familiar object's memory. This, in turn, influences the degree to which an animal distinguishes between familiar and novel objects in a subsequent novel object recognition (NOR) test. As such, a single familiarization session may not sufficiently reduce dopamine activity, allowing the memory of a familiar object to maintain its salience and potentially attenuating NOR. In contrast, multiple familiarization sessions could lead to more pronounced dopamine activity suppression, strengthening NOR. Our data in mice reveals that, compared to a single session, multiple sessions result in decreased VTA dopamine neuron activation, as indicated by c-Fos measurements, and enhanced novelty discrimination. Critically, when VTA dopamine neurons are chemogenetically inhibited during a single familiarization session, NOR improves, mirroring the effects of multiple familiarization sessions. In summary, our findings highlight the pivotal function of dopamine neurons in familiarity and suggest a role in modulating the salience of memory content.

Striatal Dopamine Signals and Reward Learning

We are constantly bombarded by sensory information and constantly making decisions on how to act. In order to optimally adapt behavior, we must judge which sequences of sensory inputs and actions lead to successful outcomes in specific circumstances. Neuronal circuits of the basal ganglia have been strongly implicated in action selection, as well as the learning and execution of goal-directed behaviors, with accumulating evidence supporting the hypothesis that midbrain dopamine neurons might encode a reward signal useful for learning. Here, we review evidence suggesting that midbrain dopaminergic neurons signal reward prediction error, driving synaptic plasticity in the striatum underlying learning. We focus on phasic increases in action potential firing of midbrain dopamine neurons in response to unexpected rewards. These dopamine neurons prominently innervate the dorsal and ventral striatum. In the striatum, the released dopamine binds to dopamine receptors, where it regulates the plasticity of glutamatergic synapses. The increase of striatal dopamine accompanying an unexpected reward activates dopamine type 1 receptors (D1Rs) initiating a signaling cascade that promotes long-term potentiation of recently active glutamatergic input onto striatonigral neurons. Sensorimotor-evoked glutamatergic input, which is active immediately before reward delivery will thus be strengthened onto neurons in the striatum expressing D1Rs. In turn, these neurons cause disinhibition of brainstem motor centers and disinhibition of the motor thalamus, thus promoting motor output to reinforce rewarded stimulus-action outcomes. Although many details of the hypothesis need further investigation, altogether, it seems likely that dopamine signals in the striatum might underlie important aspects of goal-directed reward-based learning.

Surprise and novelty in the brain

Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.

Excitatory and Inhibitory Signaling in the Nucleus Accumbens Encode Different Aspects of a Pavlovian Cue in Sign Tracking and Goal Tracking Rats

When a Pavlovian cue is presented separately from its associated reward, some animals will acquire a sign tracking (ST) response - approach and/or interaction with the cue - while others will acquire a goal tracking response - approach to the site of reward. We have previously shown that cue-evoked excitations in the nucleus accumbens (NAc) encode the vigor of both behaviors; in contrast, reward-related responses diverge over the course of training, possibly reflecting neurochemical differences between sign tracker and goal tracker individuals. However, a substantial subset of neurons in the NAc exhibit inhibitory, rather than excitatory, cue-evoked responses, and the evolution of their signaling during Pavlovian conditioning remains unknown. Using single-neuron recordings in behaving rats, we show that NAc neurons with cue-evoked inhibitions have distinct coding properties from neurons with cue-evoked excitations. Cue-evoked inhibitions become more numerous over the course of training and, like excitations, may encode the vigor of sign tracking and goal tracking behavior. However, the responses of cue-inhibited neurons do not evolve differently between sign tracker and goal tracker individuals. Moreover, cue-evoked inhibitions, unlike excitations, are insensitive to extinction of the cue-reward relationship. Finally, we show that cue-evoked excitations are greatly diminished by reward devaluation, while inhibitory cue responses are virtually unaffected. Overall, these findings converge with existing evidence that cue-excited neurons in NAc, but not cue-inhibited neurons, are profoundly sensitive to the same behavior variations that are often associated with changes in dopamine release.

The respective activation and silencing of striatal direct and indirect pathway neurons support behavior encoding

The basal ganglia are known to control actions and modulate movements. Neuronal activity in the two efferent pathways of the dorsal striatum is critical for appropriate behavioral control. Previous evidence has led to divergent conclusions on the respective engagement of both pathways during actions. Using calcium imaging to evaluate how neurons in the direct and indirect pathways encode behaviors during self-paced spontaneous explorations in an open field, we observed that the two striatal pathways exhibit distinct tuning properties. Supervised learning algorithms revealed that direct pathway neurons encode behaviors through their activation, whereas indirect pathway neurons exhibit behavior-specific silencing. These properties remain stable for weeks. Our findings highlight a complementary encoding of behaviors with congruent activations in the direct pathway encoding multiple accessible behaviors in a given context, and in the indirect pathway encoding the suppression of competing behaviors. This model reconciles previous conflicting conclusions on motor encoding in the striatum.

Changes in dorsomedial striatum activity mediate expression of goal-directed vs. habit-like cue-induced cocaine seeking

A preclinical model of cue exposure therapy, cue extinction, reduces cue-induced cocaine seeking when drug seeking is goal-directed but not habitual. Goal-directed and habitual behaviors differentially rely on the dorsomedial striatum (DMS) and dorsolateral striatum (DLS), but the effects of cue extinction on dorsal striatal responses to cue-induced drug seeking are unknown. We used fiber photometry to examine how dorsal striatal intracellular calcium and extracellular dopamine activity differs between goal-directed and habitual cue-induced cocaine seeking and how it is impacted by cue extinction. Rats trained to self-administer cocaine paired with an audiovisual cue on schedules of reinforcement that promote goal-directed or habitual cocaine seeking had different patterns of dorsal striatal calcium and dopamine responses to cue-reinforced lever presses. Cue extinction reduced calcium and dopamine responses during subsequent drug seeking in the DMS, but not in the DLS. Therefore, cue extinction may reduce goal-directed behavior through its effects on the DMS, whereas habitual behavior and the DLS are unaffected.

Inhibition of dopamine neurons prevents incentive value encoding of a reward cue: With revelations from deep phenotyping

The survival of an organism is dependent on their ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the nucleus accumbens is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCut revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient towards and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.

Significance Statement

Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCut revealed that cue-directed behaviors do not emerge without VTA dopamine. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of VTA dopamine during cue presentation to encode the incentive value of reward cues.

Novelty selectively permits learning-associated plasticity in ventral tegmental-hippocampal-prefrontal circuitry

Modifying established behavior in novel situations is essential, and patients with neuropsychiatric disorders often lack this flexibility. Understanding how novelty affects behavioral flexibility therefore has therapeutic potential. Here, novelty differentially impacts connectivity within the ventral tegmental-hippocampal-medial prefrontal (VTA-HPC-mPFC) circuit, thereby enhancing the ability of mice to overcome established behavioral bias and adapt to new rules. Circuit connectivity was measured by local field potential (LFP) coherence. As mice exposed to novelty learned to overcome previously established spatial bias, the ventral HPC (vHPC) strengthens its coherence with the VTA and mPFC in theta frequency (4-8 Hz). Novelty or learning did not affect circuits involving the dorsal HPC (dHPC). Without novelty, however, mice continued following established spatial bias and connectivity strength remained stable in the VTA-HPC-mPFC circuit. Pharmacologically blocking dopamine D1-receptors (D1Rs) in the vHPC abolished the behavioral and physiological impacts of novelty. Thus, novelty promotes behavioral adaptation by permitting learning-associated plasticity in the vHPC-mPFC and VTA-vHPC circuit, a process mediated by D1Rs in the vHPC.

Locomotor and olfactory responses in dopamine neurons of the Drosophila superior-lateral brain

The Drosophila brain contains about 50 distinct morphological types of dopamine neurons.^1,2,3,4 Physiological studies of Drosophila dopamine neurons have been largely limited to one brain region, the mushroom body,^{5,6,7,8,9,10,11,12,13} where they are implicated in learning.^{14,15,16,17,18} By comparison, we know little about the physiology of other Drosophila dopamine neurons. Interestingly, a recent whole-brain imaging study found that dopamine neuron activity in several fly brain regions is correlated with locomotion.¹⁹ This is notable because many dopamine neurons in the rodent brain are also correlated with locomotion or other movements^{20,21,22,23,24,25,26,27,28,29,30}; however, most rodent studies have focused on learned and rewarded behaviors, and few have investigated dopamine neuron activity during spontaneous (self-timed) movements. In this study, we monitored dopamine neurons in the Drosophila brain during self-timed locomotor movements, focusing on several previously uncharacterized cell types that arborize in the superior-lateral brain, specifically the lateral horn and superior-lateral protocerebrum. We found that activity of all of these dopamine neurons correlated with spontaneous fluctuations in walking speed, with different cell types showing different speed correlations. Some dopamine neurons also responded to odors, but these responses were suppressed by repeated odor encounters. Finally, we found that the same identifiable dopamine neuron can encode different combinations of locomotion and odor in different individuals. If these dopamine neurons promote synaptic plasticity-like the dopamine neurons of the mushroom body-then, their tuning profiles would imply that plasticity depends on a flexible integration of sensory signals, motor signals, and recent experience.

Integrated world modeling theory expanded: Implications for the future of consciousness

Integrated world modeling theory (IWMT) is a synthetic theory of consciousness that uses the free energy principle and active inference (FEP-AI) framework to combine insights from integrated information theory (IIT) and global neuronal workspace theory (GNWT). Here, I first review philosophical principles and neural systems contributing to IWMT's integrative perspective. I then go on to describe predictive processing models of brains and their connections to machine learning architectures, with particular emphasis on autoencoders (perceptual and active inference), turbo-codes (establishment of shared latent spaces for multi-modal integration and inferential synergy), and graph neural networks (spatial and somatic modeling and control). Future directions for IIT and GNWT are considered by exploring ways in which modules and workspaces may be evaluated as both complexes of integrated information and arenas for iterated Bayesian model selection. Based on these considerations, I suggest novel ways in which integrated information might be estimated using concepts from probabilistic graphical models, flow networks, and game theory. Mechanistic and computational principles are also considered with respect to the ongoing debate between IIT and GNWT regarding the physical substrates of different kinds of conscious and unconscious phenomena. I further explore how these ideas might relate to the "Bayesian blur problem," or how it is that a seemingly discrete experience can be generated from probabilistic modeling, with some consideration of analogies from quantum mechanics as potentially revealing different varieties of inferential dynamics. I go on to describe potential means of addressing critiques of causal structure theories based on network unfolding, and the seeming absurdity of conscious expander graphs (without cybernetic symbol grounding). Finally, I discuss future directions for work centered on attentional selection and the evolutionary origins of consciousness as facilitated "unlimited associative learning." While not quite solving the Hard problem, this article expands on IWMT as a unifying model of consciousness and the potential future evolution of minds.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Salience memories formed by value, novelty and aversiveness jointly shape object responses in the prefrontal cortex and basal ganglia

Ecological fitness depends on maintaining object histories to guide future interactions. Recent evidence shows that value memory changes passive visual responses to objects in ventrolateral prefrontal cortex (vlPFC) and substantia nigra reticulata (SNr). However, it is not known whether this effect is limited to reward history and if not how cross-domain representations are organized within the same or different neural populations in this corticobasal circuitry. To address this issue, visual responses of the same neurons across appetitive, aversive and novelty domains were recorded in vlPFC and SNr. Results showed that changes in visual responses across domains happened in the same rather than separate populations and were related to salience rather than valence of objects. Furthermore, while SNr preferentially encoded outcome related salience memory, vlPFC encoded salience memory across all domains in a correlated fashion, consistent with its role as an information hub to guide behavior.

Phosphorylation Signals Downstream of Dopamine Receptors in Emotional Behaviors: Association with Preference and Avoidance

Dopamine regulates emotional behaviors, including rewarding and aversive behaviors, through the mesolimbic dopaminergic pathway, which projects dopamine neurons from the ventral tegmental area to the nucleus accumbens (NAc). Protein phosphorylation is critical for intracellular signaling pathways and physiological functions, which are regulated by neurotransmitters in the brain. Previous studies have demonstrated that dopamine stimulated the phosphorylation of intracellular substrates, such as receptors, ion channels, and transcription factors, to regulate neuronal excitability and synaptic plasticity through dopamine receptors. We also established a novel database called KANPHOS that provides information on phosphorylation signals downstream of monoamines identified by our kinase substrate screening methods, including dopamine, in addition to those reported in the literature. Recent advances in proteomics techniques have enabled us to clarify the mechanisms through which dopamine controls rewarding and aversive behaviors through signal pathways in the NAc. In this review, we discuss the intracellular phosphorylation signals regulated by dopamine in these two emotional behaviors.

Dopamine signaling in the nucleus accumbens core mediates latent inhibition

Studies investigating the neural mechanisms by which associations between cues and predicted outcomes control behavior often use associative learning frameworks to understand the neural control of behavior. These frameworks do not always account for the full range of effects that novelty can have on behavior and future associative learning. Here, in mice, we show that dopamine in the nucleus accumbens core is evoked by novel, neutral stimuli, and the trajectory of this response over time tracked habituation to these stimuli. Habituation to novel cues before associative learning reduced future associative learning, a phenomenon known as latent inhibition. Crucially, trial-by-trial dopamine response patterns tracked this phenomenon. Optogenetic manipulation of dopamine responses to the cue during the habituation period bidirectionally influenced future associative learning. Thus, dopamine signaling in the nucleus accumbens core has a causal role in novelty-based learning in a way that cannot be predicted based on purely associative factors.

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning

A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multiple trials. Here we demonstrate that such a gradual shift occurs both at the level of dopaminergic cellular activity and dopamine release in the ventral striatum in mice. Our results establish a long-sought link between dopaminergic activity and the TD learning algorithm, providing fundamental insights into how the brain associates cues and rewards that are separated in time.

Electrophysiological Evidence of Enhanced Processing of Novel Pornographic Images in Individuals With Tendencies Toward Problematic Internet Pornography Use

Novelty seeking is regarded as a core feature in substance use disorders. However, few studies thus far have investigated this feature in problematic Internet pornography use (PIPU). The main aim of the present study was to examine group differences in electrophysiological activity associated with novelty processing in participants with high tendencies toward PIPU vs. low tendencies using event-related potentials (ERPs). Twenty-seven participants with high tendencies toward PIPU and 25 with low tendencies toward PIPU completed a modified three-stimulus oddball task while electroencephalogram (EEG) was recorded. Participants were instructed to detect neutral target stimuli from distracting stimuli. The distracting stimuli contained a familiar sexual stimulus and a set of novel sexual stimuli. The novel-familiar difference waves were calculated to identify specific group difference in novelty effect. While both groups demonstrated a sustained novelty effect in the late positive potential (LPP) within the 500–800 ms time windows, the novelty effect was greater in the high tendencies toward PIPU group than in the low tendencies toward PIPU group. This result suggests that individuals with high tendencies toward PIPU allocate more attentional resources for novelty processing. Enhanced brain responding to novel sexual stimuli may facilitate pornographic consumption and play an essential role in the development and maintenance of PIPU.

Dopamine errors drive excitatory and inhibitory components of backward conditioning in an outcome-specific manner

For over two decades, phasic activity in midbrain dopamine neurons was considered synonymous with the prediction error in temporal-difference reinforcement learning.^1-4 Central to this proposal is the notion that reward-predictive stimuli become endowed with the scalar value of predicted rewards. When these cues are subsequently encountered, their predictive value is compared to the value of the actual reward received, allowing for the calculation of prediction errors.^5,6 Phasic firing of dopamine neurons was proposed to reflect this computation,^1,2 facilitating the backpropagation of value from the predicted reward to the reward-predictive stimulus, thus reducing future prediction errors. There are two critical assumptions of this proposal: (1) that dopamine errors can only facilitate learning about scalar value and not more complex features of predicted rewards, and (2) that the dopamine signal can only be involved in anticipatory cue-reward learning in which cues or actions precede rewards. Recent work^7-15 has challenged the first assumption, demonstrating that phasic dopamine signals across species are involved in learning about more complex features of the predicted outcomes, in a manner that transcends this value computation. Here, we tested the validity of the second assumption. Specifically, we examined whether phasic midbrain dopamine activity would be necessary for backward conditioning-when a neutral cue reliably follows a rewarding outcome.^16-20 Using a specific Pavlovian-to-instrumental transfer (PIT) procedure,^21-23 we show rats learn both excitatory and inhibitory components of a backward association, and that this association entails knowledge of the specific identity of the reward and cue. We demonstrate that brief optogenetic inhibition of VTA_DA neurons timed to the transition between the reward and cue reduces both of these components of backward conditioning. These findings suggest VTA_DA neurons are capable of facilitating associations between contiguously occurring events, regardless of the content of those events. We conclude that these data may be in line with suggestions that the VTA_DA error acts as a universal teaching signal. This may provide insight into why dopamine function has been implicated in myriad psychological disorders that are characterized by very distinct reinforcement-learning deficits.

Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning

Learning associations between cues and rewards require the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well trained animals. However, these value-related dopamine responses are absent during early training sessions when cues signal differences in the reward rate. These findings suggest cue-evoked dopamine release conveys differences between outcomes only after extensive training, though it is unclear whether this is unique to when cues signal differences in reward rate, or whether this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we used a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout learning. While female rats exhibited higher levels of conditioned responding, a faster latency to respond, and elevated post-reward head entries relative to male rats, there were no sex differences in the dopamine response to cues. Multiple training sessions were required before cue-evoked dopamine release signaled differences in reward size. Reward-evoked dopamine release scaled with reward size, though females displayed lower reward-evoked dopamine responses relative to males. Conditioned responding related to the decrease in the peak reward-evoked dopamine response and not to cue-evoked dopamine release. Collectively, these data illustrate sex differences in behavioral responding as well as in reward-evoked dopamine release during Pavlovian learning.

Hippocampal Disinhibition Reduces Contextual and Elemental Fear Conditioning While Sparing the Acquisition of Latent Inhibition

Hippocampal neural disinhibition, i.e., reduced GABAergic inhibition, is a key feature of schizophrenia pathophysiology. The hippocampus is an important part of the neural circuitry that controls fear conditioning and can also modulate prefrontal and striatal mechanisms, including dopamine signaling, which play a role in salience modulation. Consequently, hippocampal neural disinhibition may contribute to impairments in fear conditioning and salience modulation reported in schizophrenia. Therefore, we examined the effect of ventral hippocampus (VH) disinhibition in male rats on fear conditioning and salience modulation, as reflected by latent inhibition (LI), in a conditioned emotional response (CER) procedure. A flashing light was used as the conditioned stimulus (CS), and conditioned suppression was used to index conditioned fear. In experiment 1, VH disinhibition via infusion of the GABA-A receptor antagonist picrotoxin before CS pre-exposure and conditioning markedly reduced fear conditioning to both the CS and context; LI was evident in saline-infused controls but could not be detected in picrotoxin-infused rats because of the low level of fear conditioning to the CS. In experiment 2, VH picrotoxin infusions only before CS pre-exposure did not affect the acquisition of fear conditioning or LI. Together, these findings indicate that VH neural disinhibition disrupts contextual and elemental fear conditioning, without affecting the acquisition of LI. The disruption of fear conditioning resembles aversive conditioning deficits reported in schizophrenia and may reflect a disruption of neural processing both within the hippocampus and in projection sites of the hippocampus.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

Slowly evolving dopaminergic activity modulates the moment-to-moment probability of reward-related self-timed movements

Clues from human movement disorders have long suggested that the neurotransmitter dopamine plays a role in motor control, but how the endogenous dopaminergic system influences movement is unknown. Here we examined the relationship between dopaminergic signaling and the timing of reward-related movements in mice. Animals were trained to initiate licking after a self-timed interval following a start-timing cue; reward was delivered in response to movements initiated after a criterion time. The movement time was variable from trial-to-trial, as expected from previous studies. Surprisingly, dopaminergic signals ramped-up over seconds between the start-timing cue and the self-timed movement, with variable dynamics that predicted the movement/reward time on single trials. Steeply rising signals preceded early lick-initiation, whereas slowly rising signals preceded later initiation. Higher baseline signals also predicted earlier self-timed movements. Optogenetic activation of dopamine neurons during self-timing did not trigger immediate movements, but rather caused systematic early-shifting of movement initiation, whereas inhibition caused late-shifting, as if modulating the probability of movement. Consistent with this view, the dynamics of the endogenous dopaminergic signals quantitatively predicted the moment-by-moment probability of movement initiation on single trials. We propose that ramping dopaminergic signals, likely encoding dynamic reward expectation, can modulate the decision of when to move.

Dissociable contributions of phasic dopamine activity to reward and prediction

Sensory cues that precede reward acquire predictive (expected value) and incentive (drive reward-seeking action) properties. Mesolimbic dopamine neurons' responses to sensory cues correlate with both expected value and reward-seeking action. This has led to the proposal that phasic dopamine responses may be sufficient to inform value-based decisions, elicit actions, and/or induce motivational states; however, causal tests are incomplete. Here, we show that direct dopamine neuron stimulation, both calibrated to physiological and greater intensities, at the time of reward can be sufficient to induce and maintain reward seeking (reinforcing) although replacement of a cue with stimulation is insufficient to induce reward seeking or act as an informative cue. Stimulation of descending cortical inputs, one synapse upstream, are sufficient for reinforcement and cues to future reward. Thus, physiological activation of mesolimbic dopamine neurons can be sufficient for reinforcing properties of reward without being sufficient for the predictive and incentive properties of cues.

Dopamine release and its control over early Pavlovian learning differs between the NAc core and medial NAc shell

Dopamine neurons respond to cues to reflect the value of associated outcomes. These cue-evoked dopamine responses can encode the relative rate of reward in rats with extensive Pavlovian training. Specifically, a cue that always follows the previous reward by a short delay (high reward rate) evokes a larger dopamine response in the nucleus accumbens (NAc) core relative to a distinct cue that always follows the prior reward by a long delay (low reward rate). However, it was unclear if these reward rate dopamine signals are evident during early Pavlovian training sessions and across NAc subregions. To address this, we performed fast-scan cyclic voltammetry recordings of dopamine levels to track the pattern of cue- and reward-evoked dopamine signals in the NAc core and medial NAc shell. We identified regional differences in the progression of cue-evoked dopamine signals across training. However, the dopamine response to cues did not reflect the reward rate in either the NAc core or the medial NAc shell during early training sessions. Pharmacological experiments found that dopamine-sensitive conditioned responding emerged in the NAc core before the medial NAc shell. Together, these findings illustrate regional differences in NAc dopamine release and its control over behavior during early Pavlovian learning.

Neural Representation of Costs and Rewards in Decision Making

Decision making is crucial for animal survival because the choices they make based on their current situation could influence their future rewards and could have potential costs. This review summarises recent developments in decision making, discusses how rewards and costs could be encoded in the brain, and how different options are compared such that the most optimal one is chosen. The reward and cost are mainly encoded by the forebrain structures (e.g., anterior cingulate cortex, orbitofrontal cortex), and their value is updated through learning. The recent development on dopamine and the lateral habenula's role in reporting prediction errors and instructing learning will be emphasised. The importance of dopamine in powering the choice and accounting for the internal state will also be discussed. While the orbitofrontal cortex is the place where the state values are stored, the anterior cingulate cortex is more important when the environment is volatile. All of these structures compare different attributes of the task simultaneously, and the local competition of different neuronal networks allows for the selection of the most appropriate one. Therefore, the total value of the task is not encoded as a scalar quantity in the brain but, instead, as an emergent phenomenon, arising from the computation at different brain regions.

Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making

Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.