Multiple Dopamine Systems: Weal and Woe of Dopamine

Ventral pallidum GABA and glutamate neurons drive approach and avoidance through distinct modulation of VTA cell types

The ventral pallidum (VP) contains GABA and glutamate neurons projecting to ventral tegmental area (VTA) whose stimulation drives approach and avoidance, respectively. Yet little is known about the mechanisms by which VP cell types shape VTA activity and drive behavior. Here, we found that both VP GABA and glutamate neurons were activated during approach to reward or by delivery of an aversive stimulus. Stimulation of VP GABA neurons inhibited VTA GABA, but activated dopamine and glutamate neurons. Remarkably, stimulation-evoked activation was behavior-contingent such that VTA recruitment was inhibited when evoked by the subject's own action. Conversely, VP glutamate neurons activated VTA GABA, as well as dopamine and glutamate neurons, despite driving aversion. However, VP glutamate neurons evoked dopamine in aversion-associated ventromedial nucleus accumbens (NAc), but reduced dopamine release in reward-associated dorsomedial NAc. These findings show how heterogeneous VP projections to VTA can be engaged to shape approach and avoidance behaviors.

Sensory encoding and memory in the mushroom body: signals, noise, and variability

To survive in changing environments, animals need to learn to associate specific sensory stimuli with positive or negative valence. How do they form stimulus-specific memories to distinguish between positively/negatively associated stimuli and other irrelevant stimuli? Solving this task is one of the functions of the mushroom body, the associative memory center in insect brains. Here we summarize recent work on sensory encoding and memory in the Drosophila mushroom body, highlighting general principles such as pattern separation, sparse coding, noise and variability, coincidence detection, and spatially localized neuromodulation, and placing the mushroom body in comparative perspective with mammalian memory systems.

Evolution of neural circuitry and cognition

Neural circuits govern the interface between the external environment, internal cues and outwardly directed behaviours. To process multiple environmental stimuli and integrate these with internal state requires considerable neural computation. Expansion in neural network size, most readily represented by whole brain size, has historically been linked to behavioural complexity, or the predominance of cognitive behaviours. Yet, it is largely unclear which aspects of circuit variation impact variation in performance. A key question in the field of evolutionary neurobiology is therefore how neural circuits evolve to allow improved behavioural performance or innovation. We discuss this question by first exploring how volumetric changes in brain areas reflect actual neural circuit change. We explore three major axes of neural circuit evolution-replication, restructuring and reconditioning of cells and circuits-and discuss how these could relate to broader phenotypes and behavioural variation. This discussion touches on the relevant uses and limitations of volumetrics, while advocating a more circuit-based view of cognition. We then use this framework to showcase an example from the insect brain, the multi-sensory integration and internal processing that is shared between the mushroom bodies and central complex. We end by identifying future trends in this research area, which promise to advance the field of evolutionary neurobiology.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

Songbird mesostriatal dopamine pathways are spatially segregated before the onset of vocal learning

Diverse dopamine (DA) pathways send distinct reinforcement signals to different striatal regions. In adult songbirds, a DA pathway from the ventral tegmental area (VTA) to Area X, the striatal nucleus of the song system, carries singing-related performance error signals important for learning. Meanwhile, a parallel DA pathway to a medial striatal area (MST) arises from a distinct group of neighboring DA neurons that lack connectivity to song circuits and do not encode song error. To test if the structural and functional segregation of these two pathways depends on singing experience, we carried out anatomical studies early in development before the onset of song learning. We find that distinct VTA neurons project to either Area X or MST in juvenile birds before the onset of substantial vocal practice. Quantitative comparisons of early juveniles (30-35 days post hatch), late juveniles (60-65 dph), and adult (>90 dph) brains revealed an outsized expansion of Area X-projecting neurons relative to MST-projecting neurons in VTA over development. These results show that a mesostriatal DA system dedicated to social communication can exist and be spatially segregated before the onset of vocal practice and associated sensorimotor experience.

Multi-timescale reinforcement learning in the brain

To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents2-6 and at characterizing the firing of dopamine neurons in the midbrain7-9. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, controlled by the discount factor. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, a mechanistic basis for the empirical observation that humans and animals use non-exponential discounts in many situations10-14, and open new avenues for the design of more efficient reinforcement learning algorithms.

Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system

The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.

Ventral pallidum GABA and glutamate neurons drive approach and avoidance through distinct modulation of VTA cell types

The ventral pallidum (VP) contains GABA and glutamate (Glut) neurons projecting to ventral tegmental area (VTA) whose stimulation drives approach and avoidance, respectively. Yet little is known about the cell-type-specific mechanisms by which VP projections to VTA drive behavior. Here, we found that both VP GABA and Glut neurons were activated during approach to reward or delivery of an aversive stimulus. Stimulation of VP GABA neurons inhibited VTA GABA, but activated dopamine (DA) and glutamate neurons. Remarkably, this cell-type-specific recruitment was behavior-contingent such that VTA recruitment was inhibited when evoked by the subject's own action. Conversely, VP Glut neurons activated VTA GABA, as well as DA and Glut neurons, despite driving aversion. However, VP Glut neurons evoked DA in reward-associated ventromedial nucleus accumbens (NAc), but reduced DA in aversion-associated dorsomedial NAc. These findings show how heterogeneous VP cell types can engage VTA cell types to shape approach and avoidance behaviors.

Striosomes and Matrisomes: Scaffolds for Dynamic Coupling of Volition and Action

Striosomes form neurochemically specialized compartments of the striatum embedded in a large matrix made up of modules called matrisomes. Striosome-matrix architecture is multiplexed with the canonical direct-indirect organization of the striatum. Striosomal functions remain to be fully clarified, but key information is emerging. First, striosomes powerfully innervate nigral dopamine-containing neurons and can completely shut down their activity, with a following rebound excitation. Second, striosomes receive limbic and cognition-related corticostriatal afferents and are dynamically modulated in relation to value-based actions. Third, striosomes are spatially interspersed among matrisomes and interneurons and are influenced by local and global neuromodulatory and oscillatory activities. Fourth, striosomes tune engagement and the motivation to perform reinforcement learning, to manifest stereotypical behaviors, and to navigate valence conflicts and valence discriminations. We suggest that, at an algorithmic level, striosomes could serve as distributed scaffolds to provide formats of the striatal computations generated through development and refined through learning. We propose that striosomes affect subjective states. By transforming corticothalamic and other inputs to the functional formats of the striatum, they could implement state transitions in nigro-striato-nigral circuits to affect bodily and cognitive actions according to internal motives whose functions are compromised in neuropsychiatric conditions.

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.

Reappraising the role of dopamine in adolescent risk-taking behavior

Adolescence is characterized by increased risk-taking, which is often ascribed to developmental changes in dopaminergic signaling. Popular models propose that these behaviors are caused by dopamine-induced hypersensitivity to rewards, which overrides adolescents' immature self-control mechanisms. However, these models are often based on oversimplified notions about the workings and functions of dopamine. Here we discuss the relationship between changes in the dopaminergic system and adolescent behavior in light of current theories/models about the functions of dopamine. We show that dopamine is linked to learning, adaptive decision-making under uncertainty, and increased motivation to work for rewards. Thus, changes in the dopaminergic system contribute to the maturation of cognitive control through various mechanisms, contrary to the false dichotomy between reward processing and self-control. Finally, we note that dopamine interacts with a number of other neuromodulator systems, which also change during adolescence, but that have been largely ignored in the field of adolescent development. A full understanding of adolescent behavior will require these neurochemicals and their interactions with dopamine to be taken into account.

Hierarchical architecture of dopaminergic circuits enables second-order conditioning in Drosophila

Dopaminergic neurons with distinct projection patterns and physiological properties compose memory subsystems in a brain. However, it is poorly understood whether or how they interact during complex learning. Here, we identify a feedforward circuit formed between dopamine subsystems and show that it is essential for second-order conditioning, an ethologically important form of higher-order associative learning. The Drosophila mushroom body comprises a series of dopaminergic compartments, each of which exhibits distinct memory dynamics. We find that a slow and stable memory compartment can serve as an effective 'teacher' by instructing other faster and transient memory compartments via a single key interneuron, which we identify by connectome analysis and neurotransmitter prediction. This excitatory interneuron acquires enhanced response to reward-predicting odor after first-order conditioning and, upon activation, evokes dopamine release in the 'student' compartments. These hierarchical connections between dopamine subsystems explain distinct properties of first- and second-order memory long known by behavioral psychologists.

Selective Vulnerability of the Locus Coeruleus Noradrenergic System and its Role in Modulation of Neuroinflammation, Cognition, and Neurodegeneration

Locus coeruleus (LC) noradrenergic (NE) neurons supply the main adrenergic input to the forebrain. NE is a dual modulator of cognition and neuroinflammation. NE neurons of the LC are particularly vulnerable to degeneration both with normal aging and in neurodegenerative disorders. Consequences of this vulnerability can be observed in both cognitive impairment and dysregulation of neuroinflammation. LC NE neurons are pacemaker neurons that are active during waking and arousal and are responsive to stressors in the environment. Chronic overactivation is thought to be a major contributor to the vulnerability of these neurons. Here we review what is known about the mechanisms underlying this neuronal vulnerability and combinations of environmental and genetic factors that contribute to confer risk to these important brainstem neuromodulatory and immunomodulatory neurons. Finally, we discuss proposed and potential interventions that may reduce the overall risk for LC NE neuronal degeneration.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Lateral axonal modulation is required for stimulus-specific olfactory conditioning in Drosophila

Effective and stimulus-specific learning is essential for animals' survival. Two major mechanisms are known to aid stimulus specificity of associative learning. One is accurate stimulus-specific representations in neurons. The second is a limited effective temporal window for the reinforcing signals to induce neuromodulation after sensory stimuli. However, these mechanisms are often imperfect in preventing unspecific associations; different sensory stimuli can be represented by overlapping populations of neurons, and more importantly, the reinforcing signals alone can induce neuromodulation even without coincident sensory-evoked neuronal activity. Here, we report a crucial neuromodulatory mechanism that counteracts both limitations and is thereby essential for stimulus specificity of learning. In Drosophila, olfactory signals are sparsely represented by cholinergic Kenyon cells (KCs), which receive dopaminergic reinforcing input. We find that KCs have numerous axo-axonic connections mediated by the muscarinic type-B receptor (mAChR-B). By using functional imaging and optogenetic approaches, we show that these axo-axonic connections suppress both odor-evoked calcium responses and dopamine-evoked cAMP signals in neighboring KCs. Strikingly, behavior experiments demonstrate that mAChR-B knockdown in KCs impairs olfactory learning by inducing undesired changes to the valence of an odor that was not associated with the reinforcer. Thus, this local neuromodulation acts in concert with sparse sensory representations and global dopaminergic modulation to achieve effective and accurate memory formation.

Neuroprotection in late life attention-deficit/hyperactivity disorder: A review of pharmacotherapy and phenotype across the lifespan

For decades, psychostimulants have been the gold standard pharmaceutical treatment for attention-deficit/hyperactivity disorder (ADHD). In the United States, an astounding 9% of all boys and 4% of girls will be prescribed stimulant drugs at some point during their childhood. Recent meta-analyses have revealed that individuals with ADHD have reduced brain volume loss later in life (>60 y.o.) compared to the normal aging brain, which suggests that either ADHD or its treatment may be neuroprotective. Crucially, these neuroprotective effects were significant in brain regions (e.g., hippocampus, amygdala) where severe volume loss is linked to cognitive impairment and Alzheimer's disease. Historically, the ADHD diagnosis and its pharmacotherapy came about nearly simultaneously, making it difficult to evaluate their effects in isolation. Certain evidence suggests that psychostimulants may normalize structural brain changes typically observed in the ADHD brain. If ADHD itself is neuroprotective, perhaps exercising the brain, then psychostimulants may not be recommended across the lifespan. Alternatively, if stimulant drugs are neuroprotective, then this class of medications may warrant further investigation for their therapeutic effects. Here, we take a bottom-up holistic approach to review the psychopharmacology of ADHD in the context of recent models of attention. We suggest that future studies are greatly needed to better appreciate the interactions amongst an ADHD diagnosis, stimulant treatment across the lifespan, and structure-function alterations in the aging brain.

Dopamine

Dopamine was first described by George Barger, James Ewens, and Henry Dale in 1910 as an epinephrine-like monoamine compound. Initially believed to be a mere precursor of norepinephrine, it was mostly ignored for the next four decades (Figure 1A). However, in the 1950s Kathleen Montagu showed that dopamine occurred in the brain by itself, and a series of studies by Arvid Carlsson and collaborators demonstrated that dopamine is a bona fide neurotransmitter, a finding that would earn Carlsson the 2000 Nobel Prize in Physiology and Medicine. In a landmark experiment, he pharmacologically blocked all dopamine neurotransmission in rabbits, which rendered them completely paralyzed, and then fully recovered their behavior with an injection of the dopamine precursor L-DOPA, demonstrating that dopamine was essential for self-initiated movement (Figure 1B). A similar effect was quickly reproduced by Oleg Hornykiewicz and collaborators in human Parkinsonian patients. Within a few years, dopamine jumped from relative obscurity to being critical for life as we know it.

A functional topography within the cholinergic basal forebrain for encoding sensory cues and behavioral reinforcement outcomes

Basal forebrain cholinergic neurons (BFCNs) project throughout the cortex to regulate arousal, stimulus salience, plasticity, and learning. Although often treated as a monolithic structure, the basal forebrain features distinct connectivity along its rostrocaudal axis that could impart regional differences in BFCN processing. Here, we performed simultaneous bulk calcium imaging from rostral and caudal BFCNs over a 1-month period of variable reinforcement learning in mice. BFCNs in both regions showed equivalently weak responses to unconditioned visual stimuli and anticipated rewards. Rostral BFCNs in the horizontal limb of the diagonal band were more responsive to reward omission, more accurately classified behavioral outcomes, and more closely tracked fluctuations in pupil-indexed global brain state. Caudal tail BFCNs in globus pallidus and substantia innominata were more responsive to unconditioned auditory stimuli, orofacial movements, aversive reinforcement, and showed robust associative plasticity for punishment-predicting cues. These results identify a functional topography that diversifies cholinergic modulatory signals broadcast to downstream brain regions.

Sox6 expression distinguishes dorsally and ventrally biased dopamine neurons in the substantia nigra with distinctive properties and embryonic origins

Dopamine (DA) neurons in the ventral tier of the substantia nigra pars compacta (SNc) degenerate prominently in Parkinson's disease, while those in the dorsal tier are relatively spared. Defining the molecular, functional, and developmental characteristics of each SNc tier is crucial to understand their distinct susceptibility. We demonstrate that Sox6 expression distinguishes ventrally and dorsally biased DA neuron populations in the SNc. The Sox6+ population in the ventral SNc includes an Aldh1a1+ subset and is enriched in gene pathways that underpin vulnerability. Sox6+ neurons project to the dorsal striatum and show activity correlated with acceleration. Sox6- neurons project to the medial, ventral, and caudal striatum and respond to rewards. Moreover, we show that this adult division is encoded early in development. Overall, our work demonstrates a dual origin of the SNc that results in DA neuron cohorts with distinct molecular profiles, projections, and functions.

Context-dependent representations of movement in Drosophila dopaminergic reinforcement pathways

Dopamine plays a central role in motivating and modifying behavior, serving to invigorate current behavioral performance and guide future actions through learning. Here we examine how this single neuromodulator can contribute to such diverse forms of behavioral modulation. By recording from the dopaminergic reinforcement pathways of the Drosophila mushroom body during active odor navigation, we reveal how their ongoing motor-associated activity relates to goal-directed behavior. We found that dopaminergic neurons correlate with different behavioral variables depending on the specific navigational strategy of an animal, such that the activity of these neurons preferentially reflects the actions most relevant to odor pursuit. Furthermore, we show that these motor correlates are translated to ongoing dopamine release, and acutely perturbing dopaminergic signaling alters the strength of odor tracking. Context-dependent representations of movement and reinforcement cues are thus multiplexed within the mushroom body dopaminergic pathways, enabling them to coordinately influence both ongoing and future behavior.

Models of heterogeneous dopamine signaling in an insect learning and memory center

The Drosophila mushroom body exhibits dopamine dependent synaptic plasticity that underlies the acquisition of associative memories. Recordings of dopamine neurons in this system have identified signals related to external reinforcement such as reward and punishment. However, other factors including locomotion, novelty, reward expectation, and internal state have also recently been shown to modulate dopamine neurons. This heterogeneity is at odds with typical modeling approaches in which these neurons are assumed to encode a global, scalar error signal. How is dopamine dependent plasticity coordinated in the presence of such heterogeneity? We develop a modeling approach that infers a pattern of dopamine activity sufficient to solve defined behavioral tasks, given architectural constraints informed by knowledge of mushroom body circuitry. Model dopamine neurons exhibit diverse tuning to task parameters while nonetheless producing coherent learned behaviors. Notably, reward prediction error emerges as a mode of population activity distributed across these neurons. Our results provide a mechanistic framework that accounts for the heterogeneity of dopamine activity during learning and behavior.

Dopamine neurons across the animal kingdom are involved in the formation of associative memories. While numerous studies have recorded activity in these neurons related to external and predicted rewards, the diversity of these neurons’ activity and their tuning to non-reward-related quantities such as novelty, movement, and internal state have proved challenging to account for in traditional modeling approaches. Using a well-characterized model system for learning and memory, the mushroom body of Drosophila fruit flies, Jiang and Litwin-Kumar provide an account of the diversity of signals across dopamine neurons. They show that models optimized to solve tasks like those encountered by flies exhibit heterogeneous activity across dopamine neurons, but nonetheless this activity is sufficient for the system to solve the tasks. The models will be useful to generate testable hypotheses about dopamine neuron activity across different experimental conditions.

Acute and Sub-Chronic Exposure to Artificial Sweeteners at the Highest Environmentally Relevant Concentration Induce Less Cardiovascular Physiology Alterations in Zebrafish Larvae

Artificial sweeteners are widely used food ingredients in beverages and drinks to lower calorie intake which in turn helps prevent lifestyle diseases such as obesity. However, as their popularity has increased, the release of artificial sweetener to the aquatic environment has also increased at a tremendous rate. Thus, our study aims to systematically explore the potential cardiovascular physiology alterations caused by eight commercial artificial sweeteners, including acesulfame-K, alitame, aspartame, sodium cyclamate, dulcin, neotame, saccharine and sucralose, at the highest environmentally relevant concentration on cardiovascular performance using zebrafish (Danio rerio) as a model system. Embryonic zebrafish were exposed to the eight artificial sweeteners at 100 ppb and their cardiovascular performance (heart rate, ejection fraction, fractional shortening, stroke volume, cardiac output, heartbeat variability, and blood flow velocity) was measured and compared. Overall, our finding supports the safety of artificial sweetener exposure. However, several finding like a significant increase in the heart rate and heart rate variability after incubation in several artificial sweeteners are noteworthy. Biomarker testing also revealed that saccharine significantly increase the dopamine level in zebrafish larvae, which is might be the reason for the cardiac physiology changes observed after saccharine exposure.

Noradrenergic correlates of chronic cocaine craving: neuromelanin and functional brain imaging

Preclinical studies have implicated noradrenergic (NA) dysfunction in cocaine addiction. In particular, the NA system plays a central role in motivated behavior and may partake in the regulation of craving and drug use. Yet, human studies of the NA system are scarce, likely hampered by the difficulty in precisely localizing the locus coeruleus (LC). Here, we used neuromelanin imaging to localize the LC and quantified LC neuromelanin signal (NMS) intensity in 44 current cocaine users (CU; 37 men) and 59 nondrug users (NU; 44 men). We also employed fMRI to investigate cue-induced regional responses and LC functional connectivities, as quantified by generalized psychophysiological interaction (gPPI), in CU. Imaging data were processed by published routines and the findings were evaluated with a corrected threshold. We examined how these neural measures were associated with chronic cocaine craving, as assessed by the Cocaine Craving Questionnaire (CCQ). Compared to NU, CU demonstrated higher LC NMS for all probabilistic thresholds defined of 50-90% of the peak. In contrast, NMS of the ventral tegmental area/substantia nigra (VTA/SN) did not show significant group differences. Drug as compared to neutral cues elicited higher activations of many cortical and subcortical regions, none of which were significantly correlated with CCQ score. Drug vs. neutral cues also elicited "deactivation" of bilateral parahippocampal gyri (PHG) and PHG gPPI with a wide array of cortical and subcortical regions, including the ventral striatum and, with small volume correction, the LC. Less deactivation of the PHG (r = 0.40, p = 0.008) and higher PHG-LC gPPI (r = 0.44, p = 0.003) were positively correlated with the CCQ score. In contrast, PHG-VTA/SN connectivity did not correlate with the CCQ score. Together, chronic cocaine exposure may induce higher NMS intensity, suggesting neurotoxic effects on the LC. The correlation of cue-elicited PHG LC connectivity with CCQ score suggests a noradrenergic correlate of chronic cocaine craving. Potentially compensating for memory functions as in neurodegenerative conditions, cue-elicited PHG LC circuit connectivity plays an ill-adaptive role in supporting cocaine craving.

Dopamine modulation of sensory processing and adaptive behavior in flies

Behavioral flexibility for appropriate action selection is an advantage when animals are faced with decisions that will determine their survival or death. In order to arrive at the right decision, animals evaluate information from their external environment, internal state, and past experiences. How these different signals are integrated and modulated in the brain, and how context- and state-dependent behavioral decisions are controlled are poorly understood questions. Studying the molecules that help convey and integrate such information in neural circuits is an important way to approach these questions. Many years of work in different model organisms have shown that dopamine is a critical neuromodulator for (reward based) associative learning. However, recent findings in vertebrates and invertebrates have demonstrated the complexity and heterogeneity of dopaminergic neuron populations and their functional implications in many adaptive behaviors important for survival. For example, dopaminergic neurons can integrate external sensory information, internal and behavioral states, and learned experience in the decision making circuitry. Several recent advances in methodologies and the availability of a synaptic level connectome of the whole-brain circuitry of Drosophila melanogaster make the fly an attractive system to study the roles of dopamine in decision making and state-dependent behavior. In particular, a learning and memory center—the mushroom body—is richly innervated by dopaminergic neurons that enable it to integrate multi-modal information according to state and context, and to modulate decision-making and behavior.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

The connectome of the adult Drosophila mushroom body provides insights into function

Making inferences about the computations performed by neuronal circuits from synapse-level connectivity maps is an emerging opportunity in neuroscience. The mushroom body (MB) is well positioned for developing and testing such an approach due to its conserved neuronal architecture, recently completed dense connectome, and extensive prior experimental studies of its roles in learning, memory, and activity regulation. Here, we identify new components of the MB circuit in Drosophila, including extensive visual input and MB output neurons (MBONs) with direct connections to descending neurons. We find unexpected structure in sensory inputs, in the transfer of information about different sensory modalities to MBONs, and in the modulation of that transfer by dopaminergic neurons (DANs). We provide insights into the circuitry used to integrate MB outputs, connectivity between the MB and the central complex and inputs to DANs, including feedback from MBONs. Our results provide a foundation for further theoretical and experimental work.

A Unified Framework for Dopamine Signals across Timescales

Rapid phasic activity of midbrain dopamine neurons is thought to signal reward prediction errors (RPEs), resembling temporal difference errors used in machine learning. However, recent studies describing slowly increasing dopamine signals have instead proposed that they represent state values and arise independent from somatic spiking activity. Here we developed experimental paradigms using virtual reality that disambiguate RPEs from values. We examined dopamine circuit activity at various stages, including somatic spiking, calcium signals at somata and axons, and striatal dopamine concentrations. Our results demonstrate that ramping dopamine signals are consistent with RPEs rather than value, and this ramping is observed at all stages examined. Ramping dopamine signals can be driven by a dynamic stimulus that indicates a gradual approach to a reward. We provide a unified computational understanding of rapid phasic and slowly ramping dopamine signals: dopamine neurons perform a derivative-like computation over values on a moment-by-moment basis.

Useful road maps: studying Drosophila larva's central nervous system with the help of connectomics

The larva of Drosophila melanogaster is emerging as a powerful model system for comprehensive brain-wide understanding of the circuit implementation of neural computations. With an unprecedented amount of tools in hand, including synaptic-resolution connectomics, whole-brain imaging, and genetic tools for selective targeting of single neuron types, it is possible to dissect which circuits and computations are at work behind behaviors that have an interesting level of complexity. Here we present some of the recent advances regarding multisensory integration, learning, and action selection in Drosophila larva.

Distributional Reinforcement Learning in the Brain

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Midbrain circuits of novelty processing

Novelty triggers an increase in orienting behavior that is critical to evaluate the potential salience of unknown events. As novelty becomes familiar upon repeated encounters, this increase in response rapidly habituates as a form of behavioral adaptation underlying goal-directed behaviors. Many neurodevelopmental, psychiatric and neurodegenerative disorders are associated with abnormal responses to novelty and/or familiarity, although the neuronal circuits and cellular/molecular mechanisms underlying these natural behaviors in the healthy brain are largely unknown, as is the maladaptive processes that occur to induce impairment of novelty signaling in diseased brains. In rodents, the development of cutting-edge tools that allow for measurements of real time activity dynamics in selectively identified neuronal ensembles by gene expression signatures is beginning to provide advances in understanding the neural bases of the novelty response. Accumulating evidence indicate that midbrain circuits, the majority of which linked to dopamine transmission, promote exploratory assessments and guide approach/avoidance behaviors to different types of novelty via specific projection sites. The present review article focuses on midbrain circuit analysis relevant to novelty processing and habituation with familiarity.

Heterogeneity in striatal dopamine circuits: Form and function in dynamic reward seeking

The striatal dopamine system has long been studied in the context of reward learning, motivation, and movement. Given the prominent role dopamine plays in a variety of adaptive behavioral states, as well as diseases like addiction, it is essential to understand the full complexity of dopamine neurons and the striatal systems they target. A growing number of studies are uncovering details of the heterogeneity in dopamine neuron subpopulations. Here, we review that work to synthesize current understanding of dopamine system heterogeneity across three levels, anatomical organization, functions in behavior, and modes of action, wherein we focus on signaling profiles and local mechanisms for modulation of dopamine release. Together, these studies reveal new and emerging dimensions of the striatal dopamine system, informing its contribution to dynamic motivational and decision-making processes.

Amygdala-Midbrain Connections Modulate Appetitive and Aversive Learning

The central amygdala (CeA) orchestrates adaptive responses to emotional events. While CeA substrates for defensive behaviors have been studied extensively, CeA circuits for appetitive behaviors and their relationship to threat-responsive circuits remain poorly defined. Here, we demonstrate that the CeA sends robust inhibitory projections to the lateral substantia nigra (SNL) that contribute to appetitive and aversive learning in mice. CeA→SNL neural responses to appetitive and aversive stimuli were modulated by expectation and magnitude consistent with a population-level salience signal, which was required for Pavlovian conditioned reward-seeking and defensive behaviors. CeA→SNL terminal activation elicited reinforcement when linked to voluntary actions but failed to support Pavlovian associations that rely on incentive value signals. Consistent with a disinhibitory mechanism, CeA inputs preferentially target SNL GABA neurons, and CeA→SNL and SNL dopamine neurons respond similarly to salient stimuli. Collectively, our results suggest that amygdala-nigra interactions represent a previously unappreciated mechanism for influencing emotional behaviors.

Aversion hot spots in the dopamine system

Through the development of optogenetics and other viral vector-based technologies, our view of the dopamine system has substantially advanced over the last decade. In particular, progress has been made in the reclassification of dopamine neurons based on subtypes displaying specific projections, which are associated with different features at the anatomical, molecular and behavioral level. Together, these discoveries have raised the possibility that individual groups of dopamine cells make a unique contribution to the processing of reward and aversion. Here, we review recent studies that have identified non-canonical dopamine pathways that are excited in response to aversive stimuli, including dopamine projections to the ventromedial shell of the nucleus accumbens, prefrontal cortex, tail of the striatum, and amygdala.

Summary: Order and Disorder in Brains and Behavior

The 83rd Cold Spring Harbor Symposium on Quantitative Biology on Brains and Behavior: Order and Disorder in the Nervous System explored the tremendous recent progress in neuroscience and how these advances may be used to improve brain health and address psychiatric and neurological disorders. The Symposium explored a vast array of topics from cell types to cognition. My summary focuses on a few emerging themes. Innovative techniques were ever-present, opening up new experimental possibilities. The commoditization of many state-of-the-art technologies is pushing neuroscience beyond its artisanal ways. Another important theme was "circuits in the middle": Numerous presentations dissected cell type-specific circuits that connect different levels of analysis from molecules to behavior. These new technologies have enabled curiosity-driven investigations in animals to connect more directly with preclinical and clinical studies of human brain disorders. Numerous emerging approaches were presented in human neuroscience, bolstering the hope that circuit-specific manipulations will soon provide improved treatments for brain disorders.