Slowly evolving dopaminergic activity modulates the moment-to-moment probability of reward-related self-timed movements

Value Modulation of Self-Defeating Impulsivity

BACKGROUND

Impulse control is a critical aspect of cognitive functioning. Intuitively, whether an action is executed prematurely depends on its associated reward, yet the link between value and impulsivity remains poorly understood. Three frameworks for impulsivity offer contrasting views: impulsive behavior may be valuable because it is associated with hidden internal reward (e.g., reduction of mental effort). Alternatively, it can emerge from exploration, which is disadvantageous in the short term but can yield long-term benefits. Finally, impulsivity may reflect Pavlovian bias, an inherent tendency that occurs even when its outcome is negative.

METHODS

To test these hypotheses, we trained 17 male mice to withhold licking while anticipating variable rewards. We then measured and optogenetically manipulated dopamine release in the ventral striatum.

RESULTS

We found that higher reward magnitudes correlated with increased impulsivity. This behavior was well explained by a Pavlovian bias model. Furthermore, we observed negative dopamine signals during premature licking, suggesting that in this task, impulsivity is not merely an unsuccessful attempt at obtaining a reward. Rather, it is a failure to overcome the urge to act prematurely despite knowledge of the negative consequences of such impulsive actions.

CONCLUSIONS

Our findings underscore the integral role value plays in regulating impulsivity and suggest that the dopaminergic system influences impulsivity through the mediation of value learning.

The learning primacy hypothesis of dopamine: reconsidering dopamine's dual functions

The dopaminergic modulation of striatal circuit function remains intensely studied and debated. Nevertheless, a prevalent view is that striatal dopamine serves important roles in both reinforcement learning and the performance of movements, two highly distinct processes. But this dichotomy has led to a longstanding problem of how to interpret the functional consequences of a particular dopaminergic signal-is it to learn or to move? In order to explore this ambiguity and approach a possible resolution, this review examines the key evidence for dopamine's role in learning and movement. As part of that discussion, we consider a recent body of evidence that views the common dichotomous perspective through a more nuanced lens, by suggesting a comparatively limited dopaminergic contribution to movement. This concept, which we refer to as the learning primacy hypothesis, offers a unified conceptual framework for understanding dopaminergic function.

A mechanism linking dopamine's roles in reinforcement, movement and motivation

Dopamine neurons (DANs) play seemingly distinct roles in reinforcement, 1-3 motivation, 4,5 and movement, 6,7 and DA-modulating therapies relieve symptoms across a puzzling spectrum of neurologic and psychiatric symptoms. 8 Yet, the mechanistic relationship among these roles is unknown. Here, we show DA's tripartite roles are causally linked by a process in which phasic striatal DA rapidly and persistently recalibrates the propensity to move, a measure of vigor. Using a self-timed movement task, we found that single exposures to reward-related DA transients (both endogenous and exogenously-induced) exerted one-shot updates to movement timing-but in a surprising fashion. Rather than reinforce specific movement times, DA transients quantitatively changed movement timing on the next trial, with larger transients leading to earlier movements (and smaller to later), consistent with a stochastic search process that calibrates the frequency of movement. Both abrupt and gradual changes in external and internal contingencies-such as timing criterion, reward content, and satiety state-caused changes to the amplitude of DA transients that causally altered movement timing. The rapidity and bidirectionality of the one-shot effects are difficult to reconcile with gradual synaptic plasticity, and instead point to more flexible cellular mechanisms, such as DA-dependent modulation of neuronal excitability. Our findings shed light on how natural reinforcement, as well as DA-related disorders such as Parkinson's disease, could affect behavioral vigor.

On the varieties of conscious experiences: Altered Beliefs Under Psychedelics (ALBUS)

How is it that psychedelics so profoundly impact brain and mind? According to the model of "Relaxed Beliefs Under Psychedelics" (REBUS), 5-HT2a agonism is thought to help relax prior expectations, thus making room for new perspectives and patterns. Here, we introduce an alternative (but largely compatible) perspective, proposing that REBUS effects may primarily correspond to a particular (but potentially pivotal) regime of very high levels of 5-HT2a receptor agonism. Depending on both a variety of contextual factors and the specific neural systems being considered, we suggest opposite effects may also occur in which synchronous neural activity becomes more powerful, with accompanying "Strengthened Beliefs Under Psychedelics" (SEBUS) effects. Such SEBUS effects are consistent with the enhanced meaning-making observed in psychedelic therapy (e.g. psychological insight and the noetic quality of mystical experiences), with the imposition of prior expectations on perception (e.g. hallucinations and pareidolia), and with the delusional thinking that sometimes occurs during psychedelic experiences (e.g. apophenia, paranoia, engendering of inaccurate interpretations of events, and potentially false memories). With "Altered Beliefs Under Psychedelics" (ALBUS), we propose that the manifestation of SEBUS vs. REBUS effects may vary across the dose-response curve of 5-HT2a signaling. While we explore a diverse range of sometimes complex models, our basic idea is fundamentally simple: psychedelic experiences can be understood as kinds of waking dream states of varying degrees of lucidity, with similar underlying mechanisms. We further demonstrate the utility of ALBUS by providing neurophenomenological models of psychedelics focusing on mechanisms of conscious perceptual synthesis, dreaming, and episodic memory and mental simulation.

Neuronal encoding of behaviors and instrumental learning in the dorsal striatum

The dorsal striatum is instrumental in regulating motor control and goal-directed behaviors. The classical description of the two output pathways of the dorsal striatum highlights their antagonistic control over actions. However, recent experimental evidence implicates both pathways and their coordinated activities during actions. In this review, we examine the different models proposed for striatal encoding of actions during self-paced behaviors and how they can account for evidence harvested during goal-directed behaviors. We also discuss how the activation of striatal ensembles can be reshaped and reorganized to support the formation of instrumental learning and behavioral flexibility. Future work integrating these considerations may resolve controversies regarding the control of actions by striatal networks.

Dopamine builds and reveals reward-associated latent behavioral attractors

Phasic variations in dopamine levels are interpreted as a teaching signal reinforcing rewarded behaviors. However, behavior also depends on the motivational, neuromodulatory effect of phasic dopamine. In this study, we reveal a neurodynamical principle that unifies these roles in a recurrent network-based decision architecture embodied through an action-perception loop with the task space, the MAGNet model. Dopamine optogenetic conditioning in mice was accounted for by an embodied network model in which attractors encode internal goals. Dopamine-dependent synaptic plasticity created "latent" attractors, to which dynamics converged, but only locally. Attractor basins were widened by dopamine-modulated synaptic excitability, rendering goals accessible globally, i.e. from distal positions. We validated these predictions optogenetically in mice: dopamine neuromodulation suddenly and specifically attracted animals toward rewarded locations, without off-target motor effects. We thus propose that motivational dopamine reveals dopamine-built attractors representing potential goals in a behavioral landscape.

Dopamine dynamics are dispensable for movement but promote reward responses

Dopamine signalling modes differ in kinetics and spatial patterns of receptor activation1,2. How these modes contribute to motor function, motivation and learning has long been debated3-21. Here we show that action-potential-induced dopamine release is dispensable for movement initiation but supports reward-oriented behaviour. We generated mice with dopamine-neuron-specific knockout of the release site organizer protein RIM to disrupt action-potential-induced dopamine release. In these mice, rapid in vivo dopamine dynamics were strongly impaired, but baseline dopamine persisted and fully supported spontaneous movement. Conversely, reserpine-mediated dopamine depletion or blockade of dopamine receptors disrupted movement initiation. The dopamine precursor L-DOPA reversed reserpine-induced bradykinesia without restoring fast dopamine dynamics, a result that substantiated the conclusion that these dynamics are dispensable for movement initiation. In contrast to spontaneous movement, reward-oriented behaviour was impaired in dopamine-neuron-specific RIM knockout mice. In conditioned place preference and two-odour discrimination tasks, the mice effectively learned to distinguish the cues, which indicates that reward-based learning persists after RIM ablation. However, the performance vigour was reduced. During probabilistic cue-reward association, dopamine dynamics and conditioned responses assessed through anticipatory licking were disrupted. These results demonstrate that action-potential-induced dopamine release is dispensable for motor function and subsecond precision of movement initiation but promotes motivation and performance during reward-guided behaviours.

Constraints on the subsecond modulation of striatal dynamics by physiological dopamine signaling

Dopaminergic neurons play a crucial role in associative learning, but their capacity to regulate behavior on subsecond timescales remains debated. It is thought that dopaminergic neurons drive certain behaviors by rapidly modulating striatal spiking activity; however, a view has emerged that only artificially high (that is, supra-physiological) dopamine signals alter behavior on fast timescales. This raises the possibility that moment-to-moment striatal spiking activity is not strongly shaped by dopamine signals in the physiological range. To test this, we transiently altered dopamine levels while monitoring spiking responses in the ventral striatum of behaving mice. These manipulations led to only weak changes in striatal activity, except when dopamine release exceeded reward-matched levels. These findings suggest that dopaminergic neurons normally play a minor role in the subsecond modulation of striatal dynamics in relation to other inputs and demonstrate the importance of discerning dopaminergic neuron contributions to brain function under physiological and potentially nonphysiological conditions.

Competitive integration of time and reward explains value-sensitive foraging decisions and frontal cortex ramping dynamics

Patch foraging presents a ubiquitous decision-making process in which animals decide when to abandon a resource patch of diminishing value to pursue an alternative. We developed a virtual foraging task in which mouse behavior varied systematically with patch value. Mouse behavior could be explained by a model integrating time and rewards antagonistically, scaled by a latent patience state. The model accounted for deviations from predictions of optimal foraging theory. Neural recordings throughout frontal areas revealed encoding of decision variables from the integrator model, most robustly in frontal cortex. Regression modeling followed by unsupervised clustering identified a subset of ramping neurons. These neurons' firing rates ramped up gradually (up to tens of seconds), were inhibited by rewards, and were better described as a continuous ramp than a discrete stepping process. Together, these results identify integration via frontal cortex ramping dynamics as a candidate mechanism for solving patch foraging problems.

Complementary cognitive roles for D2-MSNs and D1-MSNs during interval timing

The role of striatal pathways in cognitive processing is unclear. We studied dorsomedial striatal cognitive processing during interval timing, an elementary cognitive task that requires mice to estimate intervals of several seconds and involves working memory for temporal rules as well as attention to the passage of time. We harnessed optogenetic tagging to record from striatal D2-dopamine receptor-expressing medium spiny neurons (D2-MSNs) in the indirect pathway and from D1-dopamine receptor-expressing MSNs (D1-MSNs) in the direct pathway. We found that D2-MSNs and D1-MSNs exhibited distinct dynamics over temporal intervals as quantified by principal component analyses and trial-by-trial generalized linear models. MSN recordings helped construct and constrain a four-parameter drift-diffusion computational model. This model predicted that disrupting either D2-MSNs or D1-MSNs would increase interval timing response times and alter MSN firing. In line with this prediction, we found that optogenetic inhibition or pharmacological disruption of either D2-MSNs or D1-MSNs increased interval timing response times. Pharmacologically disrupting D2-MSNs or D1-MSNs also changed MSN dynamics and degraded trial-by-trial temporal decoding. Together, our findings demonstrate that D2-MSNs and D1-MSNs make complementary contributions to interval timing despite opposing dynamics, implying that striatal direct and indirect pathways work together to shape temporal control of action. These data provide novel insight into basal ganglia cognitive operations beyond movement and have implications for human striatal diseases and therapies targeting striatal pathways.

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

Free will: An Example of the Dopaminergic System

Neuroscience has convinced people that much of their behavior is determined by causes unknown to them and beyond their control. However, are advances in neuroscience truly a prerequisite for such beliefs? Robert Kane's theory of ultimate responsibility is libertarian theory. Its innovative nature makes it possible to discuss the neurophysiological basis of its postulates. Using the functions of the midbrain dopaminergic system as an example, this article provides an overview of this neurophysiological basis. According to Kane, if we are to be ultimately responsible for our wills as well as for our actions, some actions in our lives must lack sufficient motives and causes. These are self-forming actions. Dopamine is hypothesized to mediate self-forming action execution. Dopamine not only mediates action but also ensures synaptic plasticity in the brain, that is, learning from action; hence, dopamine changes the acting individual and provides the formation of our own wills. The basal ganglia, which are the main target of dopamine in the brain, act through parallel pathways and are involved in decision-making processes. Dopamine is also involved in the regulation of the neurodynamical properties of prefrontal cortex networks with random spiking noise. It can be assumed that the activity of the dopaminergic system is closely related to the physiological basis of free will.

Slow ramping emerges from spontaneous fluctuations in spiking neural networks

The capacity to initiate actions endogenously is critical for goal-directed behavior. Spontaneous voluntary actions are typically preceded by slow-ramping activity in medial frontal cortex that begins around two seconds before movement, which may reflect spontaneous fluctuations that influence action timing. However, the mechanisms by which these slow ramping signals emerge from single-neuron and network dynamics remain poorly understood. Here, we developed a spiking neural-network model that produces spontaneous slow ramping activity in single neurons and population activity with onsets ~2 s before threshold crossings. A key prediction of our model is that neurons that ramp together have correlated firing patterns before ramping onset. We confirmed this model-derived hypothesis in a dataset of human single neuron recordings from medial frontal cortex. Our results suggest that slow ramping signals reflect bounded spontaneous fluctuations that emerge from quasi-winner-take-all dynamics in clustered networks that are temporally stabilized by slow-acting synapses.

Structured flexibility in recurrent neural networks via neuromodulation

The goal of theoretical neuroscience is to develop models that help us better understand biological intelligence. Such models range broadly in complexity and biological detail. For example, task-optimized recurrent neural networks (RNNs) have generated hypotheses about how the brain may perform various computations, but these models typically assume a fixed weight matrix representing the synaptic connectivity between neurons. From decades of neuroscience research, we know that synaptic weights are constantly changing, controlled in part by chemicals such as neuromodulators. In this work we explore the computational implications of synaptic gain scaling, a form of neuromodulation, using task-optimized low-rank RNNs. In our neuromodulated RNN (NM-RNN) model, a neuromodulatory subnetwork outputs a low-dimensional neuromodulatory signal that dynamically scales the low-rank recurrent weights of an output-generating RNN. In empirical experiments, we find that the structured flexibility in the NM-RNN allows it to both train and generalize with a higher degree of accuracy than low-rank RNNs on a set of canonical tasks. Additionally, via theoretical analyses we show how neuromodulatory gain scaling endows networks with gating mechanisms commonly found in artificial RNNs. We end by analyzing the low-rank dynamics of trained NM-RNNs, to show how task computations are distributed.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward rewards. Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) are crucial for this process, via engagement of a reciprocally connected network with their striatal targets. Critically, it remains unknown how dopamine neuron activity itself engages dopamine signals throughout the striatum, across learning. Here, we investigated how optogenetic Pavlovian cue conditioning of VTA or SNc dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. Although VTA stimulation-evoked robust dopamine release in NAc core, shell, and DMS, predictive cues preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS. Despite the development of vigorous movement late in training, conditioned dopamine signals did not emerge in the DLS, even during Pavlovian conditioning with SNc dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show a broad dissociation in the fundamental prediction and reward-related information generated by VTA and SNc dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale adaptations across the striatal network emerge during learning to coordinate behavior.

Encoding of self-initiated actions in axon terminals of the mesocortical pathway

Significance

The initiation of goal-directed actions is a complex process involving the medial prefrontal cortex and dopaminergic inputs through the mesocortical pathway. However, it is unclear what information the mesocortical pathway conveys and how it impacts action initiation. In this study, we unveiled the indispensable role of mesocortical axon terminals in encoding the execution of movements in self-initiated actions.

Aim

To investigate the role of mesocortical axon terminals in encoding the execution of movements in self-initiated actions.

Approach

We designed a lever-press task in which mice internally determine the timing of the press, receiving a larger reward for longer waiting periods.

Results

Our study revealed that self-initiated actions depend on dopaminergic signaling mediated by D2 receptors, whereas sensory-triggered lever-press actions do not involve D2 signaling. Microprism-mediated two-photon calcium imaging further demonstrated ramping activity in mesocortical axon terminals approximately 0.5 s before the self-initiated lever press. Remarkably, the ramping patterns remained consistent whether the mice responded to cues immediately for a smaller reward or held their response for a larger reward.

Conclusions

We conclude that mesocortical dopamine axon terminals encode the timing of self-initiated actions, shedding light on a crucial aspect of the intricate neural mechanisms governing goal-directed behavior.

Striatal Dopamine Contributions to Skilled Motor Learning

Coordinated multijoint limb and digit movements-"manual dexterity"-underlie both specialized skills (e.g., playing the piano) and more mundane tasks (e.g., tying shoelaces). Impairments in dexterous skill cause significant disability, as occurs with motor cortical injury, Parkinson's disease, and a range of other pathologies. Clinical observations, as well as basic investigations, suggest that corticostriatal circuits play a critical role in learning and performing dexterous skills. Furthermore, dopaminergic signaling in these regions is implicated in synaptic plasticity and motor learning. Nonetheless, the role of striatal dopamine signaling in skilled motor learning remains poorly understood. Here, we use fiber photometry paired with a genetically encoded dopamine sensor to investigate striatal dopamine release in both male and female mice as they learn and perform a skilled reaching task. Dopamine rapidly increases during a skilled reach and peaks near pellet consumption. In the dorsolateral striatum, dopamine dynamics are faster than in the dorsomedial and ventral striatum. Across training, as reaching performance improves, dopamine signaling shifts from pellet consumption to cues that predict pellet availability, particularly in medial and ventral areas of the striatum. Furthermore, performance prediction errors are present across the striatum, with reduced dopamine release after an unsuccessful reach. These findings show that dopamine dynamics during skilled motor behaviors change with learning and are differentially regulated across striatal subregions.

Mesolimbic dopamine ramps reflect environmental timescales

Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.

Slow ramping emerges from spontaneous fluctuations in spiking neural networks

1. We reveal a mechanism for slow-ramping signals before spontaneous voluntary movements. 2. Slow synapses stabilize spontaneous fluctuations in spiking neural network. 3. We validate model predictions in human frontal cortical single-neuron recordings. 4. The model recreates the readiness potential in an EEG proxy signal. 5. Neurons that ramp together had correlated activity before ramping onset. The capacity to initiate actions endogenously is critical for goal-directed behavior. Spontaneous voluntary actions are typically preceded by slow-ramping activity in medial frontal cortex that begins around two seconds before movement, which may reflect spontaneous fluctuations that influence action timing. However, the mechanisms by which these slow ramping signals emerge from single-neuron and network dynamics remain poorly understood. Here, we developed a spiking neural-network model that produces spontaneous slow ramping activity in single neurons and population activity with onsets ∼2 seconds before threshold crossings. A key prediction of our model is that neurons that ramp together have correlated firing patterns before ramping onset. We confirmed this model-derived hypothesis in a dataset of human single neuron recordings from medial frontal cortex. Our results suggest that slow ramping signals reflect bounded spontaneous fluctuations that emerge from quasi-winner-take-all dynamics in clustered networks that are temporally stabilized by slow-acting synapses.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward acquisition of rewards. Dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNC) are crucial for this process. Dopamine neurons are embedded in a reciprocally connected network with their striatal targets, the functional organization of which remains poorly understood. Here, we investigated how learning during optogenetic Pavlovian cue conditioning of VTA or SNC dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent dopamine biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. While VTA stimulation evoked robust dopamine release in NAc core, shell, and DMS, cues predictive of this activation preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Corresponding negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS, and not the shell. Despite development of vigorous movement late in training, conditioned dopamine signals did not similarly emerge in the DLS, even during Pavlovian conditioning with SNC dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show broad dissociation in the fundamental prediction and reward-related information generated by different dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale plasticity across the striatal network emerges during Pavlovian learning to coordinate behavior.

Libet's legacy: A primer to the neuroscience of volition

The neuroscience of volition is an emerging subfield of the brain sciences, with hundreds of papers on the role of consciousness in action formation published each year. This makes the state-of-the-art in the discipline poorly accessible to newcomers and difficult to follow even for experts in the field. Here we provide a comprehensive summary of research in this field since its inception that will be useful to both groups. We also discuss important ideas that have received little coverage in the literature so far. We systematically reviewed a set of 2220 publications, with detailed consideration of almost 500 of the most relevant papers. We provide a thorough introduction to the seminal work of Benjamin Libet from the 1960s to 1980s. We also discuss common criticisms of Libet's method, including temporal introspection, the interpretation of the assumed physiological correlates of volition, and various conceptual issues. We conclude with recent advances and potential future directions in the field, highlighting modern methodological approaches to volition, as well as important recent findings.

Temporal Information Processing in the Cerebellum and Basal Ganglia

Temporal information processing in the range of a few hundred milliseconds to seconds involves the cerebellum and basal ganglia. In this chapter, we present recent studies on nonhuman primates. In the studies presented in the first half of the chapter, monkeys were trained to make eye movements when a certain amount of time had elapsed since the onset of the visual cue (time production task). The animals had to report time lapses ranging from several hundred milliseconds to a few seconds based on the color of the fixation point. In this task, the saccade latency varied with the time length to be measured and showed stochastic variability from one trial to the other. Trial-to-trial variability under the same conditions correlated well with pupil diameter and the preparatory activity in the deep cerebellar nuclei and the motor thalamus. Inactivation of these brain regions delayed saccades when asked to report subsecond intervals. These results suggest that the internal state, which changes with each trial, may cause fluctuations in cerebellar neuronal activity, thereby producing variations in self-timing. When measuring different time intervals, the preparatory activity in the cerebellum always begins approximately 500 ms before movements, regardless of the length of the time interval being measured. However, the preparatory activity in the striatum persists throughout the mandatory delay period, which can be up to 2 s, with different rate of increasing activity. Furthermore, in the striatum, the visual response and low-frequency oscillatory activity immediately before time measurement were altered by the length of the intended time interval. These results indicate that the state of the network, including the striatum, changes with the intended timing, which lead to different time courses of preparatory activity. Thus, the basal ganglia appear to be responsible for measuring time in the range of several hundred milliseconds to seconds, whereas the cerebellum is responsible for regulating self-timing variability in the subsecond range. The second half of this chapter presents studies related to periodic timing. During eye movements synchronized with alternating targets at regular intervals, different neurons in the cerebellar nuclei exhibit activity related to movement timing, predicted stimulus timing, and the temporal error of synchronization. Among these, the activity associated with target appearance is particularly enhanced during synchronized movements and may represent an internal model of the temporal structure of stimulus sequence. We also considered neural mechanism underlying the perception of periodic timing in the absence of movement. During perception of rhythm, we predict the timing of the next stimulus and focus our attention on that moment. In the missing oddball paradigm, the subjects had to detect the omission of a regularly repeated stimulus. When employed in humans, the results show that the fastest temporal limit for predicting each stimulus timing is about 0.25 s (4 Hz). In monkeys performing this task, neurons in the cerebellar nuclei, striatum, and motor thalamus exhibit periodic activity, with different time courses depending on the brain region. Since electrical stimulation or inactivation of recording sites changes the reaction time to stimulus omission, these neuronal activities must be involved in periodic temporal processing. Future research is needed to elucidate the mechanism of rhythm perception, which appears to be processed by both cortico-cerebellar and cortico-basal ganglia pathways.

Acetylcholine waves and dopamine release in the striatum

Striatal dopamine encodes reward, with recent work showing that dopamine release occurs in spatiotemporal waves. However, the mechanism of dopamine waves is unknown. Here we report that acetylcholine release in mouse striatum also exhibits wave activity, and that the spatial scale of striatal dopamine release is extended by nicotinic acetylcholine receptors. Based on these findings, and on our demonstration that single cholinergic interneurons can induce dopamine release, we hypothesized that the local reciprocal interaction between cholinergic interneurons and dopamine axons suffices to drive endogenous traveling waves. We show that the morphological and physiological properties of cholinergic interneuron - dopamine axon interactions can be modeled as a reaction-diffusion system that gives rise to traveling waves. Analytically-tractable versions of the model show that the structure and the nature of propagation of acetylcholine and dopamine traveling waves depend on their coupling, and that traveling waves can give rise to empirically observed correlations between these signals. Thus, our study provides evidence for striatal acetylcholine waves in vivo, and proposes a testable theoretical framework that predicts that the observed dopamine and acetylcholine waves are strongly coupled phenomena.

Reversal of pathological motor behavior in a model of Parkinson's disease by striatal dopamine uncaging

Motor deficits observed in Parkinson's disease (PD) are caused by the loss of dopaminergic neurons and the subsequent dopamine depletion in different brain areas. The most common therapy to treat motor symptoms for patients with this disorder is the systemic intake of L-DOPA that increases dopamine levels in all the brain, making it difficult to discern the main locus of dopaminergic action in the alleviation of motor control. Caged compounds are molecules with the ability to release neuromodulators locally in temporary controlled conditions using light. In the present study, we measured the turning behavior of unilateral dopamine-depleted mice before and after dopamine uncaging. The optical delivery of dopamine in the striatum of lesioned mice produced contralateral turning behavior that resembled, to a lesser extent, the contralateral turning behavior evoked by a systemic injection of apomorphine. Contralateral turning behavior induced by dopamine uncaging was temporarily tied to the transient elevation of dopamine concentration and was reversed when dopamine decreased to pathological levels. Remarkably, contralateral turning behavior was tuned by changing the power and frequency of light stimulation, opening the possibility to modulate dopamine fluctuations using different light stimulation protocols. Moreover, striatal dopamine uncaging recapitulated the motor effects of a low concentration of systemic L-DOPA, but with better temporal control of dopamine levels. Finally, dopamine uncaging reduced the pathological synchronization of striatal neuronal ensembles that characterize unilateral dopamine-depleted mice. We conclude that optical delivery of dopamine in the striatum resembles the motor effects induced by systemic injection of dopaminergic agonists in unilateral dopamine-depleted mice. Future experiments using this approach could help to elucidate the role of dopamine in different brain nuclei in normal and pathological conditions.

Bio-Inspired Design of Superconducting Spiking Neuron and Synapse

The imitative modelling of processes in the brain of living beings is an ambitious task. However, advances in the complexity of existing hardware brain models are limited by their low speed and high energy consumption. A superconducting circuit with Josephson junctions closely mimics the neuronal membrane with channels involved in the operation of the sodium-potassium pump. The dynamic processes in such a system are characterised by a duration of picoseconds and an energy level of attojoules. In this work, two superconducting models of a biological neuron are studied. New modes of their operation are identified, including the so-called bursting mode, which plays an important role in biological neural networks. The possibility of switching between different modes in situ is shown, providing the possibility of dynamic control of the system. A synaptic connection that mimics the short-term potentiation of a biological synapse is developed and demonstrated. Finally, the simplest two-neuron chain comprising the proposed bio-inspired components is simulated, and the prospects of superconducting hardware biosimilars are briefly discussed.

Past, present, and future of tools for dopamine detection

Dopamine (DA) is a critical neuromodulator involved in various brain functions. To understand how DA regulates neural circuits and behaviors in the physiological and pathological conditions, it is essential to have tools that enable the direct detection of DA dynamics in vivo. Recently, genetically encoded DA sensors based on G protein-coupled receptors revolutionized this field, as it allows us to track in vivo DA dynamic with unprecedented spatial-temporal resolution, high molecular specificity, and sub-second kinetics. In this review, we first summarize traditional DA detection methods. Then we focus on the development of genetically encoded DA sensors and feature its significance to understanding dopaminergic neuromodulation across diverse behaviors and species. Finally, we present our perspectives about the future direction of the next-generation DA sensors and extend their potential applications. Overall, this review offers a comprehensive perspective on the past, present, and future of DA detection tools, with important implications for the study of DA functions in health and disease.

A rise-to-threshold process for a relative-value decision

Whereas progress has been made in the identification of neural signals related to rapid, cued decisions1-3, less is known about how brains guide and terminate more ethologically relevant decisions in which an animal's own behaviour governs the options experienced over minutes4-6. Drosophila search for many seconds to minutes for egg-laying sites with high relative value7,8 and have neurons, called oviDNs, whose activity fulfills necessity and sufficiency criteria for initiating the egg-deposition motor programme9. Here we show that oviDNs express a calcium signal that (1) dips when an egg is internally prepared (ovulated), (2) drifts up and down over seconds to minutes-in a manner influenced by the relative value of substrates-as a fly determines whether to lay an egg and (3) reaches a consistent peak level just before the abdomen bend for egg deposition. This signal is apparent in the cell bodies of oviDNs in the brain and it probably reflects a behaviourally relevant rise-to-threshold process in the ventral nerve cord, where the synaptic terminals of oviDNs are located and where their output can influence behaviour. We provide perturbational evidence that the egg-deposition motor programme is initiated once this process hits a threshold and that subthreshold variation in this process regulates the time spent considering options and, ultimately, the choice taken. Finally, we identify a small recurrent circuit that feeds into oviDNs and show that activity in each of its constituent cell types is required for laying an egg. These results argue that a rise-to-threshold process regulates a relative-value, self-paced decision and provide initial insight into the underlying circuit mechanism for building this process.

Accumbens cholinergic interneurons dynamically promote dopamine release and enable motivation

Motivation to work for potential rewards is critically dependent on dopamine (DA) in the nucleus accumbens (NAc). DA release from NAc axons can be controlled by at least two distinct mechanisms: (1) action potentials propagating from DA cell bodies in the ventral tegmental area (VTA), and (2) activation of β2* nicotinic receptors by local cholinergic interneurons (CINs). How CIN activity contributes to NAc DA dynamics in behaving animals is not well understood. We monitored DA release in the NAc Core of awake, unrestrained rats using the DA sensor RdLight1, while simultaneously monitoring or manipulating CIN activity at the same location. CIN stimulation rapidly evoked DA release, and in contrast to slice preparations, this DA release showed no indication of short-term depression or receptor desensitization. The sound of unexpected food delivery evoked a brief joint increase in CIN population activity and DA release, with a second joint increase as rats approached the food. In an operant task, we observed fast ramps in CIN activity during approach behaviors, either to start the trial or to collect rewards. These CIN ramps co-occurred with DA release ramps, without corresponding changes in the firing of lateral VTA DA neurons. Finally, we examined the effects of blocking CIN influence over DA release through local NAc infusion of DHβE, a selective antagonist of β2* nicotinic receptors. DHβE dose-dependently interfered with motivated approach decisions, mimicking the effects of a DA antagonist. Our results support a key influence of CINs over motivated behavior via the local regulation of DA release.

From Prediction to Action: Dissociable Roles of Ventral Tegmental Area and Substantia Nigra Dopamine Neurons in Instrumental Reinforcement

Reward seeking requires the coordination of motor programs to achieve goals. Midbrain dopamine neurons are critical for reinforcement, and their activation is sufficient for learning about cues, actions, and outcomes. Here we examine in detail the mechanisms underlying the ability of ventral tegmental area (VTA) and substantia nigra (SNc) dopamine neurons to support instrumental learning. By exploiting numerous behavioral tasks in combination with time-limited optogenetic manipulations in male and female rats, we reveal that VTA and SNc dopamine neurons generate reinforcement through separable psychological processes. VTA dopamine neurons imbue actions and their associated cues with motivational value that allows flexible and persistent pursuit, whereas SNc dopamine neurons support time-limited, precise, action-specific learning that is nonscalable and inflexible. This architecture is reminiscent of actor-critic reinforcement learning models with VTA and SNc instructing the critic and actor, respectively. Our findings indicate that heterogeneous dopamine systems support unique forms of instrumental learning that ultimately result in disparate reward-seeking strategies.SIGNIFICANCE STATEMENT Dopamine neurons in the midbrain are essential for learning, motivation, and movement. Here we describe in detail the ability of VTA and SNc dopamine neurons to generate instrumental reinforcement, a process where an agent learns about actions they can emit to earn reward. While rats will avidly work and learn to respond for activation of VTA and SNc dopamine neurons, we find that only VTA dopamine neurons imbue actions and their associated cues with motivational value that spur continued pursuit of reward. Our data support a hypothesis that VTA and SNc dopamine neurons engage distinct psychological processes that have consequences for our understanding of these neurons in health and disease.

Unraveling the dynamics of dopamine release and its actions on target cells

The neuromodulator dopamine (DA) is essential for regulating learning, motivation, and movement. Despite its importance, however, the mechanisms by which DA influences the activity of target cells to alter behavior remain poorly understood. In this review, we describe recent methodological advances that are helping to overcome challenges that have historically hindered the field. We discuss how the employment of these methods is shedding light on the complex dynamics of extracellular DA in the brain, as well as how DA signaling alters the electrical, biochemical, and population activity of target neurons in vivo. These developments are generating novel hypotheses about the mechanisms through which DA release modifies behavior.

Spontaneous behaviour is structured by reinforcement without explicit reward

Spontaneous animal behaviour is built from action modules that are concatenated by the brain into sequences1,2. However, the neural mechanisms that guide the composition of naturalistic, self-motivated behaviour remain unknown. Here we show that dopamine systematically fluctuates in the dorsolateral striatum (DLS) as mice spontaneously express sub-second behavioural modules, despite the absence of task structure, sensory cues or exogenous reward. Photometric recordings and calibrated closed-loop optogenetic manipulations during open field behaviour demonstrate that DLS dopamine fluctuations increase sequence variation over seconds, reinforce the use of associated behavioural modules over minutes, and modulate the vigour with which modules are expressed, without directly influencing movement initiation or moment-to-moment kinematics. Although the reinforcing effects of optogenetic DLS dopamine manipulations vary across behavioural modules and individual mice, these differences are well predicted by observed variation in the relationships between endogenous dopamine and module use. Consistent with the possibility that DLS dopamine fluctuations act as a teaching signal, mice build sequences during exploration as if to maximize dopamine. Together, these findings suggest a model in which the same circuits and computations that govern action choices in structured tasks have a key role in sculpting the content of unconstrained, high-dimensional, spontaneous behaviour.

Mesolimbic dopamine adapts the rate of learning from action

Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions^1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction⁴; however, so far there has been little consideration of how direct policy learning might inform our understanding⁵. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning⁶.

Protein composition of axonal dopamine release sites in the striatum

Dopamine is an important modulator of cognition and movement. We recently found that evoked dopamine secretion is fast and relies on active zone-like release sites. Here, we used in vivo biotin identification (iBioID) proximity proteomics in mouse striatum to assess which proteins are present at these sites. Using three release site baits, we identified proteins that are enriched over the general dopamine axonal protein content, and they fell into several categories, including active zone, Ca2+ regulatory, and synaptic vesicle proteins. We also detected many proteins not previously associated with vesicular exocytosis. Knockout of the presynaptic organizer protein RIM strongly decreased the hit number obtained with iBioID, while Synaptotagmin-1 knockout did not. α-Synuclein, a protein linked to Parkinson's disease, was enriched at release sites, and its enrichment was lost in both tested mutants. We conclude that RIM organizes scaffolded dopamine release sites and provide a proteomic assessment of the composition of these sites.

Midbrain dopamine neurons signal phasic and ramping reward prediction error during goal-directed navigation

Goal-directed navigation requires learning to accurately estimate location and select optimal actions in each location. Midbrain dopamine neurons are involved in reward value learning and have been linked to reward location learning. They are therefore ideally placed to provide teaching signals for goal-directed navigation. By imaging dopamine neural activity as mice learned to actively navigate a closed-loop virtual reality corridor to obtain reward, we observe phasic and pre-reward ramping dopamine activity, which are modulated by learning stage and task engagement. A Q-learning model incorporating position inference recapitulates our results, displaying prediction errors resembling phasic and ramping dopamine neural activity. The model predicts that ramping is followed by improved task performance, which we confirm in our experimental data, indicating that the dopamine ramp may have a teaching effect. Our results suggest that midbrain dopamine neurons encode phasic and ramping reward prediction error signals to improve goal-directed navigation.

Dopamine mediates the bidirectional update of interval timing

The role of dopamine (DA) as a reward prediction error (RPE) signal in reinforcement learning (RL) tasks has been well-established over the past decades. Recent work has shown that the RPE interpretation can also account for the effects of DA on interval timing by controlling the speed of subjective time. According to this theory, the timing of the dopamine signal relative to reward delivery dictates whether subjective time speeds up or slows down: Early DA signals speed up subjective time and late signals slow it down. To test this bidirectional prediction, we reanalyzed measurements of dopaminergic neurons in the substantia nigra pars compacta of mice performing a self-timed movement task. Using the slope of ramping dopamine activity as a readout of subjective time speed, we found that trial-by-trial changes in the slope could be predicted from the timing of dopamine activity on the previous trial. This result provides a key piece of evidence supporting a unified computational theory of RL and interval timing. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Review: Subjective Time Perception, Dopamine Signaling, and Parkinsonian Slowness

The association between idiopathic Parkinson's disease, a paradigmatic dopamine-deficiency syndrome, and problems in the estimation of time has been studied experimentally for decades. I review that literature, which raises a question about whether and if dopamine deficiency relates not only to the motor slowness that is an objective and cardinal parkinsonian sign, but also to a compromised neural substrate for time perception. Why does a clinically (motorically) significant deficiency in dopamine play a role in the subjective perception of time's passage? After a discussion of a classical conception of basal ganglionic control of movement under the influence of dopamine, I describe recent work in healthy mice using optogenetics; the methodology visualizes dopaminergic neuronal firing in very short time intervals, then allows for correlation with motor behaviors in trained tasks. Moment-to-moment neuronal activity is both highly dynamic and variable, as assessed by photometry of genetically defined dopaminergic neurons. I use those animal data as context to review a large experimental experience in humans, spanning decades, that has examined subjective time perception mainly in Parkinson's disease, but also in other movement disorders. Although the human data are mixed in their findings, I argue that loss of dynamic variability in dopaminergic neuronal activity over very short intervals may be a fundamental sensory aspect in the pathophysiology of parkinsonism. An important implication is that therapeutic response in Parkinson's disease needs to be understood in terms of short-term alterations in dynamic neuronal firing, as has already been examined in novel ways—for example, in the study of real-time changes in neuronal network oscillations across very short time intervals. A finer analysis of a treatment's network effects might aid in any effort to augment clinical response to either medications or functional neurosurgical interventions in Parkinson's disease.

Different Patterns of Neural Activity Characterize Motor Skill Performance During Acquisition and Retention

Motor performance and learning have distinct behavioral and neural signatures and can be uniquely modulated by various informational and motivational factors. Contemporary frameworks describe four different motor learning mechanisms mapped onto specific neural regions which are key for motor skill acquisition: error-based learning (cerebellum), reinforcement learning (basal ganglia), cognitive strategies (prefrontal cortex), and use-dependent learning (motor cortex). However, little is known about the neural circuits engaged during skill acquisition that are modulated specifically by practice-based performance improvement and those that predict recall performance. Based on previous work, we hypothesize that brain activity during practice in primary motor cortex and basal ganglia (1) is associated with trial-by-trial practice performance and (2) is predictive of immediate recall performance. Leveraging the contemporary framework, we use a well-known task paradigm that primarily relies upon cognitive strategy, reinforcement, and use-based learning mechanisms to test our hypotheses. Forty neurotypical young adults were asked to practice a pinch force tracking task. Participants received performance feedback after each trial during practice. We used whole brain analysis of functional magnetic resonance imaging (fMRI) and behavioral performance measures (i.e., time-on-target and self-efficacy) during the practice phase to determine which brain activation patterns are (1) associated with trial-by-trial tracking performance and (2) predictive of immediate no-feedback retention performance. We observed brain activations in the frontal orbital cortex, putamen, amygdala, and insula correlated with tracking performance improvement during practice. In contrast, a different set of performance-related activated regions were observed that were associated with immediate retention performance that included the primary motor cortex, superior frontal gyrus, somatosensory cortex, angular gyrus, and parietal gyrus. Our findings demonstrate that improved practice performance and recall of a sensorimotor skill are correlated with distinct neural activity patterns during acquisition, drawing on different motor learning mechanisms during encoding. While motor performance improvements depend on both cortical and subcortical regions, motor skill recall depends primarily on prefrontal and motor cortices. We discuss possible interpretations for why our hypothesis regarding basal ganglia activity and retention performance was not supported. Understanding the different neural mechanisms engaged in motor performance and learning may inform novel interventions to enhance motor skill learning.

Multimodal detection of dopamine by sniffer cells expressing genetically encoded fluorescent sensors

Dopamine supports locomotor control and higher brain functions such as motivation and learning. Consistently, dopaminergic dysfunction is involved in a spectrum of neurological and neuropsychiatric diseases. Detailed data on dopamine dynamics is needed to understand how dopamine signals translate into cellular and behavioral responses, and to uncover pathological disturbances in dopamine-related diseases. Genetically encoded fluorescent dopamine sensors have recently enabled unprecedented monitoring of dopamine dynamics in vivo. However, these sensors' utility for in vitro and ex vivo assays remains unexplored. Here, we present a blueprint for making dopamine sniffer cells for multimodal dopamine detection. We generated sniffer cell lines with inducible expression of seven different dopamine sensors and perform a head-to-head comparison of sensor properties to guide users in sensor selection. In proof-of-principle experiments, we apply the sniffer cells to record endogenous dopamine release from cultured neurons and striatal slices, and for determining tissue dopamine content. Furthermore, we use the sniffer cells to measure dopamine uptake and release via the dopamine transporter as a radiotracer free, high-throughput alternative to electrochemical- and radiotracer-based assays. Importantly, the sniffer cell framework can readily be applied to the growing list of genetically encoded fluorescent neurotransmitter sensors.

Dopamine firing plays a dual role in coding reward prediction errors and signaling motivation in a working memory task

Little is known about how dopamine (DA) neuron firing rates behave in cognitively demanding decision-making tasks. Here, we investigated midbrain DA activity in monkeys performing a discrimination task in which the animal had to use working memory (WM) to report which of two sequentially applied vibrotactile stimuli had the higher frequency. We found that perception was altered by an internal bias, likely generated by deterioration of the representation of the first frequency during the WM period. This bias greatly controlled the DA phasic response during the two stimulation periods, confirming that DA reward prediction errors reflected stimulus perception. In contrast, tonic dopamine activity during WM was not affected by the bias and did not encode the stored frequency. More interestingly, both delay-period activity and phasic responses before the second stimulus negatively correlated with reaction times of the animals after the trial start cue and thus represented motivated behavior on a trial-by-trial basis. During WM, this motivation signal underwent a ramp-like increase. At the same time, motivation positively correlated with accuracy, especially in difficult trials, probably by decreasing the effect of the bias. Overall, our results indicate that DA activity, in addition to encoding reward prediction errors, could at the same time be involved in motivation and WM. In particular, the ramping activity during the delay period suggests a possible DA role in stabilizing sustained cortical activity, hypothetically by increasing the gain communicated to prefrontal neurons in a motivation-dependent way.