Tonic dopamine modulates exploitation of reward learning

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Mesolimbic Neural Response Dynamics Predict Future Individual Alcohol Drinking in Mice

BACKGROUND

Individual variability in response to rewarding stimuli is a striking but understudied phenomenon. The mesolimbic dopamine system is critical in encoding the reinforcing properties of both natural reward and alcohol; however, how innate or baseline differences in the response dynamics of this circuit define individual behavior and shape future vulnerability to alcohol remain unknown.

METHODS

Using naturalistic behavioral assays, a voluntary alcohol drinking paradigm, in vivo fiber photometry, in vivo electrophysiology, and chemogenetics, we investigated how differences in mesolimbic neural circuit activity contribute to the individual variability seen in reward processing and, by proxy, alcohol drinking.

RESULTS

We first characterized heterogeneous behavioral and neural responses to natural reward and defined how these baseline responses predicted future individual alcohol-drinking phenotypes in male mice. We then determined spontaneous ventral tegmental area dopamine neuron firing profiles associated with responses to natural reward that predicted alcohol drinking. Using a dual chemogenetic approach, we mimicked specific mesolimbic dopamine neuron firing activity before or during voluntary alcohol drinking to link unique neurophysiological profiles to individual phenotype. We show that hyperdopaminergic individuals exhibit a lower neuronal response to both natural reward and alcohol that predicts lower levels of alcohol consumption in the future.

CONCLUSIONS

These findings reveal unique, circuit-specific neural signatures that predict future individual vulnerability or resistance to alcohol and expand the current knowledge base on how some individuals are able to titrate their alcohol consumption whereas others go on to engage in unhealthy alcohol-drinking behaviors.

Impaired flexible reward learning in ADHD patients is associated with blunted reinforcement sensitivity and neural signals in ventral striatum and parietal cortex

Reward-based learning and decision-making are prime candidates to understand symptoms of attention deficit hyperactivity disorder (ADHD). However, only limited evidence is available regarding the neurocomputational underpinnings of the alterations seen in ADHD. This concerns flexible behavioral adaption in dynamically changing environments, which is challenging for individuals with ADHD. One previous study points to elevated choice switching in adolescent ADHD, which was accompanied by disrupted learning signals in medial prefrontal cortex. Here, we investigated young adults with ADHD (n = 17) as compared to age- and sex-matched controls (n = 17) using a probabilistic reversal learning experiment during functional magnetic resonance imaging (fMRI). The task requires continuous learning to guide flexible behavioral adaptation to changing reward contingencies. To disentangle the neurocomputational underpinnings of the behavioral data, we used reinforcement learning (RL) models, which informed the analysis of fMRI data. ADHD patients performed worse than controls particularly in trials before reversals, i.e., when reward contingencies were stable. This pattern resulted from 'noisy' choice switching regardless of previous feedback. RL modelling showed decreased reinforcement sensitivity and enhanced learning rates for negative feedback in ADHD patients. At the neural level, this was reflected in a diminished representation of choice probability in the left posterior parietal cortex in ADHD. Moreover, modelling showed a marginal reduction of learning about the unchosen option, which was paralleled by a marginal reduction in learning signals incorporating the unchosen option in the left ventral striatum. Taken together, we show that impaired flexible behavior in ADHD is due to excessive choice switching ('hyper-flexibility'), which can be detrimental or beneficial depending on the learning environment. Computationally, this resulted from blunted sensitivity to reinforcement of which we detected neural correlates in the attention-control network, specifically in the parietal cortex. These neurocomputational findings remain preliminary due to the relatively small sample size.

Striatal dopamine contributions to skilled motor learning

Coordinated multi-joint limb and digit movements - "manual dexterity" - underlie both specialized skills (e.g., playing the piano) and more mundane tasks (e.g., tying shoelaces). Impairments in dexterous skill cause significant disability, as occurs with motor cortical injury, Parkinson's Disease, and a range of other pathologies. Clinical observations, as well as basic investigations, suggest that cortico-striatal circuits play a critical role in learning and performing dexterous skills. Furthermore, dopaminergic signaling in these regions is implicated in synaptic plasticity and motor learning. Nonetheless, the role of striatal dopamine signaling in skilled motor learning remains poorly understood. Here, we use fiber photometry paired with a genetically encoded dopamine sensor to investigate striatal dopamine release as mice learn and perform a skilled reaching task. Dopamine rapidly increases during a skilled reach and peaks near pellet consumption. In dorsolateral striatum, dopamine dynamics are faster than in dorsomedial and ventral striatum. Across training, as reaching performance improves, dopamine signaling shifts from pellet consumption to cues that predict pellet availability, particularly in medial and ventral areas of striatum. Furthermore, performance prediction errors are present across the striatum, with reduced dopamine release after an unsuccessful reach. These findings show that dopamine dynamics during skilled motor behaviors change with learning and are differentially regulated across striatal subregions.

Maternal selenium dietary supplementation alters sociability and reinforcement learning deficits induced by in utero exposure to maternal immune activation in mice

Maternal immune activation (MIA) during pregnancy increases the risk for the unborn foetus to develop neurodevelopmental conditions such as autism spectrum disorder and schizophrenia later in life. MIA mouse models recapitulate behavioural and biological phenotypes relevant to both conditions, and are valuable models to test novel treatment approaches. Selenium (Se) has potent anti-inflammatory properties suggesting it may be an effective prophylactic treatment against MIA. The aim of this study was to determine if Se supplementation during pregnancy can prevent adverse effects of MIA on offspring brain and behaviour in a mouse model. Selenium was administered via drinking water (1.5 ppm) to pregnant dams from gestational day (GD) 9 to birth, and MIA was induced at GD17 using polyinosinic:polycytidylic acid (poly-I:C, 20 mg/kg via intraperitoneal injection). Foetal placenta and brain cytokine levels were assessed using a Luminex assay and brain elemental nutrients assessed using inductively coupled plasma- mass spectrometry. Adult offspring were behaviourally assessed using a reinforcement learning paradigm, the three-chamber sociability test and the open field test. MIA elevated placental IL-1β and IL-17, and Se supplementation successfully prevented this elevation. MIA caused an increase in foetal brain calcium, which was prevented by Se supplement. MIA caused in offspring a female-specific reduction in sociability, which was recovered by Se, and a male-specific reduction in social memory, which was not recovered by Se. Exposure to poly-I:C or selenium, but not both, reduced performance in the reinforcement learning task. Computational modelling indicated that this was predominantly due to increased exploratory behaviour, rather than reduced rate of learning the location of the food reward. This study demonstrates that while Se may be beneficial in ameliorating sociability deficits caused by MIA, it may have negative effects in other behavioural domains. Caution in the use of Se supplementation during pregnancy is therefore warranted.

Distinct sub-second dopamine signaling in dorsolateral striatum measured by a genetically-encoded fluorescent sensor

The development of genetically encoded dopamine sensors such as dLight has provided a new approach to measuring slow and fast dopamine dynamics both in brain slices and in vivo, possibly enabling dopamine measurements in areas like the dorsolateral striatum (DLS) where previously such recordings with fast-scan cyclic voltammetry (FSCV) were difficult. To test this, we first evaluated dLight photometry in mouse brain slices with simultaneous FSCV and found that both techniques yielded comparable results, but notable differences in responses to dopamine transporter inhibitors, including cocaine. We then used in vivo fiber photometry with dLight in mice to examine responses to cocaine in DLS. We also compared dopamine responses during Pavlovian conditioning across the striatum. We show that dopamine increases were readily detectable in DLS and describe transient dopamine kinetics, as well as slowly developing signals during conditioning. Overall, our findings indicate that dLight photometry is well suited to measuring dopamine dynamics in DLS.

Synaptotagmin-1-dependent phasic axonal dopamine release is dispensable for basic motor behaviors in mice

In Parkinson's disease (PD), motor dysfunctions only become apparent after extensive loss of DA innervation. This resilience has been hypothesized to be due to the ability of many motor behaviors to be sustained through a diffuse basal tone of DA; but experimental evidence for this is limited. Here we show that conditional deletion of the calcium sensor synaptotagmin-1 (Syt1) in DA neurons (Syt1 cKODA mice) abrogates most activity-dependent axonal DA release in the striatum and mesencephalon, leaving somatodendritic (STD) DA release intact. Strikingly, Syt1 cKODA mice showed intact performance in multiple unconditioned DA-dependent motor tasks and even in a task evaluating conditioned motivation for food. Considering that basal extracellular DA levels in the striatum were unchanged, our findings suggest that activity-dependent DA release is dispensable for such tasks and that they can be sustained by a basal tone of extracellular DA. Taken together, our findings reveal the striking resilience of DA-dependent motor functions in the context of a near-abolition of phasic DA release, shedding new light on why extensive loss of DA innervation is required to reveal motor dysfunctions in PD.

Dopamine and norepinephrine differentially mediate the exploration-exploitation tradeoff

The catecholamines dopamine (DA) and norepinephrine (NE) have been repeatedly implicated in neuropsychiatric vulnerability, in part via their roles in mediating the decision making processes. Although the two neuromodulators share a synthesis pathway and are co-activated under states of arousal, they engage in distinct circuits and roles in modulating neural activity across the brain. However, in the computational neuroscience literature, they have been assigned similar roles in modulating the latent cognitive processes of decision making, in particular the exploration-exploitation tradeoff. Revealing how each neuromodulator contributes to this explore-exploit process will be important in guiding mechanistic hypotheses emerging from computational psychiatric approaches. To understand the differences and overlaps of the roles of these two catecholamine systems in regulating exploration and exploitation, a direct comparison using the same dynamic decision making task is needed. Here, we ran mice in a restless two-armed bandit task, which encourages both exploration and exploitation. We systemically administered a nonselective DA receptor antagonist (flupenthixol), a nonselective DA receptor agonist (apomorphine), a NE beta-receptor antagonist (propranolol), and a NE beta-receptor agonist (isoproterenol), and examined changes in exploration within subjects across sessions. We found a bidirectional modulatory effect of dopamine receptor activity on the level of exploration. Increasing dopamine activity decreased exploration and decreasing dopamine activity increased exploration. Beta-noradrenergic receptor activity also modulated exploration, but the modulatory effect was mediated by sex. Reinforcement learning model parameters suggested that dopamine modulation affected exploration via decision noise and norepinephrine modulation affected exploration via outcome sensitivity. Together, these findings suggested that the mechanisms that govern the transition between exploration and exploitation are sensitive to changes in both catecholamine functions and revealed differential roles for NE and DA in mediating exploration.

Dietary fat restriction affects brain reward regions in a randomized crossover trial

BACKGROUNDWeight-loss diets often target dietary fat or carbohydrates, macronutrients that are sensed via distinct gut-brain pathways and differentially affect peripheral hormones and metabolism. However, the effects of such diet changes on the human brain are unclear. METHODSWe investigated whether selective isocaloric reductions in dietary fat or carbohydrates altered dopamine D2/3 receptor binding potential (D2BP) and neural activity in brain-reward regions in response to visual food cues in 17 inpatient adults with obesity as compared with a eucaloric baseline diet using a randomized crossover design. RESULTSOn the fifth day of dietary fat restriction, but not carbohydrate restriction, both D2BP and neural activity to food cues were decreased in brain-reward regions. After the reduced-fat diet, ad libitum intake shifted toward foods high in both fat and carbohydrates. CONCLUSIONThese results suggest that dietary fat restriction increases tonic dopamine in brain-reward regions and affects food choice in ways that may hamper diet adherence. TRIAL REGISTRATIONClinicalTrials.gov NCT00846040 FUNDING. NIDDK 1ZIADK013037.

Dopamine Transporter Deficient Rodents: Perspectives and Limitations for Neuroscience

The key element of dopamine (DA) neurotransmission is undoubtedly DA transporter (DAT), a transmembrane protein responsible for the synaptic reuptake of the mediator. Changes in DAT's function can be a key mechanism of pathological conditions associated with hyperdopaminergia. The first strain of gene-modified rodents with a lack of DAT were created more than 25 years ago. Such animals are characterized by increased levels of striatal DA, resulting in locomotor hyperactivity, increased levels of motor stereotypes, cognitive deficits, and other behavioral abnormalities. The administration of dopaminergic and pharmacological agents affecting other neurotransmitter systems can mitigate those abnormalities. The main purpose of this review is to systematize and analyze (1) known data on the consequences of changes in DAT expression in experimental animals, (2) results of pharmacological studies in these animals, and (3) to estimate the validity of animals lacking DAT as models for discovering new treatments of DA-related disorders.

The effects of time horizon and guided choices on explore-exploit decisions in rodents

Humans and animals have to balance the need for exploring new options with exploiting known options that yield good outcomes. This tradeoff is known as the explore-exploit dilemma. To better understand the neural mechanisms underlying how humans and animals address the explore-exploit dilemma, a good animal behavioral model is critical. Most previous rodents explore-exploit studies used ethologically unrealistic operant boxes and reversal learning paradigms in which the decision to abandon a bad option is confounded by the need for exploring a novel option for information collection, making it difficult to separate different drives and heuristics for exploration. In this study, we investigated how rodents make explore-exploit decisions using a spatial navigation horizon task (Wilson et al., 2014) adapted to rats to address the above limitations. We compared the rats' performance to that of humans using identical measures. We showed that rats use prior information to effectively guide exploration. In addition, rats use information-driven directed exploration like humans, but the extent to which they explore has the opposite dependance on time horizon than humans. Moreover, we found that free choices and guided choices have different influences on exploration in rodents, a finding that has not yet been tested in humans. This study reveals that the explore-exploit spatial behavior of rats is more complex than previously thought. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Maladaptive explore/exploit trade-offs in schizophrenia

Schizophrenia is a complex disorder that remains poorly understood, particularly at the systems level. In this opinion article we argue that the explore/exploit trade-off concept provides a holistic and ecologically valid framework to resolve some of the apparent paradoxes that have emerged within schizophrenia research. We review recent evidence suggesting that fundamental explore/exploit behaviors may be maladaptive in schizophrenia during physical, visual, and cognitive foraging. We also describe how theories from the broader optimal foraging literature, such as the marginal value theorem (MVT), could provide valuable insight into how aberrant processing of reward, context, and cost/effort evaluations interact to produce maladaptive responses.

Evidence for dopaminergic involvement in endogenous modulation of pain relief

Relief of ongoing pain is a potent motivator of behavior, directing actions to escape from or reduce potentially harmful stimuli. Whereas endogenous modulation of pain events is well characterized, relatively little is known about the modulation of pain relief and its corresponding neurochemical basis. Here, we studied pain modulation during a probabilistic relief-seeking task (a 'wheel of fortune' gambling task), in which people actively or passively received reduction of a tonic thermal pain stimulus. We found that relief perception was enhanced by active decisions and unpredictability, and greater in high novelty-seeking trait individuals, consistent with a model in which relief is tuned by its informational content. We then probed the roles of dopaminergic and opioidergic signaling, both of which are implicated in relief processing, by embedding the task in a double-blinded cross-over design with administration of the dopamine precursor levodopa and the opioid receptor antagonist naltrexone. We found that levodopa enhanced each of these information-specific aspects of relief modulation but no significant effects of the opioidergic manipulation. These results show that dopaminergic signaling has a key role in modulating the perception of pain relief to optimize motivation and behavior.

Software for near-real-time voltammetric tracking of tonic neurotransmitter levels in vivo

Tonic extracellular neurotransmitter concentrations are important modulators of central network homeostasis. Disruptions in these tonic levels are thought to play a role in neurologic and psychiatric disease. Therefore, ways to improve their quantification are actively being investigated. Previously published voltammetric software packages have implemented FSCV, which is not capable of measuring tonic concentrations of neurotransmitters in vivo. In this paper, custom software was developed for near-real-time tracking (scans every 10 s) of neurotransmitters’ tonic concentrations with high sensitivity and spatiotemporal resolution both in vitro and in vivo using cyclic voltammetry combined with dynamic background subtraction (M-CSWV and FSCAV). This software was designed with flexibility, speed, and user-friendliness in mind. This software enables near-real-time measurement by reducing data analysis time through an optimized modeling algorithm, and efficient memory handling makes long-term measurement possible. The software permits customization of the cyclic voltammetric waveform shape, enabling experiments to detect a specific analyte of interest. Finally, flexibility considerations allow the user to alter the fitting parameters, filtering characteristics, and size and shape of the analyte kernel, based on data obtained live during the experiment to obtain accurate measurements as experimental conditions change. Herein, the design and advantages of this near-real-time voltammetric software are described, and its use is demonstrated in in vivo experiments.

Striatal dopamine dissociates methylphenidate effects on value-based versus surprise-based reversal learning

Psychostimulants such as methylphenidate are widely used for their cognitive enhancing effects, but there is large variability in the direction and extent of these effects. We tested the hypothesis that methylphenidate enhances or impairs reward/punishment-based reversal learning depending on baseline striatal dopamine levels and corticostriatal gating of reward/punishment-related representations in stimulus-specific sensory cortex. Young healthy adults (N = 100) were scanned with functional magnetic resonance imaging during a reward/punishment reversal learning task, after intake of methylphenidate or the selective D_2/3-receptor antagonist sulpiride. Striatal dopamine synthesis capacity was indexed with [¹⁸F]DOPA positron emission tomography. Methylphenidate improved and sulpiride decreased overall accuracy and response speed. Both drugs boosted reward versus punishment learning signals to a greater degree in participants with higher dopamine synthesis capacity. By contrast, striatal and stimulus-specific sensory surprise signals were boosted in participants with lower dopamine synthesis. These results unravel the mechanisms by which methylphenidate gates both attention and reward learning.

The mechanisms underpinning the variability in methylphenidate’s effects on cognition remain unclear. Here, the authors show that such effects reflect changes in striatal dopamine-related output gating of task-relevant cortical signals, and that these changes depend on baseline dopamine synthesis capacity.

Different brain systems support learning from received and avoided pain during human pain-avoidance learning

Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.

Social Determinants of Inter-Individual Variability and Vulnerability: The Role of Dopamine

Individuals differ in their traits and preferences, which shape their interactions, their prospects for survival and their susceptibility to diseases. These correlations are well documented, yet the neurophysiological mechanisms underlying the emergence of distinct personalities and their relation to vulnerability to diseases are poorly understood. Social ties, in particular, are thought to be major modulators of personality traits and psychiatric vulnerability, yet the majority of neuroscience studies are performed on rodents in socially impoverished conditions. Rodent micro-society paradigms are therefore key experimental paradigms to understand how social life generates diversity by shaping individual traits. Dopamine circuitry is implicated at the interface between social life experiences, the expression of essential traits, and the emergence of pathologies, thus proving a possible mechanism to link these three concepts at a neuromodulatory level. Evaluating inter-individual variability in automated social testing environments shows great promise for improving our understanding of the link between social life, personality, and precision psychiatry – as well as elucidating the underlying neurophysiological mechanisms.

Impulsivity and risk-seeking as Bayesian inference under dopaminergic control

Bayesian models successfully account for several of dopamine (DA)'s effects on contextual calibration in interval timing and reward estimation. In these models, tonic levels of DA control the precision of stimulus encoding, which is weighed against contextual information when making decisions. When DA levels are high, the animal relies more heavily on the (highly precise) stimulus encoding, whereas when DA levels are low, the context affects decisions more strongly. Here, we extend this idea to intertemporal choice and probability discounting tasks. In intertemporal choice tasks, agents must choose between a small reward delivered soon and a large reward delivered later, whereas in probability discounting tasks, agents must choose between a small reward that is always delivered and a large reward that may be omitted with some probability. Beginning with the principle that animals will seek to maximize their reward rates, we show that the Bayesian model predicts a number of curious empirical findings in both tasks. First, the model predicts that higher DA levels should normally promote selection of the larger/later option, which is often taken to imply that DA decreases 'impulsivity,' and promote selection of the large/risky option, often taken to imply that DA increases 'risk-seeking.' However, if the temporal precision is sufficiently decreased, higher DA levels should have the opposite effect-promoting selection of the smaller/sooner option (higher impulsivity) and the small/safe option (lower risk-seeking). Second, high enough levels of DA can result in preference reversals. Third, selectively decreasing the temporal precision, without manipulating DA, should promote selection of the larger/later and large/risky options. Fourth, when a different post-reward delay is associated with each option, animals will not learn the option-delay contingencies, but this learning can be salvaged when the post-reward delays are made more salient. Finally, the Bayesian model predicts correlations among behavioral phenotypes: Animals that are better timers will also appear less impulsive.

The Role of Dopaminergic Genes in Probabilistic Reinforcement Learning in Schizophrenia Spectrum Disorders

Schizophrenia spectrum disorders (SZ) are characterized by impairments in probabilistic reinforcement learning (RL), which is associated with dopaminergic circuitry encompassing the prefrontal cortex and basal ganglia. However, there are no studies examining dopaminergic genes with respect to probabilistic RL in SZ. Thus, the aim of our study was to examine the impact of dopaminergic genes on performance assessed by the Probabilistic Selection Task (PST) in patients with SZ in comparison to healthy control (HC) subjects. In our study, we included 138 SZ patients and 188 HC participants. Genetic analysis was performed with respect to the following genetic polymorphisms: rs4680 in COMT, rs907094 in DARP-32, rs2734839, rs936461, rs1800497, and rs6277 in DRD2, rs747302 and rs1800955 in DRD4 and rs28363170 and rs2975226 in DAT1 genes. The probabilistic RL task was completed by 59 SZ patients and 95 HC subjects. SZ patients performed significantly worse in acquiring reinforcement contingencies during the task in comparison to HCs. We found no significant association between genetic polymorphisms and RL among SZ patients; however, among HC participants with respect to the DAT1 rs28363170 polymorphism, individuals with 10-allele repeat genotypes performed better in comparison to 9-allele repeat carriers. The present study indicates the relevance of the DAT1 rs28363170 polymorphism in RL in HC participants.

Chronic nicotine increases midbrain dopamine neuron activity and biases individual strategies towards reduced exploration in mice

Long-term exposure to nicotine alters brain circuits and induces profound changes in decision-making strategies, affecting behaviors both related and unrelated to drug seeking and consumption. Using an intracranial self-stimulation reward-based foraging task, we investigated in mice the impact of chronic nicotine on midbrain dopamine neuron activity and its consequence on the trade-off between exploitation and exploration. Model-based and archetypal analysis revealed substantial inter-individual variability in decision-making strategies, with mice passively exposed to nicotine shifting toward a more exploitative profile compared to non-exposed animals. We then mimicked the effect of chronic nicotine on the tonic activity of dopamine neurons using optogenetics, and found that photo-stimulated mice adopted a behavioral phenotype similar to that of mice exposed to chronic nicotine. Our results reveal a key role of tonic midbrain dopamine in the exploration/exploitation trade-off and highlight a potential mechanism by which nicotine affects the exploration/exploitation balance and decision-making.

Sex differences in learning from exploration

Sex-based modulation of cognitive processes could set the stage for individual differences in vulnerability to neuropsychiatric disorders. While value-based decision making processes in particular have been proposed to be influenced by sex differences, the overall correct performance in decision making tasks often show variable or minimal differences across sexes. Computational tools allow us to uncover latent variables that define different decision making approaches, even in animals with similar correct performance. Here, we quantify sex differences in mice in the latent variables underlying behavior in a classic value-based decision making task: a restless 2-armed bandit. While male and female mice had similar accuracy, they achieved this performance via different patterns of exploration. Male mice tended to make more exploratory choices overall, largely because they appeared to get 'stuck' in exploration once they had started. Female mice tended to explore less but learned more quickly during exploration. Together, these results suggest that sex exerts stronger influences on decision making during periods of learning and exploration than during stable choices. Exploration during decision making is altered in people diagnosed with addictions, depression, and neurodevelopmental disabilities, pinpointing the neural mechanisms of exploration as a highly translational avenue for conferring sex-modulated vulnerability to neuropsychiatric diagnoses.

Reinforcement learning links spontaneous cortical dopamine impulses to reward

In their pioneering study on dopamine release, Romo and Schultz speculated "...that the amount of dopamine released by unmodulated spontaneous impulse activity exerts a tonic, permissive influence on neuronal processes more actively engaged in preparation of self-initiated movements...."¹ Motivated by the suggestion of "spontaneous impulses," as well as by the "ramp up" of dopaminergic neuronal activity that occurs when rodents navigate to a reward,^2-5 we asked two questions. First, are there spontaneous impulses of dopamine that are released in cortex? Using cell-based optical sensors of extrasynaptic dopamine, [DA]_ex,⁶ we found that spontaneous dopamine impulses in cortex of naive mice occur at a rate of ∼0.01 per second. Next, can mice be trained to change the amplitude and/or timing of dopamine events triggered by internal brain dynamics, much as they can change the amplitude and timing of dopamine impulses based on an external cue?^7-9 Using a reinforcement learning paradigm based solely on rewards that were gated by feedback from real-time measurements of [DA]_ex, we found that mice can volitionally modulate their spontaneous [DA]_ex. In particular, by only the second session of daily, hour-long training, mice increased the rate of impulses of [DA]_ex, increased the amplitude of the impulses, and increased their tonic level of [DA]_ex for a reward. Critically, mice learned to reliably elicit [DA]_ex impulses prior to receiving a reward. These effects reversed when the reward was removed. We posit that spontaneous dopamine impulses may serve as a salient cognitive event in behavioral planning.

The modulation of brain network integration and arousal during exploration

There is growing interest in how neuromodulators shape brain networks. Recent neuroimaging studies provide evidence that brainstem arousal systems, such as the locus coeruleus-norepinephrine system (LC-NE), influence functional connectivity and brain network topology, suggesting they have a role in flexibly reconfiguring brain networks in order to adapt behavior and cognition to environmental demands. To date, however, the relationship between brainstem arousal systems and functional connectivity has not been assessed within the context of a task with an established relationship between arousal and behavior, with most prior studies relying on incidental variations in arousal or pharmacological manipulation and static brain networks constructed over long periods of time. These factors have likely contributed to a heterogeneity of effects across studies. To address these issues, we took advantage of the association between LC-NE-linked arousal and exploration to probe the relationships between exploratory choice, arousal—as measured indirectly via pupil diameter—and brain network dynamics. Exploration in a bandit task was associated with a shift toward fewer, more weakly connected modules that were more segregated in terms of connectivity and topology but more integrated with respect to the diversity of cognitive systems represented in each module. Functional connectivity strength decreased, and changes in connectivity were correlated with changes in pupil diameter, in line with the hypothesis that brainstem arousal systems influence the dynamic reorganization of brain networks. More broadly, we argue that carefully aligning dynamic network analyses with task designs can increase the temporal resolution at which behaviorally- and cognitively-relevant modulations can be identified, and offer these results as a proof of concept of this approach.

The Role of Glia in Addiction: Dopamine as a Modulator of Glial Responses in Addiction

Addiction is a chronic and potentially deadly disease considered a global health problem. Nevertheless, there is still no ideal treatment for its management. The alterations in the reward system are the most known pathophysiological mechanisms. Dopamine is the pivotal neurotransmitter involved in neuronal drug reward mechanisms and its neuronal mechanisms have been intensely investigated in recent years. However, neuroglial interactions and their relation to drug addiction development and maintenance of drug addiction have been understudied. Many reports have found that most neuroglial cells express dopamine receptors and that dopamine activity may induce neuroimmunomodulatory effects. Furthermore, current research has also shown that pro- and anti-inflammatory molecules modulate dopaminergic neuron activity. Thus, studying the immune mechanisms of dopamine associated with drug abuse is vital in researching new pathophysiological mechanisms and new therapeutic targets for addiction management.

Tonic dopamine, uncertainty and basal ganglia action selection

To make optimal decisions in uncertain circumstances flexible adaption of behaviour is required; exploring alternatives when the best choice is unknown, exploiting what is known when that is best. Using a computational model of the basal ganglia, we propose that switches between exploratory and exploitative decisions are mediated by the interaction between tonic dopamine and cortical input to the basal ganglia. We show that a biologically detailed action selection circuit model, endowed with dopamine dependant striatal plasticity, can optimally solve the explore-exploit problem, estimating the true underlying state of a noisy Gaussian diffusion process. Critical to the model's performance was a fluctuating level of tonic dopamine which increased under conditions of uncertainty. With an optimal range of tonic dopamine, explore-exploit decisions were mediated by the effects of tonic dopamine on the precision of the model action selection mechanism. Under conditions of uncertain reward pay-out, the model's reduced selectivity allowed disinhibition of multiple alternative actions to be explored at random. Conversely, when uncertainly about reward pay-out was low, enhanced selectivity of the action selection circuit facilitated exploitation of the high value choice. Model performance was at the level of a Kalman filter which provides an optimal solution for the task. These simulations support the idea that this subcortical neural circuit may have evolved to facilitate decision making in non-stationary reward environments. The model generates several experimental predictions with relevance to abnormal decision making in neuropsychiatric and neurological disease.

A New Paradigm for Training Hyperactive Dopamine Transporter Knockout Rats: Influence of Novel Stimuli on Object Recognition

Attention deficit hyperactivity disorder (ADHD) is believed to be connected with a high level of hyperactivity caused by alterations of the control of dopaminergic transmission in the brain. The strain of hyperdopaminergic dopamine transporter knockout (DAT-KO) rats represents an optimal model for investigating ADHD-related pathological mechanisms. The goal of this work was to study the influence of the overactivated dopamine system in the brain on a motor cognitive task fulfillment. The DAT-KO rats were trained to learn an object recognition task and store it in long-term memory. We found that DAT-KO rats can learn to move an object and retrieve food from the rewarded familiar objects and not to move the non-rewarded novel objects. However, we observed that the time of task performance and the distances traveled were significantly increased in DAT-KO rats in comparison with wild-type controls. Both groups of rats explored the novel objects longer than the familiar cubes. However, unlike controls, DAT-KO rats explored novel objects significantly longer and with fewer errors, since they preferred not to move the non-rewarded novel objects. After a 3 months' interval that followed the training period, they were able to retain the learned skills in memory and to efficiently retrieve them. The data obtained indicate that DAT-KO rats have a deficiency in learning the cognitive task, but their hyperactivity does not prevent the ability to learn a non-spatial cognitive task under the presentation of novel stimuli. The longer exploration of novel objects during training may ensure persistent learning of the task paradigm. These findings may serve as a base for developing new ADHD learning paradigms.

Reward motivation and cognitive flexibility in tau null-mutation mice

The reduction of tau or hyperphosphorylated tau (p-tau) has been proposed as a therapeutic strategy for Alzheimer's disease (AD) and frontotemporal dementia (FTD). Cognitive decline and sleep-wake dysregulation seen in AD and FTD patients are mimicked in transgenic and null-mutation mouse models of tauopathy. Alterations in the reward system are additional symptoms of AD and FTD. However, the role of tau in reward processes is not well understood. The present study aimed to examine reward and reward-motivated cognitive processes in male and female tau knockout (tau-/-) and wild-type mice using progressive ratio and reversal learning tasks. Tau-/- mice were heavier, ate more in the home cage, and reached criterion in operant lever training faster than wild-type mice. Tau-/- mice had a higher breakpoint in progressive ratio but were unimpaired in reversal learning or reward sensitivity. These data indicate that tau loss of function alters reward processing. This may help to explain aberrant reward-related behaviors in tauopathy patients and highlights a potentially important area for consideration in the development of anti-tau therapies.

Rational inattention and tonic dopamine

Slow-timescale (tonic) changes in dopamine (DA) contribute to a wide variety of processes in reinforcement learning, interval timing, and other domains. Furthermore, changes in tonic DA exert distinct effects depending on when they occur (e.g., during learning vs. performance) and what task the subject is performing (e.g., operant vs. classical conditioning). Two influential theories of tonic DA-the average reward theory and the Bayesian theory in which DA controls precision-have each been successful at explaining a subset of empirical findings. But how the same DA signal performs two seemingly distinct functions without creating crosstalk is not well understood. Here we reconcile the two theories under the unifying framework of 'rational inattention,' which (1) conceptually links average reward and precision, (2) outlines how DA manipulations affect this relationship, and in so doing, (3) captures new empirical phenomena. In brief, rational inattention asserts that agents can increase their precision in a task (and thus improve their performance) by paying a cognitive cost. Crucially, whether this cost is worth paying depends on average reward availability, reported by DA. The monotonic relationship between average reward and precision means that the DA signal contains the information necessary to retrieve the precision. When this information is needed after the task is performed, as presumed by Bayesian inference, acute manipulations of DA will bias behavior in predictable ways. We show how this framework reconciles a remarkably large collection of experimental findings. In reinforcement learning, the rational inattention framework predicts that learning from positive and negative feedback should be enhanced in high and low DA states, respectively, and that DA should tip the exploration-exploitation balance toward exploitation. In interval timing, this framework predicts that DA should increase the speed of the internal clock and decrease the extent of interference by other temporal stimuli during temporal reproduction (the central tendency effect). Finally, rational inattention makes the new predictions that these effects should be critically dependent on the controllability of rewards, that post-reward delays in intertemporal choice tasks should be underestimated, and that average reward manipulations should affect the speed of the clock-thus capturing empirical findings that are unexplained by either theory alone. Our results suggest that a common computational repertoire may underlie the seemingly heterogeneous roles of DA.

Learning poly-synaptic paths with traveling waves

Traveling waves are commonly observed across the brain. While previous studies have suggested the role of traveling waves in learning, the mechanism remains unclear. We adopted a computational approach to investigate the effect of traveling waves on synaptic plasticity. Our results indicate that traveling waves facilitate the learning of poly-synaptic network paths when combined with a reward-dependent local synaptic plasticity rule. We also demonstrate that traveling waves expedite finding the shortest paths and learning nonlinear input/output mapping, such as exclusive or (XOR) function.

There are approximately 10¹¹ neurons with 10¹⁴ connections in the human brain. Information transmission among neurons in this large network is considered crucial for our behavior. To achieve this, multiple synaptic connections along a poly-synaptic network path must be adjusted coherently during learning. Because the previously proposed reward-dependent synaptic plasticity rule requires coactivation of presynaptic and postsynaptic neurons, learning can fail if a subset of neurons along a distant network path is inactive at the beginning of learning. We suggest that traveling waves that are initiated at an information source can mitigate this problem. We performed computer simulations of spiking neural networks with reward-dependent local synaptic plasticity rules and traveling waves. Our results show that this combination facilitates the learning and refinement of synaptic network paths. We argue that these features are a general biological strategy for maintaining and optimizing our brain function. Our research provides new insights into how complex neural networks in the brain form during learning and memory consolidation.

Attention-deficit/hyperactivity disorder and the explore/exploit trade-off

The ability to maximize rewards and minimize the costs of obtaining them is vital to making advantageous explore/exploit decisions. Exploratory decisions are theorized to be greater among individuals with attention-deficit/hyperactivity disorder (ADHD), potentially due to deficient catecholamine transmission. Here, we examined the effects of ADHD status and methylphenidate, a common ADHD medication, on explore/exploit decisions using a 6-armed bandit task. We hypothesized that ADHD participants would make more exploratory decisions than controls, and that MPH would reduce group differences. On separate study days, adults with (n = 26) and without (n = 23) ADHD completed the bandit task at baseline, and after methylphenidate or placebo in counter-balanced order. Explore/exploit decisions were modeled using reinforcement learning algorithms. ADHD participants made more exploratory decisions (i.e., chose options without the highest expected reward value) and earned fewer points than controls in all three study days, and methylphenidate did not affect these outcomes. Baseline exploratory choices were positively associated with hyperactive ADHD symptoms across all participants. These results support several theoretical models of increased exploratory choices in ADHD and suggest the unexplained variance in ADHD decisions may be due to less value tracking. The inability to suppress actions with little to no reward value may be a key feature of hyperactive ADHD symptoms.

Lapses in perceptual decisions reflect exploration

Perceptual decision-makers often display a constant rate of errors independent of evidence strength. These ‘lapses’ are treated as a nuisance arising from noise tangential to the decision, e.g. inattention or motor errors. Here, we use a multisensory decision task in rats to demonstrate that these explanations cannot account for lapses’ stimulus dependence. We propose a novel explanation: lapses reflect a strategic trade-off between exploiting known rewarding actions and exploring uncertain ones. We tested this model’s predictions by selectively manipulating one action’s reward magnitude or probability. As uniquely predicted by this model, changes were restricted to lapses associated with that action. Finally, we show that lapses are a powerful tool for assigning decision-related computations to neural structures based on disruption experiments (here, posterior striatum and secondary motor cortex). These results suggest that lapses reflect an integral component of decision-making and are informative about action values in normal and disrupted brain states.

An open-source device for measuring food intake and operant behavior in rodent home-cages

Feeding is critical for survival, and disruption in the mechanisms that govern food intake underlies disorders such as obesity and anorexia nervosa. It is important to understand both food intake and food motivation to reveal mechanisms underlying feeding disorders. Operant behavioral testing can be used to measure the motivational component to feeding, but most food intake monitoring systems do not measure operant behavior. Here, we present a new solution for monitoring both food intake and motivation in rodent home-cages: the Feeding Experimentation Device version 3 (FED3). FED3 measures food intake and operant behavior in rodent home-cages, enabling longitudinal studies of feeding behavior with minimal experimenter intervention. It has a programmable output for synchronizing behavior with optogenetic stimulation or neural recordings. Finally, FED3 design files are open-source and freely available, allowing researchers to modify FED3 to suit their needs.

Distinct roles for dopamine clearance mechanisms in regulating behavioral flexibility

Dopamine plays a crucial role in adaptive behavior, and dysfunctional dopamine is implicated in multiple psychiatric conditions characterized by inflexible or inconsistent choices. However, the precise relationship between dopamine and flexible decision making remains unclear. One reason is that, while many studies have focused on the activity of dopamine neurons, efficient dopamine signaling also relies on clearance mechanisms, notably the dopamine transporter (DAT), which predominates in striatum, and catechol-O-methyltransferase (COMT), which predominates in cortex. The exact locus, extent, and timescale of the effects of DAT and COMT are uncertain. Moreover, there is limited data on how acute disruption of either mechanism affects flexible decision making strategies mediated by cortico-striatal networks. To address these issues, we combined pharmacological modulation of DAT and COMT with electrochemistry and behavior in mice. DAT blockade, but not COMT inhibition, regulated sub-second dopamine release in the nucleus accumbens core, but surprisingly neither clearance mechanism affected evoked release in prelimbic cortex. This was not due to a lack of sensitivity, as both amphetamine and atomoxetine changed the kinetics of sub-second release. In a multi-step decision making task where mice had to respond to reversals in either reward probabilities or the choice sequence to reach the goal, DAT blockade selectively impaired, and COMT inhibition improved, performance after reward reversals, but neither manipulation affected the adaptation of choices after action-state transition reversals. Together, our data suggest that DAT and COMT shape specific aspects of behavioral flexibility by regulating different aspects of the kinetics of striatal and cortical dopamine, respectively.

Perseverative stereotypic behavior of Epac2 KO mice in a reward-based decision making task

Successfully navigating dynamic environments requires balancing the decision to stay at an optimal choice with that to switch to an alternative to acquire new knowledge. However, the genetic factors and cellular activity shaping this "stay or switch" action decision remains largely unidentified. Here we find that mice carrying a deletion of the exchange protein directly activated by cAMP 2 (Epac2) gene, a putative autism locus, exhibit perseverative "stay" behavior in a dynamic foraging task. Anatomical analysis found that the loss of Epac2 resulted in a significant decrease in the density of PV-expressing interneurons in the ventrolateral orbitofrontal cortex (OFC) and dorsal striatum (dSTR). Further, in vitro whole cell patch clamp recordings of PV⁺ GABAergic interneurons in the dSTR revealed altered neural activity in Epac2 KO mice in response to dopamine. Our findings highlight a potential role of Epac2 in structural changes and neural responses of PV-expressing GABAergic interneurons in the ventrolateral OFC and dSTR during value-based reinforcement learning and link Epac2 function to abnormal decision-making processes and perseverative behaviors seen in autism.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

Impaired context-sensitive adjustment of behaviour in Parkinson's disease patients tested on and off medication: An fMRI study

The brain's sensitivity to and accentuation of unpredicted over predicted sensory signals plays a fundamental role in learning. According to recent theoretical models of the predictive coding framework, dopamine is responsible for balancing the interplay between bottom-up input and top-down predictions by controlling the precision of surprise signals that guide learning. Using functional MRI, we investigated whether patients with Parkinson's disease (PD) show impaired learning from prediction errors requiring either adaptation or stabilisation of current predictions. Moreover, we were interested in whether deficits in learning over a specific time scale would be accompanied by altered surprise responses in dopamine-related brain structures. To this end, twenty-one PD patients tested on and off dopaminergic medication and twenty-one healthy controls performed a digit prediction paradigm. During the task, violations of sequence-based predictions either signalled the need to update or to stabilise the current prediction and, thus, to react to them or ignore them, respectively. To investigate contextual adaptation to prediction errors, the probability (or its inverse, surprise) of the violations fluctuated across the experiment. When the probability of prediction errors over a specific time scale increased, healthy controls but not PD patients off medication became more flexible, i.e., error rates at violations requiring a motor response decreased in controls but increased in patients. On the neural level, this learning deficit in patients was accompanied by reduced signalling in the substantia nigra and the caudate nucleus. In contrast, differences between the groups regarding the probabilistic modulation of behaviour and neural responses were much less pronounced at prediction errors requiring only stabilisation but no adaptation. Interestingly, dopaminergic medication could neither improve learning from prediction errors nor restore the physiological, neurotypical pattern. Our findings point to a pivotal role of dysfunctions of the substantia nigra and caudate nucleus in deficits in learning from flexibility-demanding prediction errors in PD. Moreover, the data witness poor effects of dopaminergic medication on learning in PD.

Dopaminergic signaling of uncertainty and the aetiology of gambling addiction

Although there is increasing clinical recognition of behavioral addictions, of which gambling disorder is the prototype example, there is a limited understanding of the psychological properties of (non-substance-related) behaviors that enable them to become 'addictive' in a way that is comparable to drugs of abuse. According to an influential application of reinforcement learning to substance addictions, the direct effects of drugs to release dopamine can create a perpetual escalation of incentive salience. This article focusses on reward uncertainty, which is proposed to be the core feature of gambling that creates the capacity for addiction. We describe the neuro-dynamics of the dopamine response to uncertainty that may allow a similar escalation of incentive salience, and its relevance to behavioral addictions. We review translational evidence from both preclinical animal models and human clinical research, including studies in people with gambling disorder. Further, we describe the evidence for 1) the effects of the omission of expected reward as a stressor and to promote sensitization, 2) the effect of the resolution of reward uncertainty as a source of value, 3) structural characteristics of modern Electronic Gaming Machines (EGMs) in leveraging these mechanisms, 4) analogies to the aberrant salience hypothesis of psychosis for creating and maintaining gambling-related cognitive distortions. This neurobiologically-inspired model has implications for harm profiling of other putative behavioral addictions, as well as offering avenues for enhancing neurological, pharmacological and psychological treatments for gambling disorder, and harm reduction strategies for EGM design.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

Signals of anticipation of reward and of mean reward rates in the human brain

Theoretical models of dopamine function stemming from reinforcement learning theory have emphasized the importance of prediction errors, which signal changes in the expectation of impending rewards. Much less is known about the effects of mean reward rates, which may be of motivational significance due to their role in computing the optimal effort put into exploiting reward opportunities. Here, we used a reinforcement learning model to design three functional neuroimaging studies and disentangle the effects of changes in reward expectations and mean reward rates, showing recruitment of specific regions in the brainstem regardless of prediction errors. While changes in reward expectations activated ventral striatal areas as in previous studies, mean reward rates preferentially modulated the substantia nigra/ventral tegmental area, deep layers of the superior colliculi, and a posterior pontomesencephalic region. These brainstem structures may work together to set motivation and attentional efforts levels according to perceived reward opportunities.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

Dopamine restores cognitive motivation in Parkinson's disease

Disorders of motivation, such as apathy, are common in Parkinson's disease, and a key feature of such disorders is a greater aversion to effort. In humans, the experience of cognitive effort is ubiquitous, and cognitive apathy has traditionally been considered distinct and separable from other subtypes. Surprisingly, however, the neurobiology of cognitive motivation is poorly understood. In particular, although dopamine has a well-characterized role in incentivizing physically effortful behaviour, a critical, unresolved issue is whether its facilitatory role generalizes to other domains. Here, we asked how dopamine modulates the willingness of patients with Parkinson's disease to invest cognitive effort in return for reward. We tested 20 patients with idiopathic Parkinson's disease across two counterbalanced sessions-ON and OFF their usual dopaminergic medication-and compared their performance to 20 healthy age-matched controls. We applied a novel task in which we manipulated cognitive effort as the number of rapid serial visual presentation streams to which participants had to attend. After training participants to ceiling performance, we then asked them to choose between a low-effort/low-reward baseline option, and a higher-effort/higher-reward offer. Computational models of choice behaviour revealed four key results. First, patients OFF medication were significantly less cognitively motivated than controls, as manifest by steeper cognitive effort discounting functions in the former group. Second, dopaminergic therapy improved this deficit, such that choices in patients ON medication were indistinguishable from controls. Third, differences in motivation were also accompanied by independent changes in the stochasticity of individuals' decisions, such that dopamine reduced the variability in choice behaviour. Finally, choices on our task correlated uniquely with the subscale of the Dimensional Apathy Scale that specifically indexes cognitive motivation, which suggests a close relationship between our laboratory measure of cognitive effort discounting and subjective reports of day-to-day cognitive apathy. Importantly, participants' choices were not confounded by temporal discounting, probability discounting, physical demand, or varying task performance. These results are the first to reveal the central role of dopamine in overcoming cognitive effort costs. They provide an insight into the computational mechanisms underlying cognitive apathy in Parkinson's disease, and demonstrate its amenability to dopaminergic therapy. More broadly, they offer important empirical support for prominent frameworks proposing a domain-general role for dopamine in value-based decision-making, and provide a critical link between dopamine and multidimensional theories of apathy.

Spontaneous eye blink rate predicts individual differences in exploration and exploitation during reinforcement learning

Spontaneous eye blink rate (sEBR) has been linked to striatal dopamine function and to how individuals make value-based choices after a period of reinforcement learning (RL). While sEBR is thought to reflect how individuals learn from the negative outcomes of their choices, this idea has not been tested explicitly. This study assessed how individual differences in sEBR relate to learning by focusing on the cognitive processes that drive RL. Using Bayesian latent mixture modelling to quantify the mapping between RL behaviour and its underlying cognitive processes, we were able to differentiate low and high sEBR individuals at the level of these cognitive processes. Further inspection of these cognitive processes indicated that sEBR uniquely indexed explore-exploit tendencies during RL: lower sEBR predicted exploitative choices for high valued options, whereas higher sEBR predicted exploration of lower value options. This relationship was additionally supported by a network analysis where, notably, no link was observed between sEBR and how individuals learned from negative outcomes. Our findings challenge the notion that sEBR predicts learning from negative outcomes during RL, and suggest that sEBR predicts individual explore-exploit tendencies. These then influence value sensitivity during choices to support successful performance when facing uncertain reward.

Dopamine-Dependent Loss Aversion during Effort-Based Decision-Making

From psychology to economics, there has been substantial interest in how costs (e.g., delay, risk) are represented asymmetrically during decision-making when attempting to gain reward or avoid punishment. For example, in decision-making under risk, individuals show a tendency to prefer to avoid punishment rather than to acquire the equivalent reward (loss aversion). Although the cost of physical effort has recently received significant attention, it remains unclear whether loss aversion exists during effort-based decision-making. On the one hand, loss aversion may be hardwired due to asymmetric evolutionary pressure on losses and gains and therefore exists across decision-making contexts. On the other hand, distinct brain regions are involved with different decision costs, making it questionable whether similar asymmetries exist. Here, we demonstrate that young healthy human participants (females, 16; males, 6) exhibit loss aversion during effort-based decision-making by exerting more physical effort to avoid punishment than to gain a same-size reward. Next, we show that medicated Parkinson's disease (PD) patients (females, 9; males, 9) show a reduction in loss aversion compared with age-matched control subjects (females, 11; males, 9). Behavioral and computational analysis revealed that people with PD exerted similar physical effort in return for a reward but were less willing to produce effort to avoid punishment. Therefore, loss aversion is present during effort-based decision-making and can be modulated by altered dopaminergic state. This finding could have important implications for our understanding of clinical disorders that show a reduced willingness to exert effort in the pursuit of reward.SIGNIFICANCE STATEMENT Loss aversion-preferring to avoid punishment rather than to acquire equivalent reward-is an important concept in decision-making under risk. However, little is known about whether loss aversion also exists during decisions where the cost is physical effort. This is surprising given that motor cost shapes human behavior, and a reduced willingness to exert effort is a characteristic of many clinical disorders. Here, we show that healthy human individuals exert more effort to minimize punishment than to maximize reward (loss aversion). We also demonstrate that medicated Parkinson's disease patients exert similar effort to gain reward but less effort to avoid punishment when compared with healthy age-matched control subjects. This indicates that dopamine-dependent loss aversion is crucial for explaining effort-based decision-making.

Attentional blink and putative noninvasive dopamine markers: Two experiments to consolidate possible associations

Adaptive behavioral control involves a balance between top-down persistence and flexible updating of goals under changing demands. According to the metacontrol state model (MSM), this balance emerges from the interaction between the frontal and the striatal dopaminergic system. The attentional blink (AB) task has been argued to tap into the interaction between persistence and flexibility, as it reflects overpersistence—the too-exclusive allocation of attentional resources to the processing of the first of two consecutive targets. Notably, previous studies are inconclusive about the association between the AB and noninvasive proxies of dopamine including the spontaneous eye blink rate (sEBR), which allegedly assesses striatal dopamine levels. We aimed to substantiate and extend previous attempts to predict individual sizes of the AB in two separate experiments with larger sample sizes (N = 71 & N = 65) by means of noninvasive behavioral and physiological proxies of dopamine (DA), such as sEBR and mood measures, which are likely to reflect striatal dopamine levels, and color discrimination, which has been argued to tap into the frontal dopamine levels. Our findings did not confirm the prediction that AB size covaries with sEBR, mood, or color discrimination. The implications of this inconsistency with previous observations are discussed.

Distinct roles of striatal direct and indirect pathways in value-based decision making

The striatum is critically involved in value-based decision making. However, it is unclear how striatal direct and indirect pathways work together to make optimal choices in a dynamic and uncertain environment. Here, we examined the effects of selectively inactivating D1 receptor (D1R)- or D2 receptor (D2R)-expressing dorsal striatal neurons (corresponding to direct- and indirect-pathway neurons, respectively) on mouse choice behavior in a reversal task with progressively increasing reversal frequency and a dynamic two-armed bandit task. Inactivation of either D1R- or D2R-expressing striatal neurons impaired performance in both tasks, but the pattern of altered choice behavior differed between the two animal groups. A reinforcement learning model-based analysis indicated that inactivation of D1R- and D2R-expressing striatal neurons selectively impairs value-dependent action selection and value learning, respectively. Our results suggest differential contributions of striatal direct and indirect pathways to two distinct steps in value-based decision making.

Pattern of dopamine signaling during aversive events predicts active avoidance learning

Learning to avoid aversive outcomes is an adaptive strategy to limit one's future exposure to stressful events. However, there is considerable variance in active avoidance learning across a population. The mesolimbic dopamine system contributes to behaviors elicited by aversive stimuli, although it is unclear if the heterogeneity in active avoidance learning is explained by differences in dopamine transmission. Furthermore, it is not known how dopamine signals evolve throughout active avoidance learning. To address these questions, we performed voltammetry recordings of dopamine release in the ventral medial striatum throughout training on inescapable footshock and signaled active avoidance tasks. This approach revealed differences in the pattern of dopamine signaling during the aversive cue and the safety period that corresponded to subsequent task performance. Dopamine transmission throughout the footshock bout did not predict performance but rather was modulated by the prior stress exposure. Additionally, we demonstrate that dopamine encodes a safety prediction error signal, which illustrates that ventral medial striatal dopamine release conveys a learning signal during both appetitive and aversive conditions.

Ready, set, explore! Event-related potentials reveal the time-course of exploratory decisions

The decision trade-off between exploiting the known and exploring the unknown has been studied using a variety of approaches and techniques. Surprisingly, electroencephalography (EEG) has been underused in this area of study, even though its high temporal resolution has the potential to reveal the time-course of exploratory decisions. We addressed this issue by recording EEG data while participants tried to win as many points as possible in a two-choice gambling task called a two-armed bandit. After using a computational model to classify responses as either exploitations or explorations, we examined event-related potentials locked to two events preceding decisions to exploit/explore: the arrival of feedback, and the subsequent appearance of the next trial's choice stimuli. In particular, we examined the feedback-locked P300 component, thought to index a phasic release of norepinephrine (a neural interrupt signal), and the reward positivity, thought to index a phasic release of dopamine (a neural prediction error signal). We observed an exploration-dependent enhancement of the P300 only, suggesting a critical role of norepinephrine (but not dopamine) in triggering decisions to explore. Similarly, we examined the N200/P300 components evoked by the appearance of the choice stimuli. In this case, exploration was characterized by an enhancement of the N200, but not P300, a result we attribute to increased response conflict. These results demonstrate the usefulness of combining computational and EEG methodologies, and suggest that exploratory decisions are preceded by two characterizing events: a feedback-locked neural interrupt signal (enhanced P300), and a choice-locked increase in response conflict (enhanced N200).

Dopamine blockade impairs the exploration-exploitation trade-off in rats

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.