A Neural Circuit Mechanism for the Involvements of Dopamine in Effort-Related Choices: Decay of Learned Values, Secondary Effects of Depletion, and Calculation of Temporal Difference Error

Valence-dependent dopaminergic modulation during reversal learning in Parkinson's disease: A neurocomputational approach

Reinforcement learning, crucial for behavior in dynamic environments, is driven by rewards and punishments, modulated by dopamine (DA) changes. This study explores the dopaminergic system's influence on learning, particularly in Parkinson's disease (PD), where medication leads to impaired adaptability. Highlighting the role of tonic DA in signaling the valence of actions, this research investigates how DA affects response vigor and decision-making in PD. DA not only influences reward and punishment learning but also indicates the cognitive effort level and risk propensity in actions, which are essential for understanding and managing PD symptoms. In this work, we adapt our existing neurocomputational model of basal ganglia (BG) to simulate two reversal learning tasks proposed by Cools et al. We first optimized a Hebb rule for both probabilistic and deterministic reversal learning, conducted a sensitivity analysis (SA) on parameters related to DA effect, and compared performances between three groups: PD-ON, PD-OFF, and control subjects. In our deterministic task simulation, we explored switch error rates after unexpected task switches and found a U-shaped relationship between tonic DA levels and switch error frequency. Through SA, we classify these three groups. Then, assuming that the valence of the stimulus affects the tonic levels of DA, we were able to reproduce the results by Cools et al. As for the probabilistic task simulation, our results are in line with clinical data, showing similar trends with PD-ON, characterized by higher tonic DA levels that are correlated with increased difficulty in both acquisition and reversal tasks. Our study proposes a new hypothesis: valence, signaled by tonic DA levels, influences learning in PD, confirming the uncorrelation between phasic and tonic DA changes. This hypothesis challenges existing paradigms and opens new avenues for understanding cognitive processes in PD, particularly in reversal learning tasks.

Mesolimbic dopamine ramps reflect environmental timescales

Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.

Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Glutamate inputs send prediction error of reward but not negative value of aversive stimuli to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs) but the mechanisms underlying RPE computation, particularly contributions of different neurotransmitters, remain poorly understood. Here we used a genetically-encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons. We found that glutamate inputs exhibit virtually all of the characteristics of RPE, rather than conveying a specific component of RPE computation such as reward or expectation. Notably, while glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli toward more positive responses, while excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Phasic Dopamine Changes and Hebbian Mechanisms during Probabilistic Reversal Learning in Striatal Circuits: A Computational Study

Cognitive flexibility is essential to modify our behavior in a non-stationary environment and is often explored by reversal learning tasks. The basal ganglia (BG) dopaminergic system, under a top-down control of the pre-frontal cortex, is known to be involved in flexible action selection through reinforcement learning. However, how adaptive dopamine changes regulate this process and learning mechanisms for training the striatal synapses remain open questions. The current study uses a neurocomputational model of the BG, based on dopamine-dependent direct (Go) and indirect (NoGo) pathways, to investigate reinforcement learning in a probabilistic environment through a task that associates different stimuli to different actions. Here, we investigated: the efficacy of several versions of the Hebb rule, based on covariance between pre- and post-synaptic neurons, as well as the required control in phasic dopamine changes crucial to achieving a proper reversal learning. Furthermore, an original mechanism for modulating the phasic dopamine changes is proposed, assuming that the expected reward probability is coded by the activity of the winner Go neuron before a reward/punishment takes place. Simulations show that this original formulation for an automatic phasic dopamine control allows the achievement of a good flexible reversal even in difficult conditions. The current outcomes may contribute to understanding the mechanisms for active control of dopamine changes during flexible behavior. In perspective, it may be applied in neuropsychiatric or neurological disorders, such as Parkinson’s or schizophrenia, in which reinforcement learning is impaired.

Diminished positive affect and traumatic stress: A biobehavioral review and commentary on trauma affective neuroscience

Post-traumatic stress manifests in disturbed affect and emotion, including exaggerated severity and frequency of negative valence emotions, e.g., fear, anxiety, anger, shame, and guilt. However, another core feature of common post-trauma psychopathologies, i.e. post-traumatic stress disorder (PTSD) and major depression, is diminished positive affect, or reduced frequency and intensity of positive emotions and affective states such as happiness, joy, love, interest, and desire/capacity for interpersonal affiliation. There remains a stark imbalance in the degree to which the neuroscience of each affective domain has been probed and characterized in PTSD, with our knowledge of post-trauma diminished positive affect remaining comparatively underdeveloped. This remains a prominent barrier to realizing the clinical breakthroughs likely to be afforded by the increasing availability of neuroscience assessment and intervention tools. In this review and commentary, the author summarizes the modest extant neuroimaging literature that has probed diminished positive affect in PTSD using reward processing behavioral paradigms, first briefly reviewing and outlining the neurocircuitry implicated in reward and positive emotion and its interrelationship with negative emotion and negative valence circuitry. Specific research guidelines are then offered to best and most efficiently develop the knowledge base in this area in a way that is clinically translatable and will exert a positive impact on routine clinical care. The author concludes with the prediction that the development of an integrated, bivalent theoretical and predictive model of how trauma impacts affective neurocircuitry to promote post-trauma psychopathology will ultimately lead to breakthroughs in how trauma treatments are conceptualized mechanistically and developed pragmatically.