Networks of VTA Neurons Encode Real-Time Information about Uncertain Numbers of Actions Executed to Earn a Reward

Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making

Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.

Dopamine neurons evaluate natural fluctuations in performance quality

Many motor skills are learned by comparing ongoing behavior to internal performance benchmarks. Dopamine neurons encode performance error in behavioral paradigms where error is externally induced, but it remains unknown whether dopamine also signals the quality of natural performance fluctuations. Here, we record dopamine neurons in singing birds and examine how spontaneous dopamine spiking activity correlates with natural fluctuations in ongoing song. Antidromically identified basal ganglia-projecting dopamine neurons correlate with recent, and not future, song variations, consistent with a role in evaluation, not production. Furthermore, maximal dopamine spiking occurs at a single vocal target, consistent with either actively maintaining the existing song or shifting the song to a nearby form. These data show that spontaneous dopamine spiking can evaluate natural behavioral fluctuations unperturbed by experimental events such as cues or rewards.

Learning and producing skilled behavior requires an internal measure of performance. Duffy et al. examine dopamine neurons’ relationship to natural song in singing birds. Spontaneous dopamine activity correlates with song fluctuations in a manner consistent with evaluation of natural behavioral variations, independent of external perturbations, cues, or rewards.

A recurrent neural network framework for flexible and adaptive decision making based on sequence learning

The brain makes flexible and adaptive responses in a complicated and ever-changing environment for an organism's survival. To achieve this, the brain needs to understand the contingencies between its sensory inputs, actions, and rewards. This is analogous to the statistical inference that has been extensively studied in the natural language processing field, where recent developments of recurrent neural networks have found many successes. We wonder whether these neural networks, the gated recurrent unit (GRU) networks in particular, reflect how the brain solves the contingency problem. Therefore, we build a GRU network framework inspired by the statistical learning approach of NLP and test it with four exemplar behavior tasks previously used in empirical studies. The network models are trained to predict future events based on past events, both comprising sensory, action, and reward events. We show the networks can successfully reproduce animal and human behavior. The networks generalize the training, perform Bayesian inference in novel conditions, and adapt their choices when event contingencies vary. Importantly, units in the network encode task variables and exhibit activity patterns that match previous neurophysiology findings. Our results suggest that the neural network approach based on statistical sequence learning may reflect the brain's computational principle underlying flexible and adaptive behaviors and serve as a useful approach to understand the brain.

Dopamine Modulation of Prefrontal Cortex Activity Is Manifold and Operates at Multiple Temporal and Spatial Scales

Although the function of dopamine in subcortical structures is largely limited to reward and movement, dopamine neurotransmission in the prefrontal cortex (PFC) is critical to a multitude of temporally and functionally diverse processes, such as attention, working memory, behavioral flexibility, action planning, and sustained motivational and affective states. How does dopamine influence computation of these temporally complex functions? We find causative links between sustained and burst patterns of phasic dopamine neuron activation and modulation of medial PFC neuronal activity at multiple spatiotemporal scales. These include a multidirectional and weak impact on individual neuron rate activity but a robust influence on coordinated ensemble activity, gamma oscillations, and gamma-theta coupling that persisted for minutes. In addition, PFC network responses to burst pattern of dopamine firing were selectively strengthened in behaviorally active states. This multiplex mode of modulation by dopamine input may enable PFC to compute and generate spatiotemporally diverse and specialized outputs.

Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity

Animals infer when and where a reward is available from experience with informative sensory stimuli and their own actions. In vertebrates, this is thought to depend upon the release of dopamine from midbrain dopaminergic neurons. Studies of the role of dopamine have focused almost exclusively on their encoding of informative sensory stimuli; however, many dopaminergic neurons are active just prior to movement initiation, even in the absence of sensory stimuli. How should current frameworks for understanding the role of dopamine incorporate these observations? To address this question, we review recent anatomical and functional evidence for action-related dopamine signaling. We conclude by proposing a framework in which dopaminergic neurons encode subjective signals of action initiation to solve an internal credit assignment problem.

Resonance in the Mouse Ventral Tegmental Area Dopaminergic Network Induced by Regular and Poisson Distributed Optogenetic Stimulation in-vitro

Neurons in many brain regions exhibit spontaneous, intrinsic rhythmic firing activity. This rhythmic firing activity may determine the way in which these neurons respond to extrinsic synaptic inputs. We hypothesized that neurons should be most responsive to inputs at the frequency of the intrinsic oscillation frequency. We addressed this question in the ventral tegmental area (VTA), a dopaminergic nucleus in the midbrain. VTA neurons have a unique propensity to exhibit spontaneous intrinsic rhythmic activity in the 1-5 Hz frequency range, which persists in the in-vitro brain slice, and form a network of weakly coupled oscillators. Here, we combine in-vitro simultaneous recording of action potentials from a 60 channel multi-electrode-array with cell-type-specific optogenetic stimulation of the VTA dopamine neurons. We investigated how VTA neurons respond to wide-band stochastic (Poisson input) as well as regular laser pulses. Strong synchrony was induced between the laser input and the spike timing of the neurons, both for regular pulse trains and Poisson pulse trains. For rhythmically pulsed input, the neurons demonstrated resonant behavior with the strongest phase locking at their intrinsic oscillation frequency, but also at half and double the intrinsic oscillation frequency. Stochastic Poisson pulse stimulation provided a more effective stimulation of the entire population, yet we observed resonance at lower frequencies (approximately half the oscillation frequency) than the neurons' intrinsic oscillation frequency. The non-linear filter characteristics of dopamine neurons could allow the VTA to predict precisely timed future rewards based on past sensory inputs, a crucial component of reward prediction error signaling. In addition, these filter characteristics could contribute to a pacemaker role for the VTA in synchronizing activity with other regions like the prefrontal cortex and the hippocampus.

Neurobiological links between stress and anxiety

Stress and anxiety have intertwined behavioral and neural underpinnings. These commonalities are critical for understanding each state, as well as their mutual interactions. Grasping the mechanisms underlying this bidirectional relationship will have major clinical implications for managing a wide range of psychopathologies. After briefly defining key concepts for the study of stress and anxiety in pre-clinical models, we present circuit, as well as cellular and molecular mechanisms involved in either or both stress and anxiety. First, we review studies on divergent circuits of the basolateral amygdala (BLA) underlying emotional valence processing and anxiety-like behaviors, and how norepinephrine inputs from the locus coeruleus (LC) to the BLA are responsible for acute-stress induced anxiety. We then describe recent studies revealing a new role for mitochondrial function within the nucleus accumbens (NAc), defining individual trait anxiety in rodents, and participating in the link between stress and anxiety. Next, we report findings on the impact of anxiety on reward encoding through alteration of circuit dynamic synchronicity. Finally, we present work unravelling a new role for hypothalamic corticotropin-releasing hormone (CRH) neurons in controlling anxiety-like and stress-induce behaviors. Altogether, the research reviewed here reveals circuits sharing subcortical nodes and underlying the processing of both stress and anxiety. Understanding the neural overlap between these two psychobiological states, might provide alternative strategies to manage disorders such as post-traumatic stress disorder (PTSD).

Modulation of Functional Connectivity Between Dopamine Neurons of the Rat Ventral Tegmental Area in vitro

Micro Electrode Arrays were used to simultaneously record spontaneous extracellular action potentials from 10 to 30 dopamine neurons in acute brain slices from the lateral Ventral Tegmental Area (VTA) of the rat. The spike train of an individual neuron was used to characterize the firing pattern: firing rate, firing irregularity and oscillation frequency. Functional connectivity between a pair of neurons was quantified by the Paired Phase Consistency (PPC), taking the oscillation frequency as reference. Under baseline conditions the PPC was significantly different from zero and 42 of the 386 pairs of VTA neurons showed significant coupling. Fifty percent of the recorded dopamine neurons were part of the coupled VTA network. Raising extracellular potassium from 3.5 to 5 mM increased the mean firing rate of the dopamine neurons by 45%. The same increase could be induced by bath application of 300 μm glutamate. High potassium reduced the PPC, but it did not change during the glutamate application. Our findings imply that manipulating excitability has distinct and specific consequences for functional connectivity in the VTA network that cannot be directly predicted from the changes in neuronal firing rates. Functional connectivity reflects the spatial organization and synchronization of the VTA output and thus represents a unique element of the message that is sent to the mesolimbic projection area. It adds a dimension to pharmacological manipulation of the VTA micro circuit that might help to understand the pharmacological (side) effects of e.g., anti-psychotic drugs.

The Psychopharmacology of Effort-Related Decision Making: Dopamine, Adenosine, and Insights into the Neurochemistry of Motivation

Effort-based decision making is studied using tasks that offer choices between high-effort options leading to more highly valued reinforcers versus low-effort/low-reward options. These tasks have been used to study the involvement of neural systems, including mesolimbic dopamine and related circuits, in effort-related aspects of motivation. Moreover, such tasks are useful as animal models of some of the motivational symptoms that are seen in people with depression, schizophrenia, Parkinson's disease, and other disorders. The present review will discuss the pharmacology of effort-related decision making and will focus on the use of these tasks for the development of drug treatments for motivational dysfunction. Research has identified pharmacological conditions that can alter effort-based choice and serve as models for depression-related symptoms (e.g., the vesicular monoamine transport-2 inhibitor tetrabenazine and proinflammatory cytokines). Furthermore, tests of effort-based choice have identified compounds that are particularly useful for stimulating high-effort work output and reversing the deficits induced by tetrabenazine and cytokines. These studies indicate that drugs that act by facilitating dopamine transmission, as well as adenosine A_2A antagonists, are relatively effective at reversing effort-related impairments. Studies of effort-based choice may lead to the identification of drug targets that could be useful for treating motivational treatments that are resistant to commonly used antidepressants such as serotonin transport inhibitors.

The timing of action determines reward prediction signals in identified midbrain dopamine neurons

Animals adapt behavior in response to informative sensory cues using multiple brain circuits. The activity of midbrain dopamine (mDA) neurons is thought to convey a critical teaching signal: reward prediction error (RPE). Although RPE signals are thought to be essential to learning, little is known about the dynamic changes in mDA neuron activity as animals learn about novel sensory cues and appetitive rewards. Here we describe a large dataset of cell-attached recordings of identified dopaminergic neurons as naïve mice learned a novel cue-reward association. During learning mDA neuron activity results from summation of sensory cue-related and movement initiation-related response components. These components are both a function of reward expectation yet dissociable. Learning produces an increasingly precise coordination of action initiation following sensory cues that results in apparent RPE correlates. Our data thus provide new insights into circuit mechanisms underlying a critical computation in a highly conserved learning circuit.

Firing of Putative Dopamine Neurons in Ventral Tegmental Area Is Modulated by Probability of Success during Performance of a Stop-Change Task

Response inhibition, the ability to refrain from unwanted actions, is an essential component of complex behavior and is often impaired across numerous neuropsychiatric disorders such as addiction, attention-deficit hyperactivity disorder (ADHD), schizophrenia, and obsessive-compulsive disorder. Accordingly, much research has been devoted to characterizing brain regions responsible for the regulation of response inhibition. The stop-signal task, a task in which animals are required to inhibit a prepotent response in the presence of a STOP cue, is one of the most well-studied tasks of response inhibition. While pharmacological evidence suggests that dopamine (DA) contributes to the regulation of response inhibition, what is exactly encoded by DA neurons during performance of response inhibition tasks is unknown. To address this issue, we recorded from single units in the ventral tegmental area (VTA), while rats performed a stop-change task. We found that putative DA neurons fired less and higher to cues and reward on STOP trials relative to GO trials, respectively, and that firing was reduced during errors. These results suggest that DA neurons in VTA encode the uncertainty associated with the probability of obtaining reward on difficult trials instead of the saliency associated with STOP cues or the need to resolve conflict between competing responses during response inhibition.

Dopamine, Effort-Based Choice, and Behavioral Economics: Basic and Translational Research

Operant behavior is not only regulated by factors related to the quality or quantity of reinforcement, but also by the work requirements inherent in performing instrumental actions. Moreover, organisms often make effort-related decisions involving economic choices such as cost/benefit analyses. Effort-based decision making is studied using behavioral procedures that offer choices between high-effort options leading to relatively preferred reinforcers vs. low effort/low reward choices. Several neural systems, including the mesolimbic dopamine (DA) system and other brain circuits, are involved in regulating effort-related aspects of motivation. Considerable evidence indicates that mesolimbic DA transmission exerts a bi-directional control over exertion of effort on instrumental behavior tasks. Interference with DA transmission produces a low-effort bias in animals tested on effort-based choice tasks, while increasing DA transmission with drugs such as DA transport blockers tends to enhance selection of high-effort options. The results from these pharmacology studies are corroborated by the findings from recent articles using optogenetic, chemogenetic and physiological techniques. In addition to providing important information about the neural regulation of motivated behavior, effort-based choice tasks are useful for developing animal models of some of the motivational symptoms that are seen in people with various psychiatric and neurological disorders (e.g., depression, schizophrenia, Parkinson’s disease). Studies of effort-based decision making may ultimately contribute to the development of novel drug treatments for motivational dysfunction.

Risk of punishment influences discrete and coordinated encoding of reward-guided actions by prefrontal cortex and VTA neurons

Actions motivated by rewards are often associated with risk of punishment. Little is known about the neural representation of punishment risk during reward-seeking behavior. We modeled this circumstance in rats by designing a task where actions were consistently rewarded but probabilistically punished. Spike activity and local field potentials were recorded during task performance simultaneously from VTA and mPFC, two reciprocally connected regions implicated in reward-seeking and aversive behaviors. At the single unit level, we found that ensembles of putative dopamine and non-dopamine VTA neurons and mPFC neurons encode the relationship between action and punishment. At the network level, we found that coherent theta oscillations synchronize VTA and mPFC in a bottom-up direction, effectively phase-modulating the neuronal spike activity in the two regions during punishment-free actions. This synchrony declined as a function of punishment probability, suggesting that during reward-seeking actions, risk of punishment diminishes VTA-driven neural synchrony between the two regions.