Social modulation of learned behavior by dopamine in the basal ganglia: Insights from songbirds

A neural network model for timing control with reinforcement

How do humans and animals perform trial-and-error learning when the space of possibilities is infinite? In a previous study, we used an interval timing production task and discovered an updating strategy in which the agent adjusted the behavioral and neuronal noise for exploration. In the experiment, human subjects proactively generated a series of timed motor outputs. Positive or negative feedback was provided after each response based on the timing accuracy. We found that the sequential motor timing varied at two temporal scales: long-term correlation around the target interval due to memory drifts and short-term adjustments of timing variability according to feedback. We have previously described these two key features of timing variability with an augmented Gaussian process, termed reward-sensitive Gaussian process (RSGP). In a nutshell, the temporal covariance of the timing variable was updated based on the feedback history to recreate the two behavioral characteristics mentioned above. However, the RSGP was mainly descriptive and lacked a neurobiological basis of how the reward feedback can be used by a neural circuit to adjust motor variability. Here we provide a mechanistic model and simulate the process by borrowing the architecture of recurrent neural networks (RNNs). While recurrent connection provided the long-term serial correlation in motor timing, to facilitate reward-driven short-term variations, we introduced reward-dependent variability in the network connectivity, inspired by the stochastic nature of synaptic transmission in the brain. Our model was able to recursively generate an output sequence incorporating internal variability and external reinforcement in a Bayesian framework. We show that the model can generate the temporal structure of the motor variability as a basis for exploration and exploitation trade-off. Unlike other neural network models that search for unique network connectivity for the best match between the model prediction and observation, this model can estimate the uncertainty associated with each outcome and thus did a better job in teasing apart adjustable task-relevant variability from unexplained variability. The proposed artificial neural network model parallels the mechanisms of information processing in neural systems and can extend the framework of brain-inspired reinforcement learning (RL) in continuous state control.

Neural correlates of vocal initiation in the VTA/SNc of juvenile male zebra finches

Initiation and execution of complex learned vocalizations such as human speech and birdsong depend on multiple brain circuits. In songbirds, neurons in the motor cortices and basal ganglia circuitry exhibit preparatory activity before initiation of song, and that activity is thought to play an important role in successful song performance. However, it remains unknown where a start signal for song is represented in the brain and how such a signal would lead to appropriate vocal initiation. To test whether neurons in the midbrain ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) show activity related to song initiation, we carried out extracellular recordings of VTA/SNc single units in singing juvenile male zebra finches. We found that a subset of VTA/SNc units exhibit phasic activity precisely time-locked to the onset of the song bout, and that the activity occurred specifically at the beginning of song. These findings suggest that phasic activity in the VTA/SNc represents a start signal that triggers song vocalization.

Capturing the Effects of Domestication on Vocal Learning Complexity

Domesticated and vocal learning species can serve as informative model organisms for the reduction of reactive aggression and emergence of speech in our lineage. Amidst mounting evidence that domestication modifies vocal repertoires across different species, we focus on the domesticated Bengalese finch, which has a more complex song than the wild-type white-rumped munia. Our explanation for this effect revolves around the glutamate neurotransmitter system. Glutamate signaling (i) is implicated in birdsong learning, (ii) controls dopamine activity in neural circuits crucial for vocal learning, (iii) is disproportionately targeted in the evolution of domesticates, and (iv) regulates stress responses and aggressive behaviors attenuated under domestication. We propose that attenuated excitation of stress-related neural circuits potentiates vocal learning via altered dopaminergic signaling.

Social context-dependent singing alters molecular markers of synaptic plasticity signaling in finch basal ganglia Area X

Vocal communication is a crucial skill required throughout life. However, there is a critical gap in our understanding of the underlying molecular brain mechanisms, thereby motivating our use of the zebra finch songbird model. Adult male zebra finches show differences in neural activity patterns in song-dedicated brain nuclei when they sing in two distinct social contexts: a male singing by himself (undirected, UD) and a male singing to a female (female-directed, FD). In our prior work, we showed that in song-dedicated basal ganglia Area X, protein levels of a N-methyl-D-aspartate receptor subtype 2B (NMDAR2B) increased with more UD song and decreased with more FD song. We hypothesized that molecules downstream of this receptor would show differential protein expression levels in Area X between UD and FD song. Specifically, we investigated calcium/calmodulin dependent protein kinase II beta (CaMKIIB), homer scaffold protein 1 (HOMER1), serine/threonine protein kinase (Akt), and mechanistic target of rapamycin kinase (mTOR) following singing and non-singing states in Area X. We show relationships between social context and protein levels. HOMER1 protein levels decreased with time spent singing FD song, and mTOR protein levels decreased with the amount of and time spent singing FD song. For both HOMER1 and mTOR, there were no differences with the amount of UD song. With time spent singing UD, CaMKIIB protein levels trended in a U-shaped curve whereas Akt protein levels trended down. Both molecules showed no change with FD song. Our results support differential involvement of molecules in synaptic plasticity pathways between UD and FD song behaviors.

Differential effects of Foxp2 disruption in distinct motor circuits

Disruptions of the FOXP2 gene cause a speech and language disorder involving difficulties in sequencing orofacial movements. FOXP2 is expressed in cortico-striatal and cortico-cerebellar circuits important for fine motor skills, and affected individuals show abnormalities in these brain regions. We selectively disrupted Foxp2 in the cerebellar Purkinje cells, striatum or cortex of mice and assessed the effects on skilled motor behaviour using an operant lever-pressing task. Foxp2 loss in each region impacted behaviour differently, with striatal and Purkinje cell disruptions affecting the variability and the speed of lever-press sequences, respectively. Mice lacking Foxp2 in Purkinje cells showed a prominent phenotype involving slowed lever pressing as well as deficits in skilled locomotion. In vivo recordings from Purkinje cells uncovered an increased simple spike firing rate and decreased modulation of firing during limb movements. This was caused by increased intrinsic excitability rather than changes in excitatory or inhibitory inputs. Our findings show that Foxp2 can modulate different aspects of motor behaviour in distinct brain regions, and uncover an unknown role for Foxp2 in the modulation of Purkinje cell activity that severely impacts skilled movements.

Neurotensin and neurotensin receptor 1 mRNA expression in song-control regions changes during development in male zebra finches

Learned vocalizations are important for communication in some vertebrate taxa. The neural circuitry for the learning and production of vocalizations is well known in songbirds, many of which learn songs initially during a critical period early in life. Dopamine is essential for motor learning, including song learning, and dopamine-related measures change throughout development in song-control regions such as HVC, the lateral magnocellular nucleus of the anterior nidopallium (LMAN), Area X, and the robust nucleus of the arcopallium (RA). In mammals, the neuropeptide neurotensin strongly interacts with dopamine signaling. This study investigated a potential role for the neurotensin system in song learning by examining how neurotensin (Nts) and neurotensin receptor 1 (Ntsr1) expression change throughout development. Nts and Ntsr1 mRNA expression was analyzed in song-control regions of male zebra finches in four stages of the song learning process: pre-subsong (25 days posthatch; dph), subsong (45 dph), plastic song (60 dph), and crystallized song (130 dph). Nts expression in LMAN during the subsong stage was lower compared to other time points. Ntsr1 expression was highest in HVC, Area X, and RA during the pre-subsong stage. Opposite and complementary expression patterns for the two genes in song nuclei and across the whole brain suggest distinct roles for regions that produce and receive Nts. The expression changes at crucial time points for song development are similar to changes observed in dopamine studies and suggest Nts may be involved in the process of vocal learning. © 2018 Wiley Periodicals, Inc. Develop Neurobiol 78: 671-686, 2018.

The Role of Variability in Motor Learning

Trial-to-trial variability in the execution of movements and motor skills is ubiquitous and widely considered to be the unwanted consequence of a noisy nervous system. However, recent studies have suggested that motor variability may also be a feature of how sensorimotor systems operate and learn. This view, rooted in reinforcement learning theory, equates motor variability with purposeful exploration of motor space that, when coupled with reinforcement, can drive motor learning. Here we review studies that explore the relationship between motor variability and motor learning in both humans and animal models. We discuss neural circuit mechanisms that underlie the generation and regulation of motor variability and consider the implications that this work has for our understanding of motor learning.

Dopaminergic Contributions to Vocal Learning

Although the brain relies on auditory information to calibrate vocal behavior, the neural substrates of vocal learning remain unclear. Here we demonstrate that lesions of the dopaminergic inputs to a basal ganglia nucleus in a songbird species (Bengalese finches, Lonchura striata var. domestica) greatly reduced the magnitude of vocal learning driven by disruptive auditory feedback in a negative reinforcement task. These lesions produced no measureable effects on the quality of vocal performance or the amount of song produced. Our results suggest that dopaminergic inputs to the basal ganglia selectively mediate reinforcement-driven vocal plasticity. In contrast, dopaminergic lesions produced no measurable effects on the birds' ability to restore song acoustics to baseline following the cessation of reinforcement training, suggesting that different forms of vocal plasticity may use different neural mechanisms.

SIGNIFICANCE STATEMENT

During skill learning, the brain relies on sensory feedback to improve motor performance. However, the neural basis of sensorimotor learning is poorly understood. Here, we investigate the role of the neurotransmitter dopamine in regulating vocal learning in the Bengalese finch, a songbird with an extremely precise singing behavior that can nevertheless be reshaped dramatically by auditory feedback. Our findings show that reduction of dopamine inputs to a region of the songbird basal ganglia greatly impairs vocal learning but has no detectable effect on vocal performance. These results suggest a specific role for dopamine in regulating vocal plasticity.

Living without DAT: Loss and compensation of the dopamine transporter gene in sauropsids (birds and reptiles)

The dopamine transporter (DAT) is a major regulator of synaptic dopamine (DA) availability. It plays key roles in motor control and motor learning, memory formation, and reward-seeking behavior, is a major target of cocaine and methamphetamines, and has been assumed to be conserved among vertebrates. We have found, however, that birds, crocodiles, and lizards lack the DAT gene. We also found that the unprecedented loss of this important gene is compensated for by the expression of the noradrenaline transporter (NAT) gene, and not the serotonin transporter genes, in dopaminergic cells, which explains the peculiar pharmacology of the DA reuptake activity previously noted in bird striatum. This unexpected pattern contrasts with that of ancestral vertebrates (e.g. fish) and mammals, where the NAT gene is selectively expressed in noradrenergic cells. DA circuits in birds/reptiles and mammals thus operate with an analogous reuptake mechanism exerted by different genes, bringing new insights into gene expression regulation in dopaminergic cells and the evolution of a key molecular player in reward and addiction pathways.

Social intentions in Parkinson's disease patients: A kinematic study

Dysfunction of the dopaminergic system leads to motor, cognitive and motivational symptoms in brain disorders such as Parkinson's disease (PD). Moreover, the dopaminergic system plays an important role in social interactions. The dopaminergic input to the basal ganglia (BG) thought to integrate social cues during the planning and execution of voluntary movements remains, however, largely unexplored. Since PD provides a model to assess this function in humans, our study aimed to investigate the effects of social intentions on actions in non-demented PDpatients receiving dopamine replacement therapy (Levodopa = l-Dopa) and in neurologically healthy control participants. Patients' ability to modulate motor patterning depending on the intention motivating the action to be performed was evaluated both in "on" (with l-Dopa) and "off" (without l-Dopa) states. Participants were instructed to reach for and to grasp an object; they were then told to hand it to another person (social condition) or to place it on a concave frame (individual condition). A 'passive-observer' condition, which was similar to the 'individual' condition except for the presence of an onlooker who simply observed the scene, was also assessed to exclude the possibility that differences might be due to the presence of another person. Movement kinematics were recorded using a three-dimensional motion analysis system. Study results demonstrated that the controls and the PD patients in an 'on' state adopted different kinematic patterning for the 'social' and the 'individual' conditions; the PD patients in the 'off' state, instead, were unable to kinematically differentiate between the two conditions. These results suggest that l-Dopa treatment has positive effects on translating social intentions into specific motor patterns in PD patients.