A hypothesis for basal ganglia-dependent reinforcement learning in the songbird

Daily singing of adult songbirds functions to maintain song performance independently of auditory feedback and age

Many songbirds learn to produce songs through vocal practice in early life and continue to sing daily throughout their lifetime. While it is well-known that adult songbirds sing as part of their mating rituals, the functions of singing behavior outside of reproductive contexts remain unclear. Here, we investigated this issue in adult male zebra finches by suppressing their daily singing for two weeks and examining the effects on song performance. We found that singing suppression decreased the pitch, amplitude, and duration of songs, and that those song features substantially recovered through subsequent free singing. These reversible song changes were not dependent on auditory feedback or the age of the birds, contrasting with the adult song plasticity that has been reported previously. These results demonstrate that adult song structure is not stable without daily singing, and suggest that adult songbirds maintain song performance by preventing song changes through physical act of daily singing throughout their life. Such daily singing likely functions as vocal training to maintain the song production system in optimal conditions for song performance in reproductive contexts, similar to how human singers and athletes practice daily to maintain their performance.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

Dynamics of striatal action selection and reinforcement learning

Spiny projection neurons (SPNs) in dorsal striatum are often proposed as a locus of reinforcement learning in the basal ganglia. Here, we identify and resolve a fundamental inconsistency between striatal reinforcement learning models and known SPN synaptic plasticity rules. Direct-pathway (dSPN) and indirect-pathway (iSPN) neurons, which promote and suppress actions, respectively, exhibit synaptic plasticity that reinforces activity associated with elevated or suppressed dopamine release. We show that iSPN plasticity prevents successful learning, as it reinforces activity patterns associated with negative outcomes. However, this pathological behavior is reversed if functionally opponent dSPNs and iSPNs, which promote and suppress the current behavior, are simultaneously activated by efferent input following action selection. This prediction is supported by striatal recordings and contrasts with prior models of SPN representations. In our model, learning and action selection signals can be multiplexed without interference, enabling learning algorithms beyond those of standard temporal difference models.

Songbird mesostriatal dopamine pathways are spatially segregated before the onset of vocal learning

Diverse dopamine (DA) pathways send distinct reinforcement signals to different striatal regions. In adult songbirds, a DA pathway from the ventral tegmental area (VTA) to Area X, the striatal nucleus of the song system, carries singing-related performance error signals important for learning. Meanwhile, a parallel DA pathway to a medial striatal area (MST) arises from a distinct group of neighboring DA neurons that lack connectivity to song circuits and do not encode song error. To test if the structural and functional segregation of these two pathways depends on singing experience, we carried out anatomical studies early in development before the onset of song learning. We find that distinct VTA neurons project to either Area X or MST in juvenile birds before the onset of substantial vocal practice. Quantitative comparisons of early juveniles (30-35 days post hatch), late juveniles (60-65 dph), and adult (>90 dph) brains revealed an outsized expansion of Area X-projecting neurons relative to MST-projecting neurons in VTA over development. These results show that a mesostriatal DA system dedicated to social communication can exist and be spatially segregated before the onset of vocal practice and associated sensorimotor experience.

Differential effects of reward and punishment on reinforcement-based motor learning and generalization

Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.

Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.

Dynamic top-down biasing implements rapid adaptive changes to individual movements

Complex behaviors depend on the coordinated activity of neural ensembles in interconnected brain areas. The behavioral function of such coordination, often measured as co-fluctuations in neural activity across areas, is poorly understood. One hypothesis is that rapidly varying co-fluctuations may be a signature of moment-by-moment task-relevant influences of one area on another. We tested this possibility for error-corrective adaptation of birdsong, a form of motor learning which has been hypothesized to depend on the top-down influence of a higher-order area, LMAN (lateral magnocellular nucleus of the anterior nidopallium), in shaping moment-by-moment output from a primary motor area, RA (robust nucleus of the arcopallium). In paired recordings of LMAN and RA in singing birds, we discovered a neural signature of a top-down influence of LMAN on RA, quantified as an LMAN-leading co-fluctuation in activity between these areas. During learning, this co-fluctuation strengthened in a premotor temporal window linked to the specific movement, sequential context, and acoustic modification associated with learning. Moreover, transient perturbation of LMAN activity specifically within this premotor window caused rapid occlusion of pitch modifications, consistent with LMAN conveying a temporally localized motor-biasing signal. Combined, our results reveal a dynamic top-down influence of LMAN on RA that varies on the rapid timescale of individual movements and is flexibly linked to contexts associated with learning. This finding indicates that inter-area co-fluctuations can be a signature of dynamic top-down influences that support complex behavior and its adaptation.

Self-organization of songbird neural sequences during social isolation

Behaviors emerge via a combination of experience and innate predispositions. As the brain matures, it undergoes major changes in cellular, network, and functional properties that can be due to sensory experience as well as developmental processes. In normal birdsong learning, neural sequences emerge to control song syllables learned from a tutor. Here, we disambiguate the role of tutor experience and development in neural sequence formation by delaying exposure to a tutor. Using functional calcium imaging, we observe neural sequences in the absence of tutoring, demonstrating that tutor experience is not necessary for the formation of sequences. However, after exposure to a tutor, pre-existing sequences can become tightly associated with new song syllables. Since we delayed tutoring, only half our birds learned new syllables following tutor exposure. The birds that failed to learn were the birds in which pre-tutoring neural sequences were most 'crystallized,' that is, already tightly associated with their (untutored) song.

Dopaminergic nuclei in the chick midbrain express serotonin receptor subfamily genes

Serotonin (5-hydroxytryptamine, 5-HT) is a phylogenetically conserved modulator of numerous aspects of neural functions. Serotonergic neurons in the dorsal and median raphe nucleus provide ascending innervation to the entire forebrain and midbrain. Another important neural modulatory system exists in the midbrain, the dopaminergic system, which is associated to reward processing and motivation control. Dopaminergic neurons are distributed and clustered in the brain, classically designated as groups A8-A16. Among them, groups A8-A10 associated with reward processing and motivation control are located in the midbrain and projected to the forebrain. Recently, midbrain dopaminergic neurons were shown to be innervated by serotonergic neurons and modulated by 5-HT, with the crosstalk between serotonergic and dopaminergic systems attracting increased attention. In birds, previous studies revealed that midbrain dopaminergic neurons are located in the A8-A10 homologous clusters. However, the detailed distribution of dopaminergic neurons and the crosstalk between serotonergic and dopaminergic systems in the bird are poorly understood. To improve the understanding of the regulation of the dopaminergic by the serotonergic system, we performed in situ hybridization in the chick brainstem. We prepared RNA probes for chick orthologues of dopaminergic neuron-related genes; tyrosine hydroxylase (TH) and dopa decarboxylase (DDC), noradrenaline related genes; noradrenaline transporter (NAT) and dopamine beta-hydroxylase (DBH), and serotonin receptor genes; 5-HTR1A, 5-HTR1B, 5-HTR1D, 5-HTR1E, 5-HTR1F, 5-HTR2A, 5-HTR2B, 5-HTR2C, 5-HTR3A, 5-HTR4, 5-HTR5A, and 5-HTR7. We confirmed that the expression of tyrosine hydroxylase (TH) and NAT was well matched in all chick dopaminergic nuclei examined. This supported that the compensation of the function of dopamine transporter (DAT) by NAT is a general property of avian dopaminergic neurons. Furthermore, we showed that 5-HTR1A and 5-HTR1B were expressed in midbrain dopaminergic nuclei, suggesting the serotonergic regulation of the dopaminergic system via these receptors in chicks. Our findings will help us understand the interactions between the dopaminergic and serotonergic systems in birds at the molecular level.

Motor constellation theory: A model of infants' phonological development

Every normally developing human infant solves the difficult problem of mapping their native-language phonology, but the neural mechanisms underpinning this behavior remain poorly understood. Here, motor constellation theory, an integrative neurophonological model, is presented, with the goal of explicating this issue. It is assumed that infants' motor-auditory phonological mapping takes place through infants' orosensory "reaching" for phonological elements observed in the language-specific ambient phonology, via reference to kinesthetic feedback from motor systems (e.g., articulators), and auditory feedback from resulting speech and speech-like sounds. Attempts are regulated by basal ganglion-cerebellar speech neural circuitry, and successful attempts at reproduction are enforced through dopaminergic signaling. Early in life, the pace of anatomical development constrains mapping such that complete language-specific phonological mapping is prohibited by infants' undeveloped supralaryngeal vocal tract and undescended larynx; constraints gradually dissolve with age, enabling adult phonology. Where appropriate, reference is made to findings from animal and clinical models. Some implications for future modeling and simulation efforts, as well as clinical settings, are also discussed.

Birdsong

Have your ever felt as happy as a lark, feathered your nest or taken someone under your wing? As we watch birds, we cannot help but be struck by their uncannily familiar behaviors - singing, nest building, caring for their young - to name just a few. Songbirds - the oscine suborder of perching birds that constitute roughly half (∼4,000) of all known avian species - are noted for the songs that males and sometimes both sexes in this group sing to court mates and defend territory from rivals. Birdsongs contain several to many acoustically distinct syllables, typically organized into a stereotyped phrase, and span the same audio bandwidth that we exploit for speech and music, making them easy for us to hear and appreciate. Consequently, eavesdropping humans long ago detected the most striking parallel between songbirds and humans: juvenile songbirds learn to sing in a manner similar to a child learning to speak.

Thalamocortical contribution to flexible learning in neural systems

Animal brains evolved to optimize behavior in dynamic environments, flexibly selecting actions that maximize future rewards in different contexts. A large body of experimental work indicates that such optimization changes the wiring of neural circuits, appropriately mapping environmental input onto behavioral outputs. A major unsolved scientific question is how optimal wiring adjustments, which must target the connections responsible for rewards, can be accomplished when the relation between sensory inputs, action taken, and environmental context with rewards is ambiguous. The credit assignment problem can be categorized into context-independent structural credit assignment and context-dependent continual learning. In this perspective, we survey prior approaches to these two problems and advance the notion that the brain's specialized neural architectures provide efficient solutions. Within this framework, the thalamus with its cortical and basal ganglia interactions serves as a systems-level solution to credit assignment. Specifically, we propose that thalamocortical interaction is the locus of meta-learning where the thalamus provides cortical control functions that parametrize the cortical activity association space. By selecting among these control functions, the basal ganglia hierarchically guide thalamocortical plasticity across two timescales to enable meta-learning. The faster timescale establishes contextual associations to enable behavioral flexibility, while the slower one enables generalization to new contexts.

Shared mechanisms of auditory and non-auditory vocal learning in the songbird brain

Songbirds and humans share the ability to adaptively modify their vocalizations based on sensory feedback. Prior studies have focused primarily on the role that auditory feedback plays in shaping vocal output throughout life. In contrast, it is unclear how non-auditory information drives vocal plasticity. Here, we first used a reinforcement learning paradigm to establish that somatosensory feedback (cutaneous electrical stimulation) can drive vocal learning in adult songbirds. We then assessed the role of a songbird basal ganglia thalamocortical pathway critical to auditory vocal learning in this novel form of vocal plasticity. We found that both this circuit and its dopaminergic inputs are necessary for non-auditory vocal learning, demonstrating that this pathway is critical for guiding adaptive vocal changes based on both auditory and somatosensory signals. The ability of this circuit to use both auditory and somatosensory information to guide vocal learning may reflect a general principle for the neural systems that support vocal plasticity across species.

Learning, fast and slow

Animals can learn efficiently from a single experience and change their future behavior in response. However, in other instances, animals learn very slowly, requiring thousands of experiences. Here, I survey tasks involving fast and slow learning and consider some hypotheses for what differentiates the underlying neural mechanisms. It has been proposed that fast learning relies on neural representations that favor efficient Hebbian modification of synapses. These efficient representations may be encoded in the genome, resulting in a repertoire of fast learning that differs across species. Alternatively, the required neural representations may be acquired from experience through a slow process of unsupervised learning from the environment.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

A test of context and sex-dependent dopaminergic effects on the behavior of a gregarious bird, the common waxbill Estrilda astrild

The Dopaminergic (DAergic) system has well known influences on behavioral and cognitive functions. Previous work with common waxbills (Estrilda astrild) reported context-specific DAergic effects that could have been due to social environment. Manipulating the dopamine D2-like receptor family (D2R) pathways had opposed effects on behavior depending on whether waxbills were tested alone or in a small cage with a mirror as social stimulus. Since waxbills are highly gregarious, it was hypothesized that being alone or perceiving to have a companion might explain this context-dependence. To test context-dependent DAergic effects, we compared behavioral effects of D2R manipulation in waxbills in the same familiar environment, but either alone or with a familiar, same-sex companion. We found that D2R agonism decreased movement and feeding, similarly to previous results when testing waxbills alone. However, contrary to the hypothesis of dependence on social context, we found that the behavioral effects of the D2R agonist were unchanged when waxbills were tested with a companion. The context-dependence reported earlier might thus be due to other factors, such as the stress of being in a novel environment (small cage) or with an unfamiliar social stimulus (mirror image). In tests with a companion, we also found a sex-specific social effect of D2R manipulation: D2R blocking tended to decrease aggression in males but to increase in females. Together with past work, our results suggest that DAergic effects on behavior involve different types of context- or sex-dependence.

Songbird subthalamic neurons project to dopaminergic midbrain and exhibit singing-related activity

Skill learning requires motor output to be evaluated against internal performance benchmarks. In songbirds, ventral tegmental area (VTA) dopamine neurons (DA) signal performance errors important for learning, but it remains unclear which brain regions project to VTA and how these inputs may contribute to DA error signaling. Here, we find that the songbird subthalamic nucleus (STN) projects to VTA and that STN microstimulation can excite VTA neurons. We also discover that STN receives inputs from motor cortical, auditory cortical, and ventral pallidal brain regions previously implicated in song evaluation. In the first neural recordings from songbird STN, we discover that the activity of most STN neurons is associated with body movements and not singing, but a small fraction of neurons exhibits precise song timing and performance error signals. Our results place the STN in a pathway important for song learning, but not song production, and expand the territories of songbird brain potentially associated with song learning.NEW & NOTEWORTHY Songbird subthalamic (STN) neurons exhibit singing-related signals and are interconnected with the motor cortical nucleus, auditory pallium, ventral pallidum, and ventral tegmental area, areas important for song generation and learning.

Delays in the Reticulospinal System Are Associated With a Reduced Capacity to Learn a Simulated Feeding Task in Older Adults

Learning declines with age. Recent evidence indicates that the brainstem may play an important role in learning and motor skill acquisition. Our objective was to determine if delays in the reticular formation, measured via the startle reflex, correspond to age-related deficits in learning and retention. We hypothesized that delays in the startle reflex would be linearly correlated to learning and retention deficits in older adults. To determine if associations were unique to the reticulospinal system, we also evaluated corticospinal contributions with transcranial magnetic stimulation. Our results showed a linear relationship between startle onset latency and percent learning and retention but no relationship between active or passive motor-evoked potential onsets or peak-to-peak amplitude. These results lay the foundation for further study to evaluate if (1) the reticular formation is a subcortical facilitator of skill acquisition and (2) processing delays in the reticular formation contribute to age-related learning deficits.

High-throughput automated methods for classical and operant conditioning of Drosophila larvae

Learning which stimuli (classical conditioning) or which actions (operant conditioning) predict rewards or punishments can improve chances of survival. However, the circuit mechanisms that underlie distinct types of associative learning are still not fully understood. Automated, high-throughput paradigms for studying different types of associative learning, combined with manipulation of specific neurons in freely behaving animals, can help advance this field. The Drosophila melanogaster larva is a tractable model system for studying the circuit basis of behaviour, but many forms of associative learning have not yet been demonstrated in this animal. Here, we developed a high-throughput (i.e. multi-larva) training system that combines real-time behaviour detection of freely moving larvae with targeted opto- and thermogenetic stimulation of tracked animals. Both stimuli are controlled in either open- or closed-loop, and delivered with high temporal and spatial precision. Using this tracker, we show for the first time that Drosophila larvae can perform classical conditioning with no overlap between sensory stimuli (i.e. trace conditioning). We also demonstrate that larvae are capable of operant conditioning by inducing a bend direction preference through optogenetic activation of reward-encoding serotonergic neurons. Our results extend the known associative learning capacities of Drosophila larvae. Our automated training rig will facilitate the study of many different forms of associative learning and the identification of the neural circuits that underpin them.

Vocal Learning and Behaviors in Birds and Human Bilinguals: Parallels, Divergences and Directions for Research

Comparisons between the communication systems of humans and animals are instrumental in contextualizing speech and language into an evolutionary and biological framework and for illuminating mechanisms of human communication. As a complement to previous work that compares developmental vocal learning and use among humans and songbirds, in this article we highlight phenomena associated with vocal learning subsequent to the development of primary vocalizations (i.e., the primary language (L1) in humans and the primary song (S1) in songbirds). By framing avian “second-song” (S2) learning and use within the human second-language (L2) context, we lay the groundwork for a scientifically-rich dialogue between disciplines. We begin by summarizing basic birdsong research, focusing on how songs are learned and on constraints on learning. We then consider commonalities in vocal learning across humans and birds, in particular the timing and neural mechanisms of learning, variability of input, and variability of outcomes. For S2 and L2 learning outcomes, we address the respective roles of age, entrenchment, and social interactions. We proceed to orient current and future birdsong inquiry around foundational features of human bilingualism: L1 effects on the L2, L1 attrition, and L1<–>L2 switching. Throughout, we highlight characteristics that are shared across species as well as the need for caution in interpreting birdsong research. Thus, from multiple instructive perspectives, our interdisciplinary dialogue sheds light on biological and experiential principles of L2 acquisition that are informed by birdsong research, and leverages well-studied characteristics of bilingualism in order to clarify, contextualize, and further explore S2 learning and use in songbirds.

Encoding of acquired sound-sequence salience by auditory cortical offset responses

Behaviorally relevant sounds are often composed of distinct acoustic units organized into specific temporal sequences. The meaning of such sound sequences can therefore be fully recognized only when they have terminated. However, the neural mechanisms underlying the perception of sound sequences remain unclear. Here, we use two-photon calcium imaging in the auditory cortex of behaving mice to test the hypothesis that neural responses to termination of sound sequences ("Off-responses") encode their acoustic history and behavioral salience. We find that auditory cortical Off-responses encode preceding sound sequences and that learning to associate a sound sequence with a reward induces enhancement of Off-responses relative to responses during the sound sequence ("On-responses"). Furthermore, learning enhances network-level discriminability of sound sequences by Off-responses. Last, learning-induced plasticity of Off-responses but not On-responses lasts to the next day. These findings identify auditory cortical Off-responses as a key neural signature of acquired sound-sequence salience.

Neural dynamics underlying birdsong practice and performance

Musical and athletic skills are learned and maintained through intensive practice to enable precise and reliable performance for an audience. Consequently, understanding such complex behaviours requires insight into how the brain functions during both practice and performance. Male zebra finches learn to produce courtship songs that are more varied when alone and more stereotyped in the presence of females1. These differences are thought to reflect song practice and performance, respectively2,3, providing a useful system in which to explore how neurons encode and regulate motor variability in these two states. Here we show that calcium signals in ensembles of spiny neurons (SNs) in the basal ganglia are highly variable relative to their cortical afferents during song practice. By contrast, SN calcium signals are strongly suppressed during female-directed performance, and optogenetically suppressing SNs during practice strongly reduces vocal variability. Unsupervised learning methods4,5 show that specific SN activity patterns map onto distinct song practice variants. Finally, we establish that noradrenergic signalling reduces vocal variability by directly suppressing SN activity. Thus, SN ensembles encode and drive vocal exploration during practice, and the noradrenergic suppression of SN activity promotes stereotyped and precise song performance for an audience.

Seasonal regulation of singing-driven gene expression associated with song plasticity in the canary, an open-ended vocal learner

Songbirds are one of the few animal taxa that possess vocal learning abilities. Different species of songbirds exhibit species-specific learning programs during song acquisition. Songbirds with open-ended vocal learning capacity, such as the canary, modify their songs during adulthood. Nevertheless, the neural molecular mechanisms underlying open-ended vocal learning are not fully understood. We investigated the singing-driven expression of neural activity-dependent genes (Arc, Egr1, c-fos, Nr4a1, Sik1, Dusp6, and Gadd45β) in the canary to examine a potential relationship between the gene expression level and the degree of seasonal vocal plasticity at different ages. The expression of these genes was differently regulated throughout the critical period of vocal learning in the zebra finch, a closed-ended song learner. In the canary, the neural activity-dependent genes were induced by singing in the song nuclei throughout the year. However, in the vocal motor nucleus, the robust nucleus of the arcopallium (RA), all genes were regulated with a higher induction rate by singing in the fall than in the spring. The singing-driven expression of these genes showed a similar induction rate in the fall between the first year juvenile and the second year adult canaries, suggesting a seasonal, not age-dependent, regulation of the neural activity-dependent genes. By measuring seasonal vocal plasticity and singing-driven gene expression, we found that in RA, the induction intensity of the neural activity-dependent genes was correlated with the state of vocal plasticity. These results demonstrate a correlation between vocal plasticity and the singing-driven expression of neural activity-dependent genes in RA through song development, regardless of whether a songbird species possesses an open- or closed-ended vocal learning capacity.

Deafening-induced rapid changes to spine synaptic connectivity in the adult avian vocal basal ganglia

The basal ganglia have been implicated in auditory-dependent vocal learning and plasticity in human and songbirds, but the underlying neural phenotype remains to be clarified. Here, using confocal imaging and three-dimensional electron microscopy, we investigated striatal structural plasticity in response to hearing loss in Area X, the avian vocal basal ganglia, in adult male zebra finch (Taeniopygia guttata). We observed a rapid elongation of dendritic spines, by approximately 13%, by day 3 after deafening, and a considerable increase in spine synapse density, by approximately 61%, by day 14 after deafening, compared with the controls with an intact cochlea. These findings reveal structural sensitivity of Area X to auditory deprivation and suggest that this striatal plasticity might contribute to deafening-induced changes to learned vocal behavior.

What Is the Role of Thalamostriatal Circuits in Learning Vocal Sequences?

Basal ganglia (BG) circuits integrate sensory and motor-related information from the cortex, thalamus, and midbrain to guide learning and production of motor sequences. Birdsong, like speech, is comprised of precisely sequenced vocal elements. Learning song sequences during development relies on Area X, a vocalization related region in the medial striatum of the songbird BG. Area X receives inputs from cortical-like pallial song circuits and midbrain dopaminergic circuits and sends projections to the thalamus. It has recently been shown that thalamic circuits also send substantial projections back to Area X. Here, we outline a gated-reinforcement learning model for how Area X may use signals conveyed by thalamostriatal inputs to direct song learning. Integrating conceptual advances from recent mammalian and songbird literature, we hypothesize that thalamostriatal pathways convey signals linked to song syllable onsets and offsets and influence striatal circuit plasticity via regulation of cholinergic interneurons (ChIs). We suggest that syllable sequence associated vocal-motor information from the thalamus drive precisely timed pauses in ChIs activity in Area X. When integrated with concurrent corticostriatal and dopaminergic input, this circuit helps regulate plasticity on medium spiny neurons (MSNs) and the learning of syllable sequences. We discuss new approaches that can be applied to test core ideas of this model and how associated insights may provide a framework for understanding the function of BG circuits in learning motor sequences.

Optimal plasticity for memory maintenance during ongoing synaptic change

Synaptic connections in many brain circuits fluctuate, exhibiting substantial turnover and remodelling over hours to days. Surprisingly, experiments show that most of this flux in connectivity persists in the absence of learning or known plasticity signals. How can neural circuits retain learned information despite a large proportion of ongoing and potentially disruptive synaptic changes? We address this question from first principles by analysing how much compensatory plasticity would be required to optimally counteract ongoing fluctuations, regardless of whether fluctuations are random or systematic. Remarkably, we find that the answer is largely independent of plasticity mechanisms and circuit architectures: compensatory plasticity should be at most equal in magnitude to fluctuations, and often less, in direct agreement with previously unexplained experimental observations. Moreover, our analysis shows that a high proportion of learning-independent synaptic change is consistent with plasticity mechanisms that accurately compute error gradients.

A memory-driven auditory program ensures selective and precise vocal imitation in zebra finches

In the vocal learning model, the juvenile first memorizes a model sound, and the imprinted memory gradually converts into vocal-motor output during the sensorimotor integration. However, early acquired memory may not precisely represent the fine structures of a model sound. How do juveniles ensure precise model imitation? Here we show that juvenile songbirds develop an auditory learning program by actively and attentively engaging with tutor’s singing during the sensorimotor phase. The listening/approaching behavior requires previously acquired model memory and the individual variability of approaching behavior correlates with the precision of tutor song imitation. Moreover, it is modulated by dopamine and associated with forebrain regions for sensory processing. Overall, precise vocal learning may involve two steps of auditory processing: a passive imprinting of model memory occurs during the early sensory period; the previously acquired memory then guides an active and selective engagement of the re-exposed model to fine tune model imitation.

Wan-Chun Liu et al. demonstrate that the sensory phase of vocal learning in zebra finches is split across two stages: (1) passive listening and formation of a memory, and (2) active listening and behavioral engagement of juveniles with adult tutors. Furthermore, they show that approach behavior is correlated with song imitation quality, and immediate early gene expression in the caudal medial nidopallium linked to auditory behavior.

The basal ganglia control the detailed kinematics of learned motor skills

The basal ganglia are known to influence action selection and modulation of movement vigor, but whether and how they contribute to specifying the kinematics of learned motor skills is not understood. Here, we probe this question by recording and manipulating basal ganglia activity in rats trained to generate complex task-specific movement patterns with rich kinematic structure. We find that the sensorimotor arm of the basal ganglia circuit is crucial for generating the detailed movement patterns underlying the acquired motor skills. Furthermore, the neural representations in the striatum, and the control function they subserve, do not depend on inputs from the motor cortex. Taken together, these results extend our understanding of the basal ganglia by showing that they can specify and control the fine-grained details of learned motor skills through their interactions with lower-level motor circuits.

Unilateral vocal nerve resection alters neurogenesis in the avian song system in a region-specific manner

New neurons born in the adult brain undergo a critical period soon after migration to their site of incorporation. During this time, the behavior of the animal may influence the survival or culling of these cells. In the songbird song system, earlier work suggested that adult-born neurons may be retained in the song motor pathway nucleus HVC with respect to motor progression toward a target song during juvenile song learning, seasonal song restructuring, and experimentally manipulated song variability. However, it is not known whether the quality of song per se, without progressive improvement, may also influence new neuron survival. To test this idea, we experimentally altered song acoustic structure by unilateral denervation of the syrinx, causing a poor quality song. We found no effect of aberrant song on numbers of new neurons in HVC, suggesting that song quality does not influence new neuron culling in this region. However, aberrant song resulted in the loss of left-side dominance in new neurons in the auditory region caudomedial nidopallium (NCM), and a bilateral decrease in new neurons in the basal ganglia nucleus Area X. Thus new neuron culling may be influenced by behavioral feedback in accordance with the function of new neurons within that region. We propose that studying the effects of singing behaviors on new neurons across multiple brain regions that differentially subserve singing may give rise to general rules underlying the regulation of new neuron survival across taxa and brain regions more broadly.

Low-dimensional learned feature spaces quantify individual and group differences in vocal repertoires

Increases in the scale and complexity of behavioral data pose an increasing challenge for data analysis. A common strategy involves replacing entire behaviors with small numbers of handpicked, domain-specific features, but this approach suffers from several crucial limitations. For example, handpicked features may miss important dimensions of variability, and correlations among them complicate statistical testing. Here, by contrast, we apply the variational autoencoder (VAE), an unsupervised learning method, to learn features directly from data and quantify the vocal behavior of two model species: the laboratory mouse and the zebra finch. The VAE converges on a parsimonious representation that outperforms handpicked features on a variety of common analysis tasks, enables the measurement of moment-by-moment vocal variability on the timescale of tens of milliseconds in the zebra finch, provides strong evidence that mouse ultrasonic vocalizations do not cluster as is commonly believed, and captures the similarity of tutor and pupil birdsong with qualitatively higher fidelity than previous approaches. In all, we demonstrate the utility of modern unsupervised learning approaches to the quantification of complex and high-dimensional vocal behavior.

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

Motor learning is a core aspect of human life and appears to be ubiquitous throughout the animal kingdom. Dopamine, a neuromodulator with a multifaceted role in synaptic plasticity, may be a key signaling molecule for motor skill learning. Though typically studied in the context of reward-based associative learning, dopamine appears to be necessary for some types of motor learning. Mesencephalic dopamine structures are highly conserved among vertebrates, as are some of their primary targets within the basal ganglia, a subcortical circuit important for motor learning and motor control. With a focus on the benefits of cross-species comparisons, this review examines how "model-free" and "model-based" computational frameworks for understanding dopamine's role in associative learning may be applied to motor learning. The hypotheses that dopamine could drive motor learning either by functioning as a reward prediction error, through passive facilitating of normal basal ganglia activity, or through other mechanisms are examined in light of new studies using humans, rodents, and songbirds. Additionally, new paradigms that could enhance our understanding of dopamine's role in motor learning by bridging the gap between the theoretical literature on motor learning in humans and other species are discussed.

Frozen algorithms: how the brain's wiring facilitates learning

Synapses and neural connectivity are plastic and shaped by experience. But to what extent does connectivity itself influence the ability of a neural circuit to learn? Insights from optimization theory and AI shed light on how learning can be implemented in neural circuits. Though abstract in their nature, learning algorithms provide a principled set of hypotheses on the necessary ingredients for learning in neural circuits. These include the kinds of signals and circuit motifs that enable learning from experience, as well as an appreciation of the constraints that make learning challenging in a biological setting. Remarkably, some simple connectivity patterns can boost the efficiency of relatively crude learning rules, showing how the brain can use anatomy to compensate for the biological constraints of known synaptic plasticity mechanisms. Modern connectomics provides rich data for exploring this principle, and may reveal how brain connectivity is constrained by the requirement to learn efficiently.

Addition of new neurons and the emergence of a local neural circuit for precise timing

During development, neurons arrive at local brain areas in an extended period of time, but how they form local neural circuits is unknown. Here we computationally model the emergence of a network for precise timing in the premotor nucleus HVC in songbird. We show that new projection neurons, added to HVC post hatch at early stages of song development, are recruited to the end of a growing feedforward network. High spontaneous activity of the new neurons makes them the prime targets for recruitment in a self-organized process via synaptic plasticity. Once recruited, the new neurons fire readily at precise times, and they become mature. Neurons that are not recruited become silent and replaced by new immature neurons. Our model incorporates realistic HVC features such as interneurons, spatial distributions of neurons, and distributed axonal delays. The model predicts that the birth order of the projection neurons correlates with their burst timing during the song.

Functions of local neural circuits depend on their specific network structures, but how the networks are wired is unknown. We show that such structures can emerge during development through a self-organized process, during which the network is wired by neuron-by-neuron recruitment. This growth is facilitated by steady supply of immature neurons, which are highly excitable and plastic. We suggest that neuron maturation dynamics is an integral part of constructing local neural circuits.

Behavioral and neural correlates of speech motor sequence learning in stuttering and neurotypical speakers: an fMRI investigation

Stuttering is a neurodevelopmental disorder characterized by impaired production of coordinated articulatory movements needed for fluent speech. It is currently unknown whether these abnormal production characteristics reflect disruptions to brain mechanisms underlying the acquisition and/or execution of speech motor sequences. To dissociate learning and control processes, we used a motor sequence learning paradigm to examine the behavioral and neural correlates of learning to produce novel phoneme sequences in adults who stutter (AWS) and neurotypical controls. Participants intensively practiced producing pseudowords containing non-native consonant clusters (e.g., "gvasf") over two days. The behavioral results indicated that although the two experimental groups showed comparable learning trajectories, AWS performed significantly worse on the task prior to and after speech motor practice. Using functional magnetic resonance imaging (fMRI), the authors compared brain activity during articulation of the practiced words and a set of novel pseudowords (matched in phonetic complexity). FMRI analyses revealed no differences between AWS and controls in cortical or subcortical regions; both groups showed comparable increases in activation in left-lateralized brain areas implicated in phonological working memory and speech motor planning during production of the novel sequences compared to the practiced sequences. Moreover, activation in left-lateralized basal ganglia sites was negatively correlated with in-scanner mean disfluency in AWS. Collectively, these findings demonstrate that AWS exhibit no deficit in constructing new speech motor sequences but do show impaired execution of these sequences before and after they have been acquired and consolidated.

Autism-linked gene FoxP1 selectively regulates the cultural transmission of learned vocalizations

Autism spectrum disorders (ASDs) are characterized by impaired learning of social skills and language. Memories of how parents and other social models behave are used to guide behavioral learning. How ASD-linked genes affect the intertwined aspects of observational learning and behavioral imitation is not known. Here, we examine how disrupted expression of the ASD gene FOXP1, which causes severe impairments in speech and language learning, affects the cultural transmission of birdsong between adult and juvenile zebra finches. FoxP1 is widely expressed in striatal-projecting forebrain mirror neurons. Knockdown of FoxP1 in this circuit prevents juvenile birds from forming memories of an adult song model but does not interrupt learning how to vocally imitate a previously memorized song. This selective learning deficit is associated with potent disruptions to experience-dependent structural and synaptic plasticity in mirror neurons. Thus, FoxP1 regulates the ability to form memories essential to the cultural transmission of behavior.

Variable but not random: temporal pattern coding in a songbird brain area necessary for song modification

Practice of a complex motor gesture involves motor exploration to attain a better match to target, but little is known about the neural code for such exploration. We examine spiking in a premotor area of the songbird brain critical for song modification and quantify correlations between spiking and time in the motor sequence. While isolated spikes code for time in song during performance of song to a female bird, extended strings of spiking and silence, particularly bursts, code for time in song during undirected (solo) singing, or "practice." Bursts code for particular times in song with more information than individual spikes, and this spike-spike synergy is significantly higher during undirected singing. The observed pattern information cannot be accounted for by a Poisson model with a matched time-varying rate, indicating that the precise timing of spikes in both bursts in undirected singing and isolated spikes in directed singing code for song with a temporal code. Temporal coding during practice supports the hypothesis that lateral magnocellular nucleus of the anterior nidopallium neurons actively guide song modification at local instances in time.NEW & NOTEWORTHY This paper shows that bursts of spikes in the songbird brain during practice carry information about the output motor pattern. The brain's code for song changes with social context, in performance versus practice. Synergistic combinations of spiking and silence code for time in the bird's song. This is one of the first uses of information theory to quantify neural information about a motor output. This activity may guide changes to the song.

Reinforcement regulates timing variability in thalamus

Learning reduces variability but variability can facilitate learning. This paradoxical relationship has made it challenging to tease apart sources of variability that degrade performance from those that improve it. We tackled this question in a context-dependent timing task requiring humans and monkeys to flexibly produce different time intervals with different effectors. We identified two opposing factors contributing to timing variability: slow memory fluctuation that degrades performance and reward-dependent exploratory behavior that improves performance. Signatures of these opposing factors were evident across populations of neurons in the dorsomedial frontal cortex (DMFC), DMFC-projecting neurons in the ventrolateral thalamus, and putative target of DMFC in the caudate. However, only in the thalamus were the performance-optimizing regulation of variability aligned to the slow performance-degrading memory fluctuations. These findings reveal how variability caused by exploratory behavior might help to mitigate other undesirable sources of variability and highlight a potential role for thalamocortical projections in this process.

Sensory substitution reveals a manipulation bias

Sensory substitution is a promising therapeutic approach for replacing a missing or diseased sensory organ by translating inaccessible information into another sensory modality. However, many substitution systems are not well accepted by subjects. To explore the effect of sensory substitution on voluntary action repertoires and their associated affective valence, we study deaf songbirds to which we provide visual feedback as a substitute of auditory feedback. Surprisingly, deaf birds respond appetitively to song-contingent binary visual stimuli. They skillfully adapt their songs to increase the rate of visual stimuli, showing that auditory feedback is not required for making targeted changes to vocal repertoires. We find that visually instructed song learning is basal-ganglia dependent. Because hearing birds respond aversively to the same visual stimuli, sensory substitution reveals a preference for actions that elicit sensory feedback over actions that do not, suggesting that substitution systems should be designed to exploit the drive to manipulate.

An avian cortical circuit for chunking tutor song syllables into simple vocal-motor units

How are brain circuits constructed to achieve complex goals? The brains of young songbirds develop motor circuits that achieve the goal of imitating a specific tutor song to which they are exposed. Here, we set out to examine how song-generating circuits may be influenced early in song learning by a cortical region (NIf) at the interface between auditory and motor systems. Single-unit recordings reveal that, during juvenile babbling, NIf neurons burst at syllable onsets, with some neurons exhibiting selectivity for particular emerging syllable types. When juvenile birds listen to their tutor, NIf neurons are also activated at tutor syllable onsets, and are often selective for particular syllable types. We examine a simple computational model in which tutor exposure imprints the correct number of syllable patterns as ensembles in an interconnected NIf network. These ensembles are then reactivated during singing to train a set of syllable sequences in the motor network.

Young songbirds learn to imitate their parents’ songs. Here, the authors find that, in baby birds, neurons in a brain region at the interface of auditory and motor circuits signal the onsets of song syllables during both tutoring and babbling, suggesting a specific neural mechanism for vocal imitation.

Actor-critic reinforcement learning in the songbird

It feels rewarding to ace your opponent on match point. Here, we propose common mechanisms underlie reward and performance learning. First, when a singing bird unexpectedly hits the right note, its dopamine (DA) neurons are activated as when a thirsty monkey receives an unexpected juice reward. Second, these DA signals reinforce vocal variations much as they reinforce stimulus-response associations. Third, limbic inputs to DA neurons signal the predicted quality of song syllables much like they signal the predicted reward value of a place or a stimulus during foraging. Finally, songbirds may solve difficult problems in reinforcement learning - such as credit assignment and catastrophic forgetting - with node perturbation and consolidation of reinforced vocal patterns in motor cortical circuits. Consolidation occurs downstream of a canonical 'actor-critic' circuit motif that learns to maximize performance quality in essentially the same way it learns to maximize reward: by computing and learning from prediction errors.

Regulation of vocal precision by noradrenergic modulation of a motor nucleus

Norepinephrine (NE) function is often implicated in regulating arousal levels. Recent theory suggests that the noradrenergic system also regulates the optimization of behavior with respect to reward maximization by controlling a switch between exploration and exploitation of the specific actions that yield greatest utility. We show in the songbird that NE can act directly on a cortical motor area and cause a switch between exploratory and exploitative behavior.

Spatial planning with long visual range benefits escape from visual predators in complex naturalistic environments

It is uncontroversial that land animals have more elaborated cognitive abilities than their aquatic counterparts such as fish. Yet there is no apparent a-priori reason for this. A key cognitive faculty is planning. We show that in visually guided predator-prey interactions, planning provides a significant advantage, but only on land. During animal evolution, the water-to-land transition resulted in a massive increase in visual range. Simulations of behavior identify a specific type of terrestrial habitat, clustered open and closed areas (savanna-like), where the advantage of planning peaks. Our computational experiments demonstrate how this patchy terrestrial structure, in combination with enhanced visual range, can reveal and hide agents as a function of their movement and create a selective benefit for imagining, evaluating, and selecting among possible future scenarios-in short, for planning. The vertebrate invasion of land may have been an important step in their cognitive evolution.

Manipulations of sensory experiences during development reveal mechanisms underlying vocal learning biases in zebra finches

Biological predispositions in learning can bias and constrain the cultural evolution of social and communicative behaviors (e.g., speech and birdsong), and lead to the emergence of behavioral and cultural "universals." For example, surveys of laboratory and wild populations of zebra finches (Taeniopygia guttata) document consistent patterning of vocal elements ("syllables") with respect to their acoustic properties (e.g., duration, mean frequency). Furthermore, such universal patterns are also produced by birds that are experimentally tutored with songs containing randomly sequenced syllables ("tutored birds"). Despite extensive demonstrations of learning biases, much remains to be uncovered about the nature of biological predispositions that bias song learning and production in songbirds. Here, we examined the degree to which "innate" auditory templates and/or biases in vocal motor production contribute to vocal learning biases and production in zebra finches. Such contributions can be revealed by examining acoustic patterns in the songs of birds raised without sensory exposure to song ("untutored birds") or of birds that are unable to hear from early in development ("early-deafened birds"). We observed that untutored zebra finches and early-deafened zebra finches produce songs with positional variation in some acoustic features (e.g., mean frequency) that resemble universal patterns observed in tutored birds. Similar to tutored birds, early-deafened birds also produced song motifs with alternation in acoustic features across adjacent syllables. That universal acoustic patterns are observed in the songs of both untutored and early-deafened birds highlights the contribution motor production biases to the emergence of universals in culturally transmitted behaviors.

Experts, but not novices, exhibit StartReact indicating experts use the reticulospinal system more than novices

Motor skill acquisition utilizes a wide array of neural structures; however, few articles evaluate how the relative contributions of these structures shift over the course of learning. Recent evidence from rodents and songbirds suggests there is a transfer from cortical to subcortical structures following intense, repetitive training. Evidence from humans indicate that the reticulospinal system is modulated over the course of skill acquisition and may be a subcortical facilitator of learning. The objective of this study was to evaluate how reticulospinal contributions are modulated by task expertise. Reticulospinal contributions were assessed using StartReact (SR). We hypothesized that expert typists would show SR during an individuated, keystroke task but SR would be absent in novices. Expert (75.2 ± 9.8 WPM) and novice typists (41.6 ± 8.2 WPM) were evaluated during an individuated, keystroke movements. In experts, SR was present but was absent in novices. Together, these results suggest that experts use reticulospinal contributions more for movement than novices indicating that the reticular formation becomes increasingly important for movement execution in highly trained, skilled tasks even those that require individuated movement of the fingers.

BOLD differences normally attributed to inhibitory control predict symptoms, not task-directed inhibitory control in ADHD

Background

Altered brain activity that has been observed in attention deficit hyperactivity disorder (ADHD) while performing cognitive control tasks like the stop signal task (SST) has generally been interpreted as reflecting either weak (under-active) or compensatory (over-active) versions of the same functions as in healthy controls. If so, then regional activities that correlate with the efficiency of inhibitory control (i.e. stop signal reaction time, SSRT) in healthy subjects should also correlate with SSRT in ADHD. Here we test the alternate hypothesis that BOLD (blood-oxygen-level-dependent) differences might instead reflect the redirection of neural processing resources normally used for task-directed inhibitory control, towards actively managing symptomatic behaviour. If so, then activities that correlate with SSRT in TD should instead correlate with inattentive and hyperactive symptoms in ADHD.

Methods

We used fMRI (functional magnetic resonance imaging) in 14 typically developing (TD) and 14 ADHD adolescents performing the SST, and in a replication sample of 14 healthy adults. First, we identified significant group BOLD differences during all phases of activity in the SST (i.e. warning, response, reactive inhibition, error detection and post-error slowing). Next, we correlated these phases of activity with SSRT in TD and with SSRT, inattentive and hyperactive symptom scores in ADHD. We then identified whole brain significant correlations in regions of significant group difference in activity.

Results

Only three regions of significant group difference were correlated with SSRT in TD and replication groups (left and right inferior frontal gyri (IFG) during error detection and hypothalamus during post-error slowing). Consistent with regions of altered activity managing symptomatic behaviour instead of task-directed behaviour, left IFG correlated with greater inattentive score, right IFG correlated with lower hyperactive score and hypothalamus correlated with greater inattentive score and oppositely correlated with SSRT compared to TD.

Conclusions

Stimuli that elicit task-directed integration of neural processing in healthy subjects instead appear to be directing integrated function towards managing symptomatic behaviour in ADHD. The ability of the current approach to determine whether altered neural activities reflect comparable functions in ADHD and control groups has broad implications for the development and monitoring of therapeutic interventions.

The neurobiology of innate, volitional and learned vocalizations in mammals and birds

Vocalization is an ancient vertebrate trait essential to many forms of communication, ranging from courtship calls to free verse. Vocalizations may be entirely innate and evoked by sexual cues or emotional state, as with many types of calls made in primates, rodents and birds; volitional, as with innate calls that, following extensive training, can be evoked by arbitrary sensory cues in non-human primates and corvid songbirds; or learned, acoustically flexible and complex, as with human speech and the courtship songs of oscine songbirds. This review compares and contrasts the neural mechanisms underlying innate, volitional and learned vocalizations, with an emphasis on functional studies in primates, rodents and songbirds. This comparison reveals both highly conserved and convergent mechanisms of vocal production in these different groups, despite their often vast phylogenetic separation. This similarity of central mechanisms for different forms of vocal production presents experimentalists with useful avenues for gaining detailed mechanistic insight into how vocalizations are employed for social and sexual signalling, and how they can be modified through experience to yield new vocal repertoires customized to the individual's social group. This article is part of the theme issue 'What can animal communication teach us about human language?'