Slow or sudden: Re-interpreting the learning curve for modern systems neuroscience

Stimulus uncertainty and relative reward rates determine adaptive responding in perceptual decision-making

In dynamic environments, animals must select actions based on sensory input as well as expected positive and negative consequences. This type of behavior is typically studied using perceptual decision making (PDM) tasks. The arguably most influential framework for describing the cognitive processes underlying PDM is signal detection theory (SDT). One central assumption of SDT is that observers make perceptual decisions by comparing sensory evidence to a static decision criterion. However, mounting evidence suggests that the criterion is in fact highly dynamic and that observers adjust it flexibly according to task demands. Nevertheless, the mechanisms by which observers integrate stimulus and reward information for adaptive criterion learning remain not well understood. Here, we systematically investigated the factors influencing criterion setting at the single-trial level. To that end, we first specified three SDT-based models that learn either from reward, reward omission, or both. Next, by concomitantly manipulating stimulus and reward probabilities, we constructed experimental conditions in which these models make divergent predictions. Finally, we subjected rats and pigeons to a PDM task comprising these conditions. We find that subjects adopted decision criteria that maximize total reward in all experimental conditions. Detailed behavioral analyses reveal that criterion learning is driven by the integration of rewards, not reward omissions, and that reward integration is influenced by two additional factors: first, the degree of stimulus uncertainty, and second, the difference in the relative reward rates (rather than the absolute reward rates) between the choice alternatives. A model incorporating these factors accounts well for criterion dynamics across experimental conditions for both species and links signal detection theory to a learning mechanism operating at the level of single trials which, in the steady state, produces behavior similar to the matching law, a central tenet of learning theory.

Revealing hidden knowledge in amnestic mice

Alzheimer's disease (AD) is a form of dementia in which memory and cognitive decline is thought to arise from underlying neurodegeneration. These cognitive impairments, however, are transient when they first appear and can fluctuate across disease progression. Here, we investigate the neural mechanisms underlying fluctuations of performance in amnestic mice. We trained APP/PS1+ mice on an auditory go/no-go task that dissociated learning of task contingencies (knowledge) from its more variable expression under reinforcement (performance). APP/PS1+ exhibited significant performance deficits compared to control mice. Using large-scale two-photon imaging of 6,216 excitatory neurons in 8 mice, we found that auditory cortical networks were more suppressed, less selective to the sensory cues, and exhibited aberrant higher-order encoding of reward prediction compared to control mice. A small sub-population of neurons, however, displayed the opposite phenotype, reflecting a potential compensatory mechanism. Volumetric analysis demonstrated that deficits were concentrated near Aβ plaques. Strikingly, we found that these cortical deficits were reversed almost instantaneously on probe (non-reinforced) trials when APP/PS1+ performed as well as control mice, providing neural evidence for intact stimulus-action knowledge despite variable ongoing performance. A biologically-plausible reinforcement learning model recapitulated these results and showed that synaptic weights from sensory-to-decision neurons were preserved (i.e. intact stimulus-action knowledge) despite poor performance that was due to inadequate contextual scaling (i.e. impaired performance). Our results suggest that the amnestic phenotype is transient, contextual, and endogenously reversible, with the underlying neural circuits retaining the underlying stimulus-action associations. Thus, memory deficits commonly observed in amnestic mouse models, and potentially at early stages of dementia in humans, relate more to contextual drivers of performance rather than degeneration of the underlying memory traces.

Pre-existing visual responses in a projection-defined dopamine population explain individual learning trajectories

A key challenge of learning a new task is that the environment is high dimensional-there are many different sensory features and possible actions, with typically only a small reward-relevant subset. Although animals can learn to perform complex tasks that involve arbitrary associations between stimuli, actions, and rewards,1,2,3,4,5,6 a consistent and striking result across varied experimental paradigms is that in initially acquiring such tasks, large differences between individuals are apparent in the learning process.7,8,9,10,11,12 What neural mechanisms contribute to initial task acquisition, and why do some individuals learn a new task much more quickly than others? To address these questions, we recorded longitudinally from dopaminergic (DA) axon terminals in mice learning a visual decision-making task.7 Across striatum, DA responses tracked idiosyncratic and side-specific learning trajectories, consistent with widespread reward prediction error coding across DA terminals. However, even before any rewards were delivered, contralateral-side-specific visual responses were present in DA terminals, primarily in the dorsomedial striatum (DMS). These pre-existing responses predicted the extent of learning for contralateral stimuli. Moreover, activation of these terminals improved contralateral performance. Thus, the initial conditions of a projection-specific and feature-specific DA signal help explain individual learning trajectories. More broadly, this work suggests that functional heterogeneity across DA projections may serve to bias target regions toward learning about different subsets of task features, providing a potential mechanism to address the dimensionality of the initial task learning problem.

Reward timescale controls the rate of behavioural and dopaminergic learning

Learning the causes of rewards is necessary for survival. Thus, it is critical to understand the mechanisms of such a vital biological process. Cue-reward learning is controlled by mesolimbic dopamine and improves with spacing of cue-reward pairings. However, whether a mathematical rule governs such improvements in learning rate, and if so, whether a unifying mechanism captures this rule and dopamine dynamics during learning remain unknown. Here, we investigate the behavioral, algorithmic, and dopaminergic mechanisms governing cue-reward learning rate. Across a range of conditions in mice, we show a strong, mathematically proportional relationship between both behavioral and dopaminergic learning rates and the duration between rewards. Due to this relationship, removing up to 19 out of 20 cue-reward pairings over a fixed duration has no influence on overall learning. These findings are explained by a dopamine-based model of retrospective learning, thereby providing a unified account of the biological mechanisms of learning.

Rapid emergence of latent knowledge in the sensory cortex drives learning

Rapid learning confers significant advantages to animals in ecological environments. Despite the need for speed, animals appear to only slowly learn to associate rewarded actions with predictive cues1-4. This slow learning is thought to be supported by a gradual expansion of predictive cue representation in the sensory cortex2,5. However, evidence is growing that animals learn more rapidly than classical performance measures suggest6-8, challenging the prevailing model of sensory cortical plasticity. Here, we investigated the relationship between learning and sensory cortical representations. We trained mice on an auditory go/no-go task that dissociated the rapid acquisition of task contingencies (learning) from its slower expression (performance)7. Optogenetic silencing demonstrated that the auditory cortex (AC) drives both rapid learning and slower performance gains but becomes dispensable at expert. Rather than enhancement or expansion of cue representations9, two-photon calcium imaging of AC excitatory neurons throughout learning revealed two higher-order signals that were causal to learning and performance. First, a reward prediction (RP) signal emerged rapidly within tens of trials, was present after action-related errors only early in training, and faded at expert levels. Strikingly, silencing at the time of the RP signal impaired rapid learning, suggesting it serves an associative and teaching role. Second, a distinct cell ensemble encoded and controlled licking suppression that drove the slower performance improvements. These two ensembles were spatially clustered but uncoupled from underlying sensory representations, indicating a higher-order functional segregation within AC. Our results reveal that the sensory cortex manifests higher-order computations that separably drive rapid learning and slower performance improvements, reshaping our understanding of the fundamental role of the sensory cortex.

Investigating individual learning behaviour of dogs during a yes/no detection task

Detection dogs are frequently tested for their ability to detect a variety of targets. It is crucial to comprehend the processes for odour learning and the consequences of training on an expanding set of target scents on performance. To properly evaluate their ability to identify the target, the only true measure is the dogs' initial response to novel sources, since this excludes learning effects. In this study, we evaluated the individual learning processes of three detection dogs that were pre-trained to differentially respond to a faecal sample of a mare in oestrus (S+) and a faecal sample of the same mare in di-oestrus (S-). After reaching criterion during a test with known training samples, the dogs were tested for generalization to a novel source. Average responses to S+ and S- were calculated as a function of presentation sequence, and Signal Detection Theory was used to further analyse characteristic differences in learning. The results of this study suggest that the ability of individual scent detection dogs to learn within an olfactory discrimination test varies considerably. The information obtained in this study could be helpful for mitigation training. We show that through careful monitoring of individual learning processes, the strategy each dog followed becomes apparent: especially the observations on the dogs' responses to first encounters with novel sample sources. This provides us with more detailed information than the more traditional sensitivity and specificity measures and allows us to better predict the dog's capabilities.

Revealing abrupt transitions from goal-directed to habitual behavior

A fundamental tenet of animal behavior is that decision-making involves multiple 'controllers.' Initially, behavior is goal-directed, driven by desired outcomes, shifting later to habitual control, where cues trigger actions independent of motivational state. Clark Hull's question from 1943 still resonates today: "Is this transition abrupt, or is it gradual and progressive?"1 Despite a century-long belief in gradual transitions, this question remains unanswered2,3 as current methods cannot disambiguate goal-directed versus habitual control in real-time. Here, we introduce a novel 'volitional engagement' approach, motivating animals by palatability rather than biological need. Offering less palatable water in the home cage4,5 reduced motivation to 'work' for plain water in an auditory discrimination task when compared to water-restricted animals. Using quantitative behavior and computational modeling6, we found that palatability-driven animals learned to discriminate as quickly as water-restricted animals but exhibited state-like fluctuations when responding to the reward-predicting cue-reflecting goal-directed behavior. These fluctuations spontaneously and abruptly ceased after thousands of trials, with animals now always responding to the reward-predicting cue. In line with habitual control, post-transition behavior displayed motor automaticity, decreased error sensitivity (assessed via pupillary responses), and insensitivity to outcome devaluation. Bilateral lesions of the habit-related dorsolateral striatum7 blocked transitions to habitual behavior. Thus, 'volitional engagement' reveals spontaneous and abrupt transitions from goal-directed to habitual behavior, suggesting the involvement of a higher-level process that arbitrates between the two.

Inhibitory neurons control the consolidation of neural assemblies via adaptation to selective stimuli

Brain circuits display modular architecture at different scales of organization. Such neural assemblies are typically associated to functional specialization but the mechanisms leading to their emergence and consolidation still remain elusive. In this paper we investigate the role of inhibition in structuring new neural assemblies driven by the entrainment to various inputs. In particular, we focus on the role of partially synchronized dynamics for the creation and maintenance of structural modules in neural circuits by considering a network of excitatory and inhibitory [Formula: see text]-neurons with plastic Hebbian synapses. The learning process consists of an entrainment to temporally alternating stimuli that are applied to separate regions of the network. This entrainment leads to the emergence of modular structures. Contrary to common practice in artificial neural networks-where the acquired weights are typically frozen after the learning session-we allow for synaptic adaptation even after the learning phase. We find that the presence of inhibitory neurons in the network is crucial for the emergence and the post-learning consolidation of the modular structures. Indeed networks made of purely excitatory neurons or of neurons not respecting Dale's principle are unable to form or to maintain the modular architecture induced by the stimuli. We also demonstrate that the number of inhibitory neurons in the network is directly related to the maximal number of neural assemblies that can be consolidated, supporting the idea that inhibition has a direct impact on the memory capacity of the neural network.

Sound localization acuity of the common marmoset (Callithrix jacchus)

The common marmoset (Callithrix jacchus) is a small arboreal New World primate which has emerged as a promising model in auditory neuroscience. One potentially useful application of this model system is in the study of the neural mechanism underlying spatial hearing in primate species, as the marmosets need to localize sounds to orient their head to events of interest and identify their vocalizing conspecifics that are not visible. However, interpretation of neurophysiological data on sound localization requires an understanding of perceptual abilities, and the sound localization behavior of marmosets has not been well studied. The present experiment measured sound localization acuity using an operant conditioning procedure in which marmosets were trained to discriminate changes in sound location in the horizontal (azimuth) or vertical (elevation) dimension. Our results showed that the minimum audible angle (MAA) for horizontal and vertical discrimination was 13.17° and 12.53°, respectively, for 2 to 32 kHz Gaussian noise. Removing the monaural spectral cues tended to increase the horizontal localization acuity (11.31°). Marmosets have larger horizontal MAA (15.54°) in the rear than the front. Removing the high-frequency (> 26 kHz) region of the head-related transfer function (HRTF) affected vertical acuity mildly (15.76°), but removing the first notch (12-26 kHz) region of HRTF substantially reduced the vertical acuity (89.01°). In summary, our findings indicate that marmosets' spatial acuity is on par with other species of similar head size and field of best vision, and they do not appear to use monaural spectral cues for horizontal discrimination but rely heavily on first notch region of HRTF for vertical discrimination.

Vocalization categorization behavior explained by a feature-based auditory categorization model

Vocal animals produce multiple categories of calls with high between- and within-subject variability, over which listeners must generalize to accomplish call categorization. The behavioral strategies and neural mechanisms that support this ability to generalize are largely unexplored. We previously proposed a theoretical model that accomplished call categorization by detecting features of intermediate complexity that best contrasted each call category from all other categories. We further demonstrated that some neural responses in the primary auditory cortex were consistent with such a model. Here, we asked whether a feature-based model could predict call categorization behavior. We trained both the model and guinea pigs (GPs) on call categorization tasks using natural calls. We then tested categorization by the model and GPs using temporally and spectrally altered calls. Both the model and GPs were surprisingly resilient to temporal manipulations, but sensitive to moderate frequency shifts. Critically, the model predicted about 50% of the variance in GP behavior. By adopting different model training strategies and examining features that contributed to solving specific tasks, we could gain insight into possible strategies used by animals to categorize calls. Our results validate a model that uses the detection of intermediate-complexity contrastive features to accomplish call categorization.