Learning, fast and slow

Rapid emergence of latent knowledge in the sensory cortex drives learning

Rapid learning confers significant advantages on animals in ecological environments. Despite the need for speed, animals appear to only slowly learn to associate rewarded actions with predictive cues1-4. This slow learning is thought to be supported by gradual changes to cue representation in the sensory cortex2,5. However, evidence is growing that animals learn more rapidly than classical performance measures suggest6,7, challenging the prevailing model of sensory cortical plasticity. Here we investigated the relationship between learning and sensory cortical representations. We trained mice on an auditory go/no-go task that dissociated the rapid acquisition of task contingencies (learning) from its slower expression (performance)7. Optogenetic silencing demonstrated that the auditory cortex drives both rapid learning and slower performance gains but becomes dispensable once mice achieve 'expert' performance. Instead of enhanced cue representations8, two-photon calcium imaging of auditory cortical neurons throughout learning revealed two higher-order signals that were causal to learning and performance. A reward-prediction signal emerged rapidly within tens of trials, was present after action-related errors early in training, and faded in expert mice. Silencing at the time of this signal impaired rapid learning, suggesting that it serves an associative role. A distinct cell ensemble encoded and controlled licking suppression that drove slower performance improvements. These ensembles were spatially clustered but uncoupled from sensory representations, indicating higher-order functional segregation within auditory cortex. Our results reveal that the sensory cortex manifests higher-order computations that separably drive rapid learning and slower performance improvements, reshaping our understanding of the fundamental role of the sensory cortex.

Integrating innate and learned behavior through brain circuits

Understanding how innate predispositions and learned experiences interact to shape behavior is a central question in systems neuroscience. Traditionally, innate behaviors, that is, those present without prior learning and governed by evolutionarily conserved neural circuits, have been studied separately from learned behaviors, which depend on experience and neural plasticity. This division has led to a compartmentalized view of behavior and neural circuit organization. Increasing evidence suggests that innate and learned behaviors are not independent, but rather deeply intertwined, with plasticity evident even in circuits classically considered 'innate'. In this opinion, we highlight examples across species that illustrate the dynamic interaction between these behavioral domains and discuss the implications for unifying theoretical and empirical frameworks. We argue that a more integrative approach, namely one that acknowledges the reciprocal influences of innate and learned processes, is essential for advancing our understanding of how neuronal activity drives complex behaviors.

Navigation strategies in Caenorhabditis elegans are differentially altered by learning

Learned olfactory-guided navigation is a powerful platform for studying how a brain generates goal-directed behaviors. However, the quantitative changes that occur in sensorimotor transformations and the underlying neural circuit substrates to generate such learning-dependent navigation is still unclear. Here we investigate learned sensorimotor processing for navigation in the nematode Caenorhabditis elegans by measuring and modeling experience-dependent odor and salt chemotaxis. We then explore the neural basis of learned odor navigation through perturbation experiments. We develop a novel statistical model to characterize how the worm employs two behavioral strategies: a biased random walk and weathervaning. We infer weights on these strategies and characterize sensorimotor kernels that govern them by fitting our model to the worm's time-varying navigation trajectories and precise sensory experiences. After olfactory learning, the fitted odor kernels reflect how appetitive and aversive trained worms up- and down-regulate both strategies, respectively. The model predicts an animal's past olfactory learning experience with  > 90% accuracy given finite observations, outperforming a classical chemotaxis metric. The model trained on natural odors further predicts the animals' learning-dependent response to optogenetically induced odor perception. Our measurements and model show that behavioral variability is altered by learning-trained worms exhibit less variable navigation than naive ones. Genetically disrupting individual interneuron classes downstream of an odor-sensing neuron reveals that learned navigation strategies are distributed in the network. Together, we present a flexible navigation algorithm that is supported by distributed neural computation in a compact brain.

Fast updating feedback from piriform cortex to the olfactory bulb relays multimodal identity and reward contingency signals during rule-reversal

While animals readily adjust their behavior to adapt to relevant changes in the environment, the neural pathways enabling these changes remain largely unknown. Here, using multiphoton imaging, we investigate whether feedback from the piriform cortex to the olfactory bulb supports such behavioral flexibility. To this end, we engage head-fixed male mice in a multimodal rule-reversal task guided by olfactory and auditory cues. Both odor and, surprisingly, the sound cues trigger responses in the cortical bulbar feedback axons which precede the behavioral report. Responses to the same sensory cue are strongly modulated upon changes in stimulus-reward contingency (rule-reversals). The re-shaping of individual bouton responses occurs within seconds of the rule-reversal events and is correlated with changes in behavior. Optogenetic perturbation of cortical feedback within the bulb disrupts the behavioral performance. Our results indicate that the piriform-to-olfactory bulb feedback axons carry stimulus identity and reward contingency signals which are rapidly re-formatted according to changes in the behavioral context.

ONIX: a unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge we developed ONIX, an open-source data acquisition system with high data throughput (2 GB s-1) and low closed-loop latencies (<1 ms) that uses a 0.3-mm thin tether to minimize behavioral impact. Head position and rotation are tracked in three dimensions and used to drive active commutation without torque measurements. ONIX can acquire data from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, three-dimensional trackers and other data sources. We performed uninterrupted, long (~7 h) neural recordings in mice as they traversed complex three-dimensional terrain, and multiday sleep-tracking recordings (~55 h). ONIX enabled exploration with similar mobility as nonimplanted animals, in contrast to conventional tethered systems, which have restricted movement. By combining long recordings with full mobility, our technology will enable progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

Hierarchical behavior control by a single class of interneurons

Animal behavior is organized into nested temporal patterns that span multiple timescales. This behavior hierarchy is believed to arise from a hierarchical neural architecture: Neurons near the top of the hierarchy are involved in planning, selecting, initiating, and maintaining motor programs, whereas those near the bottom of the hierarchy act in concert to produce fine spatiotemporal motor activity. In Caenorhabditis elegans, behavior on a long timescale emerges from ordered and flexible transitions between different behavioral states, such as forward, reversal, and turn. On a short timescale, different parts of the animal body coordinate fast rhythmic bending sequences to produce directional movements. Here, we show that Sublateral Anterior A (SAA), a class of interneurons that enable cross-communication between dorsal and ventral head motor neurons, play a dual role in shaping behavioral dynamics on different timescales. On a short timescale, SAA regulate and stabilize rhythmic bending activity during forward movements. On a long timescale, the same neurons suppress spontaneous reversals and facilitate reversal termination by inhibiting Ring Interneuron M (RIM), an integrating neuron that helps maintain a behavioral state. These results suggest that feedback from a lower-level cell assembly to a higher-level command center is essential for bridging behavioral dynamics at different levels.

Understanding the neural basis of natural intelligence

Understanding the neural basis of natural intelligence necessitates a paradigm shift: from strict reductionism toward embracing complexity and diversity. New tools and theories enable us to tackle this challenge, providing unprecedented access to neural dynamics and behavior across time, contexts, and species. Principles for intelligent behavior and learning in the natural world are now, more than ever, within reach.

Reward timescale controls the rate of behavioural and dopaminergic learning

Learning the causes of rewards is necessary for survival. Thus, it is critical to understand the mechanisms of such a vital biological process. Cue-reward learning is controlled by mesolimbic dopamine and improves with spacing of cue-reward pairings. However, whether a mathematical rule governs such improvements in learning rate, and if so, whether a unifying mechanism captures this rule and dopamine dynamics during learning remain unknown. Here, we investigate the behavioral, algorithmic, and dopaminergic mechanisms governing cue-reward learning rate. Across a range of conditions in mice, we show a strong, mathematically proportional relationship between both behavioral and dopaminergic learning rates and the duration between rewards. Due to this relationship, removing up to 19 out of 20 cue-reward pairings over a fixed duration has no influence on overall learning. These findings are explained by a dopamine-based model of retrospective learning, thereby providing a unified account of the biological mechanisms of learning.

Rapid emergence of latent knowledge in the sensory cortex drives learning

Rapid learning confers significant advantages to animals in ecological environments. Despite the need for speed, animals appear to only slowly learn to associate rewarded actions with predictive cues1-4. This slow learning is thought to be supported by a gradual expansion of predictive cue representation in the sensory cortex2,5. However, evidence is growing that animals learn more rapidly than classical performance measures suggest6-8, challenging the prevailing model of sensory cortical plasticity. Here, we investigated the relationship between learning and sensory cortical representations. We trained mice on an auditory go/no-go task that dissociated the rapid acquisition of task contingencies (learning) from its slower expression (performance)7. Optogenetic silencing demonstrated that the auditory cortex (AC) drives both rapid learning and slower performance gains but becomes dispensable at expert. Rather than enhancement or expansion of cue representations9, two-photon calcium imaging of AC excitatory neurons throughout learning revealed two higher-order signals that were causal to learning and performance. First, a reward prediction (RP) signal emerged rapidly within tens of trials, was present after action-related errors only early in training, and faded at expert levels. Strikingly, silencing at the time of the RP signal impaired rapid learning, suggesting it serves an associative and teaching role. Second, a distinct cell ensemble encoded and controlled licking suppression that drove the slower performance improvements. These two ensembles were spatially clustered but uncoupled from underlying sensory representations, indicating a higher-order functional segregation within AC. Our results reveal that the sensory cortex manifests higher-order computations that separably drive rapid learning and slower performance improvements, reshaping our understanding of the fundamental role of the sensory cortex.

Behavioral strategy shapes activation of the Vip-Sst disinhibitory circuit in visual cortex

In complex environments, animals can adopt diverse strategies to find rewards. How distinct strategies differentially engage brain circuits is not well understood. Here, we investigate this question, focusing on the cortical Vip-Sst disinhibitory circuit between vasoactive intestinal peptide-postive (Vip) interneurons and somatostatin-positive (Sst) interneurons. We characterize the behavioral strategies used by mice during a visual change detection task. Using a dynamic logistic regression model, we find that individual mice use mixtures of a visual comparison strategy and a statistical timing strategy. Separately, mice also have periods of task engagement and disengagement. Two-photon calcium imaging shows large strategy-dependent differences in neural activity in excitatory, Sst inhibitory, and Vip inhibitory cells in response to both image changes and image omissions. In contrast, task engagement has limited effects on neural population activity. We find that the diversity of neural correlates of strategy can be understood parsimoniously as the increased activation of the Vip-Sst disinhibitory circuit during the visual comparison strategy, which facilitates task-appropriate responses.

From innate to instructed: A new look at perceptual decision-making

Understanding how subjects perceive sensory stimuli in their environment and use this information to guide appropriate actions is a major challenge in neuroscience. To study perceptual decision-making in animals, researchers use tasks that either probe spontaneous responses to stimuli (often described as "naturalistic") or train animals to associate stimuli with experimenter-defined responses. Spontaneous decisions rely on animals' pre-existing knowledge, while trained tasks offer greater versatility, albeit often at the cost of extensive training. Here, we review emerging approaches to investigate perceptual decision-making using both spontaneous and trained behaviors, highlighting their strengths and limitations. Additionally, we propose how trained decision-making tasks could be improved to achieve faster learning and a more generalizable understanding of task rules.

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Behavior contains rich structure across many timescales, but there is a dearth of methods to identify relevant components, especially over the longer periods required for learning and decision-making. Inspired by the goals and techniques of genome-wide association studies, we present a data-driven method-the choice-wide behavioral association study: CBAS-that systematically identifies such behavioral features. CBAS uses a powerful, resampling-based, method of multiple comparisons correction to identify sequences of actions or choices that either differ significantly between groups or significantly correlate with a covariate of interest. We apply CBAS to different tasks and species (flies, rats, and humans) and find, in all instances, that it provides interpretable information about each behavioral task.

Task interference as a neuronal basis for the cost of cognitive flexibility

Humans and animals have an impressive ability to juggle multiple tasks in a constantly changing environment. This flexibility, however, leads to decreased performance under uncertain task conditions. Here, we combined monkey electrophysiology, human psychophysics, and artificial neural network modeling to investigate the neuronal mechanisms of this performance cost. We developed a behavioural paradigm to measure and influence participants' decision-making and perception in two distinct perceptual tasks. Our data revealed that both humans and monkeys, unlike an artificial neural network trained for the same tasks, make less accurate perceptual decisions when the task is uncertain. We generated a mechanistic hypothesis by comparing this neural network trained to produce correct choices with another network trained to replicate the participants' choices. We hypothesized, and confirmed with further behavioural, physiological, and causal experiments, that the cost of task flexibility comes from what we term task interference. Under uncertain conditions, interference between different tasks causes errors because it results in a stronger representation of irrelevant task features and entangled neuronal representations of different features. Our results suggest a tantalizing, general hypothesis: that cognitive capacity limitations, both in health and disease, stem from interference between neural representations of different stimuli, tasks, or memories.

Adaptive algorithms for shaping behavior

Dogs and laboratory mice are commonly trained to perform complex tasks by guiding them through a curriculum of simpler tasks ('shaping'). What are the principles behind effective shaping strategies? Here, we propose a machine learning framework for shaping animal behavior, where an autonomous teacher agent decides its student's task based on the student's transcript of successes and failures on previously assigned tasks. Using autonomous teachers that plan a curriculum in a common sequence learning task, we show that near-optimal shaping algorithms adaptively alternate between simpler and harder tasks to carefully balance reinforcement and extinction. Based on this intuition, we derive an adaptive shaping heuristic with minimal parameters, which we show is near-optimal on the sequence learning task and robustly trains deep reinforcement learning agents on navigation tasks that involve sparse, delayed rewards. Extensions to continuous curricula are explored. Our work provides a starting point towards a general computational framework for shaping animal behavior.

Fast updating feedback from piriform cortex to the olfactory bulb relays multimodal reward contingency signals during rule-reversal

While animals readily adjust their behavior to adapt to relevant changes in the environment, the neural pathways enabling these changes remain largely unknown. Here, using multiphoton imaging, we investigated whether feedback from the piriform cortex to the olfactory bulb supports such behavioral flexibility. To this end, we engaged head-fixed mice in a multimodal rule-reversal task guided by olfactory and auditory cues. Both odor and, surprisingly, the sound cues triggered cortical bulbar feedback responses which preceded the behavioral report. Responses to the same sensory cue were strongly modulated upon changes in stimulus-reward contingency (rule reversals). The re-shaping of individual bouton responses occurred within seconds of the rule-reversal events and was correlated with changes in the behavior. Optogenetic perturbation of cortical feedback within the bulb disrupted the behavioral performance. Our results indicate that the piriform-to-olfactory bulb feedback carries reward contingency signals and is rapidly re-formatted according to changes in the behavioral context.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

Competitive and cooperative games for probing the neural basis of social decision-making in animals

In a social environment, it is essential for animals to consider the behavior of others when making decisions. To quantitatively assess such social decisions, games offer unique advantages. Games may have competitive and cooperative components, modeling situations with antagonistic and shared objectives between players. Games can be analyzed by mathematical frameworks, including game theory and reinforcement learning, such that an animal's choice behavior can be compared against the optimal strategy. However, so far games have been underappreciated in neuroscience research, particularly for rodent studies. In this review, we survey the varieties of competitive and cooperative games that have been tested, contrasting strategies employed by non-human primates and birds with rodents. We provide examples of how games can be used to uncover neural mechanisms and explore species-specific behavioral differences. We assess critically the limitations of current paradigms and propose improvements. Together, the synthesis of current literature highlights the advantages of using games to probe the neural basis of social decisions for neuroscience studies.

Behavior is movement only but how to interpret it? Problems and pitfalls in translational neuroscience-a 40-year experience

Translational research in behavioral neuroscience seeks causes and remedies for human mental health problems in animals, following leads imposed by clinical research in psychiatry. This endeavor faces several problems because scientists must read and interpret animal movements to represent human perceptions, mood, and memory processes. Yet, it is still not known how mammalian brains bundle all these processes into a highly compressed motor output in the brain stem and spinal cord, but without that knowledge, translational research remains aimless. Based on some four decades of experience in the field, the article identifies sources of interpretation problems and illustrates typical translational pitfalls. (1) The sensory world of mice is different. Smell, hearing, and tactile whisker sensations dominate in rodents, while visual input is comparatively small. In humans, the relations are reversed. (2) Mouse and human brains are equated inappropriately: the association cortex makes up a large portion of the human neocortex, while it is relatively small in rodents. The predominant associative cortex in rodents is the hippocampus itself, orchestrating chiefly inputs from secondary sensorimotor areas and generating species-typical motor patterns that are not easily reconciled with putative human hippocampal functions. (3) Translational interpretation of studies of memory or emotionality often neglects the ecology of mice, an extremely small species surviving by freezing or flight reactions that do not need much cognitive processing. (4) Further misinterpretations arise from confounding neuronal properties with system properties, and from rigid mechanistic thinking unaware that many experimentally induced changes in the brain do partially reflect unpredictable compensatory plasticity. (5) Based on observing hippocampal lesion effects in mice indoors and outdoors, the article offers a simplistic general model of hippocampal functions in relation to hypothalamic input and output, placing hypothalamus and the supraspinal motor system at the top of a cerebral hierarchy. (6) Many translational problems could be avoided by inclusion of simple species-typical behaviors as end-points comparable to human cognitive or executive processing, and to rely more on artificial intelligence for recognizing patterns not classifiable by traditional psychological concepts.