Continuous Whole-Body 3D Kinematic Recordings across the Rodent Behavioral Repertoire

Deciphering social traits and pathophysiological conditions from natural behaviors in common marmosets

Nonhuman primates (NHPs) are indispensable animal models by virtue of the continuity of behavioral repertoires across primates, including humans. However, behavioral assessment at the laboratory level has so far been limited. Employing the application of three-dimensional (3D) pose estimation and the optimal integration of subsequent analytic methodologies, we demonstrate that our artificial intelligence (AI)-based approach has successfully deciphered the ethological, cognitive, and pathological traits of common marmosets from their natural behaviors. By applying multiple deep neural networks trained with large-scale datasets, we established an evaluation system that could reconstruct and estimate the 3D poses of the marmosets, a small NHP that is suitable for analyzing complex natural behaviors in laboratory setups. We further developed downstream analytic methodologies to quantify a variety of behavioral parameters beyond motion kinematics. We revealed the distinct parental roles of male and female marmosets through automated detections of food-sharing behaviors using a spatial-temporal filter on 3D poses. Employing a recurrent neural network to analyze 3D pose time series data during social interactions, we additionally discovered that marmosets adjusted their behaviors based on others' internal state, which is not directly observable but can be inferred from the sequence of others' actions. Moreover, a fully unsupervised approach enabled us to detect progressively appearing symptomatic behaviors over a year in a Parkinson's disease model. The high-throughput and versatile nature of an AI-driven approach to analyze natural behaviors will open a new avenue for neuroscience research dealing with big-data analyses of social and pathophysiological behaviors in NHPs.

Spiking activity in the visual thalamus is coupled to pupil dynamics across temporal scales

The processing of sensory information, even at early stages, is influenced by the internal state of the animal. Internal states, such as arousal, are often characterized by relating neural activity to a single "level" of arousal, defined by a behavioral indicator such as pupil size. In this study, we expand the understanding of arousal-related modulations in sensory systems by uncovering multiple timescales of pupil dynamics and their relationship to neural activity. Specifically, we observed a robust coupling between spiking activity in the mouse dorsolateral geniculate nucleus (dLGN) of the thalamus and pupil dynamics across timescales spanning a few seconds to several minutes. Throughout all these timescales, 2 distinct spiking modes-individual tonic spikes and tightly clustered bursts of spikes-preferred opposite phases of pupil dynamics. This multi-scale coupling reveals modulations distinct from those captured by pupil size per se, locomotion, and eye movements. Furthermore, coupling persisted even during viewing of a naturalistic movie, where it contributed to differences in the encoding of visual information. We conclude that dLGN spiking activity is under the simultaneous influence of multiple arousal-related processes associated with pupil dynamics occurring over a broad range of timescales.

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

Separating cognitive and motor processes in the behaving mouse

The cognitive processes supporting complex animal behavior are closely associated with ubiquitous movements responsible for our posture, facial expressions, ability to actively sample our sensory environments, and other critical processes. These movements are strongly related to neural activity across much of the brain and are often highly correlated with ongoing cognitive processes, making it challenging to dissociate the neural dynamics that support cognitive processes from those supporting related movements. In such cases, a critical issue is whether cognitive processes are separable from related movements, or if they are driven by common neural mechanisms. Here, we demonstrate how the separability of cognitive and motor processes can be assessed, and, when separable, how the neural dynamics associated with each component can be isolated. We establish a novel two-context behavioral task in mice that involves multiple cognitive processes and show that commonly observed dynamics taken to support cognitive processes are strongly contaminated by movements. When cognitive and motor components are isolated using a novel approach for subspace decomposition, we find that they exhibit distinct dynamical trajectories. Further, properly accounting for movement revealed that largely separate populations of cells encode cognitive and motor variables, in contrast to the 'mixed selectivity' often reported. Accurately isolating the dynamics associated with particular cognitive and motor processes will be essential for developing conceptual and computational models of neural circuit function and evaluating the function of the cell types of which neural circuits are composed.

Long Timescales, Individual Differences, and Scale Invariance in Animal Behavior

The explosion of data on animal behavior in more natural contexts highlights the fact that these behaviors exhibit correlations across many timescales. However, there are major challenges in analyzing these data: records of behavior in single animals have fewer independent samples than one might expect. In pooling data from multiple animals, individual differences can mimic long-ranged temporal correlations; conversely, long-ranged correlations can lead to an overestimate of individual differences. We suggest an analysis scheme that addresses these problems directly, apply this approach to data on the spontaneous behavior of walking flies, and find evidence for scale-invariant correlations over nearly three decades in time, from seconds to one hour. Three different measures of correlation are consistent with a single underlying scaling field of dimension Δ=0.180±0.005.

Facemap: a framework for modeling neural activity based on orofacial tracking

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder for predicting neural activity. Our algorithm for tracking mouse orofacial behaviors was more accurate than existing pose estimation tools, while the processing speed was several times faster, making it a powerful tool for real-time experimental interventions. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used the keypoints as inputs to a deep neural network which predicts the activity of ~50,000 simultaneously-recorded neurons and, in visual cortex, we doubled the amount of explained variance compared to previous methods. Using this model, we found that the neuronal activity clusters that were well predicted from behavior were more spatially spread out across cortex. We also found that the deep behavioral features from the model had stereotypical, sequential dynamics that were not reversible in time. In summary, Facemap provides a stepping stone toward understanding the function of the brain-wide neural signals and their relation to behavior.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms have revolutionized the analysis of animal behavior, enabling investigators to flexibly quantify behavioral dynamics from conventional video recordings obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into the modules out of which behavior is organized. This challenge is particularly acute because keypoint data is susceptible to high frequency jitter that clustering algorithms can mistake for transitions between behavioral modules. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ("syllables") from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to effectively identify syllables whose boundaries correspond to natural sub-second discontinuities inherent to mouse behavior. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior, and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq therefore renders behavioral syllables and grammar accessible to the many researchers who use standard video to capture animal behavior.

High-THC Cannabis Smoke Impairs Incidental Memory Capacity in Spontaneous Tests of Novelty Preference for Objects and Odors in Male Rats

Working memory is an executive function that orchestrates the use of limited amounts of information, referred to as working memory capacity, in cognitive functions. Cannabis exposure impairs working memory in humans; however, it is unclear whether Cannabis facilitates or impairs rodent working memory and working memory capacity. The conflicting literature in rodent models may be at least partly because of the use of drug exposure paradigms that do not closely mirror patterns of human Cannabis use. Here, we used an incidental memory capacity paradigm where a novelty preference is assessed after a short delay in spontaneous recognition-based tests. Either object or odor-based stimuli were used in test variations with sets of identical [identical stimuli test (IST)] and different [different stimuli test (DST)] stimuli (three or six) for low-memory and high-memory loads, respectively. Additionally, we developed a human-machine hybrid behavioral quantification approach which supplements stopwatch-based scoring with supervised machine learning-based classification. After validating the spontaneous IST and DST in male rats, 6-item test versions with the hybrid quantification method were used to evaluate the impact of acute exposure to high-Δ9-tetrahydrocannabinol (THC) or high-CBD Cannabis smoke on novelty preference. Under control conditions, male rats showed novelty preference in all test variations. We found that high-THC, but not high-CBD, Cannabis smoke exposure impaired novelty preference for objects under a high-memory load. Odor-based recognition deficits were seen under both low-memory and high-memory loads only following high-THC smoke exposure. Ultimately, these data show that Cannabis smoke exposure impacts incidental memory capacity of male rats in a memory load-dependent, and stimuli-specific manner.

Artificial intelligence in epilepsy phenotyping

Artificial intelligence (AI) allows data analysis and integration at an unprecedented granularity and scale. Here we review the technological advances, challenges, and future perspectives of using AI for electro-clinical phenotyping of animal models and patients with epilepsy. In translational research, AI models accurately identify behavioral states in animal models of epilepsy, allowing identification of correlations between neural activity and interictal and ictal behavior. Clinical applications of AI-based automated and semi-automated analysis of audio and video recordings of people with epilepsy, allow significant data reduction and reliable detection and classification of major motor seizures. AI models can accurately identify electrographic biomarkers of epilepsy, such as spikes, high-frequency oscillations, and seizure patterns. Integrating AI analysis of electroencephalographic, clinical, and behavioral data will contribute to optimizing therapy for patients with epilepsy.

Deep Phenotyping of Sleep in Drosophila

Sleep is an evolutionarily conserved behavior, whose function is unknown. Here, we present a method for deep phenotyping of sleep in Drosophila, consisting of a high-resolution video imaging system, coupled with closed-loop laser perturbation to measure arousal threshold. To quantify sleep-associated microbehaviors, we trained a deep-learning network to annotate body parts in freely moving flies and developed a semi-supervised computational pipeline to classify behaviors. Quiescent flies exhibit a rich repertoire of microbehaviors, including proboscis pumping (PP) and haltere switches, which vary dynamically across the night. Using this system, we characterized the effects of optogenetically activating two putative sleep circuits. These data reveal that activating dFB neurons produces micromovements, inconsistent with sleep, while activating R5 neurons triggers PP followed by behavioral quiescence. Our findings suggest that sleep in Drosophila is polyphasic with different stages and set the stage for a rigorous analysis of sleep and other behaviors in this species.

Standardized 3D test object for multi-camera calibration during animal pose capture

Accurate capture of animal behavior and posture requires the use of multiple cameras to reconstruct three-dimensional (3D) representations. Typically, a paper ChArUco (or checker) board works well for correcting distortion and calibrating for 3D reconstruction in stereo vision. However, measuring the error in two-dimensional (2D) is also prone to bias related to the placement of the 2D board in 3D. We proposed a procedure as a visual way of validating camera placement, and it also can provide some guidance about the positioning of cameras and potential advantages of using multiple cameras. We propose the use of a 3D printable test object for validating multi-camera surround-view calibration in small animal video capture arenas. The proposed 3D printed object has no bias to a particular dimension and is designed to minimize occlusions. The use of the calibrated test object provided an estimate of 3D reconstruction accuracy. The approach reveals that for complex specimens such as mice, some view angles will be more important for accurate capture of keypoints. Our method ensures accurate 3D camera calibration for surround image capture of laboratory mice and other specimens.

Hierarchical action encoding in prefrontal cortex of freely moving macaques

Our natural behavioral repertoires include coordinated actions of characteristic types. To better understand how neural activity relates to the expression of actions and action switches, we studied macaques performing a freely moving foraging task in an open environment. We developed a novel analysis pipeline that can identify meaningful units of behavior, corresponding to recognizable actions such as sitting, walking, jumping, and climbing. On the basis of transition probabilities between these actions, we found that behavior is organized in a modular and hierarchical fashion. We found that, after regressing out many potential confounders, actions are associated with specific patterns of firing in each of six prefrontal brain regions and that, overall, encoding of action category is progressively stronger in more dorsal and more caudal prefrontal regions. Together, these results establish a link between selection of units of primate behavior on one hand and neuronal activity in prefrontal regions on the other.

Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models

Deep learning-based markerless tracking has revolutionized studies of animal behavior. Yet the generalizability of trained models tends to be limited, as new training data typically needs to be generated manually for each setup or visual environment. With each model trained from scratch, researchers track distinct landmarks and analyze the resulting kinematic data in idiosyncratic ways. Moreover, due to inherent limitations in manual annotation, only a sparse set of landmarks are typically labeled. To address these issues, we developed an approach, which we term GlowTrack, for generating orders of magnitude more training data, enabling models that generalize across experimental contexts. We describe: a) a high-throughput approach for producing hidden labels using fluorescent markers; b) a multi-camera, multi-light setup for simulating diverse visual conditions; and c) a technique for labeling many landmarks in parallel, enabling dense tracking. These advances lay a foundation for standardized behavioral pipelines and more complete scrutiny of movement.

Challenges and advanced concepts for the assessment of learning and memory function in mice

The mechanisms underlying the formation and retrieval of memories are still an active area of research and discussion. Manifold models have been proposed and refined over the years, with most assuming a dichotomy between memory processes involving non-conscious and conscious mechanisms. Despite our incomplete understanding of the underlying mechanisms, tests of memory and learning count among the most performed behavioral experiments. Here, we will discuss available protocols for testing learning and memory using the example of the most prevalent animal species in research, the laboratory mouse. A wide range of protocols has been developed in mice to test, e.g., object recognition, spatial learning, procedural memory, sequential problem solving, operant- and fear conditioning, and social recognition. Those assays are carried out with individual subjects in apparatuses such as arenas and mazes, which allow for a high degree of standardization across laboratories and straightforward data interpretation but are not without caveats and limitations. In animal research, there is growing concern about the translatability of study results and animal welfare, leading to novel approaches beyond established protocols. Here, we present some of the more recent developments and more advanced concepts in learning and memory testing, such as multi-step sequential lockboxes, assays involving groups of animals, as well as home cage-based assays supported by automated tracking solutions; and weight their potential and limitations against those of established paradigms. Shifting the focus of learning tests from the classical experimental chamber to settings which are more natural for rodents comes with a new set of challenges for behavioral researchers, but also offers the opportunity to understand memory formation and retrieval in a more conclusive way than has been attainable with conventional test protocols. We predict and embrace an increase in studies relying on methods involving a higher degree of automatization, more naturalistic- and home cage-based experimental setting as well as more integrated learning tasks in the future. We are confident these trends are suited to alleviate the burden on animal subjects and improve study designs in memory research.

Postural analysis reveals persistent changes in paper wasp foundress behavioral state after conspecific challenge

Vigilant animals detect and respond to threats in the environment, often changing posture and movement patterns. Vigilance is modulated not only by predators but also by conspecific threats. In social animals, precisely how conspecific threats alter vigilance behavior over time is relevant to long-standing hypotheses about social plasticity. We report persistent effects of a simulated conspecific challenge on behavior of wild northern paper wasp foundresses, Polistes fuscatus. During the founding phase of the colony cycle, conspecific wasps can usurp nests from the resident foundress, representing a severe threat. We used automated tracking to monitor the movement and posture of P. fuscatus foundresses in response to simulated intrusions. Wasps displayed increased movement, greater bilateral wing extension, and reduced antennal separation after the threat was removed. These changes were not observed after presentation with a wooden dowel. By rapidly adjusting individual behavior after fending off an intruder, paper wasp foundresses might invest in surveillance of potential threats, even when such threats are no longer immediately present. The prolonged vigilance-like behavioral state observed here is relevant to plasticity of social recognition processes in paper wasps.

Modeling SHANK3-associated autism spectrum disorder in Beagle dogs via CRISPR/Cas9 gene editing

Despite intensive studies in modeling neuropsychiatric disorders especially autism spectrum disorder (ASD) in animals, many challenges remain. Genetic mutant mice have contributed substantially to the current understanding of the molecular and neural circuit mechanisms underlying ASD. However, the translational value of ASD mouse models in preclinical studies is limited to certain aspects of the disease due to the apparent differences in brain and behavior between rodents and humans. Non-human primates have been used to model ASD in recent years. However, a low reproduction rate due to a long reproductive cycle and a single birth per pregnancy, and an extremely high cost prohibit a wide use of them in preclinical studies. Canine model is an appealing alternative because of its complex and effective dog-human social interactions. In contrast to non-human primates, dog has comparable drug metabolism as humans and a high reproduction rate. In this study, we aimed to model ASD in experimental dogs by manipulating the Shank3 gene as SHANK3 mutations are one of most replicated genetic defects identified from ASD patients. Using CRISPR/Cas9 gene editing, we successfully generated and characterized multiple lines of Beagle Shank3 (bShank3) mutants that have been propagated for a few generations. We developed and validated a battery of behavioral assays that can be used in controlled experimental setting for mutant dogs. bShank3 mutants exhibited distinct and robust social behavior deficits including social withdrawal and reduced social interactions with humans, and heightened anxiety in different experimental settings (n = 27 for wild-type controls and n = 44 for mutants). We demonstrate the feasibility of producing a large number of mutant animals in a reasonable time frame. The robust and unique behavioral findings support the validity and value of a canine model to investigate the pathophysiology and develop treatments for ASD and potentially other psychiatric disorders.

An automated, low-latency environment for studying the neural basis of behavior in freely moving rats

BACKGROUND

Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes).

RESULTS

To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle.

CONCLUSIONS

The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.

WormSwin: Instance segmentation of C. elegans using vision transformer

The possibility to extract motion of a single organism from video recordings at a large-scale provides means for the quantitative study of its behavior, both individual and collective. This task is particularly difficult for organisms that interact with one another, overlap, and occlude parts of their bodies in the recording. Here we propose WormSwin-an approach to extract single animal postures of Caenorhabditis elegans (C. elegans) from recordings of many organisms in a single microscope well. Based on transformer neural network architecture our method segments individual worms across a range of videos and images generated in different labs. Our solutions offers accuracy of 0.990 average precision ([Formula: see text]) and comparable results on the benchmark image dataset BBBC010. Finally, it allows to segment challenging overlapping postures of mating worms with an accuracy sufficient to track the organisms with a simple tracking heuristic. An accurate and efficient method for C. elegans segmentation opens up new opportunities for studying of its behaviors previously inaccessible due to the difficulty in the worm extraction from the video frames.

Behavioral decomposition reveals rich encoding structure employed across neocortex in rats

The cortical population code is pervaded by activity patterns evoked by movement, but it remains largely unknown how such signals relate to natural behavior or how they might support processing in sensory cortices where they have been observed. To address this we compared high-density neural recordings across four cortical regions (visual, auditory, somatosensory, motor) in relation to sensory modulation, posture, movement, and ethograms of freely foraging male rats. Momentary actions, such as rearing or turning, were represented ubiquitously and could be decoded from all sampled structures. However, more elementary and continuous features, such as pose and movement, followed region-specific organization, with neurons in visual and auditory cortices preferentially encoding mutually distinct head-orienting features in world-referenced coordinates, and somatosensory and motor cortices principally encoding the trunk and head in egocentric coordinates. The tuning properties of synaptically coupled cells also exhibited connection patterns suggestive of area-specific uses of pose and movement signals, particularly in visual and auditory regions. Together, our results indicate that ongoing behavior is encoded at multiple levels throughout the dorsal cortex, and that low-level features are differentially utilized by different regions to serve locally relevant computations.

Ventromedial prefrontal neurons represent self-states shaped by vicarious fear in male mice

Perception of fear induced by others in danger elicits complex vicarious fear responses and behavioral outputs. In rodents, observing a conspecific receive aversive stimuli leads to escape and freezing behavior. It remains unclear how these behavioral self-states in response to others in fear are neurophysiologically represented. Here, we assess such representations in the ventromedial prefrontal cortex (vmPFC), an essential site for empathy, in an observational fear (OF) paradigm in male mice. We classify the observer mouse's stereotypic behaviors during OF using a machine-learning approach. Optogenetic inhibition of the vmPFC specifically disrupts OF-induced escape behavior. In vivo Ca²⁺ imaging reveals that vmPFC neural populations represent intermingled information of other- and self-states. Distinct subpopulations are activated and suppressed by others' fear responses, simultaneously representing self-freezing states. This mixed selectivity requires inputs from the anterior cingulate cortex and the basolateral amygdala to regulate OF-induced escape behavior.

Hierarchical organization of rhesus macaque behavior

Primatologists, psychologists and neuroscientists have long hypothesized that primate behavior is highly structured. However, delineating that structure has been impossible due to the difficulties of precision behavioral tracking. Here we analyzed a dataset consisting of continuous measures of the 3D position of two male rhesus macaques (Macaca mulatta) performing three different tasks in a large unrestrained environment over several hours. Using an unsupervised embedding approach on the tracked joints, we identified commonly repeated pose patterns, which we call postures. We found that macaques' behavior is characterized by 49 distinct postures, lasting an average of 0.6 seconds. We found evidence that behavior is hierarchically organized, in that transitions between poses tend to occur within larger modules, which correspond to identifiable actions; these actions are further organized hierarchically. Our behavioral decomposition allows us to identify universal (cross-individual and cross-task) and unique (specific to each individual and task) principles of behavior. These results demonstrate the hierarchical nature of primate behavior, provide a method for the automated ethogramming of primate behavior, and provide important constraints on neural models of pose generation.

Noninvasive Tracking of Every Individual in Unmarked Mouse Groups Using Multi-Camera Fusion and Deep Learning

Accurate and efficient methods for identifying and tracking each animal in a group are needed to study complex behaviors and social interactions. Traditional tracking methods (e.g., marking each animal with dye or surgically implanting microchips) can be invasive and may have an impact on the social behavior being measured. To overcome these shortcomings, video-based methods for tracking unmarked animals, such as fruit flies and zebrafish, have been developed. However, tracking individual mice in a group remains a challenging problem because of their flexible body and complicated interaction patterns. In this study, we report the development of a multi-object tracker for mice that uses the Faster region-based convolutional neural network (R-CNN) deep learning algorithm with geometric transformations in combination with multi-camera/multi-image fusion technology. The system successfully tracked every individual in groups of unmarked mice and was applied to investigate chasing behavior. The proposed system constitutes a step forward in the noninvasive tracking of individual mice engaged in social behavior.

LabGym: Quantification of user-defined animal behaviors using learning-based holistic assessment

Quantifying animal behavior is important for biological research. Identifying behaviors is the prerequisite of quantifying them. Current computational tools for behavioral quantification typically use high-level properties such as body poses to identify the behaviors, which constrains the information available for a holistic assessment. Here we report LabGym, an open-source computational tool for quantifying animal behaviors without this constraint. In LabGym, we introduce "pattern image" to represent the animal's motion pattern, in addition to "animation" that shows all spatiotemporal details of a behavior. These two pieces of information are assessed holistically by customizable deep neural networks for accurate behavior identifications. The quantitative measurements of each behavior are then calculated. LabGym is applicable for experiments involving multiple animals, requires little programming knowledge to use, and provides visualizations of behavioral datasets. We demonstrate its efficacy in capturing subtle behavioral changes in diverse animal species.

Catalyzing next-generation Artificial Intelligence through NeuroAI

Neuroscience has long been an essential driver of progress in artificial intelligence (AI). We propose that to accelerate progress in AI, we must invest in fundamental research in NeuroAI. A core component of this is the embodied Turing test, which challenges AI animal models to interact with the sensorimotor world at skill levels akin to their living counterparts. The embodied Turing test shifts the focus from those capabilities like game playing and language that are especially well-developed or uniquely human to those capabilities - inherited from over 500 million years of evolution - that are shared with all animals. Building models that can pass the embodied Turing test will provide a roadmap for the next generation of AI.

Tracking motion kinematics and tremor with intrinsic oscillatory property of instrumental mechanics

Tracking kinematic details of motor behaviors is a foundation to study the neuronal mechanism and biology of motor control. However, most of the physiological motor behaviors and movement disorders, such as gait, balance, tremor, dystonia, and myoclonus, are highly dependent on the overall momentum of the whole-body movements. Therefore, tracking the targeted movement and overall momentum simultaneously is critical for motor control research, but it remains an unmet need. Here, we introduce the intrinsic oscillatory property (IOP), a fundamental mechanical principle of physics, as a method for motion tracking in a force plate. The overall kinetic energy of animal motions can be transformed into the oscillatory amplitudes at the designed IOP frequency of the force plate, while the target movement has its own frequency features and can be tracked simultaneously. Using action tremor as an example, we reported that force plate-based IOP approach has superior performance and reliability in detecting both tremor severity and tremor frequency, showing a lower level of coefficient of variation (CV) compared with video- and accelerometer-based motion tracking methods and their combination. Under the locomotor suppression effect of medications, therapeutic effects on tremor severity can still be quantified by dynamically adjusting the overall locomotor activity detected by IOP. We further validated IOP method in optogenetic-induced movements and natural movements, confirming that IOP can represent the intensity of general rhythmic and nonrhythmic movements, thus it can be generalized as a common approach to study kinematics.

Walking humans and running mice: perception and neural encoding of optic flow during self-motion

Locomotion produces full-field optic flow that often dominates the visual motion inputs to an observer. The perception of optic flow is in turn important for animals to guide their heading and interact with moving objects. Understanding how locomotion influences optic flow processing and perception is therefore essential to understand how animals successfully interact with their environment. Here, we review research investigating how perception and neural encoding of optic flow are altered during self-motion, focusing on locomotion. Self-motion has been found to influence estimation and sensitivity for optic flow speed and direction. Nonvisual self-motion signals also increase compensation for self-driven optic flow when parsing the visual motion of moving objects. The integration of visual and nonvisual self-motion signals largely follows principles of Bayesian inference and can improve the precision and accuracy of self-motion perception. The calibration of visual and nonvisual self-motion signals is dynamic, reflecting the changing visuomotor contingencies across different environmental contexts. Throughout this review, we consider experimental research using humans, non-human primates and mice. We highlight experimental challenges and opportunities afforded by each of these species and draw parallels between experimental findings. These findings reveal a profound influence of locomotion on optic flow processing and perception across species. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Three-dimensional unsupervised probabilistic pose reconstruction (3D-UPPER) for freely moving animals

A key step in understanding animal behaviour relies in the ability to quantify poses and movements. Methods to track body landmarks in 2D have made great progress over the last few years but accurate 3D reconstruction of freely moving animals still represents a challenge. To address this challenge here we develop the 3D-UPPER algorithm, which is fully automated, requires no a priori knowledge of the properties of the body and can also be applied to 2D data. We find that 3D-UPPER reduces by [Formula: see text] fold the error in 3D reconstruction of mouse body during freely moving behaviour compared with the traditional triangulation of 2D data. To achieve that, 3D-UPPER performs an unsupervised estimation of a Statistical Shape Model (SSM) and uses this model to constrain the viable 3D coordinates. We show, by using simulated data, that our SSM estimator is robust even in datasets containing up to 50% of poses with outliers and/or missing data. In simulated and real data SSM estimation converges rapidly, capturing behaviourally relevant changes in body shape associated with exploratory behaviours (e.g. with rearing and changes in body orientation). Altogether 3D-UPPER represents a simple tool to minimise errors in 3D reconstruction while capturing meaningful behavioural parameters.

Long-term stability of single neuron activity in the motor system

How an established behavior is retained and consistently produced by a nervous system in constant flux remains a mystery. One possible solution to ensure long-term stability in motor output is to fix the activity patterns of single neurons in the relevant circuits. Alternatively, activity in single cells could drift over time provided that the population dynamics are constrained to produce the same behavior. To arbitrate between these possibilities, we recorded single-unit activity in motor cortex and striatum continuously for several weeks as rats performed stereotyped motor behaviors-both learned and innate. We found long-term stability in single neuron activity patterns across both brain regions. A small amount of drift in neural activity, observed over weeks of recording, could be explained by concomitant changes in task-irrelevant aspects of the behavior. These results suggest that long-term stable behaviors are generated by single neuron activity patterns that are themselves highly stable.

Developmental trajectory of episodic-like memory in rats

Introduction

Episodic memory formation requires the binding of multiple associations to a coherent episodic representation, with rich detail of times, places, and contextual information. During postnatal development, the ability to recall episodic memories emerges later than other types of memory such as object recognition. However, the precise developmental trajectory of episodic memory, from weaning to adulthood has not yet been established in rats. Spontaneous object exploration tasks do not require training, and allow repeated testing of subjects, provided novel objects are used on each trial. Therefore, these tasks are ideally suited for the study of the ontogeny of episodic memory and its constituents (e.g., object, spatial, and contextual memory).

Methods

In the present study, we used four spontaneous short-term object exploration tasks over two days: object (OR), object-context (OCR), object-place (OPR), and object-place-context (OPCR) recognition to characterise the ontogeny of episodic-like memory and its components in three commonly used outbred rat strains (Lister Hooded, Long Evans Hooded, and Sprague Dawley).

Results

In longitudinal studies starting at 3-4 weeks of age, we observed that short term memory for objects was already present at the earliest time point we tested, indicating that it is established before the end of the third week of life (consistent with several other reports). Object-context memory developed during the fifth week of life, while both object-in-place and the episodic-like object-place-context memory developed around the seventh postnatal week. To control for the effects of previous experience in the development of associative memory, we confirmed these developmental trajectories using a cross-sectional protocol.

Discussion

Our work provides robust evidence for different developmental trajectories of recognition memory in rats depending on the content and/or complexity of the associations and emphasises the utility of spontaneous object exploration tasks to assess the ontogeny of memory systems with high temporal resolution.

A virtual library for behavioral performance in standard conditions-rodent spontaneous activity in an open field during repeated testing and after treatment with drugs or brain lesions

Background

Beyond their specific experiment, video records of behavior have future value—for example, as inputs for new experiments or for yet unknown types of analysis of behavior—similar to tissue or blood sample banks in life sciences where clinically derived or otherwise well-described experimental samples are stored to be available for some unknown potential future purpose.

Findings

Research using an animal model of obsessive-compulsive disorder employed a standardized paradigm where the behavior of rats in a large open field was video recorded for 55 minutes on each test. From 43 experiments, there are 19,976 such trials that amount to over 2 years of continuous recording. In addition to videos, there are 2 video-derived raw data objects: XY locomotion coordinates and plots of animal trajectory. To motivate future use, the 3 raw data objects are annotated with a general schema—one that abstracts the data records from their particular experiment while providing, at the same time, a detailed list of independent variables bearing on behavioral performance. The raw data objects are deposited as 43 datasets but constitute, functionally, a library containing 1 large dataset.

Conclusions

Size and annotation schema give the library high reuse potential: in applications using machine learning techniques, statistical evaluation of subtle factors, simulation of new experiments, or as educational resource. Ultimately, the library can serve both as the seed and as the test bed to create a machine-searchable virtual library of linked open datasets for behavioral performance in defined conditions.

Estimation of skeletal kinematics in freely moving rodents

Forming a complete picture of the relationship between neural activity and skeletal kinematics requires quantification of skeletal joint biomechanics during free behavior; however, without detailed knowledge of the underlying skeletal motion, inferring limb kinematics using surface-tracking approaches is difficult, especially for animals where the relationship between the surface and underlying skeleton changes during motion. Here we developed a videography-based method enabling detailed three-dimensional kinematic quantification of an anatomically defined skeleton in untethered freely behaving rats and mice. This skeleton-based model was constrained using anatomical principles and joint motion limits and provided skeletal pose estimates for a range of body sizes, even when limbs were occluded. Model-inferred limb positions and joint kinematics during gait and gap-crossing behaviors were verified by direct measurement of either limb placement or limb kinematics using inertial measurement units. Together we show that complex decision-making behaviors can be accurately reconstructed at the level of skeletal kinematics using our anatomically constrained model.

Automated pose estimation in primates

Understanding the behavior of primates is important for primatology, for psychology, and for biology more broadly. It is also important for biomedicine, where primates are an important model organism, and whose behavior is often an important variable of interest. Our ability to rigorously quantify behavior has, however, long been limited. On one hand, we can rigorously quantify low-information measures like preference, looking time, and reaction time; on the other, we can use more gestalt measures like behavioral categories tracked via ethogram, but at high cost and with high variability. Recent technological advances have led to a major revolution in behavioral measurement that offers affordable and scalable rigor. Specifically, digital video cameras and automated pose tracking software can provide measures of full-body position (i.e., pose) of primates over time (i.e., behavior) with high spatial and temporal resolution. Pose-tracking technology in turn can be used to infer behavioral states, such as eating, sleeping, and mating. We call this technological approach behavioral imaging. In this review, we situate the behavioral imaging revolution in the history of the study of behavior, argue for investment in and development of analytical and research techniques that can profit from the advent of the era of big behavior, and propose that primate centers and zoos will take on a more central role in relevant fields of research than they have in the past.

Background climate modulates the impact of land cover on urban surface temperature

Cities with different background climates experience different thermal environments. Many studies have investigated land cover effects on surface urban heat in individual cities. However, a quantitative understanding of how background climates modify the thermal impact of urban land covers remains elusive. Here, we characterise land cover and their impacts on land surface temperature (LST) for 54 highly populated cities using Landsat-8 imagery. Results show that urban surface characteristics and their thermal response are distinctly different across various climate regimes, with the largest difference for cities in arid climates. Cold cities show the largest seasonal variability, with the least seasonality in tropical and arid cities. In tropical, temperate, and cold climates, normalised difference built-up index (NDBI) is the strongest contributor to LST variability during warm months followed by normalised difference vegetation index (NDVI), while normalised difference bareness index (NDBaI) is the most important factor in arid climates. These findings provide a climate-sensitive basis for future land cover planning oriented at mitigating local surface warming.

Comprehensive analysis of locomotion dynamics in the protochordate Ciona intestinalis reveals how neuromodulators flexibly shape its behavioral repertoire

Vertebrate nervous systems can generate a remarkable diversity of behaviors. However, our understanding of how behaviors may have evolved in the chordate lineage is limited by the lack of neuroethological studies leveraging our closest invertebrate relatives. Here, we combine high-throughput video acquisition with pharmacological perturbations of bioamine signaling to systematically reveal the global structure of the motor behavioral repertoire in the Ciona intestinalis larvae. Most of Ciona’s postural variance can be captured by 6 basic shapes, which we term “eigencionas.” Motif analysis of postural time series revealed numerous stereotyped behavioral maneuvers including “startle-like” and “beat-and-glide.” Employing computational modeling of swimming dynamics and spatiotemporal embedding of postural features revealed that behavioral differences are generated at the levels of motor modules and the transitions between, which may in part be modulated by bioamines. Finally, we show that flexible motor module usage gives rise to diverse behaviors in response to different light stimuli.

Vertebrate nervous systems can generate a remarkable diversity of behaviors, but how did these evolve in the chordate lineage? A study of the protochordate Ciona intestinalis reveals novel insights into how a simple chordate brain uses neuromodulators to control its behavioral repertoire.

Neural mechanisms underlying the temporal organization of naturalistic animal behavior

Naturalistic animal behavior exhibits a strikingly complex organization in the temporal domain, with variability arising from at least three sources: hierarchical, contextual, and stochastic. What neural mechanisms and computational principles underlie such intricate temporal features? In this review, we provide a critical assessment of the existing behavioral and neurophysiological evidence for these sources of temporal variability in naturalistic behavior. Recent research converges on an emergent mechanistic theory of temporal variability based on attractor neural networks and metastable dynamics, arising via coordinated interactions between mesoscopic neural circuits. We highlight the crucial role played by structural heterogeneities as well as noise from mesoscopic feedback loops in regulating flexible behavior. We assess the shortcomings and missing links in the current theoretical and experimental literature and propose new directions of investigation to fill these gaps.

The Promise of Behavioral Tracking Systems for Advancing Primate Animal Welfare

Simple Summary

Computerized tracking systems for primates and other animals are one of the great inventions of the 21st century. These systems have already revolutionized the study of primatology, psychology, neuroscience, and biomedicine. Less discussed is that they also promise to greatly enhance animal welfare. Their potential benefits include identifying and reducing pain, suffering, and distress in captive populations, improving laboratory animal welfare, and applying our understanding of animal behavior to increase the “natural” behaviors in captive and wild populations, especially those under threat. We are optimistic that these changes will greatly increase the welfare of primates, including those in laboratories, zoos, primate centers, and in the wild.

Abstract

Recent years have witnessed major advances in the ability of computerized systems to track the positions of animals as they move through large and unconstrained environments. These systems have so far been a great boon in the fields of primatology, psychology, neuroscience, and biomedicine. Here, we discuss the promise of these technologies for animal welfare. Their potential benefits include identifying and reducing pain, suffering, and distress in captive populations, improving laboratory animal welfare within the context of the three Rs of animal research (reduction, refinement, and replacement), and applying our understanding of animal behavior to increase the “natural” behaviors in captive and wild populations facing human impact challenges. We note that these benefits are often incidental to the designed purpose of these tracking systems, a reflection of the fact that animal welfare is not inimical to research progress, but instead, that the aligned interests between basic research and welfare hold great promise for improvements to animal well-being.

Leaving flatland: Advances in 3D behavioral measurement

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.

Behavioral Neuroscience in the Era of Genomics: Tools and Lessons for Analyzing High-Dimensional Datasets

Behavioral neuroscience underwent a technology-driven revolution with the emergence of machine-vision and machine-learning technologies. These technological advances facilitated the generation of high-resolution, high-throughput capture and analysis of complex behaviors. Therefore, behavioral neuroscience is becoming a data-rich field. While behavioral researchers use advanced computational tools to analyze the resulting datasets, the search for robust and standardized analysis tools is still ongoing. At the same time, the field of genomics exploded with a plethora of technologies which enabled the generation of massive datasets. This growth of genomics data drove the emergence of powerful computational approaches to analyze these data. Here, we discuss the composition of a large behavioral dataset, and the differences and similarities between behavioral and genomics data. We then give examples of genomics-related tools that might be of use for behavioral analysis and discuss concepts that might emerge when considering the two fields together.

Measuring the repertoire of age-related behavioral changes in Drosophila melanogaster

Aging affects almost all aspects of an organism—its morphology, its physiology, its behavior. Isolating which biological mechanisms are regulating these changes, however, has proven difficult, potentially due to our inability to characterize the full repertoire of an animal’s behavior across the lifespan. Using data from fruit flies (D. melanogaster) we measure the full repertoire of behaviors as a function of age. We observe a sexually dimorphic pattern of changes in the behavioral repertoire during aging. Although the stereotypy of the behaviors and the complexity of the repertoire overall remains relatively unchanged, we find evidence that the observed alterations in behavior can be explained by changing the fly’s overall energy budget, suggesting potential connections between metabolism, aging, and behavior.

Aging is a ubiquitous biological phenomenon that affects many aspects of an animal’s appearance, physiology, and behavior. Our understanding of how changes in physiology lead to behavioral changes, however, has been partially limited by our ability to robustly quantify how behavior alters over timescales of days and weeks. In this study, we measure a large repertoire of behaviors of fruit flies at various ages, finding how the actions the animals perform shift with age. We observe a difference between the aging dynamics of male and female flies, and we show that many of these changes can be explained with a model of energy consumption, leading us to make predictions as to the role of metabolism in changes in aging behavior.

Automatic mapping of multiplexed social receptive fields by deep learning and GPU-accelerated 3D videography

Social interactions powerfully impact the brain and the body, but high-resolution descriptions of these important physical interactions and their neural correlates are lacking. Currently, most studies rely on labor-intensive methods such as manual annotation. Scalable and objective tracking methods are required to understand the neural circuits underlying social behavior. Here we describe a hardware/software system and analysis pipeline that combines 3D videography, deep learning, physical modeling, and GPU-accelerated robust optimization, with automatic analysis of neuronal receptive fields recorded in interacting mice. Our system ("3DDD Social Mouse Tracker") is capable of fully automatic multi-animal tracking with minimal errors (including in complete darkness) during complex, spontaneous social encounters, together with simultaneous electrophysiological recordings. We capture posture dynamics of multiple unmarked mice with high spatiotemporal precision (~2 mm, 60 frames/s). A statistical model that relates 3D behavior and neural activity reveals multiplexed 'social receptive fields' of neurons in barrel cortex. Our approach could be broadly useful for neurobehavioral studies of multiple animals interacting in complex low-light environments.

Toward a Computational Neuroethology of Vocal Communication: From Bioacoustics to Neurophysiology, Emerging Tools and Future Directions

Recently developed methods in computational neuroethology have enabled increasingly detailed and comprehensive quantification of animal movements and behavioral kinematics. Vocal communication behavior is well poised for application of similar large-scale quantification methods in the service of physiological and ethological studies. This review describes emerging techniques that can be applied to acoustic and vocal communication signals with the goal of enabling study beyond a small number of model species. We review a range of modern computational methods for bioacoustics, signal processing, and brain-behavior mapping. Along with a discussion of recent advances and techniques, we include challenges and broader goals in establishing a framework for the computational neuroethology of vocal communication.

Animal models of brain metastasis

Modeling of metastatic disease in animal models is a critical resource to study the complexity of this multi-step process in a relevant system. Available models of metastatic disease to the brain are still far from ideal but they allow to address specific aspects of the biology or mimic clinically relevant scenarios. We not only review experimental models and their potential improvements but also discuss specific answers that could be obtained from them on unsolved aspects of clinical management.

Beyond Single Discrete Responses: An Integrative and Multidimensional Analysis of Behavioral Dynamics Assisted by Machine Learning

Understanding behavioral systems as emergent systems comprising the environment and organism subsystems, include spatial dynamics as a primary dimension in natural settings. Nevertheless, under the standard approaches, the experimental analysis of behavior is based on the single response paradigm and the temporal distribution of discrete responses. Thus, the continuous analysis of spatial behavioral dynamics is a scarcely studied field. The technological advancements in computer vision have opened new methodological perspectives for the continuous sensing of spatial behavior. With the application of such advancements, recent studies suggest that there are multiple features embedded in the spatial dynamics of behavior, such as entropy, and that they are affected by programmed stimuli (e.g., schedules of reinforcement) at least as much as features related to discrete responses. Despite the progress, the characterization of behavioral systems is still segmented, and integrated data analysis and representations between discrete responses and continuous spatial behavior are exiguous in the experimental analysis of behavior. Machine learning advancements, such as t-distributed stochastic neighbor embedding and variable ranking, provide invaluable tools to crystallize an integrated approach for analyzing and representing multidimensional behavioral data. Under this rationale, the present work (1) proposes a multidisciplinary approach for the integrative and multilevel analysis of behavioral systems, (2) provides sensitive behavioral measures based on spatial dynamics and helpful data representations to study behavioral systems, and (3) reveals behavioral aspects usually ignored under the standard approaches in the experimental analysis of behavior. To exemplify and evaluate our approach, the spatial dynamics embedded in phenomena relevant to behavioral science, namely, water-seeking behavior and motivational operations, are examined, showing aspects of behavioral systems hidden until now.

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

Understanding the connection between brain and behavior in animals requires precise monitoring of their behaviors in three-dimensional (3-D) space. However, there is no available three-dimensional behavior capture system that focuses on rodents. Here, we present MouseVenue3D, an automated and low-cost system for the efficient capture of 3-D skeleton trajectories in markerless rodents. We improved the most time-consuming step in 3-D behavior capturing by developing an automatic calibration module. Then, we validated this process in behavior recognition tasks, and showed that 3-D behavioral data achieved higher accuracy than 2-D data. Subsequently, MouseVenue3D was combined with fast high-resolution miniature two-photon microscopy for synchronous neural recording and behavioral tracking in the freely-moving mouse. Finally, we successfully decoded spontaneous neuronal activity from the 3-D behavior of mice. Our findings reveal that subtle, spontaneous behavior modules are strongly correlated with spontaneous neuronal activity patterns.

Statistical neuroscience in the single trial limit

Individual neurons often produce highly variable responses over nominally identical trials, reflecting a mixture of intrinsic 'noise' and systematic changes in the animal's cognitive and behavioral state. Disentangling these sources of variability is of great scientific interest in its own right, but it is also increasingly inescapable as neuroscientists aspire to study more complex and naturalistic animal behaviors. In these settings, behavioral actions never repeat themselves exactly and may rarely do so even approximately. Thus, new statistical methods that extract reliable features of neural activity using few, if any, repeated trials are needed. Accurate statistical modeling in this severely trial-limited regime is challenging, but still possible if simplifying structure in neural data can be exploited. We review recent works that have identified different forms of simplifying structure - including shared gain modulations across neural subpopulations, temporal smoothness in neural firing rates, and correlations in responses across behavioral conditions - and exploited them to reveal novel insights into the trial-by-trial operation of neural circuits.

Anipose: A toolkit for robust markerless 3D pose estimation

Quantifying movement is critical for understanding animal behavior. Advances in computer vision now enable markerless tracking from 2D video, but most animals move in 3D. Here, we introduce Anipose, an open-source toolkit for robust markerless 3D pose estimation. Anipose is built on the 2D tracking method DeepLabCut, so users can expand their existing experimental setups to obtain accurate 3D tracking. It consists of four components: (1) a 3D calibration module, (2) filters to resolve 2D tracking errors, (3) a triangulation module that integrates temporal and spatial regularization, and (4) a pipeline to structure processing of large numbers of videos. We evaluate Anipose on a calibration board as well as mice, flies, and humans. By analyzing 3D leg kinematics tracked with Anipose, we identify a key role for joint rotation in motor control of fly walking. To help users get started with 3D tracking, we provide tutorials and documentation at http://anipose.org/.