Fast animal pose estimation using deep neural networks

PursuitNet: A deep learning model for predicting competitive pursuit-like behavior in mice

Predator-prey interactions exemplify adaptive intelligence refined by evolution, yet replicating these behaviors in artificial systems remains challenging. Here, we introduce PursuitNet, a deep learning framework specifically designed to model the competitive, real-time dynamics of pursuit-escape scenarios. Our approach is anchored by the Pursuit-Escape Confrontation (PEC) dataset, which records laboratory mice chasing a magnetically controlled robotic bait programmed to evade capture. Unlike conventional trajectory datasets, PEC emphasizes abrupt speed changes, evasive maneuvers, and continuous mutual adaptation. PursuitNet integrates a lightweight architecture that explicitly models dynamic interactions and spatial relationships using Graph Convolutional Networks, and fuses velocity and acceleration data to predict change using Temporal Convolutional Networks. In empirical evaluations, it outperforms standard models such as Social GAN and TUTR, exhibiting substantially lower displacement errors on the PEC dataset. Ablation experiments confirm that integrating spatial and temporal features is crucial for predicting the erratic turns and speed modulations inherent to pursuit-escape behavior. Beyond accurate trajectory prediction, PursuitNet simulates pursuit events that closely mirror real mouse-and-bait interactions, shedding light on how innate drives, rather than external instructions, guide adaptive decision-making. Although the framework is specialized for rapidly shifting trajectories, our findings suggest that this biologically inspired perspective can deepen understanding of predator-prey dynamics and inform the design of interactive robotics and autonomous systems.

The Tailtag: A multi-mouse tracking system to measure social dynamics in complex environments

Despite recent advances, tracking individual movements safely and reliably over extended periods, particularly within complex social groups, remains a challenge. Traditional methods like color coding, tagging, and RFID tracking, while effective, have notable practical limitations. State-of-the-art neural network-based trackers often struggle to maintain individual identities in large groups for more than a few seconds. Fiducial tags such as ArUco codes present a potential solution to enable accurate tracking and identity management. However, their application to large groups of socially interacting mice in complex, enriched environments remain an open challenge. Here, we present the Tailtag system, a novel approach designed to address this challenge. The Tailtag is a non-invasive, safe, and ergonomic tail ring embedded with an ArUco marker allowing to track individual mice in colonies of up to 20 individuals in complex environments for at least seven days without performance degradation or behavioral alteration. We provide a comprehensive parameter optimization guide and practical recommendations for marker selection, for reproducibility across diverse experimental setups. Using data collected from Tailtag-equipped mice, we revealed the formation and evolution of social groups within the colony. Our analysis identified social hub regions within the vivarium where social contacts occur at different frequencies throughout one week of recordings. We quantified interactions and avoidance patterns between specific pairs of mice within the most active social hubs. Overall, our findings indicate that while the zone preferences and peer associations among the mice change over time, certain groups and pairwise interactions consistently form within the social colony.

Anti-drift pose tracker (ADPT), a transformer-based network for robust animal pose estimation cross-species

Deep learning-based methods have advanced animal pose estimation, enhancing accuracy, and efficiency in quantifying animal behavior. However, these methods frequently experience tracking drift, where noise-induced jumps in body point estimates compromise reliability. Here, we present the anti-drift pose tracker (ADPT), a transformer-based tool that mitigates tracking drift in behavioral analysis. Extensive experiments across cross-species datasets-including proprietary mouse and monkey recordings and public Drosophila and macaque datasets-demonstrate that ADPT significantly reduces drift and surpasses existing models like DeepLabCut and SLEAP in accuracy. Moreover, ADPT achieved 93.16% identification accuracy for 10 unmarked mice and 90.36% accuracy for freely interacting unmarked mice, which can be further refined to 99.72%, enhancing both anti-drift performance and pose estimation accuracy in social interactions. With its end-to-end design, ADPT is computationally efficient and suitable for real-time analysis, offering a robust solution for reproducible animal behavior studies. The ADPT code is available at https://github.com/tangguoling/ADPT.

High-throughput markerless pose estimation and home-cage activity analysis of tree shrew using deep learning

BACKGROUND

Quantifying the rich home-cage activities of tree shrews provides a reliable basis for understanding their daily routines and building disease models. However, due to the lack of effective behavioral methods, most efforts on tree shrew behavior are limited to simple measures, resulting in the loss of much behavioral information.

METHODS

To address this issue, we present a deep learning (DL) approach to achieve markerless pose estimation and recognize multiple spontaneous behaviors of tree shrews, including drinking, eating, resting, and staying in the dark house, etc. RESULTS: This high-throughput approach can monitor the home-cage activities of 16 tree shrews simultaneously over an extended period. Additionally, we demonstrated an innovative system with reliable apparatus, paradigms, and analysis methods for investigating food grasping behavior. The median duration for each bout of grasping was 0.20 s.

CONCLUSION

This study provides an efficient tool for quantifying and understand tree shrews' natural behaviors.

A segment-based framework for explainability in animal affective computing

Recent developments in animal motion tracking and pose recognition have revolutionized the study of animal behavior. More recent efforts extend beyond tracking towards affect recognition using facial and body language analysis, with far-reaching applications in animal welfare and health. Deep learning models are the most commonly used in this context. However, their "black box" nature poses a significant challenge to explainability, which is vital for building trust and encouraging adoption among researchers. Despite its importance, the field of explainability and its quantification remains under-explored. Saliency maps are among the most widely used methods for explainability, where each pixel is assigned a significance level indicating its relevance to the neural network's decision. Although these maps are frequently used in research, they are predominantly applied qualitatively, with limited methods for quantitatively analyzing them or identifying the most suitable method for a specific task. In this paper, we propose a framework aimed at enhancing explainability in the field of animal affective computing. Assuming the availability of a classifier for a specific affective state and the ability to generate saliency maps, our approach focuses on evaluating and comparing visual explanations by emphasizing the importance of meaningful semantic parts captured as segments, which are thought to be closely linked to behavioral indicators of affective states. Furthermore, our approach introduces a quantitative scoring mechanism to assess how well the saliency maps generated by a given classifier align with predefined semantic regions. This scoring system allows for systematic, measurable comparisons of different pipelines in terms of their visual explanations within animal affective computing. Such a metric can serve as a quality indicator when developing classifiers for known biologically relevant segments or help researchers assess whether a classifier is using expected meaningful regions when exploring new potential indicators. We evaluated the framework using three datasets focused on cat and horse pain and dog emotions. Across all datasets, the generated explanations consistently revealed that the eye area is the most significant feature for the classifiers. These results highlight the potential of the explainability frameworks such as the suggested one to uncover new insights into how machines 'see' animal affective states.

Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

Extracting the relationship between high-dimensional neural recordings and complex behavior is a ubiquitous problem in neuroscience. Encoding and decoding models target the conditional distribution of neural activity given behavior and vice versa, while dimensionality reduction techniques extract low-dimensional representations thereof. Variational autoencoders (VAEs) are flexible tools for inferring such low-dimensional embeddings but struggle to accurately model arbitrary conditional distributions such as those arising in neural encoding and decoding, let alone simultaneously. Here, we present a VAE-based approach for calculating such conditional distributions. We first validate our approach on a task with known ground truth. Next, we retrieve conditional distributions over masked body parts of walking flies. Finally, we decode motor trajectories from neural activity in a monkey-reach task and query the same VAE for the encoding distribution. Our approach unifies dimensionality reduction and learning conditional distributions, allowing the scaling of common analyses in neuroscience to today's high-dimensional multi-modal datasets.

Hierarchical behavioral analysis framework as a platform for standardized quantitative identification of behaviors

Behavior is composed of modules that operate based on inherent logic. Understanding behavior and its neural mechanisms is facilitated by clear structural behavioral analysis. Here, we developed a hierarchical behavioral analysis framework (HBAF) that efficiently reveals the organizational logic of these modules by analyzing high-dimensional behavioral data. By creating a spontaneous behavior atlas for male and female mice, we discovered that spontaneous behavior patterns are hardwired, with sniffing serving as the hub node for movement transitions. The sniffing-to-grooming ratio accurately distinguished the spontaneous behavioral states in a high-throughput manner. These states are influenced by emotional status, circadian rhythms, and lighting conditions, leading to unique behavioral characteristics, spatiotemporal features, and dynamic patterns. By implementing the straightforward and achievable spontaneous behavior paradigm, HBAF enables swift and accurate assessment of animal behavioral states and bridges the gap between a theoretical understanding of the behavioral structure and practical analysis using comprehensive multidimensional behavioral information.

A real-time, multi-subject three-dimensional pose tracking system for the behavioral analysis of non-human primates

The ability to track the positions and poses of multiple animals in three-dimensional (3D) space in real time is highly desired by non-human primate (NHP) researchers in behavioral and systems neuroscience. This capability enables the analysis of social behaviors involving multiple NHPs and supports closed-loop experiments. Although several animal 3D pose tracking systems have been developed, most are difficult to deploy in new environments and lack real-time analysis capabilities. To address these limitations, we developed MarmoPose, a deep-learning-based, real-time 3D pose tracking system for multiple common marmosets, an increasingly critical NHP model in neuroscience research. This system can accurately track the 3D poses of multiple marmosets freely moving in their home cage with minimal hardware requirements. By employing a marmoset skeleton model, MarmoPose can further optimize 3D poses and estimate invisible body locations. Additionally, MarmoPose achieves high inference speeds and enables real-time closed-loop experimental control based on events detected from 3D poses.

The utility of animal models to inform the next generation of human space exploration

Animals have played a vital role in every stage of space exploration, from early sub-orbital flights to contemporary missions. New physiological and psychological challenges arise with plans to venture deeper into the solar system. Advances in chimeric and knockout animal models, along with genetic modification techniques have enhanced our ability to study the effects of microgravity in greater detail. However, increased investment in the purposeful design of habitats and payloads, as well as in AI-enhanced behavioral monitoring in orbit can better support the ethical and effective use of animals in deep space research.

vmTracking enables highly accurate multi-animal pose tracking in crowded environments

In multi-animal tracking, addressing occlusion and crowding is crucial for accurate behavioral analysis. However, in situations where occlusion and crowding generate complex interactions, achieving accurate pose tracking remains challenging. Therefore, we introduced virtual marker tracking (vmTracking), which uses virtual markers for individual identification. Virtual markers are labels derived from conventional markerless multi-animal tracking tools, such as multi-animal DeepLabCut (maDLC) and Social LEAP Estimates Animal Poses (SLEAP). Unlike physical markers, virtual markers exist only within the video and attribute features to individuals, enabling consistent identification throughout the entire video while keeping the animals markerless in reality. Using these markers as cues, annotations were applied to multi-animal videos, and tracking was conducted with single-animal DeepLabCut (saDLC) and SLEAP's single-animal method. vmTracking minimized manual corrections and annotation frames needed for training, efficiently tackling occlusion and crowding. Experiments tracking multiple mice, fish, and human dancers confirmed vmTracking's variability and applicability. These findings could enhance the precision and reliability of tracking methods used in the analysis of complex naturalistic and social behaviors in animals, providing a simpler yet more effective solution.

Marigold: a machine learning-based web app for zebrafish pose tracking

BACKGROUND

High-throughput behavioral analysis is important for drug discovery, toxicological studies, and the modeling of neurological disorders such as autism and epilepsy. Zebrafish embryos and larvae are ideal for such applications because they are spawned in large clutches, develop rapidly, feature a relatively simple nervous system, and have orthologs to many human disease genes. However, existing software for video-based behavioral analysis can be incompatible with recordings that contain dynamic backgrounds or foreign objects, lack support for multiwell formats, require expensive hardware, and/or demand considerable programming expertise. Here, we introduce Marigold, a free and open source web app for high-throughput behavioral analysis of embryonic and larval zebrafish.

RESULTS

Marigold features an intuitive graphical user interface, tracks up to 10 user-defined keypoints, supports both single- and multiwell formats, and exports a range of kinematic parameters in addition to publication-quality data visualizations. By leveraging a highly efficient, custom-designed neural network architecture, Marigold achieves reasonable training and inference speeds even on modestly powered computers lacking a discrete graphics processing unit. Notably, as a web app, Marigold does not require any installation and runs within popular web browsers on ChromeOS, Linux, macOS, and Windows. To demonstrate Marigold's utility, we used two sets of biological experiments. First, we examined novel aspects of the touch-evoked escape response in techno trousers (tnt) mutant embryos, which contain a previously described loss-of-function mutation in the gene encoding Eaat2b, a glial glutamate transporter. We identified differences and interactions between touch location (head vs. tail) and genotype. Second, we investigated the effects of feeding on larval visuomotor behavior at 5 and 7 days post-fertilization (dpf). We found differences in the number and vigor of swimming bouts between fed and unfed fish at both time points, as well as interactions between developmental stage and feeding regimen.

CONCLUSIONS

In both biological experiments presented here, the use of Marigold facilitated novel behavioral findings. Marigold's ease of use, robust pose tracking, amenability to diverse experimental paradigms, and flexibility regarding hardware requirements make it a powerful tool for analyzing zebrafish behavior, especially in low-resource settings such as course-based undergraduate research experiences. Marigold is available at: https://downeslab.github.io/marigold/ .

Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour

Automated social behaviour analysis of mice has become an increasingly popular research area in behavioural neuroscience. Recently, pose information (i.e., locations of keypoints or skeleton) has been used to interpret social behaviours of mice. Nevertheless, effective encoding and decoding of social interaction information underlying the keypoints of mice has been rarely investigated in the existing methods. In particular, it is challenging to model complex social interactions between mice due to highly deformable body shapes and ambiguous movement patterns. To deal with the interaction modelling problem, we here propose a Cross-Skeleton Interaction Graph Aggregation Network (CS-IGANet) to learn abundant dynamics of freely interacting mice, where a Cross-Skeleton Node-level Interaction module (CS-NLI) is used to model multi-level interactions (i.e., intra-, inter- and cross-skeleton interactions). Furthermore, we design a novel Interaction-Aware Transformer (IAT) to dynamically learn the graph-level representation of social behaviours and update the node-level representation, guided by our proposed interaction-aware self-attention mechanism. Finally, to enhance the representation ability of our model, an auxiliary self-supervised learning task is proposed for measuring the similarity between cross-skeleton nodes. Experimental results on the standard CRMI13-Skeleton and our PDMB-Skeleton datasets show that our proposed model outperforms several other state-of-the-art approaches.

Automated analysis of a novel object recognition test in mice using image processing and machine learning

The novel object recognition test (NORT) is one of the most commonly employed behavioral tests in experimental animals designed to evaluate an animal's interest in and recognition of novelty. However, manual procedures, which rely on researchers' observations, prevent high throughput analysis. In this study, we developed an automated analysis method for NORT utilizing machine learning-assisted exploratory behavior detection. We recorded the exploratory behavior of the mice using a video camera. The coordinates of the mouse nose and tail base in recorded video files were detected using a pre-trained machine learning model, DeepLabCut. Each video was then segmented into frame images, which were categorized into "exploratory," or "non-exploratory" frames based on manual observation. Mouse feature vectors were calculated as vectors from the nose to the vertices of the object and were utilized for SVM training. The trained SVM effectively detected exploratory behaviors, showing a strong correlation with human observer assessments. Upon application to NORT, the duration of mouse exploratory behavior towards objects predicted by the SVM exhibited a significant correlation with the assessments made by human observers. The novelty discrimination index derived from the SVM predictions also aligned well with that from human observations.

ONIX: a unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge we developed ONIX, an open-source data acquisition system with high data throughput (2 GB s-1) and low closed-loop latencies (<1 ms) that uses a 0.3-mm thin tether to minimize behavioral impact. Head position and rotation are tracked in three dimensions and used to drive active commutation without torque measurements. ONIX can acquire data from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, three-dimensional trackers and other data sources. We performed uninterrupted, long (~7 h) neural recordings in mice as they traversed complex three-dimensional terrain, and multiday sleep-tracking recordings (~55 h). ONIX enabled exploration with similar mobility as nonimplanted animals, in contrast to conventional tethered systems, which have restricted movement. By combining long recordings with full mobility, our technology will enable progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

Effective identification of Alzheimer's disease in mouse models via deep learning and motion analysis

Spatial disorientation is an early symptom of Alzheimer's disease (AD). Detecting this impairment effectively in animal models can provide valuable insights into the disease and reduce experimental burdens. We have developed a markerless motion analysis system (MMAS) using deep learning techniques for the Morris water maze test. This system allows for precise analysis of behaviors and body movements from video recordings. Using the MMAS, we identified unilateral head-turning and tail-wagging preferences in AD mice, which distinguished them from wild-type mice with greater accuracy than traditional behavioral parameters. Furthermore, the cumulative turning and wagging angles were linearly correlated with escape latency and cognitive scores, demonstrating comparable effectiveness in differentiating AD mice. These findings underscore the potential of motion analysis as an advanced method for improving the effectiveness, sensitivity, and interpretability of AD mouse identification, ultimately aiding in disease diagnosis and drug development.

How can ethology inform the neuroscience of fear, aggression and dominance?

The study of behaviour is dominated by two approaches. On the one hand, ethologists aim to understand how behaviour promotes adaptation to natural contexts. On the other, neuroscientists aim to understand the molecular, cellular, circuit and psychological origins of behaviour. These two complementary approaches must be combined to arrive at a full understanding of behaviour in its natural setting. However, methodological limitations have restricted most neuroscientific research to the study of how discrete sensory stimuli elicit simple behavioural responses under controlled laboratory conditions that are only distantly related to those encountered in real life. Fortunately, the recent advent of neural monitoring and manipulation tools adapted for use in freely behaving animals has enabled neuroscientists to incorporate naturalistic behaviours into their studies and to begin to consider ethological questions. Here, we examine the promises and pitfalls of this trend by describing how investigations of rodent fear, aggression and dominance behaviours are changing to take advantage of an ethological appreciation of behaviour. We lay out current impediments to this approach and propose a framework for the evolution of the field that will allow us to take maximal advantage of an ethological approach to neuroscience and to increase its relevance for understanding human behaviour.

Kinematic coding: Measuring information in naturalistic behaviour

Recent years have seen an explosion of interest in naturalistic behaviour and in machine learning tools for automatically tracking it. However, questions about what to measure, how to measure it, and how to relate naturalistic behaviour to neural activity and cognitive processes remain unresolved. In this Perspective, we propose a general experimental and computational framework - kinematic coding - for measuring how information about cognitive states is encoded in structured patterns of behaviour and how this information is read out by others during social interactions. This framework enables the design of new experiments and the generation of testable hypotheses that link behaviour, cognition, and neural activity at the single-trial level. Researchers can employ this framework to identify single-subject, single-trial encoding and readout computations and address meaningful questions about how information encoded in bodily motion is transmitted and communicated.

AI-powered home cage system for real-time tracking and analysis of rodent behavior

Researchers in animal behavior and neuroscience devote considerable time to observing rodents behavior and physiological responses, with AI monitoring systems reducing personnel workload. This study presents the RodentWatch (RW) system, which leverages deep learning to automatically identify experimental animal behaviors in home cage environments. A single multifunctional camera and edge device are installed inside the animal's home cage, allowing continuous real-time monitoring of the animal's behavior, position, and body temperature for extended periods. We investigated identifying the drinking and resting behaviors of rats, with recognition accuracy enhanced through contextual object labeling and modified non-maximum suppression (NMS) schemes. Two tests-a light cycle change test and a sucrose preference test-were conducted to evaluate the usability of this system in rat behavioral experiments. This system enables notable advancements in image-based behavior recognition for living rodents.

Revisiting serotonin's role in spatial memory: A call for sensitive analytical approaches

The serotonergic system is involved in various psychiatric and neurological conditions, with serotonergic drugs often used in treatment. These conditions frequently affect spatial memory, which can serve as a model of declarative memory due to well-known cellular components and advanced methods that track neural activity and behavior with high temporal resolution. However, most findings on serotonin's effects on spatial learning and memory come from studies lacking refined analytical techniques and modern approaches needed to uncover the underlying neuronal mechanisms. This In Focus review critically investigates available studies to identify areas for further exploration. It finds that well-established behavioral models could yield more insights with modern tracking and data analysis approaches, while the cellular aspects of spatial memory remain underexplored. The review highlights the complex role of serotonin in spatial memory, which holds the potential for better understanding and treating memory-related disorders.

Characterizing the structure of mouse behavior using Motion Sequencing

Spontaneous mouse behavior is composed from repeatedly used modules of movement (e.g., rearing, running or grooming) that are flexibly placed into sequences whose content evolves over time. By identifying behavioral modules and the order in which they are expressed, researchers can gain insight into the effect of drugs, genes, context, sensory stimuli and neural activity on natural behavior. Here we present a protocol for performing Motion Sequencing (MoSeq), an ethologically inspired method that uses three-dimensional machine vision and unsupervised machine learning to decompose spontaneous mouse behavior into a series of elemental modules called 'syllables'. This protocol is based upon a MoSeq pipeline that includes modules for depth video acquisition, data preprocessing and modeling, as well as a standardized set of visualization tools. Users are provided with instructions and code for building a MoSeq imaging rig and acquiring three-dimensional video of spontaneous mouse behavior for submission to the modeling framework; the outputs of this protocol include syllable labels for each frame of the video data as well as summary plots describing how often each syllable was used and how syllables transitioned from one to the other. In addition, we provide instructions for analyzing and visualizing the outputs of keypoint-MoSeq, a recently developed variant of MoSeq that can identify behavioral motifs from keypoints identified from standard (rather than depth) video. This protocol and the accompanying pipeline significantly lower the bar for users without extensive computational ethology experience to adopt this unsupervised, data-driven approach to characterize mouse behavior.

X-Tracker: Automated Analysis of Xenopus Tadpole Visual Avoidance Behavior

Xenopus laevis tadpoles exhibit an avoidance behavior when they encounter a moving visual stimulus. A visual avoidance event occurs when a moving object approaches the eye of a free-swimming animal at an approximately 90-degree angle and the animal turns in response to the encounter. Analysis of this behavior requires tracking both the free-swimming animal and the moving visual stimulus both prior to and after the encounter. Previous automated tracking software does not discriminate the moving animal from the moving stimulus, requiring time-consuming manual analysis. Here we present X-Tracker, an automated behavior tracking code that can detect and discriminate moving visual stimuli and free-swimming animals and score encounters and avoidance events. X-Tracker is as accurate as human analysis without the human time commitment. We also present software improvements to our previous visual stimulus presentation and image capture that optimize videos for automated analysis, and hardware improvements that increase the number of animal-stimulus encounters. X-Tracker is a high throughput, unbiased, and significant time-saving analysis system that will greatly facilitate visual avoidance behavior analysis of Xenopus laevis tadpoles, and potentially other free-swimming organisms. The tool is available at https://github.com/ClineLab/Tadpole-Behavior-Automation.

Miniature linear and split-belt treadmills reveal mechanisms of adaptive motor control in walking Drosophila

To navigate complex environments, walking animals must detect and overcome unexpected perturbations. One technical challenge when investigating adaptive locomotion is measuring behavioral responses to precise perturbations during naturalistic walking; another is that manipulating neural activity in sensorimotor circuits often reduces spontaneous locomotion. To overcome these obstacles, we introduce miniature treadmill systems for coercing locomotion and tracking 3D kinematics of walking Drosophila. By systematically comparing walking in three experimental setups, we show that flies compelled to walk on the linear treadmill have similar stepping kinematics to freely walking flies, while kinematics of tethered walking flies are subtly different. Genetically silencing mechanosensory neurons altered step kinematics of flies walking on the linear treadmill across all speeds. We also discovered that flies can maintain a forward heading on a split-belt treadmill by specifically adapting the step distance of their middle legs. These findings suggest that proprioceptive feedback contributes to leg motor control irrespective of walking speed and that the fly's middle legs play a specialized role in stabilizing locomotion.

Long-term tracking of social structure in groups of rats

Rodents serve as an important model for examining both individual and collective behavior. Dominance within rodent social structures can determine access to critical resources, such as food and mating opportunities. Yet, many aspects of the intricate interplay between individual behaviors and the resulting group social hierarchy, especially its evolution over time, remain unexplored. In this study, we utilized an automated tracking system that continuously monitored groups of male rats for over 250 days to enable an in-depth analysis of individual behavior and the overarching group dynamic. We describe the evolution of social structures within a group and additionally investigate how past behaviors influence the emergence of new social hierarchies when group composition and experimental area changes. Notably, we find that conventional individual and pairwise tests exhibit a weak correlation with group behavior, highlighting their limited accuracy in predicting behavioral outcomes in a collective context. These results emphasize the context-dependence of social behavior as an emergent property of interactions within a group and highlight the need to measure and quantify social behavior in more naturalistic environments.

Spinal neuron diversity scales exponentially with swim-to-limb transformation during frog metamorphosis

Vertebrates exhibit a wide range of motor behaviors, ranging from swimming to complex limb-based movements. Here we take advantage of frog metamorphosis, which captures a swim-to-limb-based movement transformation during the development of a single organism, to explore changes in the underlying spinal circuits. We find that the tadpole spinal cord contains small and largely homogeneous populations of motor neurons (MNs) and V1 interneurons (V1s) at early escape swimming stages. These neuronal populations only modestly increase in number and subtype heterogeneity with the emergence of free swimming. In contrast, during frog metamorphosis and the emergence of limb movement, there is a dramatic expansion of MN and V1 interneuron number and transcriptional heterogeneity, culminating in cohorts of neurons that exhibit striking molecular similarity to mammalian motor circuits. CRISPR/Cas9-mediated gene disruption of the limb MN and V1 determinants FoxP1 and Engrailed-1, respectively, results in severe but selective deficits in tail and limb function. Our work thus demonstrates that neural diversity scales exponentially with increasing behavioral complexity and illustrates striking evolutionary conservation in the molecular organization and function of motor circuits across species.

Combining Computational Fluid Dynamics and Experimental Data to Understand Fish Schooling Behavior

Understanding the flow physics behind fish schooling poses significant challenges due to the difficulties in directly measuring hydrodynamic performance and the three-dimensional, chaotic, and complex flow structures generated by collective moving organisms. Numerous previous simulations and experiments have utilized computational, mechanical, or robotic models to represent live fish. And existing studies of live fish schools have contributed significantly to dissecting the complexities of fish schooling. But the scarcity of combined approaches that include both computational and experimental studies, ideally of the same fish schools, has limited our ability to understand the physical factors that are involved in fish collective behavior. This underscores the necessity of developing new approaches to working directly with live fish schools. An integrated method that combines experiments on live fish schools with computational fluid dynamics (CFD) simulations represents an innovative method of studying the hydrodynamics of fish schooling. CFD techniques can deliver accurate performance measurements and high-fidelity flow characteristics for comprehensive analysis. Concurrently, experimental approaches can capture the precise locomotor kinematics of fish and offer additional flow information through particle image velocimetry (PIV) measurements, potentially enhancing the accuracy and efficiency of CFD studies via advanced data assimilation techniques. The flow patterns observed in PIV experiments with fish schools and the complex hydrodynamic interactions revealed by integrated analyses highlight the complexity of fish schooling, prompting a reevaluation of the classic Weihs model of school dynamics. The synergy between CFD models and experimental data grants us comprehensive insights into the flow dynamics of fish schools, facilitating the evaluation of their functional significance and enabling comparative studies of schooling behavior. In addition, we consider the challenges in developing integrated analytical methods and suggest promising directions for future research.

Opportunities and Challenges in Applying AI to Evolutionary Morphology

Artificial intelligence (AI) is poised to revolutionize many aspects of science, including the study of evolutionary morphology. While classical AI methods such as principal component analysis and cluster analysis have been commonplace in the study of evolutionary morphology for decades, recent years have seen increasing application of deep learning to ecology and evolutionary biology. As digitized specimen databases become increasingly prevalent and openly available, AI is offering vast new potential to circumvent long-standing barriers to rapid, big data analysis of phenotypes. Here, we review the current state of AI methods available for the study of evolutionary morphology, which are most developed in the area of data acquisition and processing. We introduce the main available AI techniques, categorizing them into 3 stages based on their order of appearance: (1) machine learning, (2) deep learning, and (3) the most recent advancements in large-scale models and multimodal learning. Next, we present case studies of existing approaches using AI for evolutionary morphology, including image capture and segmentation, feature recognition, morphometrics, and phylogenetics. We then discuss the prospectus for near-term advances in specific areas of inquiry within this field, including the potential of new AI methods that have not yet been applied to the study of morphological evolution. In particular, we note key areas where AI remains underutilized and could be used to enhance studies of evolutionary morphology. This combination of current methods and potential developments has the capacity to transform the evolutionary analysis of the organismal phenotype into evolutionary phenomics, leading to an era of "big data" that aligns the study of phenotypes with genomics and other areas of bioinformatics.

A Markovian dynamics for Caenorhabditis elegans behavior across scales

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm Caenorhabditis elegans, we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences, we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm's foraging behavior, revealing both "runs-and-pirouettes" as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies.

KineWheel-DeepLabCut Automated Paw Annotation Using Alternating Stroboscopic UV and White Light Illumination

Uncovering the relationships between neural circuits, behavior, and neural dysfunction may require rodent pose tracking. While open-source toolkits such as DeepLabCut have revolutionized markerless pose estimation using deep neural networks, the training process still requires human intervention for annotating key points of interest in video data. To further reduce human labor for neural network training, we developed a method that automatically generates annotated image datasets of rodent paw placement in a laboratory setting. It uses invisible but fluorescent markers that become temporarily visible under UV light. Through stroboscopic alternating illumination, adjacent video frames taken at 720 Hz are either UV or white light illuminated. After color filtering the UV-exposed video frames, the UV markings are identified and the paw locations are deterministically mapped. This paw information is then transferred to automatically annotate paw positions in the next white light-exposed frame that is later used for training the neural network. We demonstrate the effectiveness of our method using a KineWheel-DeepLabCut setup for the markerless tracking of the four paws of a harness-fixed mouse running on top of the transparent wheel with mirror. Our automated approach, made available open-source, achieves high-quality position annotations and significantly reduces the need for human involvement in the neural network training process, paving the way for more efficient and streamlined rodent pose tracking in neuroscience research.

A virtual rodent predicts the structure of neural activity across behaviours

Animals have exquisite control of their bodies, allowing them to perform a diverse range of behaviours. How such control is implemented by the brain, however, remains unclear. Advancing our understanding requires models that can relate principles of control to the structure of neural activity in behaving animals. Here, to facilitate this, we built a 'virtual rodent', in which an artificial neural network actuates a biomechanically realistic model of the rat1 in a physics simulator2. We used deep reinforcement learning3-5 to train the virtual agent to imitate the behaviour of freely moving rats, thus allowing us to compare neural activity recorded in real rats to the network activity of a virtual rodent mimicking their behaviour. We found that neural activity in the sensorimotor striatum and motor cortex was better predicted by the virtual rodent's network activity than by any features of the real rat's movements, consistent with both regions implementing inverse dynamics6. Furthermore, the network's latent variability predicted the structure of neural variability across behaviours and afforded robustness in a way consistent with the minimal intervention principle of optimal feedback control7. These results demonstrate how physical simulation of biomechanically realistic virtual animals can help interpret the structure of neural activity across behaviour and relate it to theoretical principles of motor control.

Deep learning pose detection model for sow locomotion

Lameness affects animal mobility, causing pain and discomfort. Lameness in early stages often goes undetected due to a lack of observation, precision, and reliability. Automated and non-invasive systems offer precision and detection ease and may improve animal welfare. This study was conducted to create a repository of images and videos of sows with different locomotion scores. Our goal is to develop a computer vision model for automatically identifying specific points on the sow's body. The automatic identification and ability to track specific body areas, will allow us to conduct kinematic studies with the aim of facilitating the detection of lameness using deep learning. The video database was collected on a pig farm with a scenario built to allow filming of sows in locomotion with different lameness scores. Two stereo cameras were used to record 2D videos images. Thirteen locomotion experts assessed the videos using the Locomotion Score System developed by Zinpro Corporation. From this annotated repository, computational models were trained and tested using the open-source deep learning-based animal pose tracking framework SLEAP (Social LEAP Estimates Animal Poses). The top-performing models were constructed using the LEAP architecture to accurately track 6 (lateral view) and 10 (dorsal view) skeleton keypoints. The architecture achieved average precisions values of 0.90 and 0.72, average distances of 6.83 and 11.37 in pixel, and similarities of 0.94 and 0.86 for the lateral and dorsal views, respectively. These computational models are proposed as a Precision Livestock Farming tool and method for identifying and estimating postures in pigs automatically and objectively. The 2D video image repository with different pig locomotion scores can be used as a tool for teaching and research. Based on our skeleton keypoint classification results, an automatic system could be developed. This could contribute to the objective assessment of locomotion scores in sows, improving their welfare.

RECORD, a high-throughput, customizable system that unveils behavioral strategies leveraged by rodents during foraging-like decision-making

Translational studies benefit from experimental designs where laboratory organisms use human-relevant behaviors. One such behavior is decision-making, however studying complex decision-making in rodents is labor-intensive and typically restricted to two levels of cost/reward. We design a fully automated, inexpensive, high-throughput framework to study decision-making across multiple levels of rewards and costs: the REward-COst in Rodent Decision-making (RECORD) system. RECORD integrates three components: 1) 3D-printed arenas, 2) custom electronic hardware, and 3) software. We validated four behavioral protocols without employing any food or water restriction, highlighting the versatility of our system. RECORD data exposes heterogeneity in decision-making both within and across individuals that is quantifiably constrained. Using oxycodone self-administration and alcohol-consumption as test cases, we reveal how analytic approaches that incorporate behavioral heterogeneity are sensitive to detecting perturbations in decision-making. RECORD is a powerful approach to studying decision-making in rodents, with features that facilitate translational studies of decision-making in psychiatric disorders.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ('syllables') from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience

The study of complex behaviors is often challenging when using manual annotation due to the absence of quantifiable behavioral definitions and the subjective nature of behavioral annotation. Integration of supervised machine learning approaches mitigates some of these issues through the inclusion of accessible and explainable model interpretation. To decrease barriers to access, and with an emphasis on accessible model explainability, we developed the open-source Simple Behavioral Analysis (SimBA) platform for behavioral neuroscientists. SimBA introduces several machine learning interpretability tools, including SHapley Additive exPlanation (SHAP) scores, that aid in creating explainable and transparent behavioral classifiers. Here we show how the addition of explainability metrics allows for quantifiable comparisons of aggressive social behavior across research groups and species, reconceptualizing behavior as a sharable reagent and providing an open-source framework. We provide an open-source, graphical user interface (GUI)-driven, well-documented package to facilitate the movement toward improved automation and sharing of behavioral classification tools across laboratories.

Automation in canine science: enhancing human capabilities and overcoming adoption barriers

The emerging field of canine science has been slow in adopting automated approaches for data analysis. However, with the dramatic increase in the volume and complexity of the collected behavioral data, this is now beginning to change. This paper aims to systematize the field of automation in canine science. We provide an examination of current automation processes and pipelines by providing a literature review of state-of-the-art studies applying automation in this field. In addition, via an empirical study with researchers in animal behavior, we explore their perceptions and attitudes toward automated approaches for better understanding barriers for a wider adoption of automation. The insights derived from this research could facilitate more effective and widespread utilization of automation within canine science, addressing current challenges and enhancing the analysis of increasingly complex and voluminous behavioral data. This could potentially revolutionize the field, allowing for more objective and quantifiable assessments of dog behavior, which would ultimately contribute to our understanding of dog-human interactions and canine welfare.

MCP: Multi-Chicken Pose Estimation Based on Transfer Learning

Poultry managers can better understand the state of poultry through poultry behavior analysis. As one of the key steps in behavior analysis, the accurate estimation of poultry posture is the focus of this research. This study mainly analyzes a top-down pose estimation method of multiple chickens. Therefore, we propose the "multi-chicken pose" (MCP), a pose estimation system for multiple chickens through deep learning. Firstly, we find the position of each chicken from the image via the chicken detector; then, an estimate of the pose of each chicken is made using a pose estimation network, which is based on transfer learning. On this basis, the pixel error (PE), root mean square error (RMSE), and image quantity distribution of key points are analyzed according to the improved chicken keypoint similarity (CKS). The experimental results show that the algorithm scores in different evaluation metrics are a mean average precision (mAP) of 0.652, a mean average recall (mAR) of 0.742, a percentage of correct keypoints (PCKs) of 0.789, and an RMSE of 17.30 pixels. To the best of our knowledge, this is the first time that transfer learning has been used for the pose estimation of multiple chickens as objects. The method can provide a new path for future poultry behavior analysis.

A toolkit for the dynamic study of air sacs in siamang and other elastic circular structures

Biological structures are defined by rigid elements, such as bones, and elastic elements, like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments provide insights into complex time-varying dynamics of biological motion. Conversely, the elastic soft-tissues of organisms, like the nose of elephant seals, or the buccal sac of frogs, are poorly studied and no computer vision methods have been proposed. This leaves major gaps in different areas of biology. In primatology, most critically, the function of air sacs is widely debated; many open questions on the role of air sacs in the evolution of animal communication, including human speech, remain unanswered. To support the dynamic study of soft-tissue structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit contains unsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting DeepLabCut) methodology to track inflation of laryngeal air sacs or other biological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis, we show that air sac inflation correlates with acoustic markers that likely inform about body size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of siamang (Symphalangus syndactylus) singing. This toolkit (https://github.com/WimPouw/AirSacTracker) aims to revitalize the study of non-skeletal morphological structures across multiple species.

From innate to instructed: A new look at perceptual decision-making

Understanding how subjects perceive sensory stimuli in their environment and use this information to guide appropriate actions is a major challenge in neuroscience. To study perceptual decision-making in animals, researchers use tasks that either probe spontaneous responses to stimuli (often described as "naturalistic") or train animals to associate stimuli with experimenter-defined responses. Spontaneous decisions rely on animals' pre-existing knowledge, while trained tasks offer greater versatility, albeit often at the cost of extensive training. Here, we review emerging approaches to investigate perceptual decision-making using both spontaneous and trained behaviors, highlighting their strengths and limitations. Additionally, we propose how trained decision-making tasks could be improved to achieve faster learning and a more generalizable understanding of task rules.

Understanding collective behavior through neurobiology

A variety of organisms exhibit collective movement, including schooling fish and flocking birds, where coordinated behavior emerges from the interactions between group members. Despite the prevalence of collective movement in nature, little is known about the neural mechanisms producing each individual's behavior within the group. Here we discuss how a neurobiological approach can enrich our understanding of collective behavior by determining the mechanisms by which individuals interact. We provide examples of sensory systems for social communication during collective movement, highlight recent discoveries about neural systems for detecting the position and actions of social partners, and discuss opportunities for future research. Understanding the neurobiology of collective behavior can provide insight into how nervous systems function in a dynamic social world.

STPoseNet: A real-time spatiotemporal network model for robust mouse pose estimation

Animal behavior analysis plays a crucial role in contemporary neuroscience research. However, the performance of the frame-by-frame approach may degrade in scenarios with occlusions or motion blur. In this study, we propose a spatiotemporal network model based on YOLOv8 to enhance the accuracy of key-point detection in mouse behavioral experimental videos. This model integrates a time-domain tracking strategy comprising two components: the first part utilizes key-point detection results from the previous frame to detect potential target locations in the subsequent frame; the second part employs Kalman filtering to analyze key-point changes prior to detection, allowing for the estimation of missing key-points. In the comparison of pose estimation results between our approach, YOLOv8, DeepLabCut and SLEAP on videos of three mouse behavioral experiments, our approach demonstrated significantly superior performance. This suggests that our method offers a new and effective means of accurately tracking and estimating pose in mice through spatiotemporal processing.

An efficient visual servo tracker for herd monitoring by UAV

It is a challenging and meaningful task to carry out UAV-based livestock monitoring in high-altitude (more than 4500 m on average) and cold regions (annual average - 4 °C) on the Qinghai Tibet Plateau. The purpose of artificial intelligence (AI) is to execute automated tasks and to solve practical problems in actual applications by combining the software technology with the hardware carrier to create integrated advanced devices. Only in this way, the maximum value of AI could be realized. In this paper, a real-time tracking system with dynamic target tracking ability is proposed. It is developed based on the tracking-by-detection architecture using YOLOv7 and Deep SORT algorithms for target detection and tracking, respectively. In response to the problems encountered in the tracking process of complex and dense scenes, our work (1) Uses optical flow to compensate the Kalman filter, to solve the problem of mismatch between the target bounding box predicted by the Kalman filter (KF) and the input when the target detection in the current frame is complex, thereby improving the prediction accuracy; (2) Using a low confidence trajectory filtering method to reduce false positive trajectories generated by Deep SORT, thereby mitigating the impact of unreliable detection on target tracking. (3) A visual servo controller has been designed for the Unmanned Aerial Vehicle (UAV) to reduce the impact of rapid movement on tracking and ensure that the target is always within the field of view of the UAV camera, thereby achieving automatic tracking tasks. Finally, the system was tested using Tibetan yaks on the Qinghai Tibet Plateau as tracking targets, and the results showed that the system has real-time multi tracking ability and ideal visual servo effect in complex and dense scenes.

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Behavior contains rich structure across many timescales, but there is a dearth of methods to identify relevant components, especially over the longer periods required for learning and decision-making. Inspired by the goals and techniques of genome-wide association studies, we present a data-driven method-the choice-wide behavioral association study: CBAS-that systematically identifies such behavioral features. CBAS uses a powerful, resampling-based, method of multiple comparisons correction to identify sequences of actions or choices that either differ significantly between groups or significantly correlate with a covariate of interest. We apply CBAS to different tasks and species (flies, rats, and humans) and find, in all instances, that it provides interpretable information about each behavioral task.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Multidimensional Analysis of a Social Behavior Identifies Regression and Phenotypic Heterogeneity in a Female Mouse Model for Rett Syndrome

Regression is a key feature of neurodevelopmental disorders such as autism spectrum disorder, Fragile X syndrome, and Rett syndrome (RTT). RTT is caused by mutations in the X-linked gene methyl-CpG-binding protein 2 (MECP2). It is characterized by an early period of typical development with subsequent regression of previously acquired motor and speech skills in girls. The syndromic phenotypes are individualistic and dynamic over time. Thus far, it has been difficult to capture these dynamics and syndromic heterogeneity in the preclinical Mecp2-heterozygous female mouse model (Het). The emergence of computational neuroethology tools allows for robust analysis of complex and dynamic behaviors to model endophenotypes in preclinical models. Toward this first step, we utilized DeepLabCut, a marker-less pose estimation software to quantify trajectory kinematics and multidimensional analysis to characterize behavioral heterogeneity in Het in the previously benchmarked, ethologically relevant social cognition task of pup retrieval. We report the identification of two distinct phenotypes of adult Het: Het that display a delay in efficiency in early days and then improve over days like wild-type mice and Het that regress and perform worse in later days. Furthermore, regression is dependent on age and behavioral context and can be detected in the initial days of retrieval. Together, the novel identification of two populations of Het suggests differential effects on neural circuitry, opens new avenues to investigate the underlying molecular and cellular mechanisms of heterogeneity, and designs better studies for stratifying therapeutics.

FABEL: Forecasting Animal Behavioral Events with Deep Learning-Based Computer Vision

Behavioral neuroscience aims to provide a connection between neural phenomena and emergent organism-level behaviors. This requires perturbing the nervous system and observing behavioral outcomes, and comparing observed post-perturbation behavior with predicted counterfactual behavior and therefore accurate behavioral forecasts. In this study we present FABEL, a deep learning method for forecasting future animal behaviors and locomotion trajectories from historical locomotion alone. We train an offline pose estimation network to predict animal body-part locations in behavioral video; then sequences of pose vectors are input to deep learning time-series forecasting models. Specifically, we train an LSTM network that predicts a future food interaction event in a specified time window, and a Temporal Fusion Transformer that predicts future trajectories of animal body-parts, which are then converted into probabilistic label forecasts. Importantly, accurate prediction of food interaction provides a basis for neurobehavioral intervention in the context of compulsive eating. We show promising results on forecasting tasks between 100 milliseconds and 5 seconds timescales. Because the model takes only behavioral video as input, it can be adapted to any behavioral task and does not require specific physiological readouts. Simultaneously, these deep learning models may serve as extensible modules that can accommodate diverse signals, such as in-vivo fluorescence imaging and electrophysiology, which may improve behavior forecasts and elucidate invervention targets for desired behavioral change.

Fantastic beasts and how to study them: rethinking experimental animal behavior

Humans have been trying to understand animal behavior at least since recorded history. Recent rapid development of new technologies has allowed us to make significant progress in understanding the physiological and molecular mechanisms underlying behavior, a key goal of neuroethology. However, there is a tradeoff when studying animal behavior and its underlying biological mechanisms: common behavior protocols in the laboratory are designed to be replicable and controlled, but they often fail to encompass the variability and breadth of natural behavior. This Commentary proposes a framework of 10 key questions that aim to guide researchers in incorporating a rich natural context into their experimental design or in choosing a new animal study system. The 10 questions cover overarching experimental considerations that can provide a template for interspecies comparisons, enable us to develop studies in new model organisms and unlock new experiments in our quest to understand behavior.

Markerless Mouse Tracking for Social Experiments

Automated behavior quantification in socially interacting animals requires accurate tracking. While many methods have been very successful and highly generalizable to different settings, issues of mistaken identities and lost information on key anatomical features are common, although they can be alleviated by increased human effort in training or post-processing. We propose a markerless video-based tool to simultaneously track two interacting mice of the same appearance in controlled settings for quantifying behaviors such as different types of sniffing, touching, and locomotion to improve tracking accuracy under these settings without increased human effort. It incorporates conventional handcrafted tracking and deep-learning-based techniques. The tool is trained on a small number of manually annotated images from a basic experimental setup and outputs body masks and coordinates of the snout and tail-base for each mouse. The method was tested on several commonly used experimental conditions including bedding in the cage and fiberoptic or headstage implants on the mice. Results obtained without any human corrections after the automated analysis showed a near elimination of identities switches and a ∼15% improvement in tracking accuracy over pure deep-learning-based pose estimation tracking approaches. Our approach can be optionally ensembled with such techniques for further improvement. Finally, we demonstrated an application of this approach in studies of social behavior of mice by quantifying and comparing interactions between pairs of mice in which some lack olfaction. Together, these results suggest that our approach could be valuable for studying group behaviors in rodents, such as social interactions.