Quantifying behavior to understand the brain

Large-scale foundation models and generative AI for BigData neuroscience

Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.

Behavioral sequences across multiple animal species in the wild share common structural features

Animal behavior can be decomposed into a sequence of discrete activity bouts over time. Analyzing the statistical structure of such behavioral sequences can provide insights into the drivers of behavioral decisions. Laboratory studies, predominantly in invertebrates, have suggested that behavioral sequences exhibit multiple timescales and long-range memory, but whether these results can be generalized to other taxa and to animals in natural settings remains unclear. By analyzing accelerometer-inferred predictions of behavioral states in three species of social mammals (meerkats, white-nosed coatis, and spotted hyenas) in the wild, we found surprisingly consistent structuring of behavioral sequences across all behavioral states, all individuals, and all study species. Behavioral bouts were characterized by decreasing hazard functions, wherein the longer a behavioral bout had progressed, the less likely it was to end within the next instant. The predictability of an animal's future behavioral state as a function of its present state always decreased as a truncated power-law for predictions made farther into the future, with very similar estimates for the power law exponent across all species. Finally, the distributions of bout durations were also heavy-tailed. Why such shared structural principles emerge remains unknown, and we explore multiple plausible explanations, including environmental nonstationarity, behavioral self-reinforcement, and the hierarchical nature of behavior. The existence of highly consistent patterns in behavioral sequences across our study species suggests that these phenomena could be widespread in nature, and points to the existence of fundamental properties of behavioral dynamics that could drive such convergent patterns.

A circadian behavioral analysis suite for real-time classification of daily rhythms in complex behaviors

Long-term analysis of animal behavior has been limited by reliance on real-time sensors or manual scoring. Existing machine learning tools can automate analysis but often fail under variable conditions or ignore temporal dynamics. We developed a scalable pipeline for continuous, real-time acquisition and classification of behavior across multiple animals and conditions. At its core is a self-supervised vision model paired with a lightweight classifier that enables robust performance with minimal manual labeling. Our system achieves expert-level performance and can operate indefinitely across diverse recording environments. As a proof-of-concept, we recorded 97 mice over 2 weeks to test whether sex hormones influence circadian behaviors. We discovered sex- and estrogen-dependent rhythms in behaviors such as digging and nesting. We introduce the Circadian Behavioral Analysis Suite (CBAS), a modular toolkit that supports high-throughput, long-timescale behavioral phenotyping, allowing for the temporal analysis of behaviors that were previously difficult or impossible to observe.

Anti-drift pose tracker (ADPT), a transformer-based network for robust animal pose estimation cross-species

Deep learning-based methods have advanced animal pose estimation, enhancing accuracy, and efficiency in quantifying animal behavior. However, these methods frequently experience tracking drift, where noise-induced jumps in body point estimates compromise reliability. Here, we present the anti-drift pose tracker (ADPT), a transformer-based tool that mitigates tracking drift in behavioral analysis. Extensive experiments across cross-species datasets-including proprietary mouse and monkey recordings and public Drosophila and macaque datasets-demonstrate that ADPT significantly reduces drift and surpasses existing models like DeepLabCut and SLEAP in accuracy. Moreover, ADPT achieved 93.16% identification accuracy for 10 unmarked mice and 90.36% accuracy for freely interacting unmarked mice, which can be further refined to 99.72%, enhancing both anti-drift performance and pose estimation accuracy in social interactions. With its end-to-end design, ADPT is computationally efficient and suitable for real-time analysis, offering a robust solution for reproducible animal behavior studies. The ADPT code is available at https://github.com/tangguoling/ADPT.

A rat model of volitional mutual social interactions

Social interactions are essential for building societies and fostering cooperation among individuals. These behaviors are governed by complex norms and signaling mechanisms promoting mutual engagement. While animal models are often used to study social behaviors, they typically focus on one individual, overlooking the role and motivation of an otherwise passive social partner. Here, we developed a model where resident and partner rats voluntarily engage in mutual social interactions. In this model, the resident initiates interaction by pressing a lever to activate cues for the partner, who responds by pressing an additional lever, leading to social interaction. To test motivation for mutual social interaction, we increased the effort required for both residents and partners either concurrently or independently. We further investigated the mechanisms underlying these interactions by manipulating the norepinephrine system both systemically and centrally during mutual social interactions. Both male and female paired rats consistently demonstrate mutual motivation to engage in social interactions, regardless of their roles. The rats effectively coordinate their actions, showing low latency and high engagement frequency even as effort demands increase. The average social score analysis identified a significant proportion of highly motivated social pairs. Manipulating the norepinephrine system selectively disrupted the distribution of highly motivated social pairs, emphasizing its role in regulating social interactions. Ablating norepinephrine terminals had no impact on motivation for food rewards, further confirming that central norepinephrine manipulation specifically affects mutual social interactions. Our findings provide insight into the fundamental behavioral and neurobiological mechanisms underlying sociability and complex social structures in rodents.

Computer vision for primate behavior analysis in the wild

Advances in computer vision and increasingly widespread video-based behavioral monitoring are currently transforming how we study animal behavior. However, there is still a gap between the prospects and practical application, especially in videos from the wild. In this Perspective, we aim to present the capabilities of current methods for behavioral analysis, while at the same time highlighting unsolved computer vision problems that are relevant to the study of animal behavior. We survey state-of-the-art methods for computer vision problems relevant to the video-based study of individualized animal behavior, including object detection, multi-animal tracking, individual identification and (inter)action understanding. We then review methods for effort-efficient learning, one of the challenges from a practical perspective. In our outlook on the emerging field of computer vision for animal behavior, we argue that the field should develop approaches to unify detection, tracking, identification and (inter)action understanding in a single, video-based framework.

MINT: A toolbox for the analysis of multivariate neural information coding and transmission

Information theory has deeply influenced the conceptualization of brain information processing and is a mainstream framework for analyzing how neural networks in the brain process information to generate behavior. Information theory tools have been initially conceived and used to study how information about sensory variables is encoded by the activity of small neural populations. However, recent multivariate information theoretic advances have enabled addressing how information is exchanged across areas and used to inform behavior. Moreover, its integration with dimensionality-reduction techniques has enabled addressing information encoding and communication by the activity of large neural populations or many brain areas, as recorded by multichannel activity measurements in functional imaging and electrophysiology. Here, we provide a Multivariate Information in Neuroscience Toolbox (MINT) that combines these new methods with statistical tools for robust estimation from limited-size empirical datasets. We demonstrate the capabilities of MINT by applying it to both simulated and real neural data recorded with electrophysiology or calcium imaging, but all MINT functions are equally applicable to other brain-activity measurement modalities. We highlight the synergistic opportunities that combining its methods afford for reverse engineering of specific information processing and flow between neural populations or areas, and for discovering how information processing functions emerge from interactions between neurons or areas. MINT works on Linux, Windows and macOS operating systems, is written in MATLAB (requires MATLAB version 2018b or newer) and depends on 4 native MATLAB toolboxes. The calculation of one possible way to compute information redundancy requires the installation and compilation of C files (made available by us also as pre-compiled files). MINT is freely available at https://github.com/panzerilab/MINT with DOI doi.org/10.5281/zenodo.13998526 and operates under a GNU GPLv3 license.

InteBOMB: Integrating generic object tracking and segmentation with pose estimation for animal behavior analysis

Advancements in animal behavior quantification methods have driven the development of computational ethology, enabling fully automated behavior analysis. Existing multi-animal pose estimation workflows rely on tracking-by-detection frameworks for either bottom-up or top-down approaches, requiring retraining to accommodate diverse animal appearances. This study introduces InteBOMB, an integrated workflow that enhances top-down approaches by incorporating generic object tracking, eliminating the need for prior knowledge of target animals while maintaining broad generalizability. InteBOMB includes two key strategies for tracking and segmentation in laboratory environments and two techniques for pose estimation in natural settings. The "background enhancement" strategy optimizes foreground-background contrastive loss, generating more discriminative correlation maps. The "online proofreading" strategy stores human-in-the-loop long-term memory and dynamic short-term memory, enabling adaptive updates to object visual features. The "automated labeling suggestion" technique reuses the visual features saved during tracking to identify representative frames for training set labeling. Additionally, the "joint behavior analysis" technique integrates these features with multimodal data, expanding the latent space for behavior classification and clustering. To evaluate the framework, six datasets of mice and six datasets of non-human primates were compiled, covering laboratory and natural scenes. Benchmarking results demonstrated a 24% improvement in zero-shot generic tracking and a 21% enhancement in joint latent space performance across datasets, highlighting the effectiveness of this approach in robust, generalizable behavior analysis.

FlyVISTA, an integrated machine learning platform for deep phenotyping of sleep in Drosophila

There is great interest in using genetically tractable organisms such as Drosophila to gain insights into the regulation and function of sleep. However, sleep phenotyping in Drosophila has largely relied on simple measures of locomotor inactivity. Here, we present FlyVISTA, a machine learning platform to perform deep phenotyping of sleep in flies. This platform comprises a high-resolution closed-loop video imaging system, coupled with a deep learning network to annotate 35 body parts, and a computational pipeline to extract behaviors from high-dimensional data. FlyVISTA reveals the distinct spatiotemporal dynamics of sleep and wake-associated microbehaviors at baseline, following administration of the sleep-inducing drug gaboxadol, and with dorsal fan-shaped body drivers. We identify a microbehavior ("haltere switch") exclusively seen during quiescence that indicates a deeper sleep stage. These results enable the rigorous analysis of sleep in Drosophila and set the stage for computational analyses of microbehaviors in quiescent animals.

WildPose: a long-range 3D wildlife motion capture system

Understanding and monitoring wildlife behavior is crucial in ecology and biomechanics, yet challenging because of the limitations of current methods. To address this issue, we introduce WildPose, a novel long-range motion capture system specifically tailored for free-ranging wildlife observation. This system combines an electronically controllable zoom-lens camera with a LiDAR to capture both 2D videos and 3D point cloud data, thereby allowing researchers to observe high-fidelity animal morphometrics, behavior and interactions in a completely remote manner. Field trials conducted in Kgalagadi Transfrontier Park (South Africa) have successfully demonstrated WildPose's ability to quantify morphological features of different species, accurately track the 3D movements of a springbok herd over time, and observe the respiratory patterns of a distant lion. By facilitating non-intrusive, long-range 3D data collection, WildPose marks a significant complementary technique in ecological and biomechanical studies, offering new possibilities for conservation efforts and animal welfare, and enriching the prospects for interdisciplinary research.

Separating cognitive and motor processes in the behaving mouse

The cognitive processes supporting complex animal behavior are closely associated with movements responsible for critical processes, such as facial expressions or the active sampling of our environments. These movements are strongly related to neural activity across much of the brain and are often highly correlated with ongoing cognitive processes. A fundamental issue for understanding the neural signatures of cognition and movements is whether cognitive processes are separable from related movements or if they are driven by common neural mechanisms. Here we demonstrate how the separability of cognitive and motor processes can be assessed and, when separable, how the neural dynamics associated with each component can be isolated. We designed a behavioral task in mice that involves multiple cognitive processes, and we show that dynamics commonly taken to support cognitive processes are strongly contaminated by movements. When cognitive and motor components are isolated using a novel approach for subspace decomposition, we find that they exhibit distinct dynamical trajectories and are encoded by largely separate populations of cells. Accurately isolating dynamics associated with particular cognitive and motor processes will be essential for developing conceptual and computational models of neural circuit function.

Hierarchical behavioral analysis framework as a platform for standardized quantitative identification of behaviors

Behavior is composed of modules that operate based on inherent logic. Understanding behavior and its neural mechanisms is facilitated by clear structural behavioral analysis. Here, we developed a hierarchical behavioral analysis framework (HBAF) that efficiently reveals the organizational logic of these modules by analyzing high-dimensional behavioral data. By creating a spontaneous behavior atlas for male and female mice, we discovered that spontaneous behavior patterns are hardwired, with sniffing serving as the hub node for movement transitions. The sniffing-to-grooming ratio accurately distinguished the spontaneous behavioral states in a high-throughput manner. These states are influenced by emotional status, circadian rhythms, and lighting conditions, leading to unique behavioral characteristics, spatiotemporal features, and dynamic patterns. By implementing the straightforward and achievable spontaneous behavior paradigm, HBAF enables swift and accurate assessment of animal behavioral states and bridges the gap between a theoretical understanding of the behavioral structure and practical analysis using comprehensive multidimensional behavioral information.

A real-time, multi-subject three-dimensional pose tracking system for the behavioral analysis of non-human primates

The ability to track the positions and poses of multiple animals in three-dimensional (3D) space in real time is highly desired by non-human primate (NHP) researchers in behavioral and systems neuroscience. This capability enables the analysis of social behaviors involving multiple NHPs and supports closed-loop experiments. Although several animal 3D pose tracking systems have been developed, most are difficult to deploy in new environments and lack real-time analysis capabilities. To address these limitations, we developed MarmoPose, a deep-learning-based, real-time 3D pose tracking system for multiple common marmosets, an increasingly critical NHP model in neuroscience research. This system can accurately track the 3D poses of multiple marmosets freely moving in their home cage with minimal hardware requirements. By employing a marmoset skeleton model, MarmoPose can further optimize 3D poses and estimate invisible body locations. Additionally, MarmoPose achieves high inference speeds and enables real-time closed-loop experimental control based on events detected from 3D poses.

The utility of animal models to inform the next generation of human space exploration

Animals have played a vital role in every stage of space exploration, from early sub-orbital flights to contemporary missions. New physiological and psychological challenges arise with plans to venture deeper into the solar system. Advances in chimeric and knockout animal models, along with genetic modification techniques have enhanced our ability to study the effects of microgravity in greater detail. However, increased investment in the purposeful design of habitats and payloads, as well as in AI-enhanced behavioral monitoring in orbit can better support the ethical and effective use of animals in deep space research.

Temporal action localisation in video data containing rabbit behavioural patterns

In this paper we present the results of a research on artificial intelligence based approaches to temporal action localisation in video recordings of rabbit behavioural patterns. When using the artificial intelligence, special attention should be paid to quality and quantity of data collected for the research. Conducting the experiments in science may take long time and involve expensive preparatory work. Artificial intelligence based approaches can be applied to different kinds of actors in the video including animals, humans, intelligent agents, etc. The peculiarities of using these approaches in specific research conditions can be of particular importance for project cost reduction. In this paper we analyze the peculiarities of using the frame-by-frame classification based approach to temporal localisation of rabbit actions in video data and propose a metric for evaluating its consistency. The analysis of existing approaches described in the literature indicates that the aforementioned approach has high accuracy (up to 99%) and F1 score of temporal action localisation (up to 0.97) thus fulfilling conditions for substantial reduction or total exclusion of manual data labeling from the process of studying actor behaviour patterns in video data collected in experimental setting. We conducted further investigation in order to determine the optimal number of manually labeled frames required to achieve 99% accuracy of automatic labeling and studied the dependence of labeling accuracy on the number of actors presented in the training data.

In vivo electrophysiology recordings and computational modeling can predict octopus arm movement

The octopus has many features that make it advantageous for revealing principles of motor circuits and control and predicting behavior. Here, an array of carbon electrodes providing single-unit electrophysiology recordings were implanted into the octopus anterior nerve cord. The number of spikes and arm movements in response to stimulation at different locations along the arm were recorded. We observed that the number of spikes occurring within the first 100 ms after stimulation were predictive of the resultant movement response. Machine learning models showed that temporal electrophysiological features could be used to predict whether an arm movement occurred with 88.64% confidence, and if it was a lateral arm movement or a grasping motion with 75.45% confidence. Both supervised and unsupervised methods were applied to gain streaming measurements of octopus arm movements and how their motor circuitry produces rich movement types in real time. For kinematic analysis, deep learning models and unsupervised dimensionality reduction identified a consistent set of features that could be used to distinguish different types of arm movements. The neural circuits and the computational models identified here generated predictions for how to evoke a particular, complex movement in an orchestrated sequence for an individual motor circuit. This study demonstrates how real-time motor behaviors can be predicted and distinguished, contributing to the development of brain-machine interfaces. The ability to accurately model and predict complex movement patterns has broad implications for advancing technologies in robotics, neuroprosthetics, and artificial intelligence, paving the way for more sophisticated and adaptable systems.

Distinct contributions of BDNF/MEK/ERK1/2 signaling pathway components to whisker-dependent tactile learning and memory

Whisker-mediated tactile perception is essential for rodent navigation, food acquisition, and social interactions. However, the molecular mechanisms underlying tactile information processing, learning, and memory have not been studied to the same extent as for other modalities. Using immunohistochemical staining, we investigated changes in regional c-Fos expression as an index of neuronal activity and phosphorylated (p)ERK1/2 as an index of ERK1/2 activity in mice trained on a tactile-cued 8-arm radial maze task. Over 12 trials, mice learned to selectively explore four baited arms covered with wire as the tactile cue while avoiding un-baited uncovered arms. The density of c-Fos+ cells was significantly higher in somatosensory cortex but not frontal cortex or amygdala of mice exposed to tactile cue - bait pairing compared to mice exposed to the same maze with all arms baited with or without tactile cues (unpaired conditions). The density of pERK1/2+ cells was also increased after paired trials 7 and 12 but not after paired trials 1 and 3 in frontal cortex, amygdala, and somatosensory cortex compared to mice exposed to the unpaired condition. The MEK1/2 inhibitor SL327 reduced c-Fos expression in frontal cortex and amygdala when applied during early trials, but impaired working memory when applied before later trials without affecting c-Fos expression. Heterozygous BDNF knockout mice exhibited impaired task learning and reduced pERK1/2 expression in frontal cortex and amygdala but not somatosensory cortex. These findings suggest that the BDNF/MEK/ERK1/2 pathway selectively promotes memory trace formation in frontal cortex and amygdala but not encoding in somatosensory cortex.

The neurobiology of parenting and infant-evoked aggression

Parenting behavior comprises a variety of adult-infant and adult-adult interactions across multiple timescales. The state transition from nonparent to parent requires an extensive reorganization of individual priorities and physiology and is facilitated by combinatorial hormone action on specific cell types that are integrated throughout interconnected and brainwide neuronal circuits. In this review, we take a comprehensive approach to integrate historical and current literature on each of these topics across multiple species, with a focus on rodents. New and emerging molecular, circuit-based, and computational technologies have recently been used to address outstanding gaps in our current framework of knowledge on infant-directed behavior. This work is raising fundamental questions about the interplay between instinctive and learned components of parenting and the mutual regulation of affiliative versus agonistic infant-directed behaviors in health and disease. Whenever possible, we point to how these technologies have helped gain novel insights and opened new avenues of research into the neurobiology of parenting. We hope this review will serve as an introduction for those new to the field, a comprehensive resource for those already studying parenting, and a guidepost for designing future studies.

ONIX: a unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge we developed ONIX, an open-source data acquisition system with high data throughput (2 GB s-1) and low closed-loop latencies (<1 ms) that uses a 0.3-mm thin tether to minimize behavioral impact. Head position and rotation are tracked in three dimensions and used to drive active commutation without torque measurements. ONIX can acquire data from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, three-dimensional trackers and other data sources. We performed uninterrupted, long (~7 h) neural recordings in mice as they traversed complex three-dimensional terrain, and multiday sleep-tracking recordings (~55 h). ONIX enabled exploration with similar mobility as nonimplanted animals, in contrast to conventional tethered systems, which have restricted movement. By combining long recordings with full mobility, our technology will enable progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

The enigmatic HCN channels: A cellular neurophysiology perspective

What physiological role does a slow hyperpolarization-activated ion channel with mixed cation selectivity play in the fast world of neuronal action potentials that are driven by depolarization? That puzzling question has piqued the curiosity of physiology enthusiasts about the hyperpolarization-activated cyclic nucleotide-gated (HCN) channels, which are widely expressed across the body and especially in neurons. In this review, we emphasize the need to assess HCN channels from the perspective of how they respond to time-varying signals, while also accounting for their interactions with other co-expressing channels and receptors. First, we illustrate how the unique structural and functional characteristics of HCN channels allow them to mediate a slow negative feedback loop in the neurons that they express in. We present the several physiological implications of this negative feedback loop to neuronal response characteristics including neuronal gain, voltage sag and rebound, temporal summation, membrane potential resonance, inductive phase lead, spike triggered average, and coincidence detection. Next, we argue that the overall impact of HCN channels on neuronal physiology critically relies on their interactions with other co-expressing channels and receptors. Interactions with other channels allow HCN channels to mediate intrinsic oscillations, earning them the "pacemaker channel" moniker, and to regulate spike frequency adaptation, plateau potentials, neurotransmitter release from presynaptic terminals, and spike initiation at the axonal initial segment. We also explore the impact of spatially non-homogeneous subcellular distributions of HCN channels in different neuronal subtypes and their interactions with other channels and receptors. Finally, we discuss how plasticity in HCN channels is widely prevalent and can mediate different encoding, homeostatic, and neuroprotective functions in a neuron. In summary, we argue that HCN channels form an important class of channels that mediate a diversity of neuronal functions owing to their unique gating kinetics that made them a puzzle in the first place.

A modular machine learning tool for holistic and fine-grained behavioral analysis

Artificial intelligence techniques offer promising avenues for exploring human body features from videos, yet no freely accessible tool has reliably provided holistic and fine-grained behavioral analyses to date. To address this, we developed a machine learning tool based on a two-level approach: a first lower-level processing using computer vision for extracting fine-grained and comprehensive behavioral features such as skeleton or facial points, gaze, and action units; a second level of machine learning classification coupled with explainability providing modularity, to determine which behavioral features are triggered by specific environments. To validate our tool, we filmed 16 participants across six conditions, varying according to the presence of a person ("Pers"), a sound ("Snd"), or silence ("Rest"), and according to emotional levels using self-referential ("Self") and control ("Ctrl") stimuli. We demonstrated the effectiveness of our approach by extracting and correcting behavior from videos using two computer vision software (OpenPose and OpenFace) and by training two algorithms (XGBoost and long short-term memory [LSTM]) to differentiate between experimental conditions. High classification rates were achieved for "Pers" conditions versus "Snd" or "Rest" (AUC = 0.8-0.9), with explainability revealing actions units and gaze as key features. Additionally, moderate classification rates were attained for "Snd" versus "Rest" (AUC = 0.7), attributed to action units, limbs and head points, as well as for "Self" versus "Ctrl" (AUC = 0.7-0.8), due to facial points. These findings were consistent with a more conventional hypothesis-driven approach. Overall, our study suggests that our tool is well suited for holistic and fine-grained behavioral analysis and offers modularity for extension into more complex naturalistic environments.

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

Unsupervised discovery of family specific vocal usage in the Mongolian gerbil

In nature, animal vocalizations can provide crucial information about identity, including kinship and hierarchy. However, lab-based vocal behavior is typically studied during brief interactions between animals with no prior social relationship, and under environmental conditions with limited ethological relevance. Here, we address this gap by establishing long-term acoustic recordings from Mongolian gerbil families, a core social group that uses an array of sonic and ultrasonic vocalizations. Three separate gerbil families were transferred to an enlarged environment and continuous 20-day audio recordings were obtained. Using a variational autoencoder (VAE) to quantify 583,237 vocalizations, we show that gerbils exhibit a more elaborate vocal repertoire than has been previously reported and that vocal repertoire usage differs significantly by family. By performing gaussian mixture model clustering on the VAE latent space, we show that families preferentially use characteristic sets of vocal clusters and that these usage preferences remain stable over weeks. Furthermore, gerbils displayed family-specific transitions between vocal clusters. Since gerbils live naturally as extended families in complex underground burrows that are adjacent to other families, these results suggest the presence of a vocal dialect which could be exploited by animals to represent kinship. These findings position the Mongolian gerbil as a compelling animal model to study the neural basis of vocal communication and demonstrates the potential for using unsupervised machine learning with uninterrupted acoustic recordings to gain insights into naturalistic animal behavior.

SAUSI: an integrative assay for measuring social aversion and motivation

Social aversion is a key feature of numerous mental health disorders such as Social Anxiety and Autism Spectrum Disorders. Nevertheless, the biobehavioral mechanisms underlying social aversion remain poorly understood. Progress in understanding the etiology of social aversion has been hindered by the lack of comprehensive tools to assess social aversion in model systems. Here, we created a new behavioral task - Selective Access to Unrestricted Social Interaction (SAUSI), which integrates elements of social motivation, hesitancy, decision-making, and free interaction to enable the wholistic assessment of social aversion in mice. Using this novel assay, we found that social isolation-induced social aversion in mice is largely driven by increases in social fear and social motivation. Deep learning analyses revealed a unique behavioral footprint underlying the socially aversive state produced by isolation, demonstrating the compatibility of modern computational approaches with SAUSI. Social aversion was further assessed using traditional assays - including the 3-chamber sociability assay and the resident intruder assay - which were sufficient to reveal fragments of a social aversion phenotype, including changes to either social motivation or social interaction, but which failed to provide a wholistic assessment of social aversion. Critically, these assays were not sufficient to reveal key components of social aversion, including social freezing and social hesitancy behaviors. Lastly, we demonstrated that SAUSI is generalizable, as it can be used to assess social aversion induced by non-social stressors, such as foot shock. Our findings debut a novel task for the behavioral toolbox - one which overcomes limitations of previous assays, allowing for both social choice as well as free interaction, and offers a new approach for assessing social aversion in rodents.

SARTAB, a scalable system for automated real-time behavior detection based on animal tracking and Region Of Interest analysis: validation on fish courtship behavior

Methods from Machine Learning (ML) and Computer Vision (CV) have proven powerful tools for quickly and accurately analyzing behavioral recordings. The computational complexity of these techniques, however, often precludes applications that require real-time analysis: for example, experiments where a stimulus must be applied in response to a particular behavior or samples must be collected soon after the behavior occurs. Here, we describe SARTAB (Scalable Automated Real-Time Analysis of Behavior), a system that achieves automated real-time behavior detection by continuously monitoring animal positions relative to behaviorally relevant Regions Of Interest (ROIs). We then show how we used this system to detect infrequent courtship behaviors in Pseudotropheus demasoni (a species of Lake Malawi African cichlid fish) to collect neural tissue samples from actively behaving individuals for multiomic profiling at single nucleus resolution. Within this experimental context, we achieve high ROI and animal detection accuracies (mAP@[.5 : .95] of 0.969 and 0.718, respectively) and 100% classification accuracy on a set of 32 manually selected behavioral clips. SARTAB is unique in that all analysis runs on low-cost, edge-deployed hardware, making it a highly scalable and energy-efficient solution for real-time experimental feedback. Although our solution was developed specifically to study cichlid courtship behavior, the intrinsic flexibility of neural network analysis ensures that our approach can be adapted to novel species, behaviors, and environments.

Integrating computer vision and molecular neurobiology to bridge the gap between behavior and the brain

The past decade of social insect research has seen rapid development in automated behavioral tracking and molecular profiling of the nervous system, two distinct but complementary lines of inquiry into phenotypic variation across individuals, colonies, populations, and species. These experimental strategies have developed largely in parallel, as automated tracking generates a continuous stream of behavioral data, while, in contrast, 'omics-based profiling provides a single 'snapshot' of the brain. Better integration of these approaches applied to studying variation in social behavior will reveal the underlying genetic and neurobiological mechanisms that shape the evolution and diversification of social life. In this review, we discuss relevant advances in both fields and propose new strategies to better elucidate the molecular and behavioral innovations that generate social life.

Analysis of behavioral flow resolves latent phenotypes

The accurate detection and quantification of rodent behavior forms a cornerstone of basic biomedical research. Current data-driven approaches, which segment free exploratory behavior into clusters, suffer from low statistical power due to multiple testing, exhibit poor transferability across experiments and fail to exploit the rich behavioral profiles of individual animals. Here we introduce a pipeline to capture each animal's behavioral flow, yielding a single metric based on all observed transitions between clusters. By stabilizing these clusters through machine learning, we ensure data transferability, while dimensionality reduction techniques facilitate detailed analysis of individual animals. We provide a large dataset of 771 behavior recordings of freely moving mice-including stress exposures, pharmacological and brain circuit interventions-to identify hidden treatment effects, reveal subtle variations on the level of individual animals and detect brain processes underlying specific interventions. Our pipeline, compatible with popular clustering methods, substantially enhances statistical power and enables predictions of an animal's future behavior.

Uncovering multiscale structure in the variability of larval zebrafish navigation

Animals chain movements into long-lived motor strategies, exhibiting variability across scales that reflects the interplay between internal states and environmental cues. To reveal structure in such variability, we build Markov models of movement sequences that bridge across timescales and enable a quantitative comparison of behavioral phenotypes among individuals. Applied to larval zebrafish responding to diverse sensory cues, we uncover a hierarchy of long-lived motor strategies, dominated by changes in orientation distinguishing cruising versus wandering strategies. Environmental cues induce preferences along these modes at the population level: while fish cruise in the light, they wander in response to aversive stimuli, or in search for appetitive prey. As our method encodes the behavioral dynamics of each individual fish in the transitions among coarse-grained motor strategies, we use it to uncover a hierarchical structure in the phenotypic variability that reflects exploration-exploitation trade-offs. Across a wide range of sensory cues, a major source of variation among fish is driven by prior and/or immediate exposure to prey that induces exploitation phenotypes. A large degree of variability that is not explained by environmental cues unravels hidden states that override the sensory context to induce contrasting exploration-exploitation phenotypes. Altogether, by extracting the timescales of motor strategies deployed during navigation, our approach exposes structure among individuals and reveals internal states tuned by prior experience.

Segmentation tracking and clustering system enables accurate multi-animal tracking of social behaviors

Accurate analysis of social behaviors in animals is hindered by methodological challenges. Here, we develop a segmentation tracking and clustering system (STCS) to address two major challenges in computational neuroethology: reliable multi-animal tracking and pose estimation under complex interaction conditions and providing interpretable insights into social differences guided by genotype information. We established a comprehensive, long-term, multi-animal-tracking dataset across various experimental settings. Benchmarking STCS against state-of-the-art tracking algorithms, we demonstrated its superior efficacy in analyzing behavioral experiments and establishing a robust tracking baseline. By analyzing the behavior of mice with autism spectrum disorder (ASD) using a novel weakly supervised clustering method under both solitary and social conditions, STCS reveals potential links between social stress and motor impairments. Benefiting from its modular and web-based design, STCS allows researchers to easily integrate the latest computer vision methods, enabling comprehensive behavior analysis services over the Internet, even from a single laptop.

Representing the dynamics of natural marmoset vocal behaviors in frontal cortex

Here, we tested the respective contributions of primate premotor and prefrontal cortex to support vocal behavior. We applied a model-based generalized linear model (GLM) analysis that better accounts for the inherent variance in natural, continuous behaviors to characterize the activity of neurons throughout the frontal cortex as freely moving marmosets engaged in conversational exchanges. While analyses revealed functional clusters of neural activity related to the different processes involved in the vocal behavior, these clusters did not map to subfields of prefrontal or premotor cortex, as has been observed in more conventional task-based paradigms. Our results suggest a distributed functional organization for the myriad neural mechanisms underlying natural social interactions and have implications for our concepts of the role that frontal cortex plays in governing ethological behaviors in primates.

Revisiting serotonin's role in spatial memory: A call for sensitive analytical approaches

The serotonergic system is involved in various psychiatric and neurological conditions, with serotonergic drugs often used in treatment. These conditions frequently affect spatial memory, which can serve as a model of declarative memory due to well-known cellular components and advanced methods that track neural activity and behavior with high temporal resolution. However, most findings on serotonin's effects on spatial learning and memory come from studies lacking refined analytical techniques and modern approaches needed to uncover the underlying neuronal mechanisms. This In Focus review critically investigates available studies to identify areas for further exploration. It finds that well-established behavioral models could yield more insights with modern tracking and data analysis approaches, while the cellular aspects of spatial memory remain underexplored. The review highlights the complex role of serotonin in spatial memory, which holds the potential for better understanding and treating memory-related disorders.

Characterizing the structure of mouse behavior using Motion Sequencing

Spontaneous mouse behavior is composed from repeatedly used modules of movement (e.g., rearing, running or grooming) that are flexibly placed into sequences whose content evolves over time. By identifying behavioral modules and the order in which they are expressed, researchers can gain insight into the effect of drugs, genes, context, sensory stimuli and neural activity on natural behavior. Here we present a protocol for performing Motion Sequencing (MoSeq), an ethologically inspired method that uses three-dimensional machine vision and unsupervised machine learning to decompose spontaneous mouse behavior into a series of elemental modules called 'syllables'. This protocol is based upon a MoSeq pipeline that includes modules for depth video acquisition, data preprocessing and modeling, as well as a standardized set of visualization tools. Users are provided with instructions and code for building a MoSeq imaging rig and acquiring three-dimensional video of spontaneous mouse behavior for submission to the modeling framework; the outputs of this protocol include syllable labels for each frame of the video data as well as summary plots describing how often each syllable was used and how syllables transitioned from one to the other. In addition, we provide instructions for analyzing and visualizing the outputs of keypoint-MoSeq, a recently developed variant of MoSeq that can identify behavioral motifs from keypoints identified from standard (rather than depth) video. This protocol and the accompanying pipeline significantly lower the bar for users without extensive computational ethology experience to adopt this unsupervised, data-driven approach to characterize mouse behavior.

Understanding resurgence and other emergent activity with the laws of allocation, induction, and covariance

Resurgence is defined as an increase in a previously extinguished target response (B1) resulting from the worsening of conditions for a more recently reinforced alternative response (B2). Worsening includes extinction or reductions in rate, amount, and immediacy of delivery of food or some other phylogenetically important event (PIE). In the first part of the article, we apply the laws of allocation, induction, and covariance to understand not only resurgence of operant activity previously covarying with the PIE (B1) but also a constellation of ontogenetic and phylogenetic activities both related to the PIE (B0) and unrelated to the PIE (BN). In the second part, we discuss how induction might be incorporated into and provide alternative processes within an existing matching-based framework, resurgence as choice (RaC). We begin to identify how this range of activities could depend on changes in the relative competitive weight (V) of all available activities (B1, B2, B0, BN) in addition to only those receiving explicit training (B1, B2). Future empirical and theoretical research is needed within this framework to provide a more complete understanding of resurgence and behavior more generally.

Single unit electrophysiology recordings and computational modeling can predict octopus arm movement

The octopus simplified nervous system holds the potential to reveal principles of motor circuits and improve brain-machine interface devices through computational modeling with machine learning and statistical analysis. Here, an array of carbon electrodes providing single-unit electrophysiology recordings were implanted into the octopus anterior nerve cord. The number of spikes and arm movements in response to stimulation at different locations along the arm were recorded. We observed that the number of spikes occurring within the first 100ms after stimulation were predictive of the resultant movement response. Computational models showed that temporal electrophysiological features could be used to predict whether an arm movement occurred with 88.64% confidence, and if it was a lateral arm movement or a grasping motion with 75.45% confidence. Both supervised and unsupervised methods were applied to gain streaming measurements of octopus arm movements and how their motor circuitry produces rich movement types in real time. Deep learning models and unsupervised dimension reduction identified a consistent set of features that could be used to distinguish different types of arm movements. These models generated predictions for how to evoke a particular, complex movement in an orchestrated sequence for an individual motor circuit.

A behavioral analysis system MCFBM enables objective inference of songbirds' attention during social interactions

Understanding animal behavior is crucial in behavioral neuroscience, aiming to unravel the mechanisms driving these behaviors. A significant milestone in this field is the analysis of behavioral reactions during social interactions. Despite their importance in social learning, the behavioral aspects of these interaction are not well understood in detail due to the lack of appropriate tools. We introduce a high-precision, marker-based motion-capture system for analyzing behavior in songbirds, accurately tracking body location and head direction in multiple freely moving finches during social interaction. Focusing on zebra finches, our analysis revealed variations in eye use based on individuals presented. We also observed behavioral changes during virtual and live presentations and a conditioned-learning paradigm. Additionally, the system effectively analyzed social interactions among mice. This system provides an efficient tool for advanced behavioral analysis in small animals and offers an objective method to infer their focus of attention.

DAMM for the detection and tracking of multiple animals within complex social and environmental settings

Accurate detection and tracking of animals across diverse environments are crucial for studying brain and behavior. Recently, computer vision techniques have become essential for high-throughput behavioral studies; however, localizing animals in complex conditions remains challenging due to intra-class visual variability and environmental diversity. These challenges hinder studies in naturalistic settings, such as when animals are partially concealed within nests. Moreover, current tools are laborious and time-consuming, requiring extensive, setup-specific annotation and training procedures. To address these challenges, we introduce the 'Detect-Any-Mouse-Model' (DAMM), an object detector for localizing mice in complex environments with minimal training. Our approach involved collecting and annotating a diverse dataset of single- and multi-housed mice in complex setups. We trained a Mask R-CNN, a popular object detector in animal studies, to perform instance segmentation and validated DAMM's performance on a collection of downstream datasets using zero-shot and few-shot inference. DAMM excels in zero-shot inference, detecting mice and even rats, in entirely unseen scenarios and further improves with minimal training. Using the SORT algorithm, we demonstrate robust tracking, competitive with keypoint-estimation-based methods. Notably, to advance and simplify behavioral studies, we release our code, model weights, and data, along with a user-friendly Python API and a Google Colab implementation.

Unsupervised discovery of family specific vocal usage in the Mongolian gerbil

In nature, animal vocalizations can provide crucial information about identity, including kinship and hierarchy. However, lab-based vocal behavior is typically studied during brief interactions between animals with no prior social relationship, and under environmental conditions with limited ethological relevance. Here, we address this gap by establishing long-term acoustic recordings from Mongolian gerbil families, a core social group that uses an array of sonic and ultrasonic vocalizations. Three separate gerbil families were transferred to an enlarged environment and continuous 20-day audio recordings were obtained. Using a variational autoencoder (VAE) to quantify 583,237 vocalizations, we show that gerbils exhibit a more elaborate vocal repertoire than has been previously reported and that vocal repertoire usage differs significantly by family. By performing gaussian mixture model clustering on the VAE latent space, we show that families preferentially use characteristic sets of vocal clusters and that these usage preferences remain stable over weeks. Furthermore, gerbils displayed family-specific transitions between vocal clusters. Since gerbils live naturally as extended families in complex underground burrows that are adjacent to other families, these results suggest the presence of a vocal dialect which could be exploited by animals to represent kinship. These findings position the Mongolian gerbil as a compelling animal model to study the neural basis of vocal communication and demonstrates the potential for using unsupervised machine learning with uninterrupted acoustic recordings to gain insights into naturalistic animal behavior.

Formalising the role of behaviour in neuroscience

We develop a mathematical approach to formally proving that certain neural computations and representations exist based on patterns observed in an organism's behaviour. To illustrate, we provide a simple set of conditions under which an ant's ability to determine how far it is from its nest would logically imply neural structures isomorphic to the natural numbers ℕ . We generalise these results to arbitrary behaviours and representations and show what mathematical characterisation of neural computation and representation is simplest while being maximally predictive of behaviour. We develop this framework in detail using a path integration example, where an organism's ability to search for its nest in the correct location implies representational structures isomorphic to two-dimensional coordinates under addition. We also study a system for processinga n b n strings common in comparative work. Our approach provides an objective way to determine what theory of a physical system is best, addressing a fundamental challenge in neuroscientific inference. These results motivate considering which neurobiological structures have the requisite formal structure and are otherwise physically plausible given relevant physical considerations such as generalisability, information density, thermodynamic stability and energetic cost.

Deep autoencoder-based behavioral pattern recognition outperforms standard statistical methods in high-dimensional zebrafish studies

Zebrafish have become an essential model organism in screening for developmental neurotoxic chemicals and their molecular targets. The success of zebrafish as a screening model is partially due to their physical characteristics including their relatively simple nervous system, rapid development, experimental tractability, and genetic diversity combined with technical advantages that allow for the generation of large amounts of high-dimensional behavioral data. These data are complex and require advanced machine learning and statistical techniques to comprehensively analyze and capture spatiotemporal responses. To accomplish this goal, we have trained semi-supervised deep autoencoders using behavior data from unexposed larval zebrafish to extract quintessential "normal" behavior. Following training, our network was evaluated using data from larvae shown to have significant changes in behavior (using a traditional statistical framework) following exposure to toxicants that include nanomaterials, aromatics, per- and polyfluoroalkyl substances (PFAS), and other environmental contaminants. Further, our model identified new chemicals (Perfluoro-n-octadecanoic acid, 8-Chloroperfluorooctylphosphonic acid, and Nonafluoropentanamide) as capable of inducing abnormal behavior at multiple chemical-concentrations pairs not captured using distance moved alone. Leveraging this deep learning model will allow for better characterization of the different exposure-induced behavioral phenotypes, facilitate improved genetic and neurobehavioral analysis in mechanistic determination studies and provide a robust framework for analyzing complex behaviors found in higher-order model systems.

A circadian behavioral analysis suite for real-time classification of daily rhythms in complex behaviors

Measuring animal behavior over long timescales has been traditionally limited to behaviors that are easily measurable with real-time sensors. More complex behaviors have been measured over time, but these approaches are considerably more challenging due to the intensive manual effort required for scoring behaviors. Recent advances in machine learning have introduced automated behavior analysis methods, but these often overlook long-term behavioral patterns and struggle with classification in varying environmental conditions. To address this, we developed a pipeline that enables continuous, parallel recording and acquisition of animal behavior for an indefinite duration. As part of this pipeline, we applied a recent breakthrough self-supervised computer vision model to reduce training bias and overfitting and to ensure classification robustness. Our system automatically classifies animal behaviors with a performance approaching that of expert-level human labelers. Critically, classification occurs continuously, across multiple animals, and in real time. As a proof-of-concept, we used our system to record behavior from 97 mice over two weeks to test the hypothesis that sex and estrogen influence circadian rhythms in nine distinct home cage behaviors. We discovered novel sex- and estrogen-dependent differences in circadian properties of several behaviors including digging and nesting rhythms. We present a generalized version of our pipeline and novel classification model, the "circadian behavioral analysis suite," (CBAS) as a user-friendly, open-source software package that allows researchers to automatically acquire and analyze behavioral rhythms with a throughput that rivals sensor-based methods, allowing for the temporal and circadian analysis of behaviors that were previously difficult or impossible to observe.

A Markovian dynamics for Caenorhabditis elegans behavior across scales

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm Caenorhabditis elegans, we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences, we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm's foraging behavior, revealing both "runs-and-pirouettes" as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies.

Exploring motion using geometric morphometrics in microscopic aquatic invertebrates: 'modes' and movement patterns during feeding in a bdelloid rotifer model species

BACKGROUND

Movement is a defining aspect of animals, but it is rarely studied using quantitative methods in microscopic invertebrates. Bdelloid rotifers are a cosmopolitan class of aquatic invertebrates of great scientific interest because of their ability to survive in very harsh environment and also because they represent a rare example of an ancient lineage that only includes asexually reproducing species. In this class, Adineta ricciae has become a model species as it is unusually easy to culture. Yet, relatively little is known of its ethology and almost nothing on how it behaves during feeding.

METHODS

To explore feeding behaviour in A. ricciae, as well as to provide an example of application of computational ethology in a microscopic invertebrate, we apply Procrustes motion analysis in combination with ordination and clustering methods to a laboratory bred sample of individuals recorded during feeding.

RESULTS

We demonstrate that movement during feeding can be accurately described in a simple two-dimensional shape space with three main 'modes' of motion. Foot telescoping, with the body kept straight, is the most frequent 'mode', but it is accompanied by periodic rotations of the foot together with bending while the foot is mostly retracted.

CONCLUSIONS

Procrustes motion analysis is a relatively simple but effective tool for describing motion during feeding in A. ricciae. The application of this method generates quantitative data that could be analysed in relation to genetic and ecological differences in a variety of experimental settings. The study provides an example that is easy to replicate in other invertebrates, including other microscopic animals whose behavioural ecology is often poorly known.

A Deep Learning Approach to Estimate Multi-Level Mental Stress From EEG Using Serious Games

Stress is revealed by the inability of individuals to cope with their environment, which is frequently evidenced by a failure to achieve their full potential in tasks or goals. This study aims to assess the feasibility of estimating the level of stress that the user is perceiving related to a specific task through an electroencephalograpic (EEG) system. This system is integrated with a Serious Game consisting of a multi-level stress driving tool, and Deep Learning (DL) neural networks are used for classification. The game involves controlling a vehicle to dodge obstacles, with the number of obstacles increasing based on complexity. Assuming that there is a direct correlation between the difficulty level of the game and the stress level of the user, a recurrent neural network (RNN) with a structure based on gated recurrent units (GRU) was used to classify the different levels of stress. The results show that the RNN model is able to predict stress levels above current state-of-the-art with up to 94% accuracy in some cases, suggesting that the use of EEG systems in combination with Serious Games and DL represents a promising technique in the prediction and classification of mental stress levels.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ('syllables') from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.

Computer vision analysis of mother-infant interaction identified efficient pup retrieval in V1b receptor knockout mice

Close contact between lactating rodent mothers and their infants is essential for effective nursing. Whether the mother's effort to retrieve the infants to their nest requires the vasopressin-signaling via V1b receptor has not been fully defined. To address this question, V1b receptor knockout (V1bKO) and control mice were analyzed in pup retrieval test. Because an exploring mother in a new test cage randomly accessed to multiple infants in changing backgrounds over time, a computer vision-based deep learning analysis was applied to continuously calculate the distances between the mother and the infants as a parameter of their relationship. In an open-field, a virgin female V1bKO mice entered fewer times into the center area and moved shorter distances than wild-type (WT). While this behavioral pattern persisted in V1bKO mother, the pup retrieval test demonstrated that total distances between a V1bKO mother and infants came closer in a shorter time than with a WT mother. Moreover, in the medial preoptic area, parts of the V1b receptor transcripts were detected in galanin- and c-fos-positive neurons following maternal stimulation by infants. This research highlights the effectiveness of deep learning analysis in evaluating the mother-infant relationship and the critical role of V1b receptor in pup retrieval during the early lactation phase.

Temporally aligned segmentation and clustering (TASC) framework for behavior time series analysis

Behavior exhibits a complex spatiotemporal structure consisting of discrete sub-behaviors, or motifs. Continuous behavior data requires segmentation and clustering to reveal these embedded motifs. The popularity of automatic behavior quantification is growing, but existing solutions are often tailored to specific needs and are not designed for the time scale and precision required in many experimental and clinical settings. Here we propose a generalized framework with an iterative approach to refine both segmentation and clustering. Temporally aligned segmentation and clustering (TASC) uses temporal linear alignment to compute distances between and align the recurring behavior motifs in a multidimensional time series, enabling precise segmentation and clustering. We introduce an alternating-step process: evaluation of temporal neighbors against current cluster centroids using linear alignment, alternating with selecting the best non-overlapping segments and their subsequent re-clustering. The framework is evaluated on semi-synthetic and real-world experimental and clinical data, demonstrating enhanced segmentation and clustering, offering a better foundation for consequent research. The framework may be used to extend existing tools in the field of behavior research and may be applied to other domains requiring high precision of time series segmentation.

Identifying behavioral links to neural dynamics of multifiber photometry recordings in a mouse social behavior network

Objective.Distributed hypothalamic-midbrain neural circuits help orchestrate complex behavioral responses during social interactions. Given rapid advances in optical imaging, it is a fundamental question how population-averaged neural activity measured by multi-fiber photometry (MFP) for calcium fluorescence signals correlates with social behaviors is a fundamental question. This paper aims to investigate the correspondence between MFP data and social behaviors.Approach:We propose a state-space analysis framework to characterize mouse MFP data based on dynamic latent variable models, which include a continuous-state linear dynamical system and a discrete-state hidden semi-Markov model. We validate these models on extensive MFP recordings during aggressive and mating behaviors in male-male and male-female interactions, respectively.Main results:Our results show that these models are capable of capturing both temporal behavioral structure and associated neural states, and produce interpretable latent states. Our approach is also validated in computer simulations in the presence of known ground truth.Significance:Overall, these analysis approaches provide a state-space framework to examine neural dynamics underlying social behaviors and reveals mechanistic insights into the relevant networks.

SuperAnimal pretrained pose estimation models for behavioral analysis

Quantification of behavior is critical in diverse applications from neuroscience, veterinary medicine to animal conservation. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present SuperAnimal, a method to develop unified foundation models that can be used on over 45 species, without additional manual labels. These models show excellent performance across six pose estimation benchmarks. We demonstrate how to fine-tune the models (if needed) on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If fine-tuned, SuperAnimal models are 10-100× more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification and kinematic analysis. Collectively, we present a data-efficient solution for animal pose estimation.

Toward a neuroscience of natural behavior

One of the most exciting new developments in systems neuroscience is the progress being made toward neurophysiological experiments that move beyond simplified laboratory settings and address the richness of natural behavior. This is enabled by technological advances such as wireless recording in freely moving animals, automated quantification of behavior, and new methods for analyzing large data sets. Beyond new empirical methods and data, however, there is also a need for new theories and concepts to interpret that data. Such theories need to address the particular challenges of natural behavior, which often differ significantly from the scenarios studied in traditional laboratory settings. Here, we discuss some strategies for developing such novel theories and concepts and some example hypotheses being proposed.

An ethologically motivated neurobiology of primate visually-guided reach-to-grasp behavior

The precision of primate visually guided reaching likely evolved to meet the many challenges faced by living in arboreal environments, yet much of what we know about the underlying primate brain organization derives from a set of highly constrained experimental paradigms. Here we review the role of vision to guide natural reach-to-grasp movements in marmoset monkey prey capture to illustrate the breadth and diversity of these behaviors in ethological contexts, the fast predictive nature of these movements [1,2], and the advantages of this particular primate model to investigate the underlying neural mechanisms in more naturalistic contexts [3]. In addition to their amenability to freely-moving neural recording methods for investigating the neural basis of dynamic ethological behaviors [4,5], marmosets have a smooth neocortical surface that facilitates imaging and array recordings [6,7] in all areas in the primate fronto-parietal network [8,9]. Together, this model organism offers novel opportunities to study the real-world interplay between primate vision and reach-to-grasp dynamics using ethologically motivated neuroscientific experimental designs.