Mapping the stereotyped behaviour of freely moving fruit flies

Representation of high-dimensional cell morphology and morphodynamics in 2D latent space

The morphology and morphodynamics of cells as important biomarkers of the cellular state are widely appreciated in both fundamental research and clinical applications. Quantification of cell morphology often requires a large number of geometric measures that form a high-dimensional feature vector. This mathematical representation creates barriers to communicating, interpreting, and visualizing data. Here, we develop a deep learning-based algorithm to project 13-dimensional (13D) morphological feature vectors into 2-dimensional (2D) morphological latent space (MLS). We show that the projection has less than 5% information loss and separates the different migration phenotypes of metastatic breast cancer cells. Using the projection, we demonstrate the phenotype-dependent motility of breast cancer cells in the 3D extracellular matrix, and the continuous cell state change upon drug treatment. We also find that dynamics in the 2D MLS quantitatively agrees with the morphodynamics of cells in the 13D feature space, preserving the diffusive power and the Lyapunov exponent of cell shape fluctuations even though the dimensional reduction projection is highly nonlinear. Our results suggest that MLS is a powerful tool to represent and understand the cell morphology and morphodynamics.

Complexity reduction by symmetry: Uncovering the minimal regulatory network for logical computation in bacteria

Symmetry principles play an important role in geometry, and physics, allowing for the reduction of complicated systems to simpler, more comprehensible models that preserve the system's features of interest. Biological systems are often highly complex and may consist of a large number of interacting parts. Using symmetry fibrations, the relevant symmetries for biological "message-passing" networks, we introduce a scheme, called Complexity Reduction by Symmetry or CoReSym, to reduce the gene regulatory networks of Escherichia coli and Bacillus subtilis bacteria to core networks in a way that preserves the dynamics and uncovers the computational capabilities of the network. Gene nodes in the original network that share isomorphic input trees are collapsed by the fibration into equivalence classes called fibers, whereby nodes that receive signals with the same "history" belong to one fiber and synchronize. Then we reduce the networks to its minimal computational core via k-core decomposition. This computational core consists of a few strongly connected components or "signal vortices," in which signals can cycle through. While between them, these "signal vortices" transmit signals in a feedforward manner. These connected components perform signal processing and decision making in the bacterial cell by employing a series of genetic toggle-switch circuits that store memory, plus oscillator circuits. These circuits act as the central computation device of the network, whose output signals then spread to the rest of the network. Our reduction method opens the door to narrow the vast complexity of biological systems to their minimal parts in a systematic way by using fundamental theoretical principles of symmetry.

Statistical signature of subtle behavioral changes in large-scale assays

The central nervous system can generate various behaviors, including motor responses, which we can observe through video recordings. Recent advances in gene manipulation, automated behavioral acquisition at scale, and machine learning enable us to causally link behaviors to their underlying neural mechanisms. Moreover, in some animals, such as the Drosophila melanogaster larva, this mapping is possible at the unprecedented scale of single neurons, allowing us to identify the neural microcircuits generating particular behaviors. These high-throughput screening efforts, linking the activation or suppression of specific neurons to behavioral patterns in millions of animals, provide a rich dataset to explore the diversity of nervous system responses to the same stimuli. However, important challenges remain in identifying subtle behaviors, including immediate and delayed responses to neural activation or suppression, and understanding these behaviors on a large scale. We here introduce several statistically robust methods for analyzing behavioral data in response to these challenges: 1) A generative physical model that regularizes the inference of larval shapes across the entire dataset. 2) An unsupervised kernel-based method for statistical testing in learned behavioral spaces aimed at detecting subtle deviations in behavior. 3) A generative model for larval behavioral sequences, providing a benchmark for identifying higher-order behavioral changes. 4) A comprehensive analysis technique using suffix trees to categorize genetic lines into clusters based on common action sequences. We showcase these methodologies through a behavioral screen focused on responses to an air puff, analyzing data from 280 716 larvae across 569 genetic lines.

A detailed quantification of larval zebrafish behavioral repertoire uncovers principles of hunting behavior

During goal-directed behavior, animals select actions from a diverse repertoire of movements. Accurately quantifying this complex and high-dimensional repertoire is essential to uncovering the underlying rules that guide the behavior. Here, we developed a low-dimensional mathematical model that accurately reproduces the complete and continuous repertoire of hunting larval zebrafish. We show that fish position and change in heading angle following a movement are coupled, such that the choice of one of them limits the other. This repertoire structure uncovered fundamental principles of movements, showing that fish rotate around an identified rotation point and then move forward or backward. Moreover, it identified a new guiding rule of the hunt: fish turn to face the prey in each movement and then move forward or backward. These results present the first low-dimensional and continuous description of movements repertoire, uncover guiding behavioral rules, and offer insights into the underlying motor control.

Mapping the landscape of social behavior

Social interaction is integral to animal behavior. However, lacking tools to describe it in quantitative and rigorous ways has limited our understanding of its structure, underlying principles, and the neuropsychiatric disorders, like autism, that perturb it. Here, we present a technique for high-resolution 3D tracking of postural dynamics and social touch in freely interacting animals, solving the challenging subject occlusion and part-assignment problems using 3D geometric reasoning, graph neural networks, and semi-supervised learning. We collected over 110 million 3D pose samples in interacting rats and mice, including seven monogenic autism rat lines. Using a multi-scale embedding approach, we identified a rich landscape of stereotyped actions, interactions, synchrony, and body contacts. This high-resolution phenotyping revealed a spectrum of changes in autism models and in response to amphetamine not resolved by conventional measurements. Our framework and large library of interactions will facilitate studies of social behaviors and their neurobiological underpinnings.

An Automated AI Framework for Quantitative Measurement of Mammalian Behavior

Despite the large amount of video data captured during ethological studies of wild mammals, there is no widely accepted method available to automatically and quantitatively measure and analyze animal behavior. We developed a framework using facial recognition and deep learning to automatically track, measure, and quantify the behavior of single or multiple individuals from 10 distinct mammalian taxa, including three species of primates, three species of bovids, three species of carnivores, and one species of equid. We used spatiotemporal information based on animal skeleton models to recognize a set of distinct behaviors such as walking, feeding, grooming, and resting, and achieved an accuracy ranging from 0.82 to 0.96. Accuracies of validation videos ranged from 0.80 to 0.99. Our study offers an innovative analytical platform for the rapid and accurate evaluation of animal behavior in both captive and field settings.

Using AI to prevent the insect apocalypse: toward new environmental risk assessment procedures

Insect populations are declining globally, with multiple potential drivers identified. However, experimental data are needed to understand their relative contributions. We highlight the sublethal effects of pesticides at field-relevant concentrations, often overlooked in standard environmental risk assessments (ERA), as significant contributors to these declines. Behavior, as an easily monitored high-level phenotype, reflects alterations at various phenotypic levels. We propose incorporating behavioral assays with AI-based analytical methods into ERA protocols to better assess the safety of molecules intended for large-scale field use. This approach aims to safeguard food supplies and protect vital ecosystems in the future.

FlyVISTA, an integrated machine learning platform for deep phenotyping of sleep in Drosophila

There is great interest in using genetically tractable organisms such as Drosophila to gain insights into the regulation and function of sleep. However, sleep phenotyping in Drosophila has largely relied on simple measures of locomotor inactivity. Here, we present FlyVISTA, a machine learning platform to perform deep phenotyping of sleep in flies. This platform comprises a high-resolution closed-loop video imaging system, coupled with a deep learning network to annotate 35 body parts, and a computational pipeline to extract behaviors from high-dimensional data. FlyVISTA reveals the distinct spatiotemporal dynamics of sleep and wake-associated microbehaviors at baseline, following administration of the sleep-inducing drug gaboxadol, and with dorsal fan-shaped body drivers. We identify a microbehavior ("haltere switch") exclusively seen during quiescence that indicates a deeper sleep stage. These results enable the rigorous analysis of sleep in Drosophila and set the stage for computational analyses of microbehaviors in quiescent animals.

A neural correlate of individual odor preference in Drosophila

Behavior varies even among genetically identical animals raised in the same environment. However, little is known about the circuit or anatomical origins of this individuality. Here, we demonstrate a neural correlate of Drosophila odor preference behavior in the olfactory sensory periphery. Namely, idiosyncratic calcium responses in projection neuron (PN) dendrites and densities of the presynaptic protein Bruchpilot in olfactory receptor neuron (ORN) axon terminals correlate with individual preferences in a choice between two aversive odorants. The ORN-PN synapse appears to be a locus of individuality where microscale variation gives rise to idiosyncratic behavior. Simulating microscale stochasticity in ORN-PN synapses of a 3062 neuron model of the antennal lobe recapitulates patterns of variation in PN calcium responses matching experiments. Conversely, stochasticity in other compartments of this circuit does not recapitulate those patterns. Our results demonstrate how physiological and microscale structural circuit variations can give rise to individual behavior, even when genetics and environment are held constant.

Hierarchical behavioral analysis framework as a platform for standardized quantitative identification of behaviors

Behavior is composed of modules that operate based on inherent logic. Understanding behavior and its neural mechanisms is facilitated by clear structural behavioral analysis. Here, we developed a hierarchical behavioral analysis framework (HBAF) that efficiently reveals the organizational logic of these modules by analyzing high-dimensional behavioral data. By creating a spontaneous behavior atlas for male and female mice, we discovered that spontaneous behavior patterns are hardwired, with sniffing serving as the hub node for movement transitions. The sniffing-to-grooming ratio accurately distinguished the spontaneous behavioral states in a high-throughput manner. These states are influenced by emotional status, circadian rhythms, and lighting conditions, leading to unique behavioral characteristics, spatiotemporal features, and dynamic patterns. By implementing the straightforward and achievable spontaneous behavior paradigm, HBAF enables swift and accurate assessment of animal behavioral states and bridges the gap between a theoretical understanding of the behavioral structure and practical analysis using comprehensive multidimensional behavioral information.

On growth and form of animal behavior

In this study we propose an architecture (bauplan) for the growth and form of behavior in vertebrates and arthropods. We show in what sense behavior is an extension of anatomy. Then we show that movement-based behavior shares linearity and modularity with the skeletal body plan, and with the Hox genes; that it mirrors the geometry of the physical environment; and that it reveals the animal's understanding of the animate and physical situation, with implications for perception, attention, emotion, and primordial cognition. First we define the primitives of movement in relational terms, as in comparative anatomy, yielding homological primitives. Then we define modules, generative rules and the architectural plan of behavior in terms of these primitives. In this way we expose the homology of behaviors, and establish a rigorous trans-phyletic comparative discipline of the morphogenesis of movement-based behavior. In morphogenesis, behavior builds up and narrows incessantly according to strict geometric rules. The same rules apply in moment-to-moment behavior, in ontogenesis, and partly also in phylogenesis. We demonstrate these rules in development, in neurological recovery, with drugs (dopamine-stimulated striatal modulation), in stressful situations, in locomotor behavior, and partly also in human pathology. The buildup of movement culminates in free, undistracted, exuberant behavior. It is observed in play, in superior animals during agonistic interactions, and in humans in higher states of functioning. Geometrization promotes the study of genetics, anatomy, and behavior within one and the same discipline. The geometrical bauplan portrays both already evolved dimensions, and prospective dimensional constraints on evolutionary behavioral innovations.

Capturing continuous, long timescale behavioral changes in Drosophila melanogaster postural data

Animal behavior spans many timescales, from short, seconds-scale actions to daily rhythms over many hours to life-long changes during aging. To access longer timescales of behavior, we continuously recorded individual Drosophila melanogaster at 100 frames per second for up to 7 days at a time in featureless arenas on sucrose-agarose media. We use the deep learning framework SLEAP to produce a full-body postural dataset for 47 individuals resulting in nearly 2 billion pose instances. We identify stereotyped behaviors such as grooming, proboscis extension, and locomotion and use the resulting ethograms to explore how the flies' behavior varies across time of day and days in the experiment. We find distinct daily patterns in all stereotyped behaviors, adding specific information about trends in different grooming modalities, proboscis extension duration, and locomotion speed to what is known about the D. melanogaster circadian cycle. Using our holistic measurements of behavior, we find that the hour after dawn is a unique time point in the flies' daily pattern of behavior, and that the behavioral composition of this hour tracks well with other indicators of health such as locomotion speed and the fraction of time spend moving vs. resting. The method, data, and analysis presented here give us a new and clearer picture of D. melanogaster behavior across timescales, revealing novel features that hint at unexplored underlying biological mechanisms.

Collective swimming pattern and synchronization of fish pairs (Gobiocypris rarus) in response to flow with different velocities

Collective behaviors in moving fish originate from social interactions, which are thought to be driven by beneficial factors, such as predator avoidance and reduced energy expenditure. Despite numerical simulations and physical experiments aiming at the hydrodynamic mechanisms and interaction rules, how shoaling is influenced by flow velocity and group size is still only partially understood. In this study, spatial distributions, kinematics, and synchronization states between pairs (smallest subsystem of a shoal) of Gobiocypris rarus were investigated in a recirculating swim tunnel with increasing flow velocities from 0.1 to 0.5 m/s (Ucrit = 0.6 m/s). Tests of single fish were also conducted as the control group. The results of spatial distributions showed that fish pairs preferred to swim in the side-by-side configuration under high flows, while under low flows the neighboring fish's positions were more uniformly distributed around the focal fish in the transverse direction. Kinematic analysis revealed that fish pairs adopted similar tail beats (i.e., frequency and Strouhal number) as single fish in low flows, while in high flows both the frequency and Strouhal number of fish pairs were slightly lower. Moreover, the synchronization rates of fish pairs were found to increase with flow velocities, suggesting that synchronized swimming may be beneficial, especially in high flows.

Tensor analysis of animal behavior by matricization and feature selection

Contemporary neurobehavior research often collects multi-dimensional tensor (MDT) data, consisting of time-series measurements for multiple features from multiple animals subjected to various perturbations. Proper analysis of the MDT data can facilitate the dissection of the underlying neural circuitry driving the behavior. However, many common approaches for MDT analysis, such as tensor decomposition, often yield results that are difficult to interpret and not directly compatible with standard multivariate analysis (MVA), which is designed for simpler, lower-dimensional data structures. To address this issue, dimensionality reduction techniques, including matricization methods such as Index Construction and Feature Concatenation, are applied to transform all or a subset of the features in the MDT into a lower-dimensional tensor, commonly a 2-dimensional tensor (2DT), that is compatible with MVA. However, the matricization methods may exclude information from the MDT features or create too many 2DT features that introduce spurious noise to the downstream analyses. Their impacts on the downstream MVA performance remain elusive. In this study, we systematically evaluated different approaches for matricization and feature selection and their impacts on MVA performance using an MDT dataset of zebrafish visual- motor response collected from wild-types (WTs) and visually-impaired mutants. We matricized the MDT dataset using various Index Construction and Feature Concatenation methods, then identified informative 2DT features using the filter and embedded methods. To evaluate these feature-selection approaches, we conducted a classification task distinguishing WT and visually-impaired zebrafish by multiple classifiers. We then assessed classification performance with cross-validation and holdout validation. We found that most classifiers performed the best when using all 2DT features matricized by Feature Concatenation and selected by the embedded method. The results also revealed unique behavioral differences between the WTs and visually-impaired mutants that were not identified by standard MVA or MDT analysis. Our results demonstrate the utility of analyzing MDT behavioral data by matricization and feature selection.

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Objective.Creating an intracortical brain computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.Approach.We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.Main results.We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network decoder with 10-12 clusters.Significance.This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

ONIX: a unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge we developed ONIX, an open-source data acquisition system with high data throughput (2 GB s-1) and low closed-loop latencies (<1 ms) that uses a 0.3-mm thin tether to minimize behavioral impact. Head position and rotation are tracked in three dimensions and used to drive active commutation without torque measurements. ONIX can acquire data from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, three-dimensional trackers and other data sources. We performed uninterrupted, long (~7 h) neural recordings in mice as they traversed complex three-dimensional terrain, and multiday sleep-tracking recordings (~55 h). ONIX enabled exploration with similar mobility as nonimplanted animals, in contrast to conventional tethered systems, which have restricted movement. By combining long recordings with full mobility, our technology will enable progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms - which include tree-based models, deep neural networks, and graphical models - differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species - fly, mouse, and human - we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

SAUSI: an integrative assay for measuring social aversion and motivation

Social aversion is a key feature of numerous mental health disorders such as Social Anxiety and Autism Spectrum Disorders. Nevertheless, the biobehavioral mechanisms underlying social aversion remain poorly understood. Progress in understanding the etiology of social aversion has been hindered by the lack of comprehensive tools to assess social aversion in model systems. Here, we created a new behavioral task - Selective Access to Unrestricted Social Interaction (SAUSI), which integrates elements of social motivation, hesitancy, decision-making, and free interaction to enable the wholistic assessment of social aversion in mice. Using this novel assay, we found that social isolation-induced social aversion in mice is largely driven by increases in social fear and social motivation. Deep learning analyses revealed a unique behavioral footprint underlying the socially aversive state produced by isolation, demonstrating the compatibility of modern computational approaches with SAUSI. Social aversion was further assessed using traditional assays - including the 3-chamber sociability assay and the resident intruder assay - which were sufficient to reveal fragments of a social aversion phenotype, including changes to either social motivation or social interaction, but which failed to provide a wholistic assessment of social aversion. Critically, these assays were not sufficient to reveal key components of social aversion, including social freezing and social hesitancy behaviors. Lastly, we demonstrated that SAUSI is generalizable, as it can be used to assess social aversion induced by non-social stressors, such as foot shock. Our findings debut a novel task for the behavioral toolbox - one which overcomes limitations of previous assays, allowing for both social choice as well as free interaction, and offers a new approach for assessing social aversion in rodents.

Heightened lateral habenula activity during stress produces brainwide and behavioral substrates of susceptibility

Some individuals are susceptible to chronic stress, and others are more resilient. While many brain regions implicated in learning are dysregulated after stress, little is known about whether and how neural teaching signals during stress differ between susceptible and resilient individuals. Here, we seek to determine if activity in the lateral habenula (LHb), which encodes a negative teaching signal, differs between susceptible and resilient mice during stress to produce different outcomes. After (but not before) chronic social defeat stress, the LHb is active when susceptible mice are in proximity of the aggressor strain. During stress, activity is higher in susceptible mice during aggressor interactions, and activation biases mice toward susceptibility. This manipulation generates a persistent and widespread increase in the balance of subcortical vs. cortical activity in susceptible mice. Taken together, our results indicate that heightened activity in the LHb during stress produces lasting brainwide and behavioral substrates of susceptibility.

Analysis of behavioral flow resolves latent phenotypes

The accurate detection and quantification of rodent behavior forms a cornerstone of basic biomedical research. Current data-driven approaches, which segment free exploratory behavior into clusters, suffer from low statistical power due to multiple testing, exhibit poor transferability across experiments and fail to exploit the rich behavioral profiles of individual animals. Here we introduce a pipeline to capture each animal's behavioral flow, yielding a single metric based on all observed transitions between clusters. By stabilizing these clusters through machine learning, we ensure data transferability, while dimensionality reduction techniques facilitate detailed analysis of individual animals. We provide a large dataset of 771 behavior recordings of freely moving mice-including stress exposures, pharmacological and brain circuit interventions-to identify hidden treatment effects, reveal subtle variations on the level of individual animals and detect brain processes underlying specific interventions. Our pipeline, compatible with popular clustering methods, substantially enhances statistical power and enables predictions of an animal's future behavior.

Hierarchical behavior control by a single class of interneurons

Animal behavior is organized into nested temporal patterns that span multiple timescales. This behavior hierarchy is believed to arise from a hierarchical neural architecture: Neurons near the top of the hierarchy are involved in planning, selecting, initiating, and maintaining motor programs, whereas those near the bottom of the hierarchy act in concert to produce fine spatiotemporal motor activity. In Caenorhabditis elegans, behavior on a long timescale emerges from ordered and flexible transitions between different behavioral states, such as forward, reversal, and turn. On a short timescale, different parts of the animal body coordinate fast rhythmic bending sequences to produce directional movements. Here, we show that Sublateral Anterior A (SAA), a class of interneurons that enable cross-communication between dorsal and ventral head motor neurons, play a dual role in shaping behavioral dynamics on different timescales. On a short timescale, SAA regulate and stabilize rhythmic bending activity during forward movements. On a long timescale, the same neurons suppress spontaneous reversals and facilitate reversal termination by inhibiting Ring Interneuron M (RIM), an integrating neuron that helps maintain a behavioral state. These results suggest that feedback from a lower-level cell assembly to a higher-level command center is essential for bridging behavioral dynamics at different levels.

Characterizing Behavioral Dynamics in Bipolar Disorder with Computational Ethology

New technologies for the quantification of behavior have revolutionized animal studies in social, cognitive, and pharmacological neurosciences. However, comparable studies in understanding human behavior, especially in psychiatry, are lacking. In this study, we utilized data-driven machine learning to analyze natural, spontaneous open-field human behaviors from people with euthymic bipolar disorder (BD) and non-BD participants. Our computational paradigm identified representations of distinct sets of actions (motifs) that capture the physical activities of both groups of participants. We propose novel measures for quantifying dynamics, variability, and stereotypy in BD behaviors. These fine-grained behavioral features reflect patterns of cognitive functions of BD and better predict BD compared with traditional ethological and psychiatric measures and action recognition approaches. This research represents a significant computational advancement in human ethology, enabling the quantification of complex behaviors in real-world conditions and opening new avenues for characterizing neuropsychiatric conditions from behavior.

Dynamic modulation of social gaze by sex and familiarity in marmoset dyads

Social communication relies on the ability to perceive and interpret the direction of others' attention, and is commonly conveyed through head orientation and gaze direction in humans and nonhuman primates. However, traditional social gaze experiments in nonhuman primates require restraining head movements, significantly limiting their natural behavioral repertoire. Here, we developed a novel framework for accurately tracking facial features and three-dimensional head gaze orientations of multiple freely moving common marmosets (Callithrix jacchus). By combining deep learning-based computer vision tools with triangulation algorithms, we were able to track the facial features of marmoset dyads within an arena. This method effectively generates dynamic 3D geometrical facial frames while overcoming common challenges like occlusion. To detect the head gaze direction, we constructed a virtual cone, oriented perpendicular to the facial frame. Using this pipeline, we quantified different types of interactive social gaze events, including partner-directed gaze and joint gaze to a shared spatial location. We observed clear effects of sex and familiarity on both interpersonal distance and gaze dynamics in marmoset dyads. Unfamiliar pairs exhibited more stereotyped patterns of arena occupancy, more sustained levels of social gaze across social distance, and increased social gaze monitoring. On the other hand, familiar pairs exhibited higher levels of joint gazes. Moreover, males displayed significantly elevated levels of gazes toward females' faces and the surrounding regions, irrespective of familiarity. Our study reveals the importance of two key social factors in driving the gaze behaviors of a prosocial primate species and lays the groundwork for a rigorous quantification of primate behaviors in naturalistic settings.

Characterizing the structure of mouse behavior using Motion Sequencing

Spontaneous mouse behavior is composed from repeatedly used modules of movement (e.g., rearing, running or grooming) that are flexibly placed into sequences whose content evolves over time. By identifying behavioral modules and the order in which they are expressed, researchers can gain insight into the effect of drugs, genes, context, sensory stimuli and neural activity on natural behavior. Here we present a protocol for performing Motion Sequencing (MoSeq), an ethologically inspired method that uses three-dimensional machine vision and unsupervised machine learning to decompose spontaneous mouse behavior into a series of elemental modules called 'syllables'. This protocol is based upon a MoSeq pipeline that includes modules for depth video acquisition, data preprocessing and modeling, as well as a standardized set of visualization tools. Users are provided with instructions and code for building a MoSeq imaging rig and acquiring three-dimensional video of spontaneous mouse behavior for submission to the modeling framework; the outputs of this protocol include syllable labels for each frame of the video data as well as summary plots describing how often each syllable was used and how syllables transitioned from one to the other. In addition, we provide instructions for analyzing and visualizing the outputs of keypoint-MoSeq, a recently developed variant of MoSeq that can identify behavioral motifs from keypoints identified from standard (rather than depth) video. This protocol and the accompanying pipeline significantly lower the bar for users without extensive computational ethology experience to adopt this unsupervised, data-driven approach to characterize mouse behavior.

An inhibitory acetylcholine receptor gates context-dependent mechanosensory processing in C. elegans

An animal's current behavior influences its response to sensory stimuli, but the molecular and circuit-level mechanisms of this context-dependent decision-making are not well understood. Caenorhabditis elegans are less likely to respond to a mechanosensory stimulus by reversing if the stimuli is received while the animal turns. Inhibitory feedback from turning associated neurons are needed for this gating. But until now, it has remained unknown precisely where in the circuit gating occurs and which specific neurons and receptors receive inhibition from the turning circuitry. Here, we use genetic manipulations, single-cell rescue experiments, and high-throughput closed-loop optogenetic perturbations during behavior to reveal the specific neuron and receptor responsible for receiving inhibition and altering sensorimotor processing. Our measurements show that an inhibitory acetylcholine-gated chloride channel comprising LGC-47 and ACC-1 expressed in neuron type RIM disrupts mechanosensory evoked reversals during turns, presumably in response to inhibitory signals from turning-associated neuron SAA.

Mapping the landscape of social behavior

Social interaction is integral to animal behavior. However, we lack tools to describe it with quantitative rigor, limiting our understanding of its principles and neuropsychiatric disorders, like autism, that perturb it. Here, we present a technique for high-resolution 3D tracking of postural dynamics and social touch in freely interacting animals, solving the challenging subject occlusion and part assignment problems using 3D geometric reasoning, graph neural networks, and semi-supervised learning. We collected over 140 million 3D postures in interacting rodents, featuring new monogenic autism rat lines lacking reports of social behavioral phenotypes. Using a novel multi-scale embedding approach, we identified a rich landscape of stereotyped actions, interactions, synchrony, and body contact. This enhanced phenotyping revealed a spectrum of changes in autism models and in response to amphetamine that were inaccessible to conventional measurements. Our framework and large library of interactions will greatly facilitate studies of social behaviors and their neurobiological underpinnings.

Mapping the developmental structure of stereotyped and individual-unique behavioral spaces in C. elegans

Developmental patterns of behavior are variably organized in time and among different individuals. However, long-term behavioral diversity was previously studied using pre-defined behavioral parameters, representing a limited fraction of the full individuality structure. Here, we continuously extract ∼1.2 billion body postures of ∼2,200 single C. elegans individuals throughout their full development time to create a complete developmental atlas of stereotyped and individual-unique behavioral spaces. Unsupervised inference of low-dimensional movement modes of each single individual identifies a dynamic developmental trajectory of stereotyped behavioral spaces and exposes unique behavioral trajectories of individuals that deviate from the stereotyped patterns. Moreover, classification of behavioral spaces within tens of neuromodulatory and environmentally perturbed populations shows plasticity in the temporal structures of stereotyped behavior and individuality. These results present a comprehensive atlas of continuous behavioral dynamics across development time and a general framework for unsupervised dissection of shared and unique developmental signatures of behavior.

An explainable map of human gastruloid morphospace reveals gastrulation failure modes and predicts teratogens

Human gastrulation is a critical stage of development where many pregnancies fail due to poorly understood mechanisms. Using the 2D gastruloid, a stem cell model of human gastrulation, we combined high-throughput drug perturbations and mathematical modelling to create an explainable map of gastruloid morphospace. This map outlines patterning outcomes in response to diverse perturbations and identifies variations in canonical patterning and failure modes. We modeled morphogen dynamics to embed simulated gastruloids into experimentally-determined morphospace to explain how developmental parameters drive patterning. Our model predicted and validated the two greatest sources of patterning variance: cell density-based modulations in Wnt signaling and SOX2 stability. Assigning these parameters as axes of morphospace imparted interpretability. To demonstrate its utility, we predicted novel teratogens that we validated in zebrafish. Overall, we show how stem cell models of development can be used to build a comprehensive and interpretable understanding of the set of developmental outcomes.

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Objective

Creating an intracortical brain-computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.

Approach

We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.

Main results

We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network (RNN) decoder with 10-12 clusters.

Significance

This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

Unsupervised decomposition of natural monkey behavior into a sequence of motion motifs

Nonhuman primates (NHPs) exhibit complex and diverse behavior that typifies advanced cognitive function and social communication, but quantitative and systematical measure of this natural nonverbal processing has been a technical challenge. Specifically, a method is required to automatically segment time series of behavior into elemental motion motifs, much like finding meaningful words in character strings. Here, we propose a solution called SyntacticMotionParser (SMP), a general-purpose unsupervised behavior parsing algorithm using a nonparametric Bayesian model. Using three-dimensional posture-tracking data from NHPs, SMP automatically outputs an optimized sequence of latent motion motifs classified into the most likely number of states. When applied to behavioral datasets from common marmosets and rhesus monkeys, SMP outperformed conventional posture-clustering models and detected a set of behavioral ethograms from publicly available data. SMP also quantified and visualized the behavioral effects of chemogenetic neural manipulations. SMP thus has the potential to dramatically improve our understanding of natural NHP behavior in a variety of contexts.

A Markovian dynamics for Caenorhabditis elegans behavior across scales

How do we capture the breadth of behavior in animal movement, from rapid body twitches to aging? Using high-resolution videos of the nematode worm Caenorhabditis elegans, we show that a single dynamics connects posture-scale fluctuations with trajectory diffusion and longer-lived behavioral states. We take short posture sequences as an instantaneous behavioral measure, fixing the sequence length for maximal prediction. Within the space of posture sequences, we construct a fine-scale, maximum entropy partition so that transitions among microstates define a high-fidelity Markov model, which we also use as a means of principled coarse-graining. We translate these dynamics into movement using resistive force theory, capturing the statistical properties of foraging trajectories. Predictive across scales, we leverage the longest-lived eigenvectors of the inferred Markov chain to perform a top-down subdivision of the worm's foraging behavior, revealing both "runs-and-pirouettes" as well as previously uncharacterized finer-scale behaviors. We use our model to investigate the relevance of these fine-scale behaviors for foraging success, recovering a trade-off between local and global search strategies.

A virtual rodent predicts the structure of neural activity across behaviours

Animals have exquisite control of their bodies, allowing them to perform a diverse range of behaviours. How such control is implemented by the brain, however, remains unclear. Advancing our understanding requires models that can relate principles of control to the structure of neural activity in behaving animals. Here, to facilitate this, we built a 'virtual rodent', in which an artificial neural network actuates a biomechanically realistic model of the rat1 in a physics simulator2. We used deep reinforcement learning3-5 to train the virtual agent to imitate the behaviour of freely moving rats, thus allowing us to compare neural activity recorded in real rats to the network activity of a virtual rodent mimicking their behaviour. We found that neural activity in the sensorimotor striatum and motor cortex was better predicted by the virtual rodent's network activity than by any features of the real rat's movements, consistent with both regions implementing inverse dynamics6. Furthermore, the network's latent variability predicted the structure of neural variability across behaviours and afforded robustness in a way consistent with the minimal intervention principle of optimal feedback control7. These results demonstrate how physical simulation of biomechanically realistic virtual animals can help interpret the structure of neural activity across behaviour and relate it to theoretical principles of motor control.

Exploring motion using geometric morphometrics in microscopic aquatic invertebrates: 'modes' and movement patterns during feeding in a bdelloid rotifer model species

BACKGROUND

Movement is a defining aspect of animals, but it is rarely studied using quantitative methods in microscopic invertebrates. Bdelloid rotifers are a cosmopolitan class of aquatic invertebrates of great scientific interest because of their ability to survive in very harsh environment and also because they represent a rare example of an ancient lineage that only includes asexually reproducing species. In this class, Adineta ricciae has become a model species as it is unusually easy to culture. Yet, relatively little is known of its ethology and almost nothing on how it behaves during feeding.

METHODS

To explore feeding behaviour in A. ricciae, as well as to provide an example of application of computational ethology in a microscopic invertebrate, we apply Procrustes motion analysis in combination with ordination and clustering methods to a laboratory bred sample of individuals recorded during feeding.

RESULTS

We demonstrate that movement during feeding can be accurately described in a simple two-dimensional shape space with three main 'modes' of motion. Foot telescoping, with the body kept straight, is the most frequent 'mode', but it is accompanied by periodic rotations of the foot together with bending while the foot is mostly retracted.

CONCLUSIONS

Procrustes motion analysis is a relatively simple but effective tool for describing motion during feeding in A. ricciae. The application of this method generates quantitative data that could be analysed in relation to genetic and ecological differences in a variety of experimental settings. The study provides an example that is easy to replicate in other invertebrates, including other microscopic animals whose behavioural ecology is often poorly known.

Bridging model and experiment in systems neuroscience with Cleo: the Closed-Loop, Electrophysiology, and Optophysiology simulation testbed

Systems neuroscience has experienced an explosion of new tools for reading and writing neural activity, enabling exciting new experiments such as all-optical or closed-loop control that effect powerful causal interventions. At the same time, improved computational models are capable of reproducing behavior and neural activity with increasing fidelity. Unfortunately, these advances have drastically increased the complexity of integrating different lines of research, resulting in the missed opportunities and untapped potential of suboptimal experiments. Experiment simulation can help bridge this gap, allowing model and experiment to better inform each other by providing a low-cost testbed for experiment design, model validation, and methods engineering. Specifically, this can be achieved by incorporating the simulation of the experimental interface into our models, but no existing tool integrates optogenetics, two-photon calcium imaging, electrode recording, and flexible closed-loop processing with neural population simulations. To address this need, we have developed Cleo: the Closed-Loop, Electrophysiology, and Optophysiology experiment simulation testbed. Cleo is a Python package enabling injection of recording and stimulation devices as well as closed-loop control with realistic latency into a Brian spiking neural network model. It is the only publicly available tool currently supporting two-photon and multi-opsin/wavelength optogenetics. To facilitate adoption and extension by the community, Cleo is open-source, modular, tested, and documented, and can export results to various data formats. Here we describe the design and features of Cleo, validate output of individual components and integrated experiments, and demonstrate its utility for advancing optogenetic techniques in prospective experiments using previously published systems neuroscience models.

An LSTM-based adversarial variational autoencoder framework for self-supervised neural decoding of behavioral choices

Objective.This paper presents data-driven solutions to address two challenges in the problem of linking neural data and behavior: (1) unsupervised analysis of behavioral data and automatic label generation from behavioral observations, and (2) extraction of subject-invariant features for the development of generalized neural decoding models.Approach. For behavioral analysis and label generation, an unsupervised method, which employs an autoencoder to transform behavioral data into a cluster-friendly feature space is presented. The model iteratively refines the assigned clusters with soft clustering assignment loss, and gradually improves the learned feature representations. To address subject variability in decoding neural activity, adversarial learning in combination with a long short-term memory-based adversarial variational autoencoder (LSTM-AVAE) model is employed. By using an adversary network to constrain the latent representations, the model captures shared information among subjects' neural activity, making it proper for cross-subject transfer learning.Main results. The proposed approach is evaluated using cortical recordings of Thy1-GCaMP6s transgenic mice obtained via widefield calcium imaging during a motivational licking behavioral experiment. The results show that the proposed model achieves an accuracy of 89.7% in cross-subject neural decoding, outperforming other well-known autoencoder-based feature learning models. These findings suggest that incorporating an adversary network eliminates subject dependency in representations, leading to improved cross-subject transfer learning performance, while also demonstrating the effectiveness of LSTM-based models in capturing the temporal dependencies within neural data.Significance. Results demonstrate the feasibility of the proposed framework in unsupervised clustering and label generation of behavioral data, as well as achieving high accuracy in cross-subject neural decoding, indicating its potentials for relating neural activity to behavior.

Heightened lateral habenula activity during stress produces brainwide and behavioral substrates of susceptibility

Some individuals are susceptible to the experience of chronic stress and others are more resilient. While many brain regions implicated in learning are dysregulated after stress, little is known about whether and how neural teaching signals during stress differ between susceptible and resilient individuals. Here, we seek to determine if activity in the lateral habenula (LHb), which encodes a negative teaching signal, differs between susceptible and resilient mice during stress to produce different outcomes. After, but not before, chronic social defeat stress (CSDS), the LHb is active when susceptible mice are in the proximity of the aggressor strain. During stress itself, LHb activity is higher in susceptible mice during aggressor proximity, and activation of the LHb during stress biases mice towards susceptibility. This manipulation generates a persistent and widespread increase in the balance of subcortical versus cortical activity in susceptible mice. Taken together, our results indicate that heightened activity in the LHb during stress produces lasting brainwide and behavioral substrates of susceptibility.

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because keypoint data are susceptible to high-frequency jitter that clustering algorithms can mistake for transitions between actions. Here we present keypoint-MoSeq, a machine learning-based platform for identifying behavioral modules ('syllables') from keypoint data without human supervision. Keypoint-MoSeq uses a generative model to distinguish keypoint noise from behavior, enabling it to identify syllables whose boundaries correspond to natural sub-second discontinuities in pose dynamics. Keypoint-MoSeq outperforms commonly used alternative clustering methods at identifying these transitions, at capturing correlations between neural activity and behavior and at classifying either solitary or social behaviors in accordance with human annotations. Keypoint-MoSeq also works in multiple species and generalizes beyond the syllable timescale, identifying fast sniff-aligned movements in mice and a spectrum of oscillatory behaviors in fruit flies. Keypoint-MoSeq, therefore, renders accessible the modular structure of behavior through standard video recordings.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce 'Lightning Pose', an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We released a cloud application that allows users to label data, train networks and process new videos directly from the browser.

Simple Behavioral Analysis (SimBA) as a platform for explainable machine learning in behavioral neuroscience

The study of complex behaviors is often challenging when using manual annotation due to the absence of quantifiable behavioral definitions and the subjective nature of behavioral annotation. Integration of supervised machine learning approaches mitigates some of these issues through the inclusion of accessible and explainable model interpretation. To decrease barriers to access, and with an emphasis on accessible model explainability, we developed the open-source Simple Behavioral Analysis (SimBA) platform for behavioral neuroscientists. SimBA introduces several machine learning interpretability tools, including SHapley Additive exPlanation (SHAP) scores, that aid in creating explainable and transparent behavioral classifiers. Here we show how the addition of explainability metrics allows for quantifiable comparisons of aggressive social behavior across research groups and species, reconceptualizing behavior as a sharable reagent and providing an open-source framework. We provide an open-source, graphical user interface (GUI)-driven, well-documented package to facilitate the movement toward improved automation and sharing of behavioral classification tools across laboratories.

Temporally aligned segmentation and clustering (TASC) framework for behavior time series analysis

Behavior exhibits a complex spatiotemporal structure consisting of discrete sub-behaviors, or motifs. Continuous behavior data requires segmentation and clustering to reveal these embedded motifs. The popularity of automatic behavior quantification is growing, but existing solutions are often tailored to specific needs and are not designed for the time scale and precision required in many experimental and clinical settings. Here we propose a generalized framework with an iterative approach to refine both segmentation and clustering. Temporally aligned segmentation and clustering (TASC) uses temporal linear alignment to compute distances between and align the recurring behavior motifs in a multidimensional time series, enabling precise segmentation and clustering. We introduce an alternating-step process: evaluation of temporal neighbors against current cluster centroids using linear alignment, alternating with selecting the best non-overlapping segments and their subsequent re-clustering. The framework is evaluated on semi-synthetic and real-world experimental and clinical data, demonstrating enhanced segmentation and clustering, offering a better foundation for consequent research. The framework may be used to extend existing tools in the field of behavior research and may be applied to other domains requiring high precision of time series segmentation.

FlyVISTA, an Integrated Machine Learning Platform for Deep Phenotyping of Sleep in Drosophila

Animal behavior depends on internal state. While subtle movements can signify significant changes in internal state, computational methods for analyzing these "microbehaviors" are lacking. Here, we present FlyVISTA, a machine-learning platform to characterize microbehaviors in freely-moving flies, which we use to perform deep phenotyping of sleep. This platform comprises a high-resolution closed-loop video imaging system, coupled with a deep-learning network to annotate 35 body parts, and a computational pipeline to extract behaviors from high-dimensional data. FlyVISTA reveals the distinct spatiotemporal dynamics of sleep-associated microbehaviors in flies. We further show that stimulation of dorsal fan-shaped body neurons induces micromovements, not sleep, whereas activating R5 ring neurons triggers rhythmic proboscis extension followed by persistent sleep. Importantly, we identify a novel microbehavior ("haltere switch") exclusively seen during quiescence that indicates a deeper sleep stage. These findings enable the rigorous analysis of sleep in Drosophila and set the stage for computational analyses of microbehaviors.

Reconfiguration of Behavioral Signals in the Anterior Cingulate Cortex Based on Emotional State

Behaviors and their execution depend on the context and emotional state in which they are performed. The contextual modulation of behavior likely relies on regions such as the anterior cingulate cortex (ACC) that multiplex information about emotional/autonomic states and behaviors. The objective of the present study was to understand how the representations of behaviors by ACC neurons become modified when performed in different emotional states. A pipeline of machine learning techniques was developed to categorize and classify complex, spontaneous behaviors in male rats from the video. This pipeline, termed Hierarchical Unsupervised Behavioural Discovery Tool (HUB-DT), discovered a range of statistically separable behaviors during a task in which motivationally significant outcomes were delivered in blocks of trials that created three unique "emotional contexts." HUB-DT was capable of detecting behaviors specific to each emotional context and was able to identify and segregate the portions of a neural signal related to a behavior and to emotional context. Overall, ∼10× as many neurons responded to behaviors in a contextually dependent versus a fixed manner, highlighting the extreme impact of emotional state on representations of behaviors that were precisely defined based on detailed analyses of limb kinematics. This type of modulation may be a key mechanism that allows the ACC to modify the behavioral output based on emotional states and contextual demands.

Visual detection of seizures in mice using supervised machine learning

Seizures are caused by abnormally synchronous brain activity that can result in changes in muscle tone, such as twitching, stiffness, limpness, or rhythmic jerking. These behavioral manifestations are clear on visual inspection and the most widely used seizure scoring systems in preclinical models, such as the Racine scale in rodents, use these behavioral patterns in semiquantitative seizure intensity scores. However, visual inspection is time-consuming, low-throughput, and partially subjective, and there is a need for rigorously quantitative approaches that are scalable. In this study, we used supervised machine learning approaches to develop automated classifiers to predict seizure severity directly from noninvasive video data. Using the PTZ-induced seizure model in mice, we trained video-only classifiers to predict ictal events, combined these events to predict an univariate seizure intensity for a recording session, as well as time-varying seizure intensity scores. Our results show, for the first time, that seizure events and overall intensity can be rigorously quantified directly from overhead video of mice in a standard open field using supervised approaches. These results enable high-throughput, noninvasive, and standardized seizure scoring for downstream applications such as neurogenetics and therapeutic discovery.

Exploratory Rearing Is Governed by Hypothalamic Melanin-Concentrating Hormone Neurons According to Locus Ceruleus

Information seeking, such as standing on tiptoes to look around in humans, is observed across animals and helps survival. Its rodent analog-unsupported rearing on hind legs-was a classic model in deciphering neural signals of cognition and is of intense renewed interest in preclinical modeling of neuropsychiatric states. Neural signals and circuits controlling this dedicated decision to seek information remain largely unknown. While studying subsecond timing of spontaneous behavioral acts and activity of melanin-concentrating hormone (MCH) neurons (MNs) in behaving male and female mice, we observed large MN activity spikes that aligned to unsupported rears. Complementary causal, loss and gain of function, analyses revealed specific control of rear frequency and duration by MNs and MCHR1 receptors. Activity in a key stress center of the brain-the locus ceruleus noradrenaline cells-rapidly inhibited MNs and required functional MCH receptors for its endogenous modulation of rearing. By defining a neural module that both tracks and controls rearing, these findings may facilitate further insights into biology of information seeking.

STPoseNet: A real-time spatiotemporal network model for robust mouse pose estimation

Animal behavior analysis plays a crucial role in contemporary neuroscience research. However, the performance of the frame-by-frame approach may degrade in scenarios with occlusions or motion blur. In this study, we propose a spatiotemporal network model based on YOLOv8 to enhance the accuracy of key-point detection in mouse behavioral experimental videos. This model integrates a time-domain tracking strategy comprising two components: the first part utilizes key-point detection results from the previous frame to detect potential target locations in the subsequent frame; the second part employs Kalman filtering to analyze key-point changes prior to detection, allowing for the estimation of missing key-points. In the comparison of pose estimation results between our approach, YOLOv8, DeepLabCut and SLEAP on videos of three mouse behavioral experiments, our approach demonstrated significantly superior performance. This suggests that our method offers a new and effective means of accurately tracking and estimating pose in mice through spatiotemporal processing.

Learning dynamical models of single and collective cell migration: a review

Single and collective cell migration are fundamental processes critical for physiological phenomena ranging from embryonic development and immune response to wound healing and cancer metastasis. To understand cell migration from a physical perspective, a broad variety of models for the underlying physical mechanisms that govern cell motility have been developed. A key challenge in the development of such models is how to connect them to experimental observations, which often exhibit complex stochastic behaviours. In this review, we discuss recent advances in data-driven theoretical approaches that directly connect with experimental data to infer dynamical models of stochastic cell migration. Leveraging advances in nanofabrication, image analysis, and tracking technology, experimental studies now provide unprecedented large datasets on cellular dynamics. In parallel, theoretical efforts have been directed towards integrating such datasets into physical models from the single cell to the tissue scale with the aim of conceptualising the emergent behaviour of cells. We first review how this inference problem has been addressed in both freely migrating and confined cells. Next, we discuss why these dynamics typically take the form of underdamped stochastic equations of motion, and how such equations can be inferred from data. We then review applications of data-driven inference and machine learning approaches to heterogeneity in cell behaviour, subcellular degrees of freedom, and to the collective dynamics of multicellular systems. Across these applications, we emphasise how data-driven methods can be integrated with physical active matter models of migrating cells, and help reveal how underlying molecular mechanisms control cell behaviour. Together, these data-driven approaches are a promising avenue for building physical models of cell migration directly from experimental data, and for providing conceptual links between different length-scales of description.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Characterization of Ultrasonic Vocalization-Modulated Neurons in Rat Motor Cortex Based on Their Activity Modulation and Axonal Projection to the Periaqueductal Gray

Vocalization, a means of social communication, is prevalent among many species, including humans. Both rats and mice use ultrasonic vocalizations (USVs) in various social contexts and affective states. The motor cortex is hypothesized to be involved in precisely controlling USVs through connections with critical regions of the brain for vocalization, such as the periaqueductal gray matter (PAG). However, it is unclear how neurons in the motor cortex are modulated during USVs. Moreover, the relationship between USV modulation of neurons and anatomical connections from the motor cortex to PAG is also not clearly understood. In this study, we first characterized the activity patterns of neurons in the primary and secondary motor cortices during emission of USVs in rats using large-scale electrophysiological recordings. We also examined the axonal projection of the motor cortex to PAG using retrograde labeling and identified two clusters of PAG-projecting neurons in the anterior and posterior parts of the motor cortex. The neural activity patterns around the emission of USVs differed between the anterior and posterior regions, which were divided based on the distribution of PAG-projecting neurons in the motor cortex. Furthermore, using optogenetic tagging, we recorded the USV modulation of PAG-projecting neurons in the posterior part of the motor cortex and found that they showed predominantly sustained excitatory responses during USVs. These results contribute to our understanding of the involvement of the motor cortex in the generation of USV at the neuronal and circuit levels.

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

Arbovirus impact on mosquito behavior: the jury is still out

Parasites can manipulate host behavior to enhance transmission, but our understanding of arbovirus-induced changes in mosquito behavior is limited. Here, we explore current knowledge on such behavioral alterations in mosquito vectors, focusing on host-seeking and blood-feeding behaviors. Reviewing studies on dengue, Zika, La Crosse, Sindbis, and West Nile viruses in Aedes or Culex mosquitoes reveals subtle yet potentially significant effects. However, assay heterogeneity and limited sample sizes challenge definitive conclusions. To enhance robustness, we propose using deep-learning tools for automated behavior quantification and stress the need for standardized assays. Additionally, conducting longitudinal studies across the extrinsic incubation period and integrating diverse traits into modeling frameworks are crucial for understanding the nuanced implications of arbovirus-induced behavioral changes for virus transmission dynamics.