Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Visual detection of seizures in mice using supervised machine learning

Seizures are caused by abnormally synchronous brain activity that can result in changes in muscle tone, such as twitching, stiffness, limpness, or rhythmic jerking. These behavioral manifestations are clear on visual inspection and the most widely used seizure scoring systems in preclinical models, such as the Racine scale in rodents, use these behavioral patterns in semiquantitative seizure intensity scores. However, visual inspection is time-consuming, low-throughput, and partially subjective, and there is a need for rigorously quantitative approaches that are scalable. In this study, we used supervised machine learning approaches to develop automated classifiers to predict seizure severity directly from noninvasive video data. Using the PTZ-induced seizure model in mice, we trained video-only classifiers to predict ictal events, combined these events to predict an univariate seizure intensity for a recording session, as well as time-varying seizure intensity scores. Our results show, for the first time, that seizure events and overall intensity can be rigorously quantified directly from overhead video of mice in a standard open field using supervised approaches. These results enable high-throughput, noninvasive, and standardized seizure scoring for downstream applications such as neurogenetics and therapeutic discovery.

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

The flow of neural activity across the neocortex during active sensory discrimination is constrained by task-specific cognitive demands, movements, and internal states. During behavior, the brain appears to sample from a broad repertoire of activation motifs. Understanding how these patterns of local and global activity are selected in relation to both spontaneous and task-dependent behavior requires in-depth study of densely sampled activity at single neuron resolution across large regions of cortex. In a significant advance toward this goal, we developed procedures to record mesoscale 2-photon Ca2+ imaging data from two novel in vivo preparations that, between them, allow for simultaneous access to nearly all 0f the mouse dorsal and lateral neocortex. As a proof of principle, we aligned neural activity with both behavioral primitives and high-level motifs to reveal the existence of large populations of neurons that coordinated their activity across cortical areas with spontaneous changes in movement and/or arousal. The methods we detail here facilitate the identification and exploration of widespread, spatially heterogeneous neural ensembles whose activity is related to diverse aspects of behavior.

Uncovering multiscale structure in the variability of larval zebrafish navigation

Animals chain movements into long-lived motor strategies, resulting in variability that ultimately reflects the interplay between internal states and environmental cues. To reveal structure in such variability, we build models that bridges across time scales that enable a quantitative comparison of behavioral phenotypes among individuals. Applied to larval zebrafish exposed to diverse sensory cues, we uncover a hierarchy of long-lived motor strategies, dominated by changes in orientation distinguishing cruising and wandering strategies. Environmental cues induce preferences along these modes at the population level: while fish cruise in the light, they wander in response to aversive (dark) stimuli or in search for prey. Our method enables us to encode the behavioral dynamics of each individual fish in the transitions among coarse-grained motor strategies. By doing so, we uncover a hierarchical structure to the phenotypic variability that corresponds to exploration-exploitation trade-offs. Within a wide range of sensory cues, a major source of variation among fish is driven by prior and immediate exposure to prey that induces exploitation phenotypes. However, a large degree of variability is unexplained by environmental cues, pointing to hidden states that override the sensory context to induce contrasting exploration-exploitation phenotypes. Altogether, our approach extracts the timescales of motor strategies deployed during navigation, exposing undiscovered structure among individuals and pointing to internal states tuned by prior experience.

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Behavior contains rich structure across many timescales, but there is a dearth of methods to identify relevant components, especially over the longer periods required for learning and decision-making. Inspired by the goals and techniques of genome-wide association studies, we present a data-driven method-the choice-wide behavioral association study: CBAS-that systematically identifies such behavioral features. CBAS uses a powerful, resampling-based, method of multiple comparisons correction to identify sequences of actions or choices that either differ significantly between groups or significantly correlate with a covariate of interest. We apply CBAS to different tasks and species (flies, rats, and humans) and find, in all instances, that it provides interpretable information about each behavioral task.

Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

FABEL: Forecasting Animal Behavioral Events with Deep Learning-Based Computer Vision

Behavioral neuroscience aims to provide a connection between neural phenomena and emergent organism-level behaviors. This requires perturbing the nervous system and observing behavioral outcomes, and comparing observed post-perturbation behavior with predicted counterfactual behavior and therefore accurate behavioral forecasts. In this study we present FABEL, a deep learning method for forecasting future animal behaviors and locomotion trajectories from historical locomotion alone. We train an offline pose estimation network to predict animal body-part locations in behavioral video; then sequences of pose vectors are input to deep learning time-series forecasting models. Specifically, we train an LSTM network that predicts a future food interaction event in a specified time window, and a Temporal Fusion Transformer that predicts future trajectories of animal body-parts, which are then converted into probabilistic label forecasts. Importantly, accurate prediction of food interaction provides a basis for neurobehavioral intervention in the context of compulsive eating. We show promising results on forecasting tasks between 100 milliseconds and 5 seconds timescales. Because the model takes only behavioral video as input, it can be adapted to any behavioral task and does not require specific physiological readouts. Simultaneously, these deep learning models may serve as extensible modules that can accommodate diverse signals, such as in-vivo fluorescence imaging and electrophysiology, which may improve behavior forecasts and elucidate invervention targets for desired behavioral change.

Pan-cortical 2-photon mesoscopic imaging and neurobehavioral alignment in awake, behaving mice

The flow of neural activity across the neocortex during active sensory discrimination is constrained by task-specific cognitive demands, movements, and internal states. During behavior, the brain appears to sample from a broad repertoire of activation motifs. Understanding how these patterns of local and global activity are selected in relation to both spontaneous and task-dependent behavior requires in-depth study of densely sampled activity at single neuron resolution across large regions of cortex. In a significant advance toward this goal, we developed procedures to record mesoscale 2-photon Ca2+ imaging data from two novel in vivo preparations that, between them, allow simultaneous access to nearly all of the mouse dorsal and lateral neocortex. As a proof of principle, we aligned neural activity with both behavioral primitives and high-level motifs to reveal the existence of large populations of neurons that coordinated their activity across cortical areas with spontaneous changes in movement and/or arousal. The methods we detail here facilitate the identification and exploration of widespread, spatially heterogeneous neural ensembles whose activity is related to diverse aspects of behavior.

DAMM for the detection and tracking of multiple animals within complex social and environmental settings

Accurate detection and tracking of animals across diverse environments are crucial for behavioral studies in various disciplines, including neuroscience. Recently, machine learning and computer vision techniques have become integral to the neuroscientist's toolkit, enabling high-throughput behavioral studies. Despite advancements in localizing individual animals in simple environments, the task remains challenging in complex conditions due to intra-class visual variability and environmental diversity. These limitations hinder studies in ethologically-relevant conditions, such as when animals are concealed within nests or in obscured environments. Moreover, current tools are laborious and time-consuming to employ, requiring extensive, setup-specific annotation and model training/validation procedures. To address these challenges, we introduce the 'Detect Any Mouse Model' (DAMM), a pretrained object detector for localizing mice in complex environments, capable of robust performance with zero to minimal additional training on new experimental setups. Our approach involves collecting and annotating a diverse dataset that encompasses single and multi-housed mice in various lighting conditions, experimental setups, and occlusion levels. We utilize the Mask R-CNN architecture for instance segmentation and validate DAMM's performance with no additional training data (zero-shot inference) and with few examples for fine-tuning (few-shot inference). DAMM excels in zero-shot inference, detecting mice, and even rats, in entirely unseen scenarios and further improves with minimal additional training. By integrating DAMM with the SORT algorithm, we demonstrate robust tracking, competitively performing with keypoint-estimation-based methods. Finally, to advance and simplify behavioral studies, we made DAMM accessible to the scientific community with a user-friendly Python API, shared model weights, and a Google Colab implementation.

SaLSa: A Combinatory Approach of Semi-Automatic Labeling and Long Short-Term Memory to Classify Behavioral Syllables

Accurately and quantitatively describing mouse behavior is an important area. Although advances in machine learning have made it possible to track their behaviors accurately, reliable classification of behavioral sequences or syllables remains a challenge. In this study, we present a novel machine learning approach, called SaLSa (a combination of semi-automatic labeling and long short-term memory-based classification), to classify behavioral syllables of mice exploring an open field. This approach consists of two major steps. First, after tracking multiple body parts, spatial and temporal features of their egocentric coordinates are extracted. A fully automated unsupervised process identifies candidates for behavioral syllables, followed by manual labeling of behavioral syllables using a graphical user interface (GUI). Second, a long short-term memory (LSTM) classifier is trained with the labeled data. We found that the classification performance was marked over 97%. It provides a performance equivalent to a state-of-the-art model while classifying some of the syllables. We applied this approach to examine how hyperactivity in a mouse model of Alzheimer's disease develops with age. When the proportion of each behavioral syllable was compared between genotypes and sexes, we found that the characteristic hyperlocomotion of female Alzheimer's disease mice emerges between four and eight months. In contrast, age-related reduction in rearing is common regardless of genotype and sex. Overall, SaLSa enables detailed characterization of mouse behavior.

Elephants and algorithms: a review of the current and future role of AI in elephant monitoring

Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behaviour and conservation strategies. Using elephants, a crucial species in Africa and Asia's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.

A developmentally defined population of neurons in the lateral septum controls responses to aversive stimuli

When interacting with their environment, animals must balance exploratory and defensive behavior to evaluate and respond to potential threats. The lateral septum (LS) is a structure in the ventral forebrain that calibrates the magnitude of behavioral responses to stress-related external stimuli, including the regulation of threat avoidance. The complex connectivity between the LS and other parts of the brain, together with its largely unexplored neuronal diversity, makes it difficult to understand how defined LS circuits control specific behaviors. Here, we describe a mouse model in which a population of neurons with a common developmental origin (Nkx2.1-lineage neurons) are absent from the LS. Using a combination of circuit tracing and behavioral analyses, we found that these neurons receive inputs from the perifornical area of the anterior hypothalamus (PeFAH) and are specifically activated in stressful contexts. Mice lacking Nkx2.1-lineage LS neurons display increased exploratory behavior even under stressful conditions. Our study extends the current knowledge about how defined neuronal populations within the LS can evaluate contextual information to select appropriate behavioral responses. This is a necessary step towards understanding the crucial role that the LS plays in neuropsychiatric conditions where defensive behavior is dysregulated, such as anxiety and aggression disorders.

Challenges and advanced concepts for the assessment of learning and memory function in mice

The mechanisms underlying the formation and retrieval of memories are still an active area of research and discussion. Manifold models have been proposed and refined over the years, with most assuming a dichotomy between memory processes involving non-conscious and conscious mechanisms. Despite our incomplete understanding of the underlying mechanisms, tests of memory and learning count among the most performed behavioral experiments. Here, we will discuss available protocols for testing learning and memory using the example of the most prevalent animal species in research, the laboratory mouse. A wide range of protocols has been developed in mice to test, e.g., object recognition, spatial learning, procedural memory, sequential problem solving, operant- and fear conditioning, and social recognition. Those assays are carried out with individual subjects in apparatuses such as arenas and mazes, which allow for a high degree of standardization across laboratories and straightforward data interpretation but are not without caveats and limitations. In animal research, there is growing concern about the translatability of study results and animal welfare, leading to novel approaches beyond established protocols. Here, we present some of the more recent developments and more advanced concepts in learning and memory testing, such as multi-step sequential lockboxes, assays involving groups of animals, as well as home cage-based assays supported by automated tracking solutions; and weight their potential and limitations against those of established paradigms. Shifting the focus of learning tests from the classical experimental chamber to settings which are more natural for rodents comes with a new set of challenges for behavioral researchers, but also offers the opportunity to understand memory formation and retrieval in a more conclusive way than has been attainable with conventional test protocols. We predict and embrace an increase in studies relying on methods involving a higher degree of automatization, more naturalistic- and home cage-based experimental setting as well as more integrated learning tasks in the future. We are confident these trends are suited to alleviate the burden on animal subjects and improve study designs in memory research.

Disentangling rodent behaviors to improve automated behavior recognition

Automated observation and analysis of behavior is important to facilitate progress in many fields of science. Recent developments in deep learning have enabled progress in object detection and tracking, but rodent behavior recognition struggles to exceed 75-80% accuracy for ethologically relevant behaviors. We investigate the main reasons why and distinguish three aspects of behavior dynamics that are difficult to automate. We isolate these aspects in an artificial dataset and reproduce effects with the state-of-the-art behavior recognition models. Having an endless amount of labeled training data with minimal input noise and representative dynamics will enable research to optimize behavior recognition architectures and get closer to human-like recognition performance for behaviors with challenging dynamics.