Real-time, low-latency closed-loop feedback using markerless posture tracking

Toward a neuroscience of natural behavior

One of the most exciting new developments in systems neuroscience is the progress being made toward neurophysiological experiments that move beyond simplified laboratory settings and address the richness of natural behavior. This is enabled by technological advances such as wireless recording in freely moving animals, automated quantification of behavior, and new methods for analyzing large data sets. Beyond new empirical methods and data, however, there is also a need for new theories and concepts to interpret that data. Such theories need to address the particular challenges of natural behavior, which often differ significantly from the scenarios studied in traditional laboratory settings. Here, we discuss some strategies for developing such novel theories and concepts and some example hypotheses being proposed.

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

Automated procedure to detect subtle motor alterations in the balance beam test in a mouse model of early Parkinson's disease

Parkinson's disease (PD) is the most common motor neurodegenerative disorder, characterised by aggregated α-synuclein (α-syn) constituting Lewy bodies. We aimed to investigate temporal changes in motor impairments in a PD mouse model induced by overexpression of α-syn with the conventional manual analysis of the balance beam test and a novel approach using machine learning algorithms to automate behavioural analysis. We combined automated animal tracking using markerless pose estimation in DeepLabCut, with automated behavioural classification in Simple Behavior Analysis. Our automated procedure was able to detect subtle motor deficits in mouse performances in the balance beam test that the manual analysis approach could not assess. The automated model revealed time-course significant differences for the "walking" behaviour in the mean interval between each behavioural bout, the median event bout duration and the classifier probability of occurrence in male PD mice, even though no statistically significant loss of tyrosine hydroxylase in the nigrostriatal system was found in either sex. These findings are valuable for early detection of motor impairment in early PD animal models. We provide a user-friendly, step-by-step guide for automated assessment of mouse performances in the balance beam test, which aims to be replicable without any significant computational and programming knowledge.

Facemap: a framework for modeling neural activity based on orofacial tracking

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder for predicting neural activity. Our algorithm for tracking mouse orofacial behaviors was more accurate than existing pose estimation tools, while the processing speed was several times faster, making it a powerful tool for real-time experimental interventions. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used the keypoints as inputs to a deep neural network which predicts the activity of ~50,000 simultaneously-recorded neurons and, in visual cortex, we doubled the amount of explained variance compared to previous methods. Using this model, we found that the neuronal activity clusters that were well predicted from behavior were more spatially spread out across cortex. We also found that the deep behavioral features from the model had stereotypical, sequential dynamics that were not reversible in time. In summary, Facemap provides a stepping stone toward understanding the function of the brain-wide neural signals and their relation to behavior.

Pose analysis in free-swimming adult zebrafish, Danio rerio : "fishy" origins of movement design

Movement requires maneuvers that generate thrust to either make turns or move the body forward in physical space. The computational space for perpetually controlling the relative position of every point on the body surface can be vast. We hypothesize the evolution of efficient design for movement that minimizes active (neural) control by leveraging the passive (reactive) forces between the body and the surrounding medium at play. To test our hypothesis, we investigate the presence of stereotypical postures during free-swimming in adult zebrafish, Danio rerio . We perform markerless tracking using DeepLabCut, a deep learning pose estimation toolkit, to track geometric relationships between body parts. To identify putative clusters of postural configurations obtained from twelve freely behaving zebrafish, we use unsupervised multivariate time-series analysis (B-SOiD machine learning software). When applied to single individuals, this method reveals a best-fit for 36 to 50 clusters in contrast 86 clusters for data pooled from all 12 animals. The centroids of each cluster obtained over 14,000 sequential frames recorded for a single fish represent an apriori classification into relatively stable "target body postures" and inter-pose "transitional postures" that lead to and away from a target pose. We use multidimensional scaling of mean parameter values for each cluster to map cluster-centroids within two dimensions of postural space. From a post-priori visual analysis, we condense neighboring postural variants into 15 superclusters or core body configurations. We develop a nomenclature specifying the anteroposterior level/s (upper, mid and lower) and degree of bending. Our results suggest that constraining bends to mainly three levels in adult zebrafish preempts the neck, fore- and hindlimb design for maneuverability in land vertebrates.

Reproducible and fully automated testing of nocifensive behavior in mice

Pain in rodents is often inferred from their withdrawal from noxious stimulation. Threshold stimulus intensity or response latency is used to quantify pain sensitivity. This usually involves applying stimuli by hand and measuring responses by eye, which limits reproducibility and throughput. We describe a device that standardizes and automates pain testing by providing computer-controlled aiming, stimulation, and response measurement. Optogenetic and thermal stimuli are applied using blue and infrared light, respectively. Precise mechanical stimulation is also demonstrated. Reflectance of red light is used to measure paw withdrawal with millisecond precision. We show that consistent stimulus delivery is crucial for resolving stimulus-dependent variations in withdrawal and for testing with sustained stimuli. Moreover, substage video reveals "spontaneous" behaviors for consideration alongside withdrawal metrics to better assess the pain experience. The entire process was automated using machine learning. RAMalgo (reproducible automated multimodal algometry) improves the standardization, comprehensiveness, and throughput of preclinical pain testing.

Protocol to investigate the neural basis for copulation posture of Drosophila using a closed-loop real-time optogenetic system

In internal fertilization animals, maintaining a copulation posture facilitates the process of transporting gametes from male to female. Here, we present a protocol to investigate the neural basis for copulation posture of fruit flies using a closed-loop real-time optogenetic system. We describe steps for using deep learning analysis to enable optogenetic manipulation of neural activity only during copulation with high efficiency. This system can be applied to various animal behaviors other than copulation. For complete details on the use and execution of this protocol, please refer to Yamanouchi et al. (2023).1.

Protocol for the integration of fiber photometry and social behavior in rodent models

Fiber photometry offers insight into cell-type-specific activity underlying social interactions. We provide a protocol for the integration of fiber photometry recordings into the analysis of social behavior in rodent models. This includes considerations during surgery, notes on synchronizing fiber photometry with behavioral recordings, advice on using multi-animal behavioral tracking software, and scripts for the analysis of fiber photometry recordings. For complete details on the use and execution of this protocol, please refer to Dawson et al. (2023).1.

Neurons in the primary visual cortex of freely moving rats encode both sensory and non-sensory task variables

Neurons in primary visual cortex (area V1) are strongly driven by both sensory stimuli and non-sensory events. However, although the representation of sensory stimuli has been well characterized, much less is known about the representation of non-sensory events. Here, we characterize the specificity and organization of non-sensory representations in rat V1 during a freely moving visual decision task. We find that single neurons encode diverse combinations of task features simultaneously and across task epochs. Despite heterogeneity at the level of single neuron response patterns, both visual and nonvisual task variables could be reliably decoded from small neural populations (5 to 40 units) throughout a trial. Interestingly, in animals trained to make an auditory decision following passive observation of a visual stimulus, some but not all task features could also be decoded from V1 activity. Our results support the view that even in V1-the earliest stage of the cortical hierarchy-bottom-up sensory information may be combined with top-down non-sensory information in a task-dependent manner.

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

Cortical network and projection neuron types that articulate serial order in a skilled motor behavior

Skilled motor behaviors require orderly coordination of multiple constituent movements with sensory cues towards achieving a goal, but the underlying brain circuit mechanisms remain unclear. Here we show that target-guided reach-grasp-to-drink (RGD) in mice involves the ordering and coordination of a set of forelimb and oral actions. Cortex-wide activity imaging of multiple glutamatergic projection neuron (PN) types uncovered a network, involving the secondary motor cortex (MOs), forelimb primary motor and somatosensory cortex, that tracked RGD movements. Photo-inhibition highlighted MOs in coordinating RGD movements. Within the MOs, population neural trajectories tracked RGD progression and single neuron activities integrated across constituent movements. Notably, MOs intratelencephalic, pyramidal tract, and corticothalamic PN activities correlated with action coordination, showed distinct neural dynamics trajectories, and differentially contributed to movement coordination. Our results delineate a cortical network and key areas, PN types, and neural dynamics therein that articulate the serial order and coordination of a skilled behavior.

A unified open-source platform for multimodal neural recording and perturbation during naturalistic behavior

Behavioral neuroscience faces two conflicting demands: long-duration recordings from large neural populations and unimpeded animal behavior. To meet this challenge, we developed ONIX, an open-source data acquisition system with high data throughput (2GB/sec) and low closed-loop latencies (<1ms) that uses a novel 0.3 mm thin tether to minimize behavioral impact. Head position and rotation are tracked in 3D and used to drive active commutation without torque measurements. ONIX can acquire from combinations of passive electrodes, Neuropixels probes, head-mounted microscopes, cameras, 3D-trackers, and other data sources. We used ONIX to perform uninterrupted, long (~7 hours) neural recordings in mice as they traversed complex 3-dimensional terrain. ONIX allowed exploration with similar mobility as non-implanted animals, in contrast to conventional tethered systems which restricted movement. By combining long recordings with full mobility, our technology will enable new progress on questions that require high-quality neural recordings during ethologically grounded behaviors.

Somatostatin peptide signaling dampens cortical circuits and promotes exploratory behavior

We sought to characterize the unique role of somatostatin (SST) in the prelimbic (PL) cortex in mice. We performed slice electrophysiology in pyramidal and GABAergic neurons to characterize the pharmacological mechanism of SST signaling and fiber photometry of GCaMP6f fluorescent calcium signals from SST neurons to characterize the activity profile of SST neurons during exploration of an elevated plus maze (EPM) and open field test (OFT). We used local delivery of a broad SST receptor (SSTR) agonist and antagonist to test causal effects of SST signaling. SSTR activation hyperpolarizes layer 2/3 pyramidal neurons, an effect that is recapitulated with optogenetic stimulation of SST neurons. SST neurons in PL are activated during EPM and OFT exploration, and SSTR agonist administration directly into the PL enhances open arm exploration in the EPM. This work describes a broad ability for SST peptide signaling to modulate microcircuits within the prefrontal cortex and related exploratory behaviors.

Open-MAC: A low-cost open-source motorized commutator for electro- and opto-physiological recordings in freely moving rodents

In vivo electro- and optophysiology experiments in rodents reveal the neural mechanisms underlying behavior and brain disorders but mostly involve a cable connection between an implant in the animal and an external recording device. Standard tethers with thin cables or non-motorized commutators require constant monitoring and often manual interference to untwist the cable. Motorized commutators offer a solution, but those few that are commercially available are expensive and often not adapted to widely used connector standards of the open-source community like 12-channel SPI. Here we introduce an open-source motorized all-in-one commutator (Open-MAC): a low-cost (240-390 EUR), low-torque motorized commutator that can operate with minimal audible noise in a torque-based mode relying on dual magnetic Hall sensors. It further includes electronics to operate in a torque-free, online pose-estimation-based mode, with future developments. Operation is controlled by an onboard microcontroller (XIAO SAMD21) powered by a USB-C cable or DC power supply. The body and movable parts are 3D-printed. Different Open-MAC versions can support electrophysiology with up to 64 recording channels using the Open-Ephys / Intan^TM recording systems as well as miniature endoscope (miniscope) recordings using the UCLA Miniscope v3/4, and can host a fibre for optogenetic modulation.

Piezo-mediated mechanosensation contributes to stabilizing copulation posture and reproductive success in Drosophila males

In internal fertilization animals, reproductive success depends on maintaining copulation until gametes are transported from male to female. In Drosophila melanogaster, mechanosensation in males likely contributes to copulation maintenance, but its molecular underpinning remains to be identified. Here we show that the mechanosensory gene piezo and its' expressing neurons are responsible for copulation maintenance. An RNA-seq database search and subsequent mutant analysis revealed the importance of piezo for maintaining male copulation posture. piezo-GAL4-positive signals were found in the sensory neurons of male genitalia bristles, and optogenetic inhibition of piezo-expressing neurons in the posterior side of the male body during copulation destabilized posture and terminated copulation. Our findings suggest that the mechanosensory system of male genitalia through Piezo channels plays a key role in copulation maintenance and indicate that Piezo may increase male fitness during copulation in flies.

Miniscope-LFOV: A large-field-of-view, single-cell-resolution, miniature microscope for wired and wire-free imaging of neural dynamics in freely behaving animals

Imaging large-population, single-cell fluorescent dynamics in freely behaving animals larger than mice remains a key endeavor of neuroscience. We present a large-field-of-view open-source miniature microscope (MiniLFOV) designed for large-scale (3.6 mm × 2.7 mm), cellular resolution neural imaging in freely behaving rats. It has an electrically adjustable working distance of up to 3.5 mm ± 100 μm, incorporates an absolute head orientation sensor, and weighs only 13.9 g. The MiniLFOV is capable of both deep brain and cortical imaging and has been validated in freely behaving rats by simultaneously imaging >1000 GCaMP7s-expressing neurons in the hippocampal CA1 layer and in head-fixed mice by simultaneously imaging ~2000 neurons in the dorsal cortex through a cranial window. The MiniLFOV also supports optional wire-free operation using a novel, wire-free data acquisition expansion board. We expect that this new open-source implementation of the UCLA Miniscope platform will enable researchers to address novel hypotheses concerning brain function in freely behaving animals.

Video-driven simulation of lower limb mechanical loading during aquatic exercises

Understanding the mechanical demands of an exercise on the musculoskeletal system is crucial to prescribe effective training or therapeutic interventions. Yet, that knowledge is currently limited in water, mostly because of the difficulty in evaluating external resistance. Here I reconcile recent advances in 3D markerless pose and mesh estimation, biomechanical simulations, and hydrodynamic modeling, to predict lower limb mechanical loading during aquatic exercises. Simulations are driven exclusively from a single video. Fluid forces were estimated within 12.5±4.1% of the peak forces determined through computational fluid dynamics analyses, at a speed three orders of magnitude greater. In silico hip and knee resultant joint forces agreed reasonably well with in vivo instrumented implant recordings (R²=0.74) downloaded from the OrthoLoad database, both in magnitude (RMSE =251±125 N) and direction (cosine similarity = 0.92±0.09). Hip flexors, glutes, adductors, and hamstrings were the main contributors to hip joint compressive forces (40.4±12.7%, 25.6±9.7%, 14.2±4.8%, 13.0±8.2%, respectively), while knee compressive forces were mostly produced by the gastrocnemius (39.1±15.9%) and vasti (29.4±13.7%). Unlike dry-land locomotion, non-hip- and non-knee-spanning muscles provided little to no offloading effect via dynamic coupling. This noninvasive method has the potential to standardize the reporting of exercise intensity, inform the design of rehabilitation protocols and improve their reproducibility.

Open-source tools for behavioral video analysis: Setup, methods, and best practices

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

Cre-dependent ACR2-expressing reporter mouse strain for efficient long-lasting inhibition of neuronal activity

Optogenetics is a powerful tool for manipulating neuronal activity by light illumination with high temporal and spatial resolution. Anion-channelrhodopsins (ACRs) are light-gated anion channels that allow researchers to efficiently inhibit neuronal activity. A blue light-sensitive ACR2 has recently been used in several in vivo studies; however, the reporter mouse strain expressing ACR2 has not yet been reported. Here, we generated a new reporter mouse strain, LSL-ACR2, in which ACR2 is expressed under the control of Cre recombinase. We crossed this strain with a noradrenergic neuron-specific driver mouse (NAT-Cre) to generate NAT-ACR2 mice. We confirmed Cre-dependent expression and function of ACR2 in the targeted neurons by immunohistochemistry and electrophysiological recordings in vitro, and confirmed physiological function using an in vivo behavioral experiment. Our results show that the LSL-ACR2 mouse strain can be applied for optogenetic inhibition of targeted neurons, particularly for long-lasting continuous inhibition, upon crossing with Cre-driver mouse strains. The LSL-ACR2 strain can be used to prepare transgenic mice with homogenous expression of ACR2 in targeted neurons with a high penetration ratio, good reproducibility, and no tissue invasion.

Multiscale imaging informs translational mouse modeling of neurological disease

Multiscale neurophysiology reveals that simple motor actions are associated with changes in neuronal firing in virtually every brain region studied. Accordingly, the assessment of focal pathology such as stroke or progressive neurodegenerative diseases must also extend widely across brain areas. To derive mechanistic information through imaging, multiple resolution scales and multimodal factors must be included, such as the structure and function of specific neurons and glial cells and the dynamics of specific neurotransmitters. Emerging multiscale methods in preclinical animal studies that span micro- to macroscale examinations fill this gap, allowing a circuit-based understanding of pathophysiological mechanisms. Combined with high-performance computation and open-source data repositories, these emerging multiscale and large field-of-view techniques include live functional ultrasound, multi- and single-photon wide-scale light microscopy, video-based miniscopes, and tissue-penetrating fiber photometry, as well as variants of post-mortem expansion microscopy. We present these technologies and outline use cases and data pipelines to uncover new knowledge within animal models of stroke, Alzheimer's disease, and movement disorders.

Automatic extraction of upper-limb kinematic activity using deep learning-based markerless tracking during deep brain stimulation implantation for Parkinson's disease: A proof of concept study

Optimal placement of deep brain stimulation (DBS) therapy for treating movement disorders routinely relies on intraoperative motor testing for target determination. However, in current practice, motor testing relies on subjective interpretation and correlation of motor and neural information. Recent advances in computer vision could improve assessment accuracy. We describe our application of deep learning-based computer vision to conduct markerless tracking for measuring motor behaviors of patients undergoing DBS surgery for the treatment of Parkinson's disease. Video recordings were acquired during intraoperative kinematic testing (N = 5 patients), as part of standard of care for accurate implantation of the DBS electrode. Kinematic data were extracted from videos post-hoc using the Python-based computer vision suite DeepLabCut. Both manual and automated (80.00% accuracy) approaches were used to extract kinematic episodes from threshold derived kinematic fluctuations. Active motor epochs were compressed by modeling upper limb deflections with a parabolic fit. A semi-supervised classification model, support vector machine (SVM), trained on the parameters defined by the parabolic fit reliably predicted movement type. Across all cases, tracking was well calibrated (i.e., reprojection pixel errors 0.016-0.041; accuracies >95%). SVM predicted classification demonstrated high accuracy (85.70%) including for two common upper limb movements, arm chain pulls (92.30%) and hand clenches (76.20%), with accuracy validated using a leave-one-out process for each patient. These results demonstrate successful capture and categorization of motor behaviors critical for assessing the optimal brain target for DBS surgery. Conventional motor testing procedures have proven informative and contributory to targeting but have largely remained subjective and inaccessible to non-Western and rural DBS centers with limited resources. This approach could automate the process and improve accuracy for neuro-motor mapping, to improve surgical targeting, optimize DBS therapy, provide accessible avenues for neuro-motor mapping and DBS implantation, and advance our understanding of the function of different brain areas.

Estimation of skeletal kinematics in freely moving rodents

Forming a complete picture of the relationship between neural activity and skeletal kinematics requires quantification of skeletal joint biomechanics during free behavior; however, without detailed knowledge of the underlying skeletal motion, inferring limb kinematics using surface-tracking approaches is difficult, especially for animals where the relationship between the surface and underlying skeleton changes during motion. Here we developed a videography-based method enabling detailed three-dimensional kinematic quantification of an anatomically defined skeleton in untethered freely behaving rats and mice. This skeleton-based model was constrained using anatomical principles and joint motion limits and provided skeletal pose estimates for a range of body sizes, even when limbs were occluded. Model-inferred limb positions and joint kinematics during gait and gap-crossing behaviors were verified by direct measurement of either limb placement or limb kinematics using inertial measurement units. Together we show that complex decision-making behaviors can be accurately reconstructed at the level of skeletal kinematics using our anatomically constrained model.

Laser stimulation of the skin for quantitative study of decision-making and motivation

Neuroeconomics studies how decision-making is guided by the value of rewards and punishments. But to date, little is known about how noxious experiences impact decisions. A challenge is the lack of an aversive stimulus that is dynamically adjustable in intensity and location, readily usable over many trials in a single experimental session, and compatible with multiple ways to measure neuronal activity. We show that skin laser stimulation used in human studies of aversion can be used for this purpose in several key animal models. We then use laser stimulation to study how neurons in the orbitofrontal cortex (OFC), an area whose many roles include guiding decisions among different rewards, encode the value of rewards and punishments. We show that some OFC neurons integrated the positive value of rewards with the negative value of aversive laser stimulation, suggesting that the OFC can play a role in more complex choices than previously appreciated.

Neuroscience Cloud Analysis As a Service: An open-source platform for scalable, reproducible data analysis

A key aspect of neuroscience research is the development of powerful, general-purpose data analyses that process large datasets. Unfortunately, modern data analyses have a hidden dependence upon complex computing infrastructure (e.g., software and hardware), which acts as an unaddressed deterrent to analysis users. Although existing analyses are increasingly shared as open-source software, the infrastructure and knowledge needed to deploy these analyses efficiently still pose significant barriers to use. In this work, we develop Neuroscience Cloud Analysis As a Service (NeuroCAAS): a fully automated open-source analysis platform offering automatic infrastructure reproducibility for any data analysis. We show how NeuroCAAS supports the design of simpler, more powerful data analyses and that many popular data analysis tools offered through NeuroCAAS outperform counterparts on typical infrastructure. Pairing rigorous infrastructure management with cloud resources, NeuroCAAS dramatically accelerates the dissemination and use of new data analyses for neuroscientific discovery.

Pi USB Cam: A Simple and Affordable DIY Solution That Enables High-Quality, High-Throughput Video Capture for Behavioral Neuroscience Research

Video recording is essential for behavioral neuroscience research, but the majority of available systems suffer from poor cost-to-functionality ratio. Commercial options frequently come at high financial cost that prohibits scalability and throughput, whereas DIY solutions often require significant expertise and time investment unaffordable to many researchers. To address this, we combined a low-cost Raspberry Pi microcomputer, DIY electronics peripherals, freely available open-source firmware, and custom 3D-printed casings to create Pi USB Cam, a simple yet powerful and highly versatile video recording solution. Pi USB Cam is constructed using affordable and widely available components and requires no expertise to build and implement. The result is a system that functions as a plug-and-play USB camera that can be easily installed in various animal testing and housing sites and is readily compatible with popular behavioral and neural recording software. Here, we provide a comprehensive parts list and step-by-step instructions for users to build and implement their own Pi USB Cam system. In a series of benchmark comparisons, Pi USB Cam was able to capture ultra-wide fields of view of behaving rats given limited object distance and produced high image quality while maintaining consistent frame rates even under low-light and no-light conditions relative to a standard, commercially available USB camera. Video recordings were easily scaled using free, open-source software. Altogether, Pi USB Cam presents an elegant yet simple solution for behavioral neuroscientists seeking an affordable and highly flexible system to enable quality video recordings.

A one-photon endoscope for simultaneous patterned optogenetic stimulation and calcium imaging in freely behaving mice

Optogenetics and calcium imaging can be combined to simultaneously stimulate and record neural activity in vivo. However, this usually requires two-photon microscopes, which are not portable nor affordable. Here we report the design and implementation of a miniaturized one-photon endoscope for performing simultaneous optogenetic stimulation and calcium imaging. By integrating digital micromirrors, the endoscope makes it possible to activate any neuron of choice within the field of view, and to apply arbitrary spatiotemporal patterns of photostimulation while imaging calcium activity. We used the endoscope to image striatal neurons from either the direct pathway or the indirect pathway in freely moving mice while activating any chosen neuron in the field of view. The endoscope also allows for the selection of neurons based on their relationship with specific animal behaviour, and to recreate the behaviour by mimicking the natural neural activity with photostimulation. The miniaturized endoscope may facilitate the study of how neural activity gives rise to behaviour in freely moving animals.

Multimodal cues displayed by submissive rats promote prosocial choices by dominants

Animals often display prosocial behaviors, performing actions that benefit others. Although prosociality is essential for social bonding and cooperation, we still know little about how animals integrate behavioral cues from those in need to make decisions that increase their well-being. To address this question, we used a two-choice task where rats can provide rewards to a conspecific in the absence of self-benefit and investigated which conditions promote prosociality by manipulating the social context of the interacting animals. Although sex or degree of familiarity did not affect prosocial choices in rats, social hierarchy revealed to be a potent modulator, with dominant decision-makers showing faster emergence and higher levels of prosocial choices toward their submissive cage mates. Leveraging quantitative analysis of multimodal social dynamics prior to choice, we identified that pairs with dominant decision-makers exhibited more proximal interactions. Interestingly, these closer interactions were driven by submissive animals that modulated their position and movement following their dominants and whose 50-kHz vocalization rate correlated with dominants' prosociality. Moreover, Granger causality revealed stronger bidirectional influences in pairs with dominant focals and submissive recipients, indicating increased behavioral coordination. Finally, multivariate analysis highlighted body language as the main information dominants use on a trial-by-trial basis to learn that their actions have effects on others. Our results provide a refined understanding of the behavioral dynamics that rats use for action-selection upon perception of socially relevant cues and navigate social decision-making.

Toward the explainability, transparency, and universality of machine learning for behavioral classification in neuroscience

The use of rigorous ethological observation via machine learning techniques to understand brain function (computational neuroethology) is a rapidly growing approach that is poised to significantly change how behavioral neuroscience is commonly performed. With the development of open-source platforms for automated tracking and behavioral recognition, these approaches are now accessible to a wide array of neuroscientists despite variations in budget and computational experience. Importantly, this adoption has moved the field toward a common understanding of behavior and brain function through the removal of manual bias and the identification of previously unknown behavioral repertoires. Although less apparent, another consequence of this movement is the introduction of analytical tools that increase the explainabilty, transparency, and universality of the machine-based behavioral classifications both within and between research groups. Here, we focus on three main applications of such machine model explainabilty tools and metrics in the drive toward behavioral (i) standardization, (ii) specialization, and (iii) explainability. We provide a perspective on the use of explainability tools in computational neuroethology, and detail why this is a necessary next step in the expansion of the field. Specifically, as a possible solution in behavioral neuroscience, we propose the use of Shapley values via Shapley Additive Explanations (SHAP) as a diagnostic resource toward explainability of human annotation, as well as supervised and unsupervised behavioral machine learning analysis.

AI-enabled, implantable, multichannel wireless telemetry for photodynamic therapy

Photodynamic therapy (PDT) offers several advantages for treating cancers, but its efficacy is highly dependent on light delivery to activate a photosensitizer. Advances in wireless technologies enable remote delivery of light to tumors, but suffer from key limitations, including low levels of tissue penetration and photosensitizer activation. Here, we introduce DeepLabCut (DLC)-informed low-power wireless telemetry with an integrated thermal/light simulation platform that overcomes the above constraints. The simulator produces an optimized combination of wavelengths and light sources, and DLC-assisted wireless telemetry uses the parameters from the simulator to enable adequate illumination of tumors through high-throughput (<20 mice) and multi-wavelength operation. Together, they establish a range of guidelines for effective PDT regimen design. In vivo Hypericin and Foscan mediated PDT, using cancer xenograft models, demonstrates substantial suppression of tumor growth, warranting further investigation in research and/or clinical settings.

Measuring Locomotor Activity and Behavioral Aspects of Rodents Living in the Home-Cage

Automatization and technological advances have led to a larger number of methods and systems to monitor and measure locomotor activity and more specific behavior of a wide variety of animal species in various environmental conditions in laboratory settings. In rodents, the majority of these systems require the animals to be temporarily taken away from their home-cage into separate observation cage environments which requires manual handling and consequently evokes distress for the animal and may alter behavioral responses. An automated high-throughput approach can overcome this problem. Therefore, this review describes existing automated methods and technologies which enable the measurement of locomotor activity and behavioral aspects of rodents in their most meaningful and stress-free laboratory environment: the home-cage. In line with the Directive 2010/63/EU and the 3R principles (replacement, reduction, refinement), this review furthermore assesses their suitability and potential for group-housed conditions as a refinement strategy, highlighting their current technological and practical limitations. It covers electrical capacitance technology and radio-frequency identification (RFID), which focus mainly on voluntary locomotor activity in both single and multiple rodents, respectively. Infrared beams and force plates expand the detection beyond locomotor activity toward basic behavioral traits but discover their full potential in individually housed rodents only. Despite the great premises of these approaches in terms of behavioral pattern recognition, more sophisticated methods, such as (RFID-assisted) video tracking technology need to be applied to enable the automated analysis of advanced behavioral aspects of individual animals in social housing conditions.

SLEAP: A deep learning system for multi-animal pose tracking

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

Multi-animal pose estimation, identification and tracking with DeepLabCut

Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking-features required for multi-animal scenarios. Furthermore, we integrate the ability to predict an animal's identity to assist tracking (in case of occlusions). We illustrate the power of this framework with four datasets varying in complexity, which we release to serve as a benchmark for future algorithm development.

Automated and Dynamic Control of Chemical Content in Droplets for Scalable Screens of Small Animals

Screening functional phenotypes in small animals is important for genetics and drug discovery. Multiphase microfluidics has great potential for enhancing throughput but has been hampered by inefficient animal encapsulation and limited control over the animal's environment in droplets. Here, a highly efficient single-animal encapsulation unit, a liquid exchanger system for controlling the droplet chemical environment dynamically, and an automation scheme for the programming and robust execution of complex protocols are demonstrated. By careful use of interfacial forces, the liquid exchanger unit allows for adding and removing chemicals from a droplet and, therefore, generating chemical gradients inaccessible in previous multiphase systems. Using Caenorhabditis elegans as an example, it is demonstrated that these advances can serve to analyze dynamic phenotyping, such as behavior and neuronal activity, perform forward genetic screen, and are scalable to manipulate animals of different sizes. This platform paves the way for large-scale screens of complex dynamic phenotypes in small animals.

Wide-angle, monocular head tracking using passive markers

BACKGROUND

Camera images can encode large amounts of visual information of an animal and its environment, enabling high fidelity 3D reconstruction of the animal and its environment using computer vision methods. Most systems, both markerless (e.g. deep learning based) and marker-based, require multiple cameras to track features across multiple points of view to enable such 3D reconstruction. However, such systems can be expensive and are challenging to set up in small animal research apparatuses.

NEW METHODS

We present an open-source, marker-based system for tracking the head of a rodent for behavioral research that requires only a single camera with a potentially wide field of view. The system features a lightweight visual target and computer vision algorithms that together enable high-accuracy tracking of the six-degree-of-freedom position and orientation of the animal's head. The system, which only requires a single camera positioned above the behavioral arena, robustly reconstructs the pose over a wide range of head angles (360° in yaw, and approximately ± 120° in roll and pitch).

RESULTS

Experiments with live animals demonstrate that the system can reliably identify rat head position and orientation. Evaluations using a commercial optical tracker device show that the system achieves accuracy that rivals commercial multi-camera systems.

COMPARISON WITH EXISTING METHODS

Our solution significantly improves upon existing monocular marker-based tracking methods, both in accuracy and in allowable range of motion.

CONCLUSIONS

The proposed system enables the study of complex behaviors by providing robust, fine-scale measurements of rodent head motions in a wide range of orientations.

Modelling behaviors relevant to brain disorders in the nonhuman primate: Are we there yet?

Recent years have seen a profound resurgence of activity with nonhuman primates (NHPs) to model human brain disorders. From marmosets to macaques, the study of NHP species offers a unique window into the function of primate-specific neural circuits that are impossible to examine in other models. Examining how these circuits manifest into the complex behaviors of primates, such as advanced cognitive and social functions, has provided enormous insights to date into the mechanisms underlying symptoms of numerous neurological and neuropsychiatric illnesses. With the recent optimization of modern techniques to manipulate and measure neural activity in vivo, such as optogenetics and calcium imaging, NHP research is more well-equipped than ever to probe the neural mechanisms underlying pathological behavior. However, methods for behavioral experimentation and analysis in NHPs have noticeably failed to keep pace with these advances. As behavior ultimately lies at the junction between preclinical findings and its translation to clinical outcomes for brain disorders, approaches to improve the integrity, reproducibility, and translatability of behavioral experiments in NHPs requires critical evaluation. In this review, we provide a unifying account of existing brain disorder models using NHPs, and provide insights into the present and emerging contributions of behavioral studies to the field.

Perspectives on Individual Animal Identification from Biology and Computer Vision

Identifying individual animals is crucial for many biological investigations. In response to some of the limitations of current identification methods, new automated computer vision approaches have emerged with strong performance. Here, we review current advances of computer vision identification techniques to provide both computer scientists and biologists with an overview of the available tools and discuss their applications. We conclude by offering recommendations for starting an animal identification project, illustrate current limitations, and propose how they might be addressed in the future.

Anipose: A toolkit for robust markerless 3D pose estimation

Quantifying movement is critical for understanding animal behavior. Advances in computer vision now enable markerless tracking from 2D video, but most animals move in 3D. Here, we introduce Anipose, an open-source toolkit for robust markerless 3D pose estimation. Anipose is built on the 2D tracking method DeepLabCut, so users can expand their existing experimental setups to obtain accurate 3D tracking. It consists of four components: (1) a 3D calibration module, (2) filters to resolve 2D tracking errors, (3) a triangulation module that integrates temporal and spatial regularization, and (4) a pipeline to structure processing of large numbers of videos. We evaluate Anipose on a calibration board as well as mice, flies, and humans. By analyzing 3D leg kinematics tracked with Anipose, we identify a key role for joint rotation in motor control of fly walking. To help users get started with 3D tracking, we provide tutorials and documentation at http://anipose.org/.

Unsupervised quantification of naturalistic animal behaviors for gaining insight into the brain

Neural computation has evolved to optimize the behaviors that enable our survival. Although much previous work in neuroscience has focused on constrained task behaviors, recent advances in computer vision are fueling a trend toward the study of naturalistic behaviors. Automated tracking of fine-scale behaviors is generating rich datasets for animal models including rodents, fruit flies, zebrafish, and worms. However, extracting meaning from these large and complex data often requires sophisticated computational techniques. Here we review the latest methods and modeling approaches providing new insights into the brain from behavior. We focus on unsupervised methods for identifying stereotyped behaviors and for resolving details of the structure and dynamics of behavioral sequences.

Closed-loop automated reaching apparatus (CLARA) for interrogating complex motor behaviors

Objective.Closed-loop neuromodulation technology is a rapidly expanding category of therapeutics for a broad range of indications. Development of these innovative neurological devices requires high-throughput systems for closed-loop stimulation of model organisms, while monitoring physiological signals and complex, naturalistic behaviors. To address this need, we developed CLARA, a closed-loop automated reaching apparatus.Approach.Using breakthroughs in computer vision, CLARA integrates fully-automated, markerless kinematic tracking of multiple features to classify animal behavior and precisely deliver neural stimulation based on behavioral outcomes. CLARA is compatible with advanced neurophysiological tools, enabling the testing of neurostimulation devices and identification of novel neurological biomarkers.Results.The CLARA system tracks unconstrained skilled reach behavior in 3D at 150 Hz without physical markers. The system fully automates trial initiation and pellet delivery and is capable of accurately delivering stimulation in response to trial outcome with short latency. Kinematic data from the CLARA system provided novel insights into the dynamics of reach consistency over the course of learning, suggesting that learning selectively improves reach failures but does not alter the kinematics of successful reaches. Additionally, using the closed-loop capabilities of CLARA, we demonstrate that vagus nerve stimulation (VNS) improves skilled reach performance and increases reach trajectory consistency in healthy animals.Significance.The CLARA system is the first mouse behavior apparatus that uses markerless pose tracking to provide real-time closed-loop stimulation in response to the outcome of an unconstrained motor task. Additionally, we demonstrate that the CLARA system was essential for our investigating the role of closed-loop VNS stimulation on motor performance in healthy animals. This approach has high translational relevance for developing neurostimulation technology based on complex human behavior.

Review of methods for animal videography using camera systems that automatically move to follow the animal

Digital photography and videography provide rich data for the study of animal behavior and are consequently widely used techniques. For fixed, unmoving cameras, the image sensor and optics govern the field of view and spatial detail. For a given sensor resolution, the optics determine a tradeoff between high magnification in which high spatial detail from a restricted field of view is obtained versus low magnification in which lower spatial detail is obtained from a larger region. In addition to this geometric resolution versus field of view tradeoff, limited light availability establishes a physical limit when imaging movement. If the animal is moving, motion blur smears the subject on the sensor during exposure. Practically, motion blur is further compounded by sensor inefficiency and noise. While these fundamental tradeoffs with stationary cameras can be sidestepped by employing multiple cameras and providing additional illumination, this may not always be desirable. An alternative that overcomes these issues of stationary cameras is to direct a high magnification camera at an animal continually as it moves. Here we review systems in which automatic tracking is used to maintain an animal in the working volume of a moving optical path. Such methods provide an opportunity to escape the tradeoff between resolution and field of view and also to reduce motion blur while still enabling automated image acquisition. We argue that further development will be useful and outline potential innovations that may improve the technology and lead to more widespread use.

Measuring and modeling the motor system with machine learning

The utility of machine learning in understanding the motor system is promising a revolution in how to collect, measure, and analyze data. The field of movement science already elegantly incorporates theory and engineering principles to guide experimental work, and in this review we discuss the growing use of machine learning: from pose estimation, kinematic analyses, dimensionality reduction, and closed-loop feedback, to its use in understanding neural correlates and untangling sensorimotor systems. We also give our perspective on new avenues, where markerless motion capture combined with biomechanical modeling and neural networks could be a new platform for hypothesis-driven research.

Improving scalability in systems neuroscience

Emerging technologies to acquire data at increasingly greater scales promise to transform discovery in systems neuroscience. However, current exponential growth in the scale of data acquisition is a double-edged sword. Scaling up data acquisition can speed up the cycle of discovery but can also misinterpret the results or possibly slow down the cycle because of challenges presented by the curse of high-dimensional data. Active, adaptive, closed-loop experimental paradigms use hardware and algorithms optimized to enable time-critical computation to provide feedback that interprets the observations and tests hypotheses to actively update the stimulus or stimulation parameters. In this perspective, we review important concepts of active and adaptive experiments and discuss how selectively constraining the dimensionality and optimizing strategies at different stages of discovery loop can help mitigate the curse of high-dimensional data. Active and adaptive closed-loop experimental paradigms can speed up discovery despite an exponentially increasing data scale, offering a road map to timely and iterative hypothesis revision and discovery in an era of exponential growth in neuroscience.

Automation of training and testing motor and related tasks in pre-clinical behavioural and rehabilitative neuroscience

Testing and training animals in motor and related tasks is a cornerstone of pre-clinical behavioural and rehabilitative neuroscience. Yet manually testing and training animals in these tasks is time consuming and analyses are often subjective. Consequently, there have been many recent advances in automating both the administration and analyses of animal behavioural training and testing. This review is an in-depth appraisal of the history of, and recent developments in, the automation of animal behavioural assays used in neuroscience. We describe the use of common locomotor and non-locomotor tasks used for motor training and testing before and after nervous system injury. This includes a discussion of how these tasks help us to understand the underlying mechanisms of neurological repair and the utility of some tasks for the delivery of rehabilitative training to enhance recovery. We propose two general approaches to automation: automating the physical administration of behavioural tasks (i.e., devices used to facilitate task training, rehabilitative training, and motor testing) and leveraging the use of machine learning in behaviour analysis to generate large volumes of unbiased and comprehensive data. The advantages and disadvantages of automating various motor tasks as well as the limitations of machine learning analyses are examined. In closing, we provide a critical appraisal of the current state of automation in animal behavioural neuroscience and a prospective on some of the advances in machine learning we believe will dramatically enhance the usefulness of these approaches for behavioural neuroscientists.

Creating and controlling visual environments using BonVision

Real-time rendering of closed-loop visual environments is important for next-generation understanding of brain function and behaviour, but is often prohibitively difficult for non-experts to implement and is limited to few laboratories worldwide. We developed BonVision as an easy-to-use open-source software for the display of virtual or augmented reality, as well as standard visual stimuli. BonVision has been tested on humans and mice, and is capable of supporting new experimental designs in other animal models of vision. As the architecture is based on the open-source Bonsai graphical programming language, BonVision benefits from native integration with experimental hardware. BonVision therefore enables easy implementation of closed-loop experiments, including real-time interaction with deep neural networks, and communication with behavioural and physiological measurement and manipulation devices.

Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

Computer vision approaches have made significant inroads into offline tracking of behavior and estimating animal poses. In particular, because of their versatility, deep-learning approaches have been gaining attention in behavioral tracking without any markers. Here, we developed an approach using DeepLabCut for real-time estimation of movement. We trained a deep-neural network (DNN) offline with high-speed video data of a mouse whisking, then transferred the trained network to work with the same mouse, whisking in real-time. With this approach, we tracked the tips of three whiskers in an arc and converted positions into a TTL output within behavioral time scales, i.e., 10.5 ms. With this approach, it is possible to trigger output based on movement of individual whiskers, or on the distance between adjacent whiskers. Flexible closed-loop systems like the one we have deployed here can complement optogenetic approaches and can be used to directly manipulate the relationship between movement and neural activity.

New Open-Source Tools: Using Bonsai for Behavioral Tracking and Closed-Loop Experiments

The ability to dynamically control a behavioral task based on real-time animal behavior is an important feature for experimental neuroscientists. However, designing automated boxes for behavioral studies requires a coordinated combination of mechanical, electronic, and software design skills which can challenge even the best engineers, and for that reason used to be out of reach for the majority of experimental neurobiology and behavioral pharmacology researchers. Due to parallel advances in open-source hardware and software developed for neuroscience researchers, by neuroscience researchers, the landscape has now changed significantly. Here, we discuss powerful approaches to the study of behavior using examples and tutorials in the Bonsai visual programming language, towards designing simple neuroscience experiments that can help researchers immediately get started. This language makes it easy for researchers, even without programming experience, to combine the operation of several open-source devices in parallel and design their own integrated custom solutions, enabling unique and flexible approaches to the study of behavior, including video tracking of behavior and closed-loop electrophysiology.

DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection

In general, animal behavior can be described as the neuronal-driven sequence of reoccurring postures through time. Most of the available current technologies focus on offline pose estimation with high spatiotemporal resolution. However, to correlate behavior with neuronal activity it is often necessary to detect and react online to behavioral expressions. Here we present DeepLabStream, a versatile closed-loop tool providing real-time pose estimation to deliver posture dependent stimulations. DeepLabStream has a temporal resolution in the millisecond range, can utilize different input, as well as output devices and can be tailored to multiple experimental designs. We employ DeepLabStream to semi-autonomously run a second-order olfactory conditioning task with freely moving mice and optogenetically label neuronal ensembles active during specific head directions.