A system for tracking whisker kinematics and whisker shape in three dimensions

Sensory adaptation in the barrel cortex during active sensation in the behaving mouse

Sensory Adaptation (SA) is a prominent aspect of how neurons respond to sensory signals, ubiquitous across species and modalities. However, SA depends on the activation state of the brain and the extent to which SA is expressed in awake, behaving animals during active sensation remains unclear. Here, we addressed this question by training head-fixed mice to detect an object using their whiskers and recording neuronal activity from barrel cortex whilst simultaneously imaging the whiskers in 3D. We found that neuronal responses decreased during the course of whisker-object touch sequences and that this was due to two factors. First, a motor effect, whereby, during a sequence of touches, later touches were mechanically weaker than early ones. Second, a sensory encoding effect, whereby neuronal tuning to touch became progressively less sensitive during the course of a touch sequence. The sensory encoding effect was whisker-specific. These results show that SA does occur during active whisker sensing and suggest that SA is fundamental to sensation during natural behaviour.

A primary sensory cortical interareal feedforward inhibitory circuit for tacto-visual integration

Tactile sensation and vision are often both utilized for the exploration of objects that are within reach though it is not known whether or how these two distinct sensory systems combine such information. Here in mice, we used a combination of stereo photogrammetry for 3D reconstruction of the whisker array, brain-wide anatomical tracing and functional connectivity analysis to explore the possibility of tacto-visual convergence in sensory space and within the circuitry of the primary visual cortex (VISp). Strikingly, we find that stimulation of the contralateral whisker array suppresses visually evoked activity in a tacto-visual sub-region of VISp whose visual space representation closely overlaps with the whisker search space. This suppression is mediated by local fast-spiking interneurons that receive a direct cortico-cortical input predominantly from layer 6 neurons located in the posterior primary somatosensory barrel cortex (SSp-bfd). These data demonstrate functional convergence within and between two primary sensory cortical areas for multisensory object detection and recognition.

Facemap: a framework for modeling neural activity based on orofacial tracking

Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracker and a deep neural network encoder for predicting neural activity. Our algorithm for tracking mouse orofacial behaviors was more accurate than existing pose estimation tools, while the processing speed was several times faster, making it a powerful tool for real-time experimental interventions. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used the keypoints as inputs to a deep neural network which predicts the activity of ~50,000 simultaneously-recorded neurons and, in visual cortex, we doubled the amount of explained variance compared to previous methods. Using this model, we found that the neuronal activity clusters that were well predicted from behavior were more spatially spread out across cortex. We also found that the deep behavioral features from the model had stereotypical, sequential dynamics that were not reversible in time. In summary, Facemap provides a stepping stone toward understanding the function of the brain-wide neural signals and their relation to behavior.

Describing whisker morphology of the Carnivora

One of the largest ecological transitions in carnivoran evolution was the shift from terrestrial to aquatic lifestyles, which has driven morphological diversity in skulls and other skeletal structures. In this paper, we investigate the association between those lifestyles and whisker morphology. However, comparing whisker morphology over a range of species is challenging since the number of whiskers and their positions on the mystacial pads vary between species. Also, each whisker will be at a different stage of growth and may have incurred damage due to wear and tear. Identifying a way to easily capture whisker morphology in a small number of whisker samples would be beneficial. Here, we describe individual and species variation in whisker morphology from two-dimensional scans in red fox, European otter and grey seal. A comparison of long, caudal whiskers shows inter-species differences most clearly. We go on to describe global whisker shape in 24 species of carnivorans, using linear approximations of curvature and taper, as well as traditional morphometric methods. We also qualitatively examine surface texture, or the presence of scales, using scanning electron micrographs. We show that gross whisker shape is highly conserved, with whisker curvature and taper obeying simple linear relationships with length. However, measures of whisker base radius, length, and maybe even curvature, can vary between species and substrate preferences. Specifically, the aquatic species in our sample have thicker, shorter whiskers that are smoother, with less scales present than those of terrestrial species. We suggest that these thicker whiskers may be stiffer and able to maintain their shape and position during underwater sensing, but being stiffer may also increase wear.

Design of a high speed rat whiskers tracking and symmetry analysis system based on FPGA

This paper presents a high-speed rat whisker tracking and symmetry analysis system based on FPGA. The system utilizes high-speed image sensors recording rat face videos at 120 and 1000 fps. The Xilinx Ultra96 single computer board is chosen as the platform to implement the system's processing system (PS) and the programmable logic (PL) part. The PL part is responsible for high-speed image processing and whisker tracking, while the PS part analyzes the symmetry of rat face using the tracking results from the PL part. With a processing speed FoM of 118.5 fps/GHz on the Xilinx Ultra96 single computer board and 275.47 fps/GHz on a laptop with Intel Core i5-11500T@1.5GHz, the presented system achieves excellent performance. The proposed whisker detection method has a precision of 98.2% when a threshold with a 4-degree error is selected, with an average error angle of 0.98 degrees across more than 10,000 video frames. Moreover, the proposed system is capable of local video processing within millisecond delays. These results demonstrate the feasibility of developing a high-speed, accurate, and efficient whisker tracking and symmetry analysis system for rat behavior research.

Leptin deficiency-caused behavioral change - A comparative analysis using EthoVision and DeepLabCut

Introduction

Obese rodents e.g., the leptin-deficient (ob/ob) mouse exhibit remarkable behavioral changes and are therefore ideal models for evaluating mental disorders resulting from obesity. In doing so, female as well as male ob/ob mice at 8, 24, and 40 weeks of age underwent two common behavioral tests, namely the Open Field test and Elevated Plus Maze, to investigate behavioral alteration in a sex- and age dependent manner. The accuracy of these tests is often dependent on the observer that can subjectively influence the data.

Methods

To avoid this bias, mice were tracked with a video system. Video files were further analyzed by the compared use of two software, namely EthoVision (EV) and DeepLabCut (DLC). In DLC a Deep Learning application forms the basis for using artificial intelligence in behavioral research in the future, also with regard to the reduction of animal numbers.

Results

After no sex and partly also no age-related differences were found, comparison revealed that both software lead to almost identical results and are therefore similar in their basic outcomes, especially in the determination of velocity and total distance movement. Moreover, we observed additional benefits of DLC compared to EV as it enabled the interpretation of more complex behavior, such as rearing and leaning, in an automated manner.

Discussion

Based on the comparable results from both software, our study can serve as a starting point for investigating behavioral alterations in preclinical studies of obesity by using DLC to optimize and probably to predict behavioral observations in the future.

California sea lions employ task-specific strategies for active touch sensing

Active sensing is the process of moving sensors to extract task-specific information. Whisker touch is often referred to as an active sensory system as whiskers are moved with purposeful control. Even though whisker movements are found in many species, it is unknown whether any animal can make task-specific movements with their whiskers. California sea lions (Zalophus californianus) make large, purposeful whisker movements and are capable of performing many whisker-related discrimination tasks. Therefore, California sea lions are an ideal species to explore the active nature of whisker touch sensing. Here, we show that California sea lions can make task-specific whisker movements. California sea lions move their whiskers with large amplitudes around object edges to judge size, make smaller, lateral stroking movements to judge texture and make very small whisker movements during a visual task. These findings, combined with the ease of training mammals and measuring whisker movements, makes whiskers an ideal system for studying mammalian perception, cognition and motor control.

Highlighted Article: California sea lions engage in task-specific active touch sensing with their whiskers to discriminate size and differentiate textures, indicating that their whiskers are truly an active sensory system.

Fast cortical dynamics encode tactile grating orientation during active touch

Touch-based object recognition relies on perception of compositional tactile features like roughness, shape, and surface orientation. However, besides roughness, it remains unclear how these different tactile features are encoded by neural activity that is linked with perception. Here, we establish a cortex-dependent perceptual task in which mice discriminate tactile gratings on the basis of orientation using only their whiskers. Multielectrode recordings in the barrel cortex reveal weak orientation tuning in average firing rates (500-ms time scale) during grating exploration despite high levels of cortical activity. Just before decision, orientation information extracted from fast cortical dynamics (100-ms time scale) more closely resembles concurrent psychophysical measurements than single neuron orientation tuning curves. This temporal code conveys both stimulus and choice/action-related information, suggesting that fast cortical dynamics during exploration of a tactile object both reflect the physical stimulus and affect the decision.

Sensorimotor strategies and neuronal representations for shape discrimination

Humans and other animals can identify objects by active touch, requiring the coordination of exploratory motion and tactile sensation. Both the motor strategies and neural representations employed could depend on the subject's goals. We developed a shape discrimination task that challenged head-fixed mice to discriminate concave from convex shapes. Behavioral decoding revealed that mice did this by comparing contacts across whiskers. In contrast, a separate group of mice performing a shape detection task simply summed up contacts over whiskers. We recorded populations of neurons in the barrel cortex, which processes whisker input, and found that individual neurons across the cortical layers encoded touch, whisker motion, and task-related signals. Sensory representations were task-specific: during shape discrimination, but not detection, neurons responded most to behaviorally relevant whiskers, overriding somatotopy. Thus, sensory cortex employs task-specific representations compatible with behaviorally relevant computations.

Body language signals for rodent social communication

Integration of social cues to initiate adaptive emotional and behavioral responses is a fundamental aspect of animal and human behavior. In humans, social communication includes prominent nonverbal components, such as social touch, gestures and facial expressions. Comparative studies investigating the neural basis of social communication in rodents has historically been centered on olfactory signals and vocalizations, with relatively less focus on non-verbal social cues. Here, we outline two exciting research directions: First, we will review recent observations pointing to a role of social facial expressions in rodents. Second, we will review observations that point to a role of 'non-canonical' rodent body language: body posture signals beyond stereotyped displays in aggressive and sexual behavior. In both sections, we will outline how social neuroscience can build on recent advances in machine learning, robotics and micro-engineering to push these research directions forward towards a holistic systems neurobiology of rodent body language.

Real-Time Closed-Loop Feedback in Behavioral Time Scales Using DeepLabCut

Computer vision approaches have made significant inroads into offline tracking of behavior and estimating animal poses. In particular, because of their versatility, deep-learning approaches have been gaining attention in behavioral tracking without any markers. Here, we developed an approach using DeepLabCut for real-time estimation of movement. We trained a deep-neural network (DNN) offline with high-speed video data of a mouse whisking, then transferred the trained network to work with the same mouse, whisking in real-time. With this approach, we tracked the tips of three whiskers in an arc and converted positions into a TTL output within behavioral time scales, i.e., 10.5 ms. With this approach, it is possible to trigger output based on movement of individual whiskers, or on the distance between adjacent whiskers. Flexible closed-loop systems like the one we have deployed here can complement optogenetic approaches and can be used to directly manipulate the relationship between movement and neural activity.

A High-Dimensional Quantification of Mouse Defensive Behaviors Reveals Enhanced Diversity and Stimulus Specificity

Instinctive defensive behaviors, consisting of stereotyped sequences of movements and postures, are an essential component of the mouse behavioral repertoire. Since defensive behaviors can be reliably triggered by threatening sensory stimuli, the selection of the most appropriate action depends on the stimulus property. However, since the mouse has a wide repertoire of motor actions, it is not clear which set of movements and postures represent the relevant action. So far, this has been empirically identified as a change in locomotion state. However, the extent to which locomotion alone captures the diversity of defensive behaviors and their sensory specificity is unknown. To tackle this problem, we developed a method to obtain a faithful 3D reconstruction of the mouse body that enabled to quantify a wide variety of motor actions. This higher dimensional description revealed that defensive behaviors are more stimulus specific than indicated by locomotion data. Thus, responses to distinct stimuli that were equivalent in terms of locomotion (e.g., freezing induced by looming and sound) could be discriminated along other dimensions. The enhanced stimulus specificity was explained by a surprising diversity. A clustering analysis revealed that distinct combinations of movements and postures, giving rise to at least 7 different behaviors, were required to account for stimulus specificity. Moreover, each stimulus evoked more than one behavior, revealing a robust one-to-many mapping between sensations and behaviors that was not apparent from locomotion data. Our results indicate that diversity and sensory specificity of mouse defensive behaviors unfold in a higher dimensional space, spanning multiple motor actions.

WhiskEras: A New Algorithm for Accurate Whisker Tracking

Rodents engage in active touch using their facial whiskers: they explore their environment by making rapid back-and-forth movements. The fast nature of whisker movements, during which whiskers often cross each other, makes it notoriously difficult to track individual whiskers of the intact whisker field. We present here a novel algorithm, WhiskEras, for tracking of whisker movements in high-speed videos of untrimmed mice, without requiring labeled data. WhiskEras consists of a pipeline of image-processing steps: first, the points that form the whisker centerlines are detected with sub-pixel accuracy. Then, these points are clustered in order to distinguish individual whiskers. Subsequently, the whiskers are parameterized so that a single whisker can be described by four parameters. The last step consists of tracking individual whiskers over time. We describe that WhiskEras performs better than other whisker-tracking algorithms on several metrics. On our four video segments, WhiskEras detected more whiskers per frame than the Biotact Whisker Tracking Tool. The signal-to-noise ratio of the output of WhiskEras was higher than that of Janelia Whisk. As a result, the correlation between reflexive whisker movements and cerebellar Purkinje cell activity appeared to be stronger than previously found using other tracking algorithms. We conclude that WhiskEras facilitates the study of sensorimotor integration by markedly improving the accuracy of whisker tracking in untrimmed mice.