Context-Aware Mouse Behavior Recognition Using Hidden Markov Models

Cross-Skeleton Interaction Graph Aggregation Network for Representation Learning of Mouse Social Behaviour

Automated social behaviour analysis of mice has become an increasingly popular research area in behavioural neuroscience. Recently, pose information (i.e., locations of keypoints or skeleton) has been used to interpret social behaviours of mice. Nevertheless, effective encoding and decoding of social interaction information underlying the keypoints of mice has been rarely investigated in the existing methods. In particular, it is challenging to model complex social interactions between mice due to highly deformable body shapes and ambiguous movement patterns. To deal with the interaction modelling problem, we here propose a Cross-Skeleton Interaction Graph Aggregation Network (CS-IGANet) to learn abundant dynamics of freely interacting mice, where a Cross-Skeleton Node-level Interaction module (CS-NLI) is used to model multi-level interactions (i.e., intra-, inter- and cross-skeleton interactions). Furthermore, we design a novel Interaction-Aware Transformer (IAT) to dynamically learn the graph-level representation of social behaviours and update the node-level representation, guided by our proposed interaction-aware self-attention mechanism. Finally, to enhance the representation ability of our model, an auxiliary self-supervised learning task is proposed for measuring the similarity between cross-skeleton nodes. Experimental results on the standard CRMI13-Skeleton and our PDMB-Skeleton datasets show that our proposed model outperforms several other state-of-the-art approaches.

A Survey on AI-Driven Mouse Behavior Analysis Applications and Solutions

The physiological similarities between mice and humans make them vital animal models in biological and medical research. This paper explores the application of artificial intelligence (AI) in analyzing mice behavior, emphasizing AI's potential to identify and classify these behaviors. Traditional methods struggle to capture subtle behavioral features, whereas AI can automatically extract quantitative features from large datasets. Consequently, this study aims to leverage AI to enhance the efficiency and accuracy of mice behavior analysis. The paper reviews various applications of mice behavior analysis, categorizes deep learning tasks based on an AI pyramid, and summarizes AI methods for addressing these tasks. The findings indicate that AI technologies are increasingly applied in mice behavior analysis, including disease detection, assessment of external stimuli effects, social behavior analysis, and neurobehavioral assessment. The selection of AI methods is crucial and must align with specific applications. Despite AI's promising potential in mice behavior analysis, challenges such as insufficient datasets and benchmarks remain. Furthermore, there is a need for a more integrated AI platform, along with standardized datasets and benchmarks, to support these analyses and further advance AI-driven mice behavior analysis.

Glial expression of Drosophila UBE3A causes spontaneous seizures that can be modulated by 5-HT signaling

Misexpression of the E3 ubiquitin ligase gene UBE3A is thought to contribute to a range of neurological disorders. In the context of Dup15q syndrome, additional genomic copies of UBE3A give rise to the autism, muscle hypotonia and spontaneous seizures characteristics of the disorder. In a Drosophila model of Dup 15q syndrome, it was recently shown that glial-driven expression of the UBE3A ortholog dube3a led to a "bang-sensitive" phenotype, where mechanical shock triggers convulsions, suggesting glial dube3a expression contributes to hyperexcitability in flies. Here we directly compare the consequences of glial- and neuronal-driven dube3a expression on motor coordination and seizure susceptibility in Drosophila. To quantify seizure-related behavioral events, we developed and trained a hidden Markov model that identified these events based on automated video tracking of fly locomotion. Both glial and neuronal driven dube3a expression led to clear motor phenotypes. However, only glial-driven dube3a expression displayed spontaneous seizure-associated immobilization events, that were clearly observed at high-temperature (38 °C). Using a tethered fly preparation amenable to electrophysiological monitoring of seizure activity, we found glial-driven dube3a flies display aberrant spontaneous spike discharges which are bilaterally synchronized. Neither neuronal-dube3a overexpressing flies, nor control flies displayed these firing patterns. We previously performed a drug screen for FDA approved compounds that can suppress bang-sensitivity in glial-driven dube3a expressing flies and identified certain 5-HT modulators as strong seizure suppressors. Here we found glial-driven dube3a flies fed the serotonin reuptake inhibitor vortioxetine and the 5-HT2A antagonist ketanserin displayed reduced immobilization and spike bursting, consistent with the previous study. Together these findings highlight the potential for glial pathophysiology to drive Dup15q syndrome-related seizure activity.

Of Mice and Mates: Automated Classification and Modelling of Mouse Behaviour in Groups Using a Single Model Across Cages

Behavioural experiments often happen in specialised arenas, but this may confound the analysis. To address this issue, we provide tools to study mice in the home-cage environment, equipping biologists with the possibility to capture the temporal aspect of the individual's behaviour and model the interaction and interdependence between cage-mates with minimal human intervention. Our main contribution is the novel Global Behaviour Model (GBM) which summarises the joint behaviour of groups of mice across cages, using a permutation matrix to match the mouse identities in each cage to the model. In support of the above, we also (a) developed the Activity Labelling Module (ALM) to automatically classify mouse behaviour from video, and (b) released two datasets, ABODe for training behaviour classifiers and IMADGE for modelling behaviour.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11263-024-02118-3.

Glial expression of Drosophila UBE3A causes spontaneous seizures modulated by 5-HT signaling

Misexpression of the E3 ubiquitin ligase UBE3A is thought to contribute to a range of neurological disorders. In the context of Dup15q syndrome, excess genomic copies of UBE3A is thought to contribute to the autism, muscle tone and spontaneous seizures characteristic of the disorder. In a Drosophila model of Dup 15q syndrome, it was recently shown glial-driven expression of the UBE3A ortholog dube3a led to a "bang-sensitive" phenotype, where mechanical shock triggers convulsions, suggesting glial dube3a expression contributes to hyperexcitability in flies. Here we directly compare the consequences of glial- and neuronal-driven dube3a expression on motor coordination and neuronal excitability in Drosophila. We utilized IowaFLI tracker and developed a hidden Markov Model to classify seizure-related immobilization. Both glial and neuronal driven dube3a expression led to clear motor phenotypes. However, only glial-driven dube3a expression displayed spontaneous immobilization events, that were exacerbated at high-temperature (38 °C). Using a tethered fly preparation we monitored flight muscle activity, we found glial-driven dube3a flies display spontaneous spike discharges which were bilaterally synchronized indicative of seizure activity. Neither control flies, nor neuronal- dube3a overexpressing flies display such firing patterns. Prior drug screen indicated bang-sensitivity in glial-driven dube3a expressing flies could be suppressed by certain 5-HT modulators. Consistent with this report, we found glial-driven dube3a flies fed the serotonin reuptake inhibitor vortioxetine and the 5HT 2A antagonist ketanserin displayed reduced immobilization and spike bursting. Together these findings highlight the potential for glial pathophysiology to drive Dup15q syndrome-related seizure activity.

Detecting and Tracking of Multiple Mice Using Part Proposal Networks

The study of mouse social behaviors has been increasingly undertaken in neuroscience research. However, automated quantification of mouse behaviors from the videos of interacting mice is still a challenging problem, where object tracking plays a key role in locating mice in their living spaces. Artificial markers are often applied for multiple mice tracking, which are intrusive and consequently interfere with the movements of mice in a dynamic environment. In this article, we propose a novel method to continuously track several mice and individual parts without requiring any specific tagging. First, we propose an efficient and robust deep-learning-based mouse part detection scheme to generate part candidates. Subsequently, we propose a novel Bayesian-inference integer linear programming (BILP) model that jointly assigns the part candidates to individual targets with necessary geometric constraints while establishing pair-wise association between the detected parts. There is no publicly available dataset in the research community that provides a quantitative test bed for part detection and tracking of multiple mice, and we here introduce a new challenging Multi-Mice PartsTrack dataset that is made of complex behaviors. Finally, we evaluate our proposed approach against several baselines on our new datasets, where the results show that our method outperforms the other state-of-the-art approaches in terms of accuracy. We also demonstrate the generalization ability of the proposed approach on tracking zebra and locust.

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

The identification of animal behavior in video is a critical but time-consuming task in many areas of research. Here, we introduce DeepAction, a deep learning-based toolbox for automatically annotating animal behavior in video. Our approach uses features extracted from raw video frames by a pretrained convolutional neural network to train a recurrent neural network classifier. We evaluate the classifier on two benchmark rodent datasets and one octopus dataset. We show that it achieves high accuracy, requires little training data, and surpasses both human agreement and most comparable existing methods. We also create a confidence score for classifier output, and show that our method provides an accurate estimate of classifier performance and reduces the time required by human annotators to review and correct automatically-produced annotations. We release our system and accompanying annotation interface as an open-source MATLAB toolbox.

Validation of a system for automatic quantitative analysis of laboratory mice behavior based on locomotor pose

Automatic recognition and accurate quantitative analysis of rodent behavior play an important role in brain neuroscience, pharmacological and toxicological. Currently, most behavior recognition systems used in experiments mainly focus on the indirect measurements of animal movement trajectories, while neglecting the changes of animal body pose that can indicate more psychological factors. Thus, this paper developed and validated an hourglass network-based behavioral quantification system (HNBQ), which uses a combination of body pose and movement parameters to quantify the activity of mice in an enclosed experimental chamber. In addition, The HNBQ was employed to record behavioral abnormalities of head scanning in the presence of food gradients in open field test (OFT). The results proved that the HNBQ in the new object recognition (NOR) experiment was highly correlated with the scores of manual observers during the latent exploration period and the cumulative exploration time. Moreover, in the OFT, HNBQ was able to capture the subtle differences in head scanning behavior of mice in the gradient experimental groups. Satisfactory results support that the combination of body pose and motor parameters can regard as a new alternative approach for quantification of animal behavior in laboratory.

Multimodal Gait Recognition for Neurodegenerative Diseases

In recent years, single modality-based gait recognition has been extensively explored in the analysis of medical images or other sensory data, and it is recognized that each of the established approaches has different strengths and weaknesses. As an important motor symptom, gait disturbance is usually used for diagnosis and evaluation of diseases; moreover, the use of multimodality analysis of the patient's walking pattern compensates for the one-sidedness of single modality gait recognition methods that only learn gait changes in a single measurement dimension. The fusion of multiple measurement resources has demonstrated promising performance in the identification of gait patterns associated with individual diseases. In this article, as a useful tool, we propose a novel hybrid model to learn the gait differences between three neurodegenerative diseases, between patients with different severity levels of Parkinson's disease, and between healthy individuals and patients, by fusing and aggregating data from multiple sensors. A spatial feature extractor (SFE) is applied to generating representative features of images or signals. In order to capture temporal information from the two modality data, a new correlative memory neural network (CorrMNN) architecture is designed for extracting temporal features. Afterward, we embed a multiswitch discriminator to associate the observations with individual state estimations. Compared with several state-of-the-art techniques, our proposed framework shows more accurate classification results.

Challenges of a small world analysis for the continuous monitoring of behavior in mice

Documenting a mouse's "real world" behavior in the "small world" of a laboratory cage with continuous video recordings offers insights into phenotypical expression of mouse genotypes, development and aging, and neurological disease. Nevertheless, there are challenges in the design of a small world, the behavior selected for analysis, and the form of the analysis used. Here we offer insights into small world analyses by describing how acute behavioral procedures can guide continuous behavioral methodology. We show how algorithms can identify behavioral acts including walking and rearing, circadian patterns of action including sleep duration and waking activity, and the organization of patterns of movement into home base activity and excursions, and how they are altered with aging. We additionally describe how specific tests can be incorporated within a mouse's living arrangement. We emphasize how machine learning can condense and organize continuous activity that extends over extended periods of time.

Multi-View Mouse Social Behaviour Recognition With Deep Graphic Model

Home-cage social behaviour analysis of mice is an invaluable tool to assess therapeutic efficacy of neurodegenerative diseases. Despite tremendous efforts made within the research community, single-camera video recordings are mainly used for such analysis. Because of the potential to create rich descriptions for mouse social behaviors, the use of multi-view video recordings for rodent observations is increasingly receiving much attention. However, identifying social behaviours from various views is still challenging due to the lack of correspondence across data sources. To address this problem, we here propose a novel multi-view latent-attention and dynamic discriminative model that jointly learns view-specific and view-shared sub-structures, where the former captures unique dynamics of each view whilst the latter encodes the interaction between the views. Furthermore, a novel multi-view latent-attention variational autoencoder model is introduced in learning the acquired features, enabling us to learn discriminative features in each view. Experimental results on the standard CRMI13 and our multi-view Parkinson's Disease Mouse Behaviour (PDMB) datasets demonstrate that our proposed model outperforms the other state of the arts technologies, has lower computational cost than the other graphical models and effectively deals with the imbalanced data problem.

Continuous Sign Language Recognition through a Context-Aware Generative Adversarial Network

Continuous sign language recognition is a weakly supervised task dealing with the identification of continuous sign gestures from video sequences, without any prior knowledge about the temporal boundaries between consecutive signs. Most of the existing methods focus mainly on the extraction of spatio-temporal visual features without exploiting text or contextual information to further improve the recognition accuracy. Moreover, the ability of deep generative models to effectively model data distribution has not been investigated yet in the field of sign language recognition. To this end, a novel approach for context-aware continuous sign language recognition using a generative adversarial network architecture, named as Sign Language Recognition Generative Adversarial Network (SLRGAN), is introduced. The proposed network architecture consists of a generator that recognizes sign language glosses by extracting spatial and temporal features from video sequences, as well as a discriminator that evaluates the quality of the generator’s predictions by modeling text information at the sentence and gloss levels. The paper also investigates the importance of contextual information on sign language conversations for both Deaf-to-Deaf and Deaf-to-hearing communication. Contextual information, in the form of hidden states extracted from the previous sentence, is fed into the bidirectional long short-term memory module of the generator to improve the recognition accuracy of the network. At the final stage, sign language translation is performed by a transformer network, which converts sign language glosses to natural language text. Our proposed method achieved word error rates of 23.4%, 2.1% and 2.26% on the RWTH-Phoenix-Weather-2014 and the Chinese Sign Language (CSL) and Greek Sign Language (GSL) Signer Independent (SI) datasets, respectively.

Research on behavior recognition based on feature fusion of automatic coder and recurrent neural network

Under the impact of COVID-19, research on behavior recognition are highly needed. In this paper, we combine the algorithm of self-adaptive coder and recurrent neural network to realize the research of behavior pattern recognition. At present, most of the research of human behavior recognition is focused on the video data, which is based on the video number. At the same time, due to the complexity of video image data, it is easy to violate personal privacy. With the rapid development of Internet of things technology, it has attracted the attention of a large number of experts and scholars. Researchers have tried to use many machine learning methods, such as random forest, support vector machine and other shallow learning methods, which perform well in the laboratory environment, but there is still a long way to go from practical application. In this paper, a recursive neural network algorithm based on long and short term memory (LSTM) is proposed to realize the recognition of behavior patterns, so as to improve the accuracy of human activity behavior recognition.

Vision tracking based on adaptive interactive fusion

Under the influence of COVID-19, detection and identification of moving targets are very important for personnel management. A lot of research work has improved the accuracy and robustness of the moving target tracking method, but the recognition accuracy of the traditional target tracking method in complex scenes (lighting changes, background interference, posture changes and other factors) is not satisfactory. In this paper, in view of the limitations of single feature representation of target objects, the method of fusion of HSV color features and edge direction features is used to identify and detect moving targets. In each frame of the tracking process, the weight of each feature is adjusted adaptively according to the proposed fusion strategy, and the position of the target is located by using the method of double template matching. Experiments show that the proposed tracking algorithm based on multi feature fusion can meet the requirements of moving target recognition in complex scenes. The method proposed in this paper has a certain reference value for personnel management under the influence of COVID-19.