Machine Learning Algorithms for Automatic Classification of Marmoset Vocalizations

A large annotated dataset of vocalizations by common marmosets

Non-human primates, our closest relatives, use a wide range of complex vocal signals for communication within their species. Previous research on marmoset (Callithrix jacchus) vocalizations has been limited by sampling rates not covering the whole hearing range and insufficient labeling for advanced analyses using Deep Neural Networks (DNNs). Here, we provide a database of common marmoset vocalizations, which were continuously recorded with a sampling rate of 96 kHz from an animal holding facility housing simultaneously ~20 marmosets in three cages. The dataset comprises more than 800,000 files, amounting to 253 hours of data collected over 40 months. Each recording lasts a few seconds and captures the marmosets' social vocalizations, encompassing their entire known vocal repertoire during the experimental period. Around 215,000 calls are annotated with the vocalization type. We offer a trained classifier to assist future investigations. Finally, we validated our dataset by sampling 700 representative recordings and cross-examining them with four experts.

Beyond bigrams: call sequencing in the common marmoset (Callithrix jacchus) vocal system

Over the last two decades, an emerging body of research has demonstrated that non-human animals exhibit the ability to combine context-specific calls into larger sequences. These structures have frequently been compared with language's syntax, whereby linguistic units are combined to form larger structures, and leveraged to argue that syntax might not be unique to language. Currently, however, the overwhelming majority of examples of call combinations are limited to simple sequences comprising just two calls which differ dramatically from the open-ended hierarchical structuring of the syntax found in language. We revisit this issue by taking a whole-repertoire approach to investigate combinatoriality in common marmosets (Callithrix jacchus). We use Markov chain models to quantify the vocal sequences produced by marmosets providing evidence for structures beyond the bigram, including three-call and even combinations of up to eight or nine calls. Our analyses of these longer vocal sequences are suggestive of potential further internal organization, including some amount of recombination, nestedness and non-adjacent dependencies. We argue that data-driven, whole-repertoire analyses are fundamental to uncovering the combinatorial complexity of non-human animals and will further facilitate meaningful comparisons with language's combinatoriality.

Applying machine learning to primate bioacoustics: Review and perspectives

This paper provides a comprehensive review of the use of computational bioacoustics as well as signal and speech processing techniques in the analysis of primate vocal communication. We explore the potential implications of machine learning and deep learning methods, from the use of simple supervised algorithms to more recent self-supervised models, for processing and analyzing large data sets obtained within the emergence of passive acoustic monitoring approaches. In addition, we discuss the importance of automated primate vocalization analysis in tackling essential questions on animal communication and highlighting the role of comparative linguistics in bioacoustic research. We also examine the challenges associated with data collection and annotation and provide insights into potential solutions. Overall, this review paper runs through a set of common or innovative perspectives and applications of machine learning for primate vocal communication analysis and outlines opportunities for future research in this rapidly developing field.

Vocal complexity in the long calls of Bornean orangutans

Vocal complexity is central to many evolutionary hypotheses about animal communication. Yet, quantifying and comparing complexity remains a challenge, particularly when vocal types are highly graded. Male Bornean orangutans (Pongo pygmaeus wurmbii) produce complex and variable "long call" vocalizations comprising multiple sound types that vary within and among individuals. Previous studies described six distinct call (or pulse) types within these complex vocalizations, but none quantified their discreteness or the ability of human observers to reliably classify them. We studied the long calls of 13 individuals to: (1) evaluate and quantify the reliability of audio-visual classification by three well-trained observers, (2) distinguish among call types using supervised classification and unsupervised clustering, and (3) compare the performance of different feature sets. Using 46 acoustic features, we used machine learning (i.e., support vector machines, affinity propagation, and fuzzy c-means) to identify call types and assess their discreteness. We additionally used Uniform Manifold Approximation and Projection (UMAP) to visualize the separation of pulses using both extracted features and spectrogram representations. Supervised approaches showed low inter-observer reliability and poor classification accuracy, indicating that pulse types were not discrete. We propose an updated pulse classification approach that is highly reproducible across observers and exhibits strong classification accuracy using support vector machines. Although the low number of call types suggests long calls are fairly simple, the continuous gradation of sounds seems to greatly boost the complexity of this system. This work responds to calls for more quantitative research to define call types and quantify gradedness in animal vocal systems and highlights the need for a more comprehensive framework for studying vocal complexity vis-à-vis graded repertoires.

Bioacoustic characterization of the black-and-white ruffed lemur (Varecia variegata) vocal repertoire

Ruffed lemurs (Varecia spp.) exhibit a unique suite of behavioral traits compared to other lemur species, which includes their fluid fission-fusion social dynamics, communal rearing of parked litters, and pronounced frugivory in their humid rainforest habitats. Given these traits, and the dense rainforests they inhabit, vocal communication may be key to maintaining social cohesion, coordinating infant care, and/or defending their high-quality food resources. Indeed, they are known for their raucous 'roar-shriek' calls. However, there has been surprisingly little research on vocal communication in Varecia species and only two previously published repertoires, both of which were qualitative descriptions of their calls. In this study, we quantitatively examined the vocal repertoire of wild black-and-white ruffed lemurs (Varecia variegata) at Mangevo, Ranomafana National Park, Madagascar. We characterized 11 call types using 33 bioacoustic parameters related to frequency, duration, tonality, and composition. We also used discriminant function analysis and hierarchical clustering to quantitatively and objectively classify call types within the black-and-white ruffed lemur vocal repertoire. The repertoire consists of both monosyllabic and multisyllabic calls that are individually given or emitted in contagious choruses. Eight of the 11 calls were also used in combination or in larger multi-call sequences. The discriminant function analysis correctly assigned call types with 87% success, though this varied greatly by call type (1-65%). Hierarchical clustering identified 3-4 robust clusters, indicating low clustering structure in the data and suggesting that V. variegata exhibits a graded vocal repertoire. Future work should consider the environmental and behavioral contexts in which calls are used to better understand the function of these call types and combinations.

The neurobiology of vocal communication in marmosets

An increasingly popular animal model for studying the neural basis of social behavior, cognition, and communication is the common marmoset (Callithrix jacchus). Interest in this New World primate across neuroscience is now being driven by their proclivity for prosociality across their repertoire, high volubility, and rapid development, as well as their amenability to naturalistic testing paradigms and freely moving neural recording and imaging technologies. The complement of these characteristics set marmosets up to be a powerful model of the primate social brain in the years to come. Here, we focus on vocal communication because it is the area that has both made the most progress and illustrates the prodigious potential of this species. We review the current state of the field with a focus on the various brain areas and networks involved in vocal perception and production, comparing the findings from marmosets to other animals, including humans.

Improving the workflow to crack Small, Unbalanced, Noisy, but Genuine (SUNG) datasets in bioacoustics: The case of bonobo calls

Despite the accumulation of data and studies, deciphering animal vocal communication remains challenging. In most cases, researchers must deal with the sparse recordings composing Small, Unbalanced, Noisy, but Genuine (SUNG) datasets. SUNG datasets are characterized by a limited number of recordings, most often noisy, and unbalanced in number between the individuals or categories of vocalizations. SUNG datasets therefore offer a valuable but inevitably distorted vision of communication systems. Adopting the best practices in their analysis is essential to effectively extract the available information and draw reliable conclusions. Here we show that the most recent advances in machine learning applied to a SUNG dataset succeed in unraveling the complex vocal repertoire of the bonobo, and we propose a workflow that can be effective with other animal species. We implement acoustic parameterization in three feature spaces and run a Supervised Uniform Manifold Approximation and Projection (S-UMAP) to evaluate how call types and individual signatures cluster in the bonobo acoustic space. We then implement three classification algorithms (Support Vector Machine, xgboost, neural networks) and their combination to explore the structure and variability of bonobo calls, as well as the robustness of the individual signature they encode. We underscore how classification performance is affected by the feature set and identify the most informative features. In addition, we highlight the need to address data leakage in the evaluation of classification performance to avoid misleading interpretations. Our results lead to identifying several practical approaches that are generalizable to any other animal communication system. To improve the reliability and replicability of vocal communication studies with SUNG datasets, we thus recommend: i) comparing several acoustic parameterizations; ii) visualizing the dataset with supervised UMAP to examine the species acoustic space; iii) adopting Support Vector Machines as the baseline classification approach; iv) explicitly evaluating data leakage and possibly implementing a mitigation strategy.

Automated acoustic detection of Geoffroy's spider monkey highlights tipping points of human disturbance

As more land is altered by human activity and more species become at risk of extinction, it is essential that we understand the requirements for conserving threatened species across human-modified landscapes. Owing to their rarity and often sparse distributions, threatened species can be difficult to study and efficient methods to sample them across wide temporal and spatial scales have been lacking. Passive acoustic monitoring (PAM) is increasingly recognized as an efficient method for collecting data on vocal species; however, the development of automated species detectors required to analyse large amounts of acoustic data is not keeping pace. Here, we collected 35 805 h of acoustic data across 341 sites in a region over 1000 km² to show that PAM, together with a newly developed automated detector, is able to successfully detect the endangered Geoffroy's spider monkey (Ateles geoffroyi), allowing us to show that Geoffroy's spider monkey was absent below a threshold of 80% forest cover and within 1 km of primary paved roads and occurred equally in old growth and secondary forests. We discuss how this methodology circumvents many of the existing issues in traditional sampling methods and can be highly successful in the study of vocally rare or threatened species. Our results provide tools and knowledge for setting targets and developing conservation strategies for the protection of Geoffroy's spider monkey.

Mammogram Image Enhancement Techniques for Online Breast Cancer Detection and Diagnosis

Breast cancer is the type of cancer with the highest incidence and global mortality of female cancers. Thus, the adaptation of modern technologies that assist in medical diagnosis in order to accelerate, automate and reduce the subjectivity of this process are of paramount importance for an efficient treatment. Therefore, this work aims to propose a robust platform to compare and evaluate the proposed strategies for improving breast ultrasound images and compare them with state-of-the-art techniques by classifying them as benign, malignant and normal. Investigations were performed on a dataset containing a total of 780 images of tumor-affected persons, divided into benign, malignant and normal. A data augmentation technique was used to scale up the corpus of images available in the chosen dataset. For this, novel image enhancement techniques were used and the Multilayer Perceptrons, k-Nearest Neighbor and Support Vector Machines algorithms were used for classification. From the promising outcomes of the conducted experiments, it was observed that the bilateral algorithm together with the SVM classifier achieved the best result for the classification of breast cancer, with an overall accuracy of 96.69% and an accuracy for the detection of malignant nodules of 95.11%. Therefore, it was found that the application of image enhancement methods can help in the detection of breast cancer at a much earlier stage with better accuracy in detection.

Utilizing DeepSqueak for automatic detection and classification of mammalian vocalizations: a case study on primate vocalizations

Bioacoustic analyses of animal vocalizations are predominantly accomplished through manual scanning, a highly subjective and time-consuming process. Thus, validated automated analyses are needed that are usable for a variety of animal species and easy to handle by non-programing specialists. This study tested and validated whether DeepSqueak, a user-friendly software, developed for rodent ultrasonic vocalizations, can be generalized to automate the detection/segmentation, clustering and classification of high-frequency/ultrasonic vocalizations of a primate species. Our validation procedure showed that the trained detectors for vocalizations of the gray mouse lemur (Microcebus murinus) can deal with different call types, individual variation and different recording quality. Implementing additional filters drastically reduced noise signals (4225 events) and call fragments (637 events), resulting in 91% correct detections (N_total = 3040). Additionally, the detectors could be used to detect the vocalizations of an evolutionary closely related species, the Goodman’s mouse lemur (M. lehilahytsara). An integrated supervised classifier classified 93% of the 2683 calls correctly to the respective call type, and the unsupervised clustering model grouped the calls into clusters matching the published human-made categories. This study shows that DeepSqueak can be successfully utilized to detect, cluster and classify high-frequency/ultrasonic vocalizations of other taxa than rodents, and suggests a validation procedure usable to evaluate further bioacoustics software.

Mapping soil pollution by using drone image recognition and machine learning at an arsenic-contaminated agricultural field

Mapping soil contamination enables the delineation of areas where protection measures are needed. Traditional soil sampling on a grid pattern followed by chemical analysis and geostatistical interpolation methods (GIMs), such as Kriging interpolation, can be costly, slow and not well-suited to highly heterogeneous soil environments. Here we propose a novel method to map soil contamination by combining high-resolution aerial imaging (HRAI) with machine learning algorithms. To support model establishment and validation, 1068 soil samples were collected from an arsenic (As) contaminated area in Zhongxiang, Hubei province, China. The average arsenic concentration was 39.88 mg/kg (SD = 213.70 mg/kg), with individual sample points determined as low risk (66.9%), medium risk (29.4%), or high risk (3.7%), respectively. Then, identified features were extracted from a HRAI image of the study area. Four machine learning algorithms were developed to predict As risk levels, including (i) support vector machine (SVM), (ii) multi-layer perceptron (MLP), (iii) random forest (RF), and (iii) extreme random forest (ERF). Among these, we found that the ERF algorithm performed best overall and that its prediction performance was generally better than that of traditional Kriging interpolation. The accuracy of ERF in test area 1 reached 0.87, performing better than RF (0.81), MLP (0.78) and SVM (0.77). The F1-score of ERF for discerning high-risk points in test area 1 was as high as 0.8. The complexity of the distribution of points with different risk levels was a decisive factor in model prediction ability. Identified features in the study area associated with fertilizer factories had the most important contribution to the ERF model. This study demonstrates that HRAI combined with machine learning has good potential to predict As soil risk levels.

A machine learning approach to infant distress calls and maternal behaviour of wild chimpanzees

Distress calls are an acoustically variable group of vocalizations ubiquitous in mammals and other animals. Their presumed function is to recruit help, but there has been much debate on whether the nature of the disturbance can be inferred from the acoustics of distress calls. We used machine learning to analyse episodes of distress calls of wild infant chimpanzees. We extracted exemplars from those distress call episodes and examined them in relation to the external event triggering them and the distance to the mother. In further steps, we tested whether the acoustic variants were associated with particular maternal responses. Our results suggest that, although infant chimpanzee distress calls are highly graded, they can convey information about discrete problems experienced by the infant and about distance to the mother, which in turn may help guide maternal parenting decisions. The extent to which mothers rely on acoustic cues alone (versus integrate other contextual-visual information) to decide upon intervening should be the focus of future research.

Validation prediction: a flexible protocol to increase efficiency of automated acoustic processing for wildlife research

Automated recognition is increasingly used to extract species detections from audio recordings; however, the time required to manually review each detection can be prohibitive. We developed a flexible protocol called "validation prediction" that uses machine learning to predict whether recognizer detections are true or false positives and can be applied to any recognizer type, ecological application, or analytical approach. Validation prediction uses a predictable relationship between recognizer score and the energy of an acoustic signal but can also incorporate any other ecological or spectral predictors (e.g., time of day, dominant frequency) that will help separate true from false-positive recognizer detections. First, we documented the relationship between recognizer score and the energy of an acoustic signal for two different recognizer algorithm types (hidden Markov models and convolutional neural networks). Next, we demonstrated our protocol using a case study of two species, the Common Nighthawk (Chordeiles minor) and Ovenbird (Seiurus aurocapilla). We reduced the number of detections that required validation by 75.7% and 42.9%, respectively, while retaining at least 98% of the true-positive detections. Validation prediction substantially improves the efficiency of using automated recognition on acoustic data sets. Our method can be of use to wildlife monitoring and research programs and will facilitate using automated recognition to mine bioacoustic data sets.

Vocal individuality and rhythm in male and female duet contributions of a nonhuman primate

Duetting, or the stereotypical, repeated and often coordinated vocalizations between 2 individuals arose independently multiple times in the Order Primates. Across primate species, there exists substantial variation in terms of timing, degree of overlap, and sex-specificity of duet contributions. There is increasing evidence that primates can modify the timing of their duet contributions relative to their partner, and this vocal flexibility may have been an important precursor to the evolution of human language. Here, we present the results of a fine-scale analysis of Gursky's spectral tarsier Tarsius spectrumgurskyae duet phrases recorded in North Sulawesi, Indonesia. Specifically, we aimed to investigate individual-level variation in the female and male contributions to the duet, quantify individual- and pair-level differences in duet timing, and measure temporal precision of duetting individuals relative to their partner. We were able to classify female duet phrases to the correct individual with an 80% accuracy using support vector machines, whereas our classification accuracy for males was lower at 64%. Females were more variable than males in terms of timing between notes. All tarsier phrases exhibited some degree of overlap between callers, and tarsiers exhibited high temporal precision in their note output relative to their partners. We provide evidence that duetting tarsier individuals can modify their note output relative to their duetting partner, and these results support the idea that flexibility in vocal exchanges-a precursor to human language-evolved early in the primate lineage and long before the emergence of modern humans.

Comparing context-dependent call sequences employing machine learning methods: an indication of syntactic structure of greater horseshoe bats

For analysis of vocal syntax, accurate classification of call sequence structures in different behavioural contexts is essential. However, an effective, intelligent program for classifying call sequences from numerous recorded sound files is still lacking. Here, we employed three machine learning algorithms (logistic regression, support vector machine and decision trees) to classify call sequences of social vocalizations of greater horseshoe bats (Rhinolophus ferrumequinum) in aggressive and distress contexts. The three machine learning algorithms obtained highly accurate classification rates (logistic regression 98%, support vector machine 97% and decision trees 96%). The algorithms also extracted three of the most important features for the classification: the transition between two adjacent syllables, the probability of occurrences of syllables in each position of a sequence, and the characteristics of a sequence. The results of statistical analysis also supported the classification of the algorithms. The study provides the first efficient method for data mining of call sequences and the possibility of linguistic parameters in animal communication. It suggests the presence of song-like syntax in the social vocalizations emitted within a non-breeding context in a bat species.

Deep convolutional network for animal sound classification and source attribution using dual audio recordings

This paper introduces an end-to-end feedforward convolutional neural network that is able to reliably classify the source and type of animal calls in a noisy environment using two streams of audio data after being trained on a dataset of modest size and imperfect labels. The data consists of audio recordings from captive marmoset monkeys housed in pairs, with several other cages nearby. The network in this paper can classify both the call type and which animal made it with a single pass through a single network using raw spectrogram images as input. The network vastly increases data analysis capacity for researchers interested in studying marmoset vocalizations, and allows data collection in the home cage, in group housed animals.

Automatic detection and classification of marmoset vocalizations using deep and recurrent neural networks

This paper investigates the methods to detect and classify marmoset vocalizations automatically using a large data set of marmoset vocalizations and deep learning techniques. For vocalization detection, neural networks-based methods, including deep neural network (DNN) and recurrent neural network with long short-term memory units, are designed and compared against a conventional rule-based detection method. For vocalization classification, three different classification algorithms are compared, including a support vector machine (SVM), DNN, and long short-term memory recurrent neural networks (LSTM-RNNs). A 1500-min audio data set containing recordings from four pairs of marmoset twins and manual annotations is employed for experiments. Two test sets are built according to whether the test samples are produced by the marmosets in the training set (test set I) or not (test set II). Experimental results show that the LSTM-RNN-based detection method outperformed others and achieved 0.92% and 1.67% frame error rate on these two test sets. Furthermore, the deep learning models obtained higher classification accuracy than the SVM model, which was 95.60% and 91.67% on the two test sets, respectively.

A "voice patch" system in the primate brain for processing vocal information?

We review behavioural and neural evidence for the processing of information contained in conspecific vocalizations (CVs) in three primate species: humans, macaques and marmosets. We focus on abilities that are present and ecologically relevant in all three species: the detection and sensitivity to CVs; and the processing of identity cues in CVs. Current evidence, although fragmentary, supports the notion of a "voice patch system" in the primate brain analogous to the face patch system of visual cortex: a series of discrete, interconnected cortical areas supporting increasingly abstract representations of the vocal input. A central question concerns the degree to which the voice patch system is conserved in evolution. We outline challenges that arise and suggesting potential avenues for comparing the organization of the voice patch system across primate brains.