Beyond speech: Exploring diversity in the human voice

Acoustic estimation of voice roughness

Roughness is a perceptual characteristic of sound that was first applied to musical consonance and dissonance, but it is increasingly recognized as a central aspect of voice quality in human and animal communication. It may be particularly important for asserting social dominance or attracting attention in urgent signals such as screams. To ensure that the results of roughness research are valid and consistent across studies, we need standard methodology for measuring it. I review the literature on roughness estimation, from classic psychoacoustics to more recent approaches, and present two collections of 602 human vocal samples whose roughness was rated by 162 listeners in perceptual experiments. Two algorithms for estimating roughness acoustically from modulation spectra are then presented and optimized to match the human ratings. One uses a bank of gammatone or Butterworth filters to obtain an auditory spectrogram, and a faster algorithm begins with a conventional spectrogram obtained with Short-Time Fourier transform; both explain ~ 50% of variance in average human ratings per stimulus. The range of modulation frequencies most relevant to roughness perception is [50, 200] Hz; this range can be selected with simple cutoff points or with a lognormal weighting function. Modulation and roughness spectrograms are proposed as visual aids for studying the dynamics of roughness in longer recordings. The described algorithms are implemented in the function modulationSpectrum() from the open-source R library soundgen. The audio recordings and their ratings are freely available from https://osf.io/gvcpx/ and can be used for benchmarking other algorithms.

Evaluation of fetal exposure to environmental noise using a computer-generated model

Acoustic noise can have profound effects on wellbeing, impacting the health of pregnant women and their fetus. Mounting evidence suggests neural memory traces are formed by auditory learning in utero. A better understanding of the fetal auditory environment is therefore critical to avoid exposure to damaging noise levels. Using anatomical data from MRI scans of pregnant patients ( N = 4 ) from 24 weeks of gestation, we develop a computational model to quantify fetal exposure to acoustic field. We obtain acoustic transfer characteristics across the human audio range and pressure maps in transverse planes passing through the uterus at 5 kHz, 10 kHz and 20 kHz, showcasing multiple scattering and modal patterns. Our calculations show that the sound transmitted in utero is attenuated by as little as 6 dB below 1 kHz, confirming results from animal studies that the maternal abdomen and pelvis do not shelter the fetus from external noise.

CoVox: A dataset of contrasting vocalizations

The human voice is remarkably versatile and can vary greatly in sound depending on how it is used. An increasing number of studies have addressed the differences and similarities between the singing and the speaking voice. However, finding adequate stimuli material that is at the same time controlled and ecologically valid is challenging, and most datasets lack variability in terms of vocal styles performed by the same voice. Here, we describe a curated stimulus set of vocalizations where 22 female singers performed the same melody excerpts in three contrasting singing styles (as a lullaby, as a pop song, and as an opera aria) and spoke the text aloud in two speaking styles (as if speaking to an adult or to an infant). All productions were made with the songs' original lyrics, in Brazilian Portuguese, and with a/lu/sound. This ecologically valid dataset of 1320 vocalizations was validated through a forced-choice lab experiment (N = 25 for each stimulus) where lay listeners could recognize the intended vocalization style with high accuracy (proportion of correct recognition superior to 69% for all styles). We also provide acoustic characterization of the stimuli, depicting clear and contrasting acoustic profiles depending on the style of vocalization. All recordings are made freely available under a Creative Commons license and can be downloaded at https://osf.io/cgexn/ .

Formant analysis of vertebrate vocalizations: achievements, pitfalls, and promises

When applied to vertebrate vocalizations, source-filter theory, initially developed for human speech, has revolutionized our understanding of animal communication, resulting in major insights into the form and function of animal sounds. However, animal calls and human nonverbal vocalizations can differ qualitatively from human speech, often having more chaotic and higher-frequency sources, making formant measurement challenging. We review the considerable achievements of the "formant revolution" in animal vocal communication research, then highlight several important methodological problems in formant analysis. We offer concrete recommendations for effectively applying source-filter theory to non-speech vocalizations and discuss promising avenues for future research in this area.Brief Formants (vocal tract resonances) play key roles in animal communication, offering researchers exciting promise but also potential pitfalls.

Exploring nonlinear phenomena in animal vocalizations through oscillator theory

Animal vocalizations comprise a rich array of complex sounds that exhibit nonlinear phenomena (NLP), which have fascinated researchers for decades. From the melodic songs of birds to the clicks and whistles of dolphins, many species have been found to produce nonlinear vocalizations, offering a valuable perspective on the mechanisms underlying sound production and potential adaptive functions. By leveraging on the principles of oscillator theory and nonlinear dynamics, animal vocalizations, which are based on coupled oscillators, can be described and conveniently classified. We review the basic ingredients for self-sustained oscillations and how different NLP can emerge. We discuss important terms in the context of oscillator theory: attractor types, phase space, bifurcations and Arnold tongue diagrams. Through a comparative analysis of observed NLP and bifurcation diagrams, our study reviews how the tools of nonlinear dynamics can provide insights into the intricate complexity of animal vocalizations, as well as into the evolutionary pressures and adaptive strategies that have shaped the diverse communication systems of the animal kingdom.This article is part of the theme issue, 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

How to analyse and manipulate nonlinear phenomena in voice recordings

We address two research applications in this methodological review: starting from an audio recording, the goal may be to characterize nonlinear phenomena (NLP) at the level of voice production or to test their perceptual effects on listeners. A crucial prerequisite for this work is the ability to detect NLP in acoustic signals, which can then be correlated with biologically relevant information about the caller and with listeners' reaction. NLP are often annotated manually, but this is labour-intensive and not very reliable, although we describe potentially helpful advanced visualization aids such as reassigned spectrograms and phasegrams. Objective acoustic features can also be useful, including general descriptives (harmonics-to-noise ratio, cepstral peak prominence, vocal roughness), statistics derived from nonlinear dynamics (correlation dimension) and NLP-specific measures (depth of modulation and subharmonics). On the perception side, playback studies can greatly benefit from tools for directly manipulating NLP in recordings. Adding frequency jumps, amplitude modulation and subharmonics is relatively straightforward. Creating biphonation, imitating chaos or removing NLP from a recording are more challenging, but feasible with parametric voice synthesis. We describe the most promising algorithms for analysing and manipulating NLP and provide detailed examples with audio files and R code in supplementary material.This article is part of the theme issue 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

Applying nonlinear dynamics to the voice: a historical perspective

The recognition that nonlinear phenomena, including subharmonics, bifurcations and deterministic chaos, are present in human and animal vocalizations is a relatively recent one. I give a brief history of this revolution in our understanding of the voice, based on interviews with some of the key players and personal experience. Most of the key concepts and mathematical principles of nonlinear dynamics were already well worked out in the early 1980s. In the early 1990s, physicist Hanspeter Herzel and colleagues in Berlin recognized that these principles are applicable to the human voice, initially to baby cries. The physics and physiology underlying many of these nonlinear phenomena had remained mysterious up until then. This insight was later generalized to animal vocalizations. Nonlinear phenomena play a relatively peripheral role in most human vocal communication but are a common feature of many animal vocalizations. The broad recognition of the existence of nonlinear vocalizations, and the quantitative study of their production and perception, has now fuelled important and exciting advances in our understanding of animal communication. I concentrate on how the core concepts came into focus, and on their initial application to an ever-wider circle of call types and species, and end with a brief prospectus for the future.This article is part of the theme issue 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

Nonlinear vocal phenomena and speech intelligibility

At some point in our evolutionary history, humans lost vocal membranes and air sacs, representing an unexpected simplification of the vocal apparatus relative to other great apes. One hypothesis is that these simplifications represent anatomical adaptations for speech because a simpler larynx provides a suitably stable and tonal vocal source with fewer nonlinear vocal phenomena (NLP). The key assumption that NLP reduce speech intelligibility is indirectly supported by studies of dysphonia, but it has not been experimentally tested. Here, we manipulate NLP in vocal stimuli ranging from single vowels to sentences, showing that the vocal source needs to be stable, but not necessarily tonal, for speech to be readily understood. When the task is to discriminate synthesized monophthong and diphthong vowels, continuous NLP (subharmonics, amplitude modulation and even deterministic chaos) actually improve vowel perception in high-pitched voices, likely because the resulting dense spectrum reveals formant transitions. Rough-sounding voices also remain highly intelligible when continuous NLP are added to recorded words and sentences. In contrast, voicing interruptions and pitch jumps dramatically reduce speech intelligibility, likely by interfering with voicing contrasts and normal intonation. We argue that NLP were not eliminated from the human vocal repertoire as we evolved for speech, but only brought under better control.This article is part of the theme issue 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

Nonlinear acoustic phenomena affect the perception of pain in human baby cries

What makes the painful cries of human babies so difficult to ignore? Vocal traits known as 'nonlinear phenomena' are prime candidates. These acoustic irregularities are common in babies' cries and are typically associated with high levels of distress or pain. Despite the vital importance of cries for a baby's survival, how these nonlinear phenomena drive pain perception in adult listeners has not previously been systematically investigated. Here, by combining acoustic analyses of cries recorded in different contexts with playback experiments using natural and synthetic cries, we show that baby cries expressing acute pain are characterized by a pronounced presence of different nonlinear phenomena, and that these nonlinear phenomena drive pain evaluation by adult listeners. While adult listeners rated all cries presenting any of these nonlinear phenomena as expressing more pain, they were particularly sensitive to the presence of chaos. Our results thus show that nonlinear phenomena, especially chaos, encode pain information in baby cries and may be critically helpful for the development of vocal-based tools for monitoring babies' needs in the context of paediatric care.This article is part of the theme issue 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

'Monkey yodels'-frequency jumps in New World monkey vocalizations greatly surpass human vocal register transitions

We investigated the causal basis of abrupt frequency jumps in a unique database of New World monkey vocalizations. We used a combination of acoustic and electroglottographic recordings in vivo, excised larynx investigations of vocal fold dynamics, and computational modelling. We particularly attended to the contribution of the vocal membranes: thin upward extensions of the vocal folds found in most primates but absent in humans. In three of the six investigated species, we observed two distinct modes of vocal fold vibration. The first, involving vocal fold vibration alone, produced low-frequency oscillations, and is analogous to that underlying human phonation. The second, incorporating the vocal membranes, resulted in much higher-frequency oscillation. Abrupt fundamental frequency shifts were observed in all three datasets. While these data are reminiscent of the rapid transitions in frequency observed in certain human singing styles (e.g. yodelling), the frequency jumps are considerably larger in the nonhuman primates studied. Our data suggest that peripheral modifications of vocal anatomy provide an important source of variability and complexity in the vocal repertoires of nonhuman primates. We further propose that the call repertoire is crucially related to a species' ability to vocalize with different laryngeal mechanisms, analogous to human vocal registers. This article is part of the theme issue 'Nonlinear phenomena in vertebrate vocalizations: mechanisms and communicative functions'.

Evaluation of fetal exposure to environmental noise using a computer-generated model

Acoustic noise can have profound effects on wellbeing, impacting the health of the pregnant mother and the development of the fetus. Mounting evidence suggests neural memory traces are formed by auditory learning in utero. A better understanding of the fetal auditory environment is therefore critical to avoid exposure to damaging noise levels. Using anatomical data from MRI scans (N = 3), we used a computational model to quantify the acoustic field inside the pregnant maternal abdomen. We obtained acoustic transfer characteristics across the human audio range and pressure maps in transverse planes passing through the uterus at 5 kHz, 10 kHz and 20 kHz, showcasing multiple scattering and modal patterns. Our calculations suggest that for all datasets, the sound transmitted in utero is attenuated by as little as 6 dB below 1 kHz, confirming results from animal studies that the maternal abdomen and pelvis do not shelter the fetus from external noise.

Vowel signatures in emotional interjections and nonlinguistic vocalizations expressing pain, disgust, and joy across languagesa)

In this comparative cross-linguistic study we test whether expressive interjections (words like ouch or yay) share similar vowel signatures across the world's languages, and whether these can be traced back to nonlinguistic vocalizations (like screams and cries) expressing the same emotions of pain, disgust, and joy. We analyze vowels in interjections from dictionaries of 131 languages (over 600 tokens) and compare these with nearly 500 vowels based on formant frequency measures from voice recordings of volitional nonlinguistic vocalizations. We show that across the globe, pain interjections feature a-like vowels and wide falling diphthongs ("ai" as in Ayyy! "aw" as in Ouch!), whereas disgust and joy interjections do not show robust vowel regularities that extend geographically. In nonlinguistic vocalizations, all emotions yield distinct vowel signatures: pain prompts open vowels such as [a], disgust schwa-like central vowels, and joy front vowels such as [i]. Our results show that pain is the only affective experience tested with a clear, robust vowel signature that is preserved between nonlinguistic vocalizations and interjections across languages. These results offer empirical evidence for iconicity in some expressive interjections. We consider potential mechanisms and origins, from evolutionary pressures and sound symbolism to colexification, proposing testable hypotheses for future research.

Principal dimensions of voice production and their role in vocal expression

How we produce and perceive voice is constrained by laryngeal physiology and biomechanics. Such constraints may present themselves as principal dimensions in the voice outcome space that are shared among speakers. This study attempts to identify such principal dimensions in the voice outcome space and the underlying laryngeal control mechanisms in a three-dimensional computational model of voice production. A large-scale voice simulation was performed with parametric variations in vocal fold geometry and stiffness, glottal gap, vocal tract shape, and subglottal pressure. Principal component analysis was applied to data combining both the physiological control parameters and voice outcome measures. The results showed three dominant dimensions accounting for at least 50% of the total variance. The first two dimensions describe respiratory-laryngeal coordination in controlling the energy balance between low- and high-frequency harmonics in the produced voice, and the third dimension describes control of the fundamental frequency. The dominance of these three dimensions suggests that voice changes along these principal dimensions are likely to be more consistently produced and perceived by most speakers than other voice changes, and thus are more likely to have emerged during evolution and be used to convey important personal information, such as emotion and larynx size.

Spectro-temporal acoustical markers differentiate speech from song across cultures

Humans produce two forms of cognitively complex vocalizations: speech and song. It is debated whether these differ based primarily on culturally specific, learned features, or if acoustical features can reliably distinguish them. We study the spectro-temporal modulation patterns of vocalizations produced by 369 people living in 21 urban, rural, and small-scale societies across six continents. Specific ranges of spectral and temporal modulations, overlapping within categories and across societies, significantly differentiate speech from song. Machine-learning classification shows that this effect is cross-culturally robust, vocalizations being reliably classified solely from their spectro-temporal features across all 21 societies. Listeners unfamiliar with the cultures classify these vocalizations using similar spectro-temporal cues as the machine learning algorithm. Finally, spectro-temporal features are better able to discriminate song from speech than a broad range of other acoustical variables, suggesting that spectro-temporal modulation-a key feature of auditory neuronal tuning-accounts for a fundamental difference between these categories.

Globally, songs and instrumental melodies are slower and higher and use more stable pitches than speech: A Registered Report

Both music and language are found in all known human societies, yet no studies have compared similarities and differences between song, speech, and instrumental music on a global scale. In this Registered Report, we analyzed two global datasets: (i) 300 annotated audio recordings representing matched sets of traditional songs, recited lyrics, conversational speech, and instrumental melodies from our 75 coauthors speaking 55 languages; and (ii) 418 previously published adult-directed song and speech recordings from 209 individuals speaking 16 languages. Of our six preregistered predictions, five were strongly supported: Relative to speech, songs use (i) higher pitch, (ii) slower temporal rate, and (iii) more stable pitches, while both songs and speech used similar (iv) pitch interval size and (v) timbral brightness. Exploratory analyses suggest that features vary along a "musi-linguistic" continuum when including instrumental melodies and recited lyrics. Our study provides strong empirical evidence of cross-cultural regularities in music and speech.

Perceptual (but not acoustic) features predict singing voice preferences

Why do we prefer some singers to others? We investigated how much singing voice preferences can be traced back to objective features of the stimuli. To do so, we asked participants to rate short excerpts of singing performances in terms of how much they liked them as well as in terms of 10 perceptual attributes (e.g.: pitch accuracy, tempo, breathiness). We modeled liking ratings based on these perceptual ratings, as well as based on acoustic features and low-level features derived from Music Information Retrieval (MIR). Mean liking ratings for each stimulus were highly correlated between Experiments 1 (online, US-based participants) and 2 (in the lab, German participants), suggesting a role for attributes of the stimuli in grounding average preferences. We show that acoustic and MIR features barely explain any variance in liking ratings; in contrast, perceptual features of the voices achieved around 43% of prediction. Inter-rater agreement in liking and perceptual ratings was low, indicating substantial (and unsurprising) individual differences in participants' preferences and perception of the stimuli. Our results indicate that singing voice preferences are not grounded in acoustic attributes of the voices per se, but in how these features are perceptually interpreted by listeners.