Spectral envelope position and shape in sustained musical instrument sounds

Cortical substrates of perceptual confusion between pitch and timbre

Pitch and timbre are two fundamental perceptual attributes of sound that help us distinguish voices in speech and appreciate music. Brightness, one of the primary dimensions of timbre, is governed by different acoustic parameters compared to pitch, but the two can be confused perceptually when varied simultaneously. Here we combine human behavior and fMRI to provide evidence of a potential neural substrate to explain this important but poorly understood perceptual confusion. We identify orderly mappings of both pitch and brightness within auditory cortex and reveal two independent lines of evidence for cortical confusion between them. First, the preferred pitch of individual voxels decreases systematically as brightness increases, and vice versa, consistent with predictions based on perceptual confusion. Second, pitch and brightness mapping share a common high-low-high gradient across auditory cortex, implying a shared trajectory of cortical activation for changes in each dimension. The results provide a cortical substrate at both local and global scales for an established auditory perceptual phenomenon that is thought to reflect efficient coding of features ubiquitous in natural sound statistics.

An elastic piezoelectric nanomembrane with double noise reduction for high-quality bandpass acoustics

Polymer piezoelectrics with high electromechanical energy conversion (HEEC) are very promising for flexible acoustoelectric devices. However, reducing thickness and improving ordered polarization and ferroelectricity while maintaining high mechanical strength pose enormous fabrication challenges for polymer piezoelectric membranes-additionally, noise management in the acoustoelectric conversion remains an open issue. Here, we present a hydro-levitation superspreading approach for fabricating polymer nanomembranes with ordered crystalline phases and sub-nanostructures on the water surface. The elastic piezoelectric nanomembrane (EPN) is only 335 nanometers thick and consists of a conductance-stable piezoelectric layer sandwiched between two elastic damping layers. Such an all-in-one EPN can reduce background noise with low autocorrelation in the environment, suppress spurious noise caused by poor circuit contact, and achieve bandpass filtering of acoustic signals at human voice frequencies. This nanomembrane holds promise in repairing the auditory system of patients with tympanic membrane perforation and in a wide range of other acoustoelectric conversion fields.

Impact of interference on vocal and instrument recognition

Voices arguably occupy a superior role in auditory processing. Specifically, studies have reported that singing voices are processed faster and more accurately and possess greater salience in musical scenes compared to instrumental sounds. However, the underlying acoustic features of this superiority and the generality of these effects remain unclear. This study investigates the impact of frequency micro-modulations (FMM) and the influence of interfering sounds on sound recognition. Thirty young participants, half with musical training, engage in three sound recognition experiments featuring short vocal and instrumental sounds in a go/no-go task. Accuracy and reaction times are measured for sounds from recorded samples and excerpts of popular music. Each sound is presented in separate versions with and without FMM, in isolation or accompanied by a piano. Recognition varies across sound categories, but no general vocal superiority emerges and no effects of FMM. When presented together with interfering sounds, all sounds exhibit degradation in recognition. However, whereas /a/ sounds stand out by showing a distinct robustness to interference (i.e., less degradation of recognition), /u/ sounds lack this robustness. Acoustical analysis implies that recognition differences can be explained by spectral similarities. Together, these results challenge the notion of general vocal superiority in auditory perception.

Relative pitch representations and invariance to timbre

Information in speech and music is often conveyed through changes in fundamental frequency (f0), perceived by humans as "relative pitch". Relative pitch judgments are complicated by two facts. First, sounds can simultaneously vary in timbre due to filtering imposed by a vocal tract or instrument body. Second, relative pitch can be extracted in two ways: by measuring changes in constituent frequency components from one sound to another, or by estimating the f0 of each sound and comparing the estimates. We examined the effects of timbral differences on relative pitch judgments, and whether any invariance to timbre depends on whether judgments are based on constituent frequencies or their f0. Listeners performed up/down and interval discrimination tasks with pairs of spoken vowels, instrument notes, or synthetic tones, synthesized to be either harmonic or inharmonic. Inharmonic sounds lack a well-defined f0, such that relative pitch must be extracted from changes in individual frequencies. Pitch judgments were less accurate when vowels/instruments were different compared to when they were the same, and were biased by the associated timbre differences. However, this bias was similar for harmonic and inharmonic sounds, and was observed even in conditions where judgments of harmonic sounds were based on f0 representations. Relative pitch judgments are thus not invariant to timbre, even when timbral variation is naturalistic, and when such judgments are based on representations of f0.

Timbral cues for learning to generalize musical instrument identity across pitch register

Timbre provides an important cue to identify musical instruments. Many timbral attributes covary with other parameters like pitch. This study explores listeners' ability to construct categories of instrumental sound sources from sounds that vary in pitch. Nonmusicians identified 11 instruments from the woodwind, brass, percussion, and plucked and bowed string families. In experiment 1, they were trained to identify instruments playing a pitch of C4, and in experiments 2 and 3, they were trained with a five-tone sequence (F#3-F#4), exposing them to the way timbre varies with pitch. Participants were required to reach a threshold of 75% correct identification in training. In the testing phase, successful listeners heard single tones (experiments 1 and 2) or three-tone sequences from (A3-D#4) (experiment 3) across each instrument's full pitch range to test their ability to generalize identification from the learned sound(s). Identification generalization over pitch varies a great deal across instruments. No significant differences were found between single-pitch and multi-pitch training or testing conditions. Identification rates can be predicted moderately well by spectrograms or modulation spectra. These results suggest that listeners use the most relevant acoustical invariance to identify musical instrument sounds, also using previous experience with the tested instruments.

A unitary model of auditory frequency change perception

Changes in the frequency content of sounds over time are arguably the most basic form of information about the behavior of sound-emitting objects. In perceptual studies, such changes have mostly been investigated separately, as aspects of either pitch or timbre. Here, we propose a unitary account of "up" and "down" subjective judgments of frequency change, based on a model combining auditory correlates of acoustic cues in a sound-specific and listener-specific manner. To do so, we introduce a generalized version of so-called Shepard tones, allowing symmetric manipulations of spectral information on a fine scale, usually associated to pitch (spectral fine structure, SFS), and on a coarse scale, usually associated timbre (spectral envelope, SE). In a series of behavioral experiments, listeners reported "up" or "down" shifts across pairs of generalized Shepard tones that differed in SFS, in SE, or in both. We observed the classic properties of Shepard tones for either SFS or SE shifts: subjective judgements followed the smallest log-frequency change direction, with cases of ambiguity and circularity. Interestingly, when both SFS and SE changes were applied concurrently (synergistically or antagonistically), we observed a trade-off between cues. Listeners were encouraged to report when they perceived "both" directions of change concurrently, but this rarely happened, suggesting a unitary percept. A computational model could accurately fit the behavioral data by combining different cues reflecting frequency changes after auditory filtering. The model revealed that cue weighting depended on the nature of the sound. When presented with harmonic sounds, listeners put more weight on SFS-related cues, whereas inharmonic sounds led to more weight on SE-related cues. Moreover, these stimulus-based factors were modulated by inter-individual differences, revealing variability across listeners in the detailed recipe for "up" and "down" judgments. We argue that frequency changes are tracked perceptually via the adaptive combination of a diverse set of cues, in a manner that is in fact similar to the derivation of other basic auditory dimensions such as spatial location.

Adaptive auditory brightness perception

Perception adapts to the properties of prior stimulation, as illustrated by phenomena such as visual color constancy or speech context effects. In the auditory domain, only little is known about adaptive processes when it comes to the attribute of auditory brightness. Here, we report an experiment that tests whether listeners adapt to spectral colorations imposed on naturalistic music and speech excerpts. Our results indicate consistent contrastive adaptation of auditory brightness judgments on a trial-by-trial basis. The pattern of results suggests that these effects tend to grow with an increase in the duration of the adaptor context but level off after around 8 trials of 2 s duration. A simple model of the response criterion yields a correlation of r = .97 with the measured data and corroborates the notion that brightness perception adapts on timescales that fall in the range of auditory short-term memory. Effects turn out to be similar for spectral filtering based on linear spectral filter slopes and filtering based on a measured transfer function from a commercially available hearing device. Overall, our findings demonstrate the adaptivity of auditory brightness perception under realistic acoustical conditions.

Status and future of modeling of musical instruments: Introduction to the JASA special issue

Over the last decades, physics-based modeling of musical instruments has seen increased attention. In 2020 and 2021, the Journal of the Acoustical Society of America accepted submissions for a special issue on the modeling of musical instruments. This article is intended as an introduction to the special issue. Our purpose is to discuss the role that modeling plays in the study of musical instruments, the kinds of things one hopes to learn from modeling studies, and how that work informs traditional experimental and theoretical studies of specific instruments. We also describe recent trends in modeling and make some observations about where we think the field is heading. Overall, our goal is to place the articles in the special issue into a context that helps the reader to better understand and appreciate the field.