Speech Categorization Reveals the Role of Early-Stage Temporal-Coherence Processing in Auditory Scene Analysis

Impact of reduced spectral resolution on temporal-coherence-based source segregation

Hearing-impaired listeners struggle to understand speech in noise, even when using cochlear implants (CIs) or hearing aids. Successful listening in noisy environments depends on the brain's ability to organize a mixture of sound sources into distinct perceptual streams (i.e., source segregation). In normal-hearing listeners, temporal coherence of sound fluctuations across frequency channels supports this process by promoting grouping of elements belonging to a single acoustic source. We hypothesized that reduced spectral resolution-a hallmark of both electric/CI (from current spread) and acoustic (from broadened tuning) hearing with sensorineural hearing loss-degrades segregation based on temporal coherence. This is because reduced frequency resolution decreases the likelihood that a single sound source dominates the activity driving any specific channel; concomitantly, it increases the correlation in activity across channels. Consistent with our hypothesis, our physiologically inspired computational model of temporal-coherence-based segregation predicts that CI current spread reduces comodulation masking release (CMR; a correlate of temporal-coherence processing) and speech intelligibility in noise. These predictions are consistent with our online behavioral data with simulated CI listening. Our model also predicts smaller CMR with increasing levels of outer-hair-cell damage. These results suggest that reduced spectral resolution relative to normal hearing impairs temporal-coherence-based segregation and speech-in-noise outcomes.

Binaural fusion: Complexities in definition and measurement

Despite the growing interest in studying binaural fusion, there is little consensus over its definition or how it is best measured. This review seeks to describe the complexities of binaural fusion, highlight measurement challenges, provide guidelines for rigorous perceptual measurements, and provide a working definition that encompasses this information. First, it is argued that binaural fusion may be multidimensional and might occur in one domain but not others, such as fusion in the spatial but not the spectral domain or vice versa. Second, binaural fusion may occur on a continuous scale rather than on a binary one. Third, binaural fusion responses are highly idiosyncratic, which could be a result of methodology, such as the specific experimental instructions, suggesting a need to explicitly report the instructions given. Fourth, it is possible that direct ("Did you hear one sound or two?") and indirect ("Where did the sound come from?" or "What was the pitch of the sound?") measurements of fusion will produce different results. In conclusion, explicit consideration of these attributes and reporting of methodology are needed for rigorous interpretation and comparison across studies and listener populations.

Hierarchical differences in the encoding of amplitude modulation in the subcortical auditory system of awake nonhuman primates

Sinusoidal amplitude modulation (SAM) is a key feature of complex sounds. Although psychophysical studies have characterized SAM perception, and neurophysiological studies in anesthetized animals report a transformation from the cochlear nucleus' (CN; brainstem) temporal code to the inferior colliculus' (IC; midbrain's) rate code, none have used awake animals or nonhuman primates to compare CN and IC's coding strategies to modulation-frequency perception. To address this, we recorded single-unit responses and compared derived neurometric measures in the CN and IC to psychometric measures of modulation frequency (MF) discrimination in macaques. IC and CN neurons often exhibited tuned responses to SAM in rate and spike-timing measures of modulation coding. Neurometric thresholds spanned a large range (2-200 Hz ΔMF). The lowest 40% of IC thresholds were less than or equal to psychometric thresholds, regardless of which code was used, whereas CN thresholds were greater than psychometric thresholds. Discrimination at 10-20 Hz could be explained by indiscriminately pooling 30 units in either structure, whereas discrimination at higher MFs was best explained by more selective pooling. This suggests that pooled CN activity was sufficient for AM discrimination. Psychometric and neurometric thresholds decreased as stimulus duration increased, but IC and CN thresholds were higher and more variable than behavior at short durations. This slower subcortical temporal integration compared with behavior was consistent with a drift diffusion model that reproduced individual differences in performance and can constrain future neurophysiological studies of temporal integration. These measures provide an account of AM perception at the neurophysiological, computational, and behavioral levels.NEW & NOTEWORTHY In everyday environments, the brain is tasked with extracting information from sound envelopes, which involves both sensory encoding and perceptual decision-making. Different neural codes for envelope representation have been characterized in midbrain and cortex, but studies of brainstem nuclei such as the cochlear nucleus (CN) have usually been conducted under anesthesia in nonprimate species. Here, we found that subcortical activity in awake monkeys and a biologically plausible perceptual decision-making model accounted for sound envelope discrimination behavior.

Peripheral Neural Synchrony in Postlingually Deafened Adult Cochlear Implant Users

OBJECTIVES

This paper reports a noninvasive method for quantifying neural synchrony in the cochlear nerve (i.e., peripheral neural synchrony) in cochlear implant (CI) users, which allows for evaluating this physiological phenomenon in human CI users for the first time in the literature. In addition, this study assessed how peripheral neural synchrony was correlated with temporal resolution acuity and speech perception outcomes measured in quiet and in noise in postlingually deafened adult CI users. It tested the hypothesis that peripheral neural synchrony was an important factor for temporal resolution acuity and speech perception outcomes in noise in postlingually deafened adult CI users.

DESIGN

Study participants included 24 postlingually deafened adult CI users with a Cochlear™ Nucleus® device. Three study participants were implanted bilaterally, and each ear was tested separately. For each of the 27 implanted ears tested in this study, 400 sweeps of the electrically evoked compound action potential (eCAP) were measured at four electrode locations across the electrode array. Peripheral neural synchrony was quantified at each electrode location using the phase-locking value (PLV), which is a measure of trial-by-trial phase coherence among eCAP sweeps/trials. Temporal resolution acuity was evaluated by measuring the within-channel gap detection threshold (GDT) using a three-alternative, forced-choice procedure in a subgroup of 20 participants (23 implanted ears). For each ear tested in these participants, GDTs were measured at two electrode locations with a large difference in PLVs. For 26 implanted ears tested in 23 participants, speech perception performance was evaluated using consonant-nucleus-consonant (CNC) word lists presented in quiet and in noise at signal to noise ratios (SNRs) of +10 and +5 dB. Linear Mixed effect Models were used to evaluate the effect of electrode location on the PLV and the effect of the PLV on GDT after controlling for the stimulation level effects. Pearson product-moment correlation tests were used to assess the correlations between PLVs, CNC word scores measured in different conditions, and the degree of noise effect on CNC word scores.

RESULTS

There was a significant effect of electrode location on the PLV after controlling for the effect of stimulation level. There was a significant effect of the PLV on GDT after controlling for the effects of stimulation level, where higher PLVs (greater synchrony) led to lower GDTs (better temporal resolution acuity). PLVs were not significantly correlated with CNC word scores measured in any listening condition or the effect of competing background noise presented at an SNR of +10 dB on CNC word scores. In contrast, there was a significant negative correlation between the PLV and the degree of noise effect on CNC word scores for a competing background noise presented at an SNR of +5 dB, where higher PLVs (greater synchrony) correlated with smaller noise effects on CNC word scores.

CONCLUSIONS

This newly developed method can be used to assess peripheral neural synchrony in CI users, a physiological phenomenon that has not been systematically evaluated in electrical hearing. Poorer peripheral neural synchrony leads to lower temporal resolution acuity and is correlated with a larger detrimental effect of competing background noise presented at an SNR of 5 dB on speech perception performance in postlingually deafened adult CI users.

Impact of Reduced Spectral Resolution on Temporal-Coherence-Based Source Segregation

Hearing-impaired listeners struggle to understand speech in noise, even when using cochlear implants (CIs) or hearing aids. Successful listening in noisy environments depends on the brain's ability to organize a mixture of sound sources into distinct perceptual streams (i.e., source segregation). In normal-hearing listeners, temporal coherence of sound fluctuations across frequency channels supports this process by promoting grouping of elements belonging to a single acoustic source. We hypothesized that reduced spectral resolution-a hallmark of both electric/CI (from current spread) and acoustic (from broadened tuning) hearing with sensorineural hearing loss-degrades segregation based on temporal coherence. This is because reduced frequency resolution decreases the likelihood that a single sound source dominates the activity driving any specific channel; concomitantly, it increases the correlation in activity across channels. Consistent with our hypothesis, predictions from a physiologically plausible model of temporal-coherence-based segregation suggest that CI current spread reduces comodulation masking release (CMR; a correlate of temporal-coherence processing) and speech intelligibility in noise. These predictions are consistent with our behavioral data with simulated CI listening. Our model also predicts smaller CMR with increasing levels of outer-hair-cell damage. These results suggest that reduced spectral resolution relative to normal hearing impairs temporal-coherence-based segregation and speech-in-noise outcomes.

Interrupted mosaic speech revisited: Gain and loss in intelligibility by stretchinga)

Our previous investigation on the effect of stretching spectrotemporally degraded and temporally interrupted speech stimuli showed remarkable intelligibility gains [Udea, Takeichi, and Wakamiya (2022). J. Acoust. Soc. Am. 152(2), 970-980]. In this previous study, however, gap durations and temporal resolution were confounded. In the current investigation, we therefore observed the intelligibility of so-called mosaic speech while dissociating the effects of interruption and temporal resolution. The intelligibility of mosaic speech (20 frequency bands and 20 ms segment duration) declined from 95% to 78% and 33% by interrupting it with 20 and 80 ms gaps. Intelligibility improved, however, to 92% and 54% (14% and 21% gains for 20 and 80 ms gaps, respectively) by stretching mosaic segments to fill silent gaps (n = 21). By contrast, the intelligibility was impoverished to a minimum of 9% (7% loss) when stretching stimuli interrupted with 160 ms gaps. Explanations based on auditory grouping, modulation unmasking, or phonemic restoration may account for the intelligibility improvement by stretching, but not for the loss. The probability summation model accounted for "U"-shaped intelligibility curves and the gain and loss of intelligibility, suggesting that perceptual unit length and speech rate may affect the intelligibility of spectrotemporally degraded speech stimuli.

Web-based psychoacoustics: Hearing screening, infrastructure, and validation

Anonymous web-based experiments are increasingly used in many domains of behavioral research. However, online studies of auditory perception, especially of psychoacoustic phenomena pertaining to low-level sensory processing, are challenging because of limited available control of the acoustics, and the inability to perform audiometry to confirm normal-hearing status of participants. Here, we outline our approach to mitigate these challenges and validate our procedures by comparing web-based measurements to lab-based data on a range of classic psychoacoustic tasks. Individual tasks were created using jsPsych, an open-source JavaScript front-end library. Dynamic sequences of psychoacoustic tasks were implemented using Django, an open-source library for web applications, and combined with consent pages, questionnaires, and debriefing pages. Subjects were recruited via Prolific, a subject recruitment platform for web-based studies. Guided by a meta-analysis of lab-based data, we developed and validated a screening procedure to select participants for (putative) normal-hearing status based on their responses in a suprathreshold task and a survey. Headphone use was standardized by supplementing procedures from prior literature with a binaural hearing task. Individuals meeting all criteria were re-invited to complete a range of classic psychoacoustic tasks. For the re-invited participants, absolute thresholds were in excellent agreement with lab-based data for fundamental frequency discrimination, gap detection, and sensitivity to interaural time delay and level difference. Furthermore, word identification scores, consonant confusion patterns, and co-modulation masking release effect also matched lab-based studies. Our results suggest that web-based psychoacoustics is a viable complement to lab-based research. Source code for our infrastructure is provided.

Peripheral neural synchrony in post-lingually deafened adult cochlear implant users

Objective

This paper reports a noninvasive method for quantifying neural synchrony in the cochlear nerve (i.e., peripheral neural synchrony) in cochlear implant (CI) users, which allows for evaluating this physiological phenomenon in human CI users for the first time in the literature. In addition, this study assessed how peripheral neural synchrony was correlated with temporal resolution acuity and speech perception outcomes measured in quiet and in noise in post-lingually deafened adult CI users. It tested the hypothesis that peripheral neural synchrony was an important factor for temporal resolution acuity and speech perception outcomes in noise in post-lingually deafened adult CI users.

Design

Study participants included 24 post-lingually deafened adult CI users with a Cochlear™ Nucleus® device. Three study participants were implanted bilaterally, and each ear was tested separately. For each of the 27 implanted ears tested in this study, 400 sweeps of the electrically evoked compound action potential (eCAP) were measured at four electrode locations across the electrode array. Peripheral neural synchrony was quantified at each electrode location using the phase locking value (PLV), which is a measure of trial-by-trial phase coherence among eCAP sweeps/trials. Temporal resolution acuity was evaluated by measuring the within-channel gap detection threshold (GDT) using a three-alternative, forced-choice procedure in a subgroup of 20 participants (23 implanted ears). For each ear tested in these participants, GDTs were measured at two electrode locations with a large difference in PLVs. For 26 implanted ears tested in 23 participants, speech perception performance was evaluated using Consonant-Nucleus-Consonant (CNC) word lists presented in quiet and in noise at signal-to-noise ratios (SNRs) of +10 and +5 dB. Linear Mixed effect Models were used to evaluate the effect of electrode location on the PLV and the effect of the PLV on GDT after controlling for the stimulation level effects. Pearson product-moment correlation tests were used to assess the correlations between PLVs, CNC word scores measured in different conditions, and the degree of noise effect on CNC word scores.

Results

There was a significant effect of electrode location on the PLV after controlling for the effect of stimulation level. There was a significant effect of the PLV on GDT after controlling for the effects of stimulation level, where higher PLVs (greater synchrony) led to lower GDTs (better temporal resolution acuity). PLVs were not significantly correlated with CNC word scores measured in any listening condition or the effect of competing background noise presented at a SNR of +10 dB on CNC word scores. In contrast, there was a significant negative correlation between the PLV and the degree of noise effect on CNC word scores for a competing background noise presented at a SNR of +5 dB, where higher PLVs (greater synchrony) correlated with smaller noise effects on CNC word scores.

Conclusions

This newly developed method can be used to assess peripheral neural synchrony in CI users, a physiological phenomenon that has not been systematically evaluated in electrical hearing. Poorer peripheral neural synchrony leads to lower temporal resolution acuity and is correlated with a larger detrimental effect of competing background noise presented at a SNR of 5 dB on speech perception performance in post-lingually deafened adult CI users.

Effect of Reverberation on Neural Responses to Natural Speech in Rabbit Auditory Midbrain: No Evidence for a Neural Dereverberation Mechanism

Reverberation is ubiquitous in everyday acoustic environments. It degrades both binaural cues and the envelope modulations of sounds and thus can impair speech perception. Still, both humans and animals can accurately perceive reverberant stimuli in most everyday settings. Previous neurophysiological and perceptual studies have suggested the existence of neural mechanisms that partially compensate for the effects of reverberation. However, these studies were limited by their use of either highly simplified stimuli or rudimentary reverberation simulations. To further characterize how reverberant stimuli are processed by the auditory system, we recorded single-unit (SU) and multiunit (MU) activity from the inferior colliculus (IC) of unanesthetized rabbits in response to natural speech utterances presented with no reverberation ("dry") and in various degrees of simulated reverberation (direct-to-reverberant energy ratios (DRRs) ranging from 9.4 to -8.2 dB). Linear stimulus reconstruction techniques (Mesgarani et al., 2009) were used to quantify the amount of speech information available in the responses of neural ensembles. We found that high-quality spectrogram reconstructions could be obtained for dry speech and in moderate reverberation from ensembles of 25 units. However, spectrogram reconstruction quality deteriorated in severe reverberation for both MUs and SUs such that the neural degradation paralleled the degradation in the stimulus spectrogram. Furthermore, spectrograms reconstructed from responses to reverberant stimuli resembled spectrograms of reverberant speech better than spectrograms of dry speech. Overall, the results provide no evidence for a dereverberation mechanism in neural responses from the rabbit IC when studied with linear reconstruction techniques.

FORUM: Remote testing for psychological and physiological acoustics

Acoustics research involving human participants typically takes place in specialized laboratory settings. Listening studies, for example, may present controlled sounds using calibrated transducers in sound-attenuating or anechoic chambers. In contrast, remote testing takes place outside of the laboratory in everyday settings (e.g., participants' homes). Remote testing could provide greater access to participants, larger sample sizes, and opportunities to characterize performance in typical listening environments at the cost of reduced control of environmental conditions, less precise calibration, and inconsistency in attentional state and/or response behaviors from relatively smaller sample sizes and unintuitive experimental tasks. The Acoustical Society of America Technical Committee on Psychological and Physiological Acoustics launched the Task Force on Remote Testing (https://tcppasa.org/remotetesting/) in May 2020 with goals of surveying approaches and platforms available to support remote testing and identifying challenges and considerations for prospective investigators. The results of this task force survey were made available online in the form of a set of Wiki pages and summarized in this report. This report outlines the state-of-the-art of remote testing in auditory-related research as of August 2021, which is based on the Wiki and a literature search of papers published in this area since 2020, and provides three case studies to demonstrate feasibility during practice.