A two‐path model of auditory modulation detection using temporal fine structure and envelope cues

Binaural detection thresholds and audio quality of speech and music signals in complex acoustic environments

Every-day acoustical environments are often complex, typically comprising one attended target sound in the presence of interfering sounds (e.g., disturbing conversations) and reverberation. Here we assessed binaural detection thresholds and (supra-threshold) binaural audio quality ratings of four distortions types: spectral ripples, non-linear saturation, intensity and spatial modifications applied to speech, guitar, and noise targets in such complex acoustic environments (CAEs). The target and (up to) two masker sounds were either co-located as if contained in a common audio stream, or were spatially separated as if originating from different sound sources. The amount of reverberation was systematically varied. Masker and reverberation had a significant effect on the distortion-detection thresholds of speech signals. Quality ratings were affected by reverberation, whereas the effect of maskers depended on the distortion. The results suggest that detection thresholds and quality ratings for distorted speech in anechoic conditions are also valid for rooms with mild reverberation, but not for moderate reverberation. Furthermore, for spectral ripples, a significant relationship between the listeners' individual detection thresholds and quality ratings was found. The current results provide baseline data for detection thresholds and audio quality ratings of different distortions of a target sound in CAEs, supporting the future development of binaural auditory models.

Speech intelligibility prediction based on modulation frequency-selective processing

Speech intelligibility models can provide insights regarding the auditory processes involved in human speech perception and communication. One successful approach to modelling speech intelligibility has been based on the analysis of the amplitude modulations present in speech as well as competing interferers. This review covers speech intelligibility models that include a modulation-frequency selective processing stage i.e., a modulation filterbank, as part of their front end. The speech-based envelope power spectrum model [sEPSM, Jørgensen and Dau (2011). J. Acoust. Soc. Am. 130(3), 1475-1487], several variants of the sEPSM including modifications with respect to temporal resolution, spectro-temporal processing and binaural processing, as well as the speech-based computational auditory signal processing and perception model [sCASP; Relaño-Iborra et al. J. Acoust. Soc. Am. 146(5), 3306-3317], which is based on an established auditory signal detection and masking model, are discussed. The key processing stages of these models for the prediction of speech intelligibility across a variety of acoustic conditions are addressed in relation to competing modeling approaches. The strengths and weaknesses of the modulation-based analysis are outlined and perspectives presented, particularly in connection with the challenge of predicting the consequences of individual hearing loss on speech intelligibility.

Double-pass consistency for amplitude- and frequency-modulation detection in normal-hearing listeners

Amplitude modulation (AM) and frequency modulation (FM) provide crucial auditory information. If FM is encoded as AM, it should be possible to give a unified account of AM and FM perception both in terms of response consistency and performance. These two aspects of behavior were estimated for normal-hearing participants using a constant-stimuli, forced-choice detection task repeated twice with the same stimuli (double pass). Sinusoidal AM or FM with rates of 2 or 20 Hz were applied to a 500-Hz pure-tone carrier and presented at detection threshold. All stimuli were masked by a modulation noise. Percent agreement of responses across passes and percent-correct detection for the two passes were used to estimate consistency and performance, respectively. These data were simulated using a model implementing peripheral processes, a central modulation filterbank, an additive internal noise, and a template-matching device. Different levels of internal noise were required to reproduce AM and FM data, but a single level could account for the 2- and 20-Hz AM data. As for FM, two levels of internal noise were needed to account for detection at slow and fast rates. Finally, the level of internal noise yielding best predictions increased with the level of the modulation-noise masker. Overall, these results suggest that different sources of internal variability are involved for AM and FM detection at low audio frequencies.

Temporal fine structure: associations with cognition and speech-in-noise recognition in adults with normal hearing or hearing impairment

OBJECTIVES

To investigate associations between sensitivity to temporal fine structure (TFS) and performance in cognitive and speech-in-noise recognition tests.

DESIGN

A binaural test of TFS sensitivity (the TFS-LF) was used. Measures of cognition included the reading span, Raven's, and text-reception threshold tests. Measures of speech recognition included the Hearing in noise (HINT) and the Hagerman matrix sentence tests in three signal processing conditions.

STUDY SAMPLE

Analyses are based on the performance of 324/317 adults with and without hearing impairment.

RESULTS

Sensitivity to TFS was significantly correlated with both the reading span test and the recognition of speech-in-noise processed using noise reduction, the latter only when limited to participants with hearing impairment. Neither association was significant when the effects of age were partialled out.

CONCLUSIONS

The findings are consistent with previous research in finding no evidence of a link between sensitivity to TFS and working memory once the effects of age had been partialled out. The results provide some evidence of an influence of signal processing strategy on the association between TFS sensitivity and speech-in-noise recognition. However, further research is necessary to assess the generalisability of the findings before any claims can be made regarding any clinical implications of these findings.

The role of cochlear place coding in the perception of frequency modulation

Natural sounds convey information via frequency and amplitude modulations (FM and AM). Humans are acutely sensitive to the slow rates of FM that are crucial for speech and music. This sensitivity has long been thought to rely on precise stimulus-driven auditory-nerve spike timing (time code), whereas a coarser code, based on variations in the cochlear place of stimulation (place code), represents faster FM rates. We tested this theory in listeners with normal and impaired hearing, spanning a wide range of place-coding fidelity. Contrary to predictions, sensitivity to both slow and fast FM correlated with place-coding fidelity. We also used incoherent AM on two carriers to simulate place coding of FM and observed poorer sensitivity at high carrier frequencies and fast rates, two properties of FM detection previously ascribed to the limits of time coding. The results suggest a unitary place-based neural code for FM across all rates and carrier frequencies.

Effects of aging on event-related potentials to single-cycle binaural beats and diotic amplitude modulation of a tone

Aim of the study is to determine whether the auditory processing of temporal fine structure (TFS) is affected with normal aging, even in the presence of normal audiometric hearing and fine cognitive state; and, if it is, to see whether a comparable effect is also observed in the processing of a diotic change in sound envelope. The event-related potentials (ERPs) to binaural beats (BBs), which are the responses of the binaural mechanisms processing TFS of a sound, and the ERPs to diotic amplitude modulation (AM) stimuli, which are the responses of the monaural mechanisms processing the changes in its envelope, were recorded from thirteen young university students and ten senior but active university professors, all with normal hearing in low frequencies. To obtain directly the specific BB responses without confounding monaural frequency change-evoked responses, we used single-cycle BB stimuli with temporary sub-threshold frequency shifts. BBs of a 250-Hz tone and diotic AM of the same tone with similar perceptual salience were presented with 2-second stimulus onset asynchrony. The N1 components of the ERPs to both stimuli displayed notable age-dependent changes in their scalp topography and significant amplitude reduction and latency prolongation in the elderly. These amplitude and latency changes were at similar rates for the two stimulus types, implying that the auditory TFS and envelope processing mechanisms are proportionally affected by physiological aging. These results may serve as control data in future studies investigating the effect of aging-associated cognitive pathologies on auditory TFS processing.

Identifying cues for tone-in-noise detection using decision variable correlation in the budgerigar (Melopsittacus undulatus)

Previous studies evaluated cues for masked tone detection using reproducible noise waveforms. Human results founded on this approach suggest that tone detection is based on combined energy and envelope (ENV) cues, but detection cues in nonhuman species are less clear. Decision variable correlation (DVC) was used to evaluate tone-in-noise detection cues in the budgerigar, an avian species with human-like behavioral sensitivity to many complex sounds. DVC quantifies a model's ability to predict trial-by-trial variance in behavioral responses. Budgerigars were behaviorally conditioned to detect 500-Hz tones in wideband (WB; 100-3000 Hz) and narrowband (NB; 452-552 Hz) noise. Behavioral responses were obtained using a single-interval, two-alternative discrimination task and two-down, one-up adaptive tracking procedures. Tone-detection thresholds in WB noise were higher than human thresholds, putatively due to broader peripheral frequency tuning, whereas NB thresholds were within ∼1 dB of human results. Budgerigar average hit and false-alarm rates across noise waveforms were consistent, highly correlated across subjects, and correlated to human results. Trial-by-trial behavioral results in NB noise were best explained by a model combining energy and ENV cues. In contrast, WB results were better predicted by ENV-based or multiple-channel energy detector models. These results suggest that budgerigars and humans use similar cues for tone-in-noise detection.

Retracted Article: Temporal fine structure: relations to cognition and aided speech recognition

Retraction statementWe, the Editor and Publisher of the International Journal of Audiology, have retracted the following article.Rachel J. Ellis, and Jerker Rönnberg. 2019. "Temporal fine structure: relations to cognition and aided speech recognition." International Journal of Audiology. doi:10.1080/14992027.2019.1672899.The authors of the above-mentioned article published in the International Journal of Audiology have identified errors in the reported analysis (relating to the inclusion of data that should have been excluded) which impact the validity of the findings. The authors have, therefore, requested that the article be retracted.We have been informed in our decision-making by our policy on publishing ethics and integrity and the COPE guidelines on retractions.The retracted article will remain online to maintain the scholarly record, but it will be digitally watermarked on each page as "Retracted".

Speech fine structure contains critical temporal cues to support speech segmentation

Segmenting the continuous speech stream into units for further perceptual and linguistic analyses is fundamental to speech recognition. The speech amplitude envelope (SE) has long been considered a fundamental temporal cue for segmenting speech. Does the temporal fine structure (TFS), a significant part of speech signals often considered to contain primarily spectral information, contribute to speech segmentation? Using magnetoencephalography, we show that the TFS entrains cortical responses between 3 and 6 Hz and demonstrate, using mutual information analysis, that (i) the temporal information in the TFS can be reconstructed from a measure of frame-to-frame spectral change and correlates with the SE and (ii) that spectral resolution is key to the extraction of such temporal information. Furthermore, we show behavioural evidence that, when the SE is temporally distorted, the TFS provides cues for speech segmentation and aids speech recognition significantly. Our findings show that it is insufficient to investigate solely the SE to understand temporal speech segmentation, as the SE and the TFS derived from a band-filtering method convey comparable, if not inseparable, temporal information. We argue for a more synthetic view of speech segmentation - the auditory system groups speech signals coherently in both temporal and spectral domains.

A Comparison of Behavioral Methods for Indexing the Auditory Processing of Temporal Fine Structure Cues

Purpose Growing evidence supports the inclusion of perceptual tests that quantify the processing of temporal fine structure (TFS) in clinical hearing assessment. Many tasks have been used to evaluate TFS in the laboratory that vary greatly in the stimuli used and whether the judgments require monaural or binaural comparisons of TFS. The purpose of this study was to compare laboratory measures of TFS for inclusion in a battery of suprathreshold auditory tests. A subset of available TFS tasks were selected on the basis of potential clinical utility and were evaluated using metrics that focus on characteristics important for clinical use. Method TFS measures were implemented in replication of studies that demonstrated clinical utility. Monaural, diotic, and dichotic measures were evaluated in 11 young listeners with normal hearing. Measures included frequency modulation (FM) tasks, harmonic frequency shift detection, interaural phase difference (TFS-low frequency), interaural time difference (ITD), monaural gap duration discrimination, and tone detection in noise with and without a difference in interaural phase (N0S0, N0Sπ). Data were compared with published results and evaluated with metrics of consistency and efficiency. Results Thresholds obtained were consistent with published data. There was no evidence of predictive relationships among the measures consistent with a homogenous group. The most stable tasks across repeated testing were TFS-low frequency, diotic and dichotic FM, and N0Sπ. Monaural and diotic FM had the lowest normalized variance and were the most efficient accounting for differences in total test duration, followed by ITD. Conclusions Despite a long stimulus duration, FM tasks dominated comparisons of consistency and efficiency. Small differences separated the dichotic tasks FM, ITD, and N0Sπ. Future comparisons following procedural optimization of the tasks will evaluate clinical efficiency in populations with impairment.

Event-related potentials to single-cycle binaural beats and diotic amplitude modulation of a tone

When two tones with slightly different frequencies are dichotically presented, binaural beats (BBs) are experienced. BBs resulting from the cycling change in interaural phase difference elicit electroencephalographic responses. Because they repeat at short periods, allowing poor recovery of the cortical responses, these steady-state responses have small amplitudes, and their various wave components intermingle and might mask each other. Using single-cycle BBs separated by relatively long inter-onset intervals would be a solution, but introducing a transient interaural frequency shift requires response subtraction which may not be acceptable for non-additive brain responses. The proposed stimulation method employs transient and monaurally subthreshold frequency shifts in opposite directions in the two ears to produce single-cycle BBs of a 250 Hz tone. These shifts are perceived as distinct BBs when presented dichotically, but remain subthreshold when presented monotically. Therefore, no frequency-shift response is elicited, and the specific BB response is obtained with no need for waveform subtraction. We recorded from 19 normal hearing participants the event-related potentials (ERPs) to single-cycle BBs and also to temporary diotic amplitude modulation (AM) with matched perceptual salience. The ERPs to single-cycle BBs presented at 2 s inter-onset intervals had N1-P2 responses with up to seven times larger amplitudes than the conventional steady-state BB responses in the literature. Significant differences were found between the scalp potential distributions of the N1 responses to BB and AM stimuli, suggesting that the cortical sites, where envelope-based level processing and temporal fine structure-based spatial processing of the stimulus take place, are not totally overlapped.

Accounting for masking of frequency modulation by amplitude modulation with the modulation filter-bank concept

Frequency modulation (FM) is assumed to be detected through amplitude modulation (AM) created by cochlear filtering for modulation rates above 10 Hz and carrier frequencies (f_c) above 4 kHz. If this is the case, a model of modulation perception based on the concept of AM filters should predict masking effects between AM and FM. To test this, masking effects of sinusoidal AM on sinusoidal FM detection thresholds were assessed on normal-hearing listeners as a function of FM rate, f_c, duration, AM rate, AM depth, and phase difference between FM and AM. The data were compared to predictions of a computational model implementing an AM filter-bank. Consistent with model predictions, AM masked FM with some AM-masking-AM features (broad tuning and effect of AM-masker depth). Similar masking was predicted and observed at f_c = 0.5 and 5 kHz for a 2 Hz AM masker, inconsistent with the notion that additional (e.g., temporal fine-structure) cues drive slow-rate FM detection at low f_c. However, masking was lower than predicted and, unlike model predictions, did not show beating or phase effects. Broadly, the modulation filter-bank concept successfully explained some AM-masking-FM effects, but could not give a complete account of both AM and FM detection.

Effects of Age and Hearing Loss on the Discrimination of Amplitude and Frequency Modulation for 2- and 10-Hz Rates

Detection of frequency modulation (FM) with rate = 10 Hz may depend on conversion of FM to amplitude modulation (AM) in the cochlea, while detection of 2-Hz FM may depend on the use of temporal fine structure (TFS) information. TFS processing may worsen with greater age and hearing loss while AM processing probably does not. A two-stage experiment was conducted to test these ideas while controlling for the effects of detection efficiency. Stage 1 measured psychometric functions for the detection of AM alone and FM alone imposed on a 1-kHz carrier, using 2- and 10-Hz rates. Stage 2 assessed the discrimination of AM from FM at the same modulation rate when the detectability of the AM alone and FM alone was equated. Discrimination was better for the 2-Hz than for the 10-Hz rate for all young normal-hearing subjects and for some older subjects with normal hearing at 1 kHz. Other older subjects with normal hearing showed no clear difference in AM-FM discrimination for the 2- and 10-Hz rates, as was the case for most older hearing-impaired subjects. The results suggest that the ability to use TFS cues is reduced for some older people and most hearing-impaired people.

The Effects of Static and Moving Spectral Ripple Sensitivity on Unaided and Aided Speech Perception in Noise

PURPOSE

This study evaluated whether certain spectral ripple conditions were more informative than others in predicting ecologically relevant unaided and aided speech outcomes.

METHOD

A quasi-experimental study design was used to evaluate 67 older adult hearing aid users with bilateral, symmetrical hearing loss. Speech perception in noise was tested under conditions of unaided and aided, auditory-only and auditory-visual, and 2 types of noise. Predictors included age, audiometric thresholds, audibility, hearing aid compression, and modulation depth detection thresholds for moving (4-Hz) or static (0-Hz) 2-cycle/octave spectral ripples applied to carriers of broadband noise or 2000-Hz low- or high-pass filtered noise.

RESULTS

A principal component analysis of the modulation detection data found that broadband and low-pass static and moving ripple detection thresholds loaded onto the first factor whereas high-pass static and moving ripple detection thresholds loaded onto a second factor. A linear mixed model revealed that audibility and the first factor (reflecting broadband and low-pass static and moving ripples) were significantly associated with speech perception performance. Similar results were found for unaided and aided speech scores. The interactions between speech conditions were not significant, suggesting that the relationship between ripples and speech perception was consistent regardless of visual cues or noise condition. High-pass ripple sensitivity was not correlated with speech understanding.

CONCLUSIONS

The results suggest that, for hearing aid users, poor speech understanding in noise and sensitivity to both static and slow-moving ripples may reflect deficits in the same underlying auditory processing mechanism. Significant factor loadings involving ripple stimuli with low-frequency content may suggest an impaired ability to use temporal fine structure information in the stimulus waveform. Support is provided for the use of spectral ripple testing to predict speech perception outcomes in clinical settings.

Sensorineural hearing loss impairs sensitivity but spares temporal integration for detection of frequency modulation

The effect of the number of modulation cycles (N) on frequency-modulation (FM) detection thresholds (FMDTs) was measured with and without interfering amplitude modulation (AM) for hearing-impaired (HI) listeners, using a 500-Hz sinusoidal carrier and FM rates of 2 and 20 Hz. The data were compared with FMDTs for normal-hearing (NH) listeners and AM detection thresholds (AMDTs) for NH and HI listeners [Wallaert, Moore, and Lorenzi (2016). J. Acoust. Soc. 139, 3088-3096; Wallaert, Moore, Ewert, and Lorenzi (2017). J. Acoust. Soc. 141, 971-980]. FMDTs were higher for HI than for NH listeners, but the effect of increasing N was similar across groups. In contrast, AMDTs were lower and the effect of increasing N was greater for HI listeners than for NH listeners. A model of temporal-envelope processing based on a modulation filter-bank and a template-matching decision strategy accounted better for the FMDTs at 20 Hz than at 2 Hz for young NH listeners and predicted greater temporal integration of FM than observed for all groups. These results suggest that different mechanisms underlie AM and FM detection at low rates and that hearing loss impairs FM-detection mechanisms, but preserves the memory and decision processes responsible for temporal integration of FM.