A method to enhance the use of interaural time differences for cochlear implants in reverberant environments

Effect of inter-aural temporal envelope differences on inter-aural time difference thresholds for amplitude modulated noise

PURPOSE

The inter-aural time difference (ITD) and inter-aural level difference (ILD) are important acoustic cues for horizontal localization and spatial release from masking. These cues are encoded based on inter-aural comparisons of tonotopically matched binaural inputs. Therefore, binaural coherence or the interaural spectro-temporal similarity is a pre-requisite for encoding ITD and ILD. The modulation depth of envelope is an important envelope characteristic that helps in encoding the envelope-ITD. However, inter-aural difference in modulation depth can result in reduced binaural coherence and poor representation of binaural cues as in the case with reverberation, noise and compression in cochlear implants and hearing aids. This study investigated the effect of inter-aural modulation depth difference on the ITD thresholds for an amplitude-modulated noise in normal hearing young adults.

METHODS

An amplitude modulated high pass filtered noise with varying modulation depth differences was presented sequentially through headphones. In one ear, the modulation depth was retained at 90% and in the other ear it varied from 90% to 50%. The ITD thresholds for modulation frequencies of 8 Hz and 16 Hz were estimated as a function of the inter-aural modulation depth difference.

RESULTS

The Friedman test findings revealed a statistically significant increase in the ITD threshold with an increase in the inter-aural modulation depth difference for 8 Hz and 16 Hz.

CONCLUSION

The results indicate that the inter-aural differences in the modulation depth negatively impact ITD perception for an amplitude-modulated high pass filtered noise.

Pitch discrimination in electric hearing with inconsistent and consistent amplitude-modulation and inter-pulse rate cues

Users of cochlear implants (CIs) struggle in situations that require selective hearing to focus on a target source while ignoring other sources. One major reason for that is the limited access to timing cues such as temporal pitch or interaural time differences (ITDs). Various approaches to improve timing-cue sensitivity while maintaining speech understanding have been proposed, among them inserting extra pulses with short inter-pulse intervals (SIPIs) into amplitude-modulated (AM) high-rate pulse trains. Indeed, SIPI rates matching the naturally occurring AM rates improve pitch discrimination. For ITD, however, low SIPI rates are required, potentially mismatching the naturally occurring AM rates and thus creating unknown pitch effects. In this study, we investigated the perceptual contribution of AM and SIPI rate to pitch discrimination in five CI listeners and with two AM depths (0.1 and 0.5). Our results show that the SIPI-rate cue generally dominated the percept for both consistent and inconsistent cues. When tested with inconsistent cues, also the AM rate contributed, however, at the large AM depth only. These findings have implications when aiming at jointly improving temporal-pitch and ITD sensitivity in a future mixed-rate stimulation approach.

Lateralization of binaural envelope cues measured with a mobile cochlear-implant research processora)

Bilateral cochlear implant (BICI) listeners do not have full access to the binaural cues that normal hearing (NH) listeners use for spatial hearing tasks such as localization. When using their unsynchronized everyday processors, BICI listeners demonstrate sensitivity to interaural level differences (ILDs) in the envelopes of sounds, but interaural time differences (ITDs) are less reliably available. It is unclear how BICI listeners use combinations of ILDs and envelope ITDs, and how much each cue contributes to perceived sound location. The CCi-MOBILE is a bilaterally synchronized research processor with the untested potential to provide spatial cues to BICI listeners. In the present study, the CCi-MOBILE was used to measure the ability of BICI listeners to perceive lateralized sound sources when single pairs of electrodes were presented amplitude-modulated stimuli with combinations of ILDs and envelope ITDs. Young NH listeners were also tested using amplitude-modulated high-frequency tones. A cue weighting analysis with six BICI and ten NH listeners revealed that ILDs contributed more than envelope ITDs to lateralization for both groups. Moreover, envelope ITDs contributed to lateralization for NH listeners but had negligible contribution for BICI listeners. These results suggest that the CCi-MOBILE is suitable for binaural testing and developing bilateral processing strategies.

Binaural consequences of speech envelope enhancement

The potential binaural consequences of two envelope-based speech enhancement strategies (broadband compression and expansion) were examined. Sensitivity to interaural time differences imposed on four single-word stimuli was measured in listeners with normal hearing and sensorineural hearing loss. While there were no consistent effects of compression or expansion across all words, some potentially interesting word-specific effects were observed.

Modulation Depth Discrimination by Cochlear Implant Users

Cochlear implants (CIs) convey the amplitude envelope of speech by modulating high-rate pulse trains. However, not all of the envelope may be necessary to perceive amplitude modulations (AMs); the effective envelope depth may be limited by forward and backward masking from the envelope peaks. Three experiments used modulated pulse trains to measure which portions of the envelope can be effectively processed by CI users as a function of AM frequency. Experiment 1 used a three-interval forced-choice task to test the ability of CI users to discriminate less-modulated pulse trains from a fully modulated standard, without controlling for loudness. The stimuli in experiment 2 were identical, but a two-interval task was used in which participants were required to choose the less-modulated interval, ignoring loudness. Catch trials, in which judgements based on level or modulation depth would give opposing answers, were included. Experiment 3 employed novel stimuli whose modulation envelope could be modified below a variable point in the dynamic range, without changing the loudness of the stimulus. Overall, results showed that substantial portions of the envelope are not accurately encoded by CI users. In experiment 1, where loudness cues were available, participants on average were insensitive to changes in the bottom 30% of their dynamic range. In experiment 2, where loudness was controlled, participants appeared insensitive to changes in the bottom 50% of the dynamic range. In experiment 3, participants were insensitive to changes in the bottom 80% of the dynamic range. We discuss potential reasons for this insensitivity and implications for CI speech-processing strategies.

The Impact of Synchronized Cochlear Implant Sampling and Stimulation on Free-Field Spatial Hearing Outcomes: Comparing the ciPDA Research Processor to Clinical Processors

OBJECTIVES

Bilateral cochlear implant (BiCI) listeners use independent processors in each ear. This independence and lack of shared hardware prevents control of the timing of sampling and stimulation across ears, which precludes the development of bilaterally-coordinated signal processing strategies. As a result, these devices potentially reduce access to binaural cues and introduce disruptive artifacts. For example, measurements from two clinical processors demonstrate that independently-running processors introduce interaural incoherence. These issues are typically avoided in the laboratory by using research processors with bilaterally-synchronized hardware. However, these research processors do not typically run in real-time and are difficult to take out into the real-world due to their benchtop nature. Hence, the question of whether just applying hardware synchronization to reduce bilateral stimulation artifacts (and thereby potentially improve functional spatial hearing performance) has been difficult to answer. The CI personal digital assistant (ciPDA) research processor, which uses one clock to drive two processors, presented an opportunity to examine whether synchronization of hardware can have an impact on spatial hearing performance.

DESIGN

Free-field sound localization and spatial release from masking (SRM) were assessed in 10 BiCI listeners using both their clinical processors and the synchronized ciPDA processor. For sound localization, localization accuracy was compared within-subject for the two processor types. For SRM, speech reception thresholds were compared for spatially separated and co-located configurations, and the amount of unmasking was compared for synchronized and unsynchronized hardware. There were no deliberate changes of the sound processing strategy on the ciPDA to restore or improve binaural cues.

RESULTS

There was no significant difference in localization accuracy between unsynchronized and synchronized hardware (p = 0.62). Speech reception thresholds were higher with the ciPDA. In addition, although five of eight participants demonstrated improved SRM with synchronized hardware, there was no significant difference in the amount of unmasking due to spatial separation between synchronized and unsynchronized hardware (p = 0.21).

CONCLUSIONS

Using processors with synchronized hardware did not yield an improvement in sound localization or SRM for all individuals, suggesting that mere synchronization of hardware is not sufficient for improving spatial hearing outcomes. Further work is needed to improve sound coding strategies to facilitate access to spatial hearing cues. This study provides a benchmark for spatial hearing performance with real-time, bilaterally-synchronized research processors.

Reweighting of Binaural Localization Cues in Bilateral Cochlear-Implant Listeners

Normal-hearing (NH) listeners rely on two binaural cues, the interaural time (ITD) and level difference (ILD), for azimuthal sound localization. Cochlear-implant (CI) listeners, however, rely almost entirely on ILDs. One reason is that present-day clinical CI stimulation strategies do not convey salient ITD cues. But even when presenting ITDs under optimal conditions using a research interface, ITD sensitivity is lower in CI compared to NH listeners. Since it has recently been shown that NH listeners change their ITD/ILD weighting when only one of the cues is consistent with visual information, such reweighting might add to CI listeners’ low perceptual contribution of ITDs, given their daily exposure to reliable ILDs but unreliable ITDs. Six bilateral CI listeners completed a multi-day lateralization training visually reinforcing ITDs, flanked by a pre- and post-measurement of ITD/ILD weights without visual reinforcement. Using direct electric stimulation, we presented 100- and 300-pps pulse trains at a single interaurally place-matched electrode pair, conveying ITDs and ILDs in various spatially consistent and inconsistent combinations. The listeners’ task was to lateralize the stimuli in a virtual environment. Additionally, ITD and ILD thresholds were measured before and after training. For 100-pps stimuli, the lateralization training increased the contribution of ITDs slightly, but significantly. Thresholds were neither affected by the training nor correlated with weights. For 300-pps stimuli, ITD weights were lower and ITD thresholds larger, but there was no effect of training. On average across test sessions, adding azimuth-dependent ITDs to stimuli containing ILDs increased the extent of lateralization for both 100- and 300-pps stimuli. The results suggest that low-rate ITD cues, robustly encoded with future CI systems, may be better exploitable for sound localization after increasing their perceptual weight via training.

High-resolution temporal weighting of interaural time differences in speech

Previous studies have shown that for high-rate click trains and low-frequency pure tones, interaural time differences (ITDs) at the onset of stimulus contribute most strongly to the overall lateralization percept (receive the largest perceptual weight). Previous studies have also shown that when these stimuli are modulated, ITDs during the rising portion of the modulation cycle receive increased perceptual weight. Baltzell, Cho, Swaminathan, and Best [(2020). J. Acoust. Soc. Am. 147, 3883-3894] measured perceptual weights for a pair of spoken words ("two" and "eight"), and found that word-initial phonemes receive larger weight than word-final phonemes, suggesting a "word-onset dominance" for speech. Generalizability of this conclusion was limited by a coarse temporal resolution and limited stimulus set. In the present study, temporal weighting functions (TWFs) were measured for four spoken words ("two," "eight," "six," and "nine"). Stimuli were partitioned into 30-ms bins, ITDs were applied independently to each bin, and lateralization judgements were obtained. TWFs were derived using a hierarchical regression model. Results suggest that "word-initial" onset dominance does not generalize across words and that TWFs depend in part on acoustic changes throughout the stimulus. Two model-based predictions were generated to account for observed TWFs, but neither could fully account for the perceptual data.

Temporal quantization deteriorates the discrimination of interaural time differences

Cochlear implants (CIs) often replace acoustic temporal fine structure by a fixed-rate pulse train. If the pulse timing is arbitrary (that is, not based on the phase information of the acoustic signal), temporal information is quantized by the pulse period. This temporal quantization is probably imperceptible with current clinical devices. However, it could result in large temporal jitter for strategies that aim to improve bilateral and bimodal CI users' perception of interaural time differences (ITDs), such as envelope enhancement. In an experiment with 16 normal-hearing listeners, it is shown that such jitter could deteriorate ITD perception for temporal quantization that corresponds to the often-used stimulation rate of 900 pulses per second (pps): the just-noticeable difference in ITD with quantization was 177 μs as compared to 129 μs without quantization. For smaller quantization step sizes, no significant deterioration of ITD perception was found. In conclusion, the binaural system can only average out the effect of temporal quantization to some extent, such that pulse timing should be well-considered. As this psychophysical procedure was somewhat unconventional, different procedural parameters were compared by simulating a number of commonly used two-down one-up adaptive procedures in Appendix B.

Improving Interaural Time Difference Sensitivity Using Short Inter-pulse Intervals with Amplitude-Modulated Pulse Trains in Bilateral Cochlear Implants

Interaural time differences (ITDs) at low frequencies are important for sound localization and spatial speech unmasking. These ITD cues are not encoded in commonly used envelope-based stimulation strategies for cochlear implants (CIs) using high pulse rates. However, ITD sensitivity can be improved by adding extra pulses with short inter-pulse intervals (SIPIs) in unmodulated high-rate trains. Here, we investigated whether this improvement also applies to amplitude-modulated (AM) high-rate pulse trains. To this end, we systematically varied the temporal position of SIPI pulses within the envelope cycle (SIPI phase), the fundamental frequency (F0) of AM (125 Hz and 250 Hz), and AM depth (from 0.1 to 0.9). Stimuli were presented at an interaurally place-matched electrode pair at a reference pulse rate of 1000 pulses/s. Participants performed an ITD-based left/right discrimination task. SIPI insertion resulted in improved ITD sensitivity throughout the range of modulation depths and for both male and female F0s. The improvements were largest for insertion at and around the envelope peak. These results are promising for conveying salient ITD cues at high pulse rates commonly used to encode speech information.

Temporal-pitch sensitivity in electric hearing with amplitude modulation and inserted pulses with short inter-pulse intervals

Listeners with cochlear implants (CIs) typically show poor sensitivity to the temporal-envelope pitch of high-rate pulse trains. Sensitivity to interaural time differences improves when adding pulses with short inter-pulse intervals (SIPIs) to high-rate pulse trains. In the current study, monaural temporal-pitch sensitivity with SIPI pulses was investigated for six CI listeners. Amplitude-modulated single-electrode stimuli, representing the coding of the fundamental frequency (F0) in the envelope of a high-rate carrier, were used. Two SIPI-insertion approaches, five modulation depths, two typical speech-F0s, and two carrier rates were tested. SIPI pulses were inserted either in every amplitude-modulation period (full-rate SIPI) to support the F0 cue or in every other amplitude-modulation period (half-rate SIPI) to circumvent a potential rate limitation at higher F0s. The results demonstrate that full-rate SIPI pulses improve temporal-pitch sensitivity across F0s and particularly at low modulation depths where envelope-pitch cues are weak. The half-rate SIPI pulses did not circumvent the limitation and further increased variability across listeners. Further, no effect of the carrier rate was found. Thus, the SIPI approach appears to be a promising approach to enhance CI listeners' access to temporal-envelope pitch cues at pulse rates used clinically.

The Effects of Dynamic-range Automatic Gain Control on Sentence Intelligibility With a Speech Masker in Simulated Cochlear Implant Listening

OBJECTIVES

"Channel-linked" and "multi-band" front-end automatic gain control (AGC) were examined as alternatives to single-band, channel-unlinked AGC in simulated bilateral cochlear implant (CI) processing. In channel-linked AGC, the same gain control signal was applied to the input signals to both of the two CIs ("channels"). In multi-band AGC, gain control acted independently on each of a number of narrow frequency regions per channel.

DESIGN

Speech intelligibility performance was measured with a single target (to the left, at -15 or -30°) and a single, symmetrically-opposed masker (to the right) at a signal-to-noise ratio (SNR) of -2 decibels. Binaural sentence intelligibility was measured as a function of whether channel linking was present and of the number of AGC bands. Analysis of variance was performed to assess condition effects on percent correct across the two spatial arrangements, both at a high and a low AGC threshold. Acoustic analysis was conducted to compare postcompressed better-ear SNR, interaural differences, and monaural within-band envelope levels across processing conditions.

RESULTS

Analyses of variance indicated significant main effects of both channel linking and number of bands at low threshold, and of channel linking at high threshold. These improvements were accompanied by several acoustic changes. Linked AGC produced a more favorable better-ear SNR and better preserved broadband interaural level difference statistics, but did not reduce dynamic range as much as unlinked AGC. Multi-band AGC sometimes improved better-ear SNR statistics and always improved broadband interaural level difference statistics whenever the AGC channels were unlinked. Multi-band AGC produced output envelope levels that were higher than single-band AGC.

CONCLUSIONS

These results favor strategies that incorporate channel-linked AGC and multi-band AGC for bilateral CIs. Linked AGC aids speech intelligibility in spatially separated speech, but reduces the degree to which dynamic range is compressed. Combining multi-band and channel-linked AGC offsets the potential impact of diminished dynamic range with linked AGC without sacrificing the intelligibility gains observed with linked AGC.

Coherent Coding of Enhanced Interaural Cues Improves Sound Localization in Noise With Bilateral Cochlear Implants

Bilateral cochlear implant (BCI) users only have very limited spatial hearing abilities. Speech coding strategies transmit interaural level differences (ILDs) but in a distorted manner. Interaural time difference (ITD) information transmission is even more limited. With these cues, most BCI users can coarsely localize a single source in quiet, but performance quickly declines in the presence of other sound. This proof-of-concept study presents a novel signal processing algorithm specific for BCIs, with the aim to improve sound localization in noise. The core part of the BCI algorithm duplicates a monophonic electrode pulse pattern and applies quasistationary natural or artificial ITDs or ILDs based on the estimated direction of the dominant source. Three experiments were conducted to evaluate different algorithm variants: Experiment 1 tested if ITD transmission alone enables BCI subjects to lateralize speech. Results showed that six out of nine BCI subjects were able to lateralize intelligible speech in quiet solely based on ITDs. Experiments 2 and 3 assessed azimuthal angle discrimination in noise with natural or modified ILDs and ITDs. Angle discrimination for frontal locations was possible with all variants, including the pure ITD case, but for lateral reference angles, it was only possible with a linearized ILD mapping. Speech intelligibility in noise, limitations, and challenges of this interaural cue transmission approach are discussed alongside suggestions for modifying and further improving the BCI algorithm.

Auditory cortex responses to interaural time differences in the envelope of low-frequency sound, recorded with MEG in young and older listeners

Interaural time and intensity differences (ITD and IID) are important cues in binaural hearing and allow for sound localization, improving speech understanding in noise and reverberation, and integrating sound sources in the auditory scene. Whereas previous research showed that the upper-frequency limit for ITD detection in the fine structure of sound declines in aging, the processing of envelope ITD in low-frequency amplitude modulated (AM) sound and the related brain responses are less understood. This study investigated the cortical processing of envelope ITD and compared the results with previous findings about the fine-structure ITD. In two experiments, participants listened to 40-Hz AM tones containing sudden changes in the envelope ITD. Multiple MEG responses were analyzed, including the auditory evoked N1 responses, elicited both by sound onsets and ITD changes, and 40-Hz responses, elicited by the AM. The first experiment with healthy young adults revealed a substantial decline in the magnitudes of the ITD change N1 response, and the 40-Hz phase resets at higher carrier frequencies, suggesting a similar frequency characteristic as observed for fine structure ITD. The amplitude of the 40-Hz ASSR declined only gradually with increasing carrier frequency, and it was excluded as a confounding factor in the decline in the ITD response. Larger responses to outward ITD changes than inward changes, here first reported for envelope ITD, were another characteristics that were similar to fine-structure ITD. A second experiment with groups of young and older listeners examined the effects of aging and concurrent noise on the cortical envelope ITD responses. One important research question was, whether binaural cues are accessible in noise. Behavioural tests showed an age-related hearing loss in the older group and decreased performance in envelope ITD detection and speech-in-noise (SIN) understanding. Binaural hearing and SIN performance were correlated with one other, but not with hearing loss. The frequency limit for envelope ITD was reduced in older listeners similarly as previously found for fine structure ITD, and older listeners were more susceptible to concurrent multi-talker noise. The similarities between responses to envelope ITD and to fine structure ITD suggest that a common cortical code exists for the envelope and fine structure ITD. The dependency on the carrier frequency suggests that envelope ITD processing at the subcortical level requires stimulus phase locking, which might be reduced in aging.

Influence of envelope waveform on ITD sensitivity of neurons in the auditory midbrain

Using single-neuron electrophysiology, we show that the precise shape of a sound’s “energy envelope” is a critical factor in determining how well midbrain neurons are able to convey information about auditory spatial cues. Consistent with human behavioral performance, sounds with rapidly rising energy and relatively long intervals between energy bursts are best at conveying spatial information. The data suggest specific sound energy patterns that might best be applied to hearing devices to aid spatial listening.

Interaural time differences (ITDs) conveyed by the modulated envelopes of high-frequency sounds can serve as a cue for localizing a sound source. Klein-Hennig et al. (J Acoust Soc Am 129: 3856, 2011) demonstrated the envelope attack (the rate at which stimulus energy in the envelope increases) and the duration of the pause (the interval between successive envelope pulses) as important factors affecting sensitivity to envelope ITDs in human listeners. Modulated sounds with rapid attacks and long pauses produce the lowest ITD discrimination thresholds. The duration of the envelope’s sustained component (sustain) and the rate at which stimulus energy falls at the offset of the envelope (decay) are only minor factors. We assessed the responses of 71 single neurons, recorded from the midbrains of 15 urethane-anesthetized tri-colored guinea pigs, to envelope shapes in which the four envelope components, i.e., attack, sustain, decay, and pause, were systematically varied. We confirmed the importance of the attack and pause components in generating ITD-sensitive responses. Analysis of neural firing rates demonstrated more neurons (49/71) show ITD sensitivity in response to “damped” stimuli (fast attack and slow decay) compared with “ramped” stimuli (slow attack and fast decay) (14/71). Furthermore, the lowest threshold for the damped stimulus (91 μs) was lower by a factor of 4 than that for the temporally reversed ramped envelope shape (407 μs). The data confirm the importance of fast attacks and optimal pause durations in generating sensitivity to ITDs conveyed in the modulated envelopes of high-frequency sounds and are incompatible with models of ITD processing based on the integration of sound energy over time.

NEW & NOTEWORTHY Using single-neuron electrophysiology, we show that the precise shape of a sound’s “energy envelope” is a critical factor in determining how well midbrain neurons are able to convey information about auditory spatial cues. Consistent with human behavioral performance, sounds with rapidly rising energy and relatively long intervals between energy bursts are best at conveying spatial information. The data suggest specific sound energy patterns that might best be applied to hearing devices to aid spatial listening.

The history and future of neural modeling for cochlear implants

This special issue of Network: Computation in Neural Systems on the topic of "Computational models of the electrically stimulated auditory system" incorporates review articles spanning a wide range of approaches to modeling cochlear implant stimulation of the auditory system. The purpose of this overview paper is to provide a historical context for the different modeling endeavors and to point toward how computational modeling could play a key role in the understanding, evaluation, and improvement of cochlear implants in the future.