Enhancing the speech envelope of continuous interleaved sampling processors for cochlear implants

Pulsatile modulation greatly enhances neural synchronization at syllable rate in children

Neural processing of the speech envelope is of crucial importance for speech perception and comprehension. This envelope processing is often investigated by measuring neural synchronization to sinusoidal amplitude-modulated stimuli at different modulation frequencies. However, it has been argued that these stimuli lack ecological validity. Pulsatile amplitude-modulated stimuli, on the other hand, are suggested to be more ecologically valid and efficient, and have increased potential to uncover the neural mechanisms behind some developmental disorders such a dyslexia. Nonetheless, pulsatile stimuli have not yet been investigated in pre-reading and beginning reading children, which is a crucial age for developmental reading research. We performed a longitudinal study to examine the potential of pulsatile stimuli in this age range. Fifty-two typically reading children were tested at three time points from the middle of their last year of kindergarten (5 years old) to the end of first grade (7 years old). Using electroencephalography, we measured neural synchronization to syllable rate and phoneme rate sinusoidal and pulsatile amplitude-modulated stimuli. Our results revealed that the pulsatile stimuli significantly enhance neural synchronization at syllable rate, compared to the sinusoidal stimuli. Additionally, the pulsatile stimuli at syllable rate elicited a different hemispheric specialization, more closely resembling natural speech envelope tracking. We postulate that using the pulsatile stimuli greatly increases EEG data acquisition efficiency compared to the common sinusoidal amplitude-modulated stimuli in research in younger children and in developmental reading research.

Neural synchronization and intervention in pre-readers who later on develop dyslexia

A growing number of studies has investigated temporal processing deficits in dyslexia. These studies largely focus on neural synchronization to speech. However, the importance of rise times for neural synchronization is often overlooked. Furthermore, targeted interventions, phonics-based and auditory, are being developed, but little is known about their impact. The current study investigated the impact of a 12-week tablet-based intervention. Children at risk for dyslexia received phonics-based training, either with (n = 31) or without (n = 31) auditory training, or engaged in active control training (n = 29). Additionally, neural synchronization and processing of rise times was longitudinally investigated in children with dyslexia (n = 26) and typical readers (n = 52) from pre-reading (5 years) to beginning reading age (7 years). The three time points in the longitudinal study correspond to intervention pre-test, post-test and consolidation, approximately 1 year after completing the intervention. At each time point neural synchronization was measured to sinusoidal stimuli and pulsatile stimuli with shortened rise times at syllable (4 Hz) and phoneme rates (20 Hz). Our results revealed no impact on neural synchronization at syllable and phoneme rate of the phonics-based and auditory training. However, we did reveal atypical hemispheric specialization at both syllable and phoneme rates in children with dyslexia. This was detected even before the onset of reading acquisition, pointing towards a possible causal rather than consequential mechanism in dyslexia. This study contributes to our understanding of the temporal processing deficits underlying the development of dyslexia, but also shows that the development of targeted interventions is still a work in progress.

Speech perception deficits and the effect of envelope-enhanced story listening combined with phonics intervention in pre-readers at risk for dyslexia

Developmental dyslexia is considered to be most effectively addressed with preventive phonics-based interventions, including grapheme-phoneme coupling and blending exercises. These intervention types require intact speech perception abilities, given their large focus on exercises with auditorily presented phonemes. Yet some children with (a risk for) dyslexia experience problems in this domain due to a poorer sensitivity to rise times, i.e., rhythmic acoustic cues present in the speech envelope. As a result, the often subtle speech perception problems could potentially constrain an optimal response to phonics-based interventions in at-risk children. The current study therefore aimed (1) to extend existing research by examining the presence of potential speech perception deficits in pre-readers at cognitive risk for dyslexia when compared to typically developing peers and (2) to explore the added value of a preventive auditory intervention for at-risk pre-readers, targeting rise time sensitivity, on speech perception and other reading-related skills. To obtain the first research objective, we longitudinally compared speech-in-noise perception between 28 5-year-old pre-readers with and 30 peers without a cognitive risk for dyslexia during the second half of the third year of kindergarten. The second research objective was addressed by exploring growth in speech perception and other reading-related skills in an independent sample of 62 at-risk 5-year-old pre-readers who all combined a 12-week preventive phonics-based intervention (GraphoGame-Flemish) with an auditory story listening intervention. In half of the sample, story recordings contained artificially enhanced rise times (GG-FL_EE group, n = 31), while in the other half, stories remained unprocessed (GG-FL_NE group, n = 31; Clinical Trial Number S60962—https://www.uzleuven.be/nl/clinical-trial-center). Results revealed a slower speech-in-noise perception growth in the at-risk compared to the non-at-risk group, due to an emerged deficit at the end of kindergarten. Concerning the auditory intervention effects, both intervention groups showed equal growth in speech-in-noise perception and other reading-related skills, suggesting no boost of envelope-enhanced story listening on top of the effect of combining GraphoGame-Flemish with listening to unprocessed stories. These findings thus provide evidence for a link between speech perception problems and dyslexia, yet do not support the potential of the auditory intervention in its current form.

Feasibility, Enjoyment, and Language Comprehension Impact of a Tablet- and GameFlow-Based Story-Listening Game for Kindergarteners: Methodological and Mixed Methods Study

BACKGROUND

Enjoyment plays a key role in the success and feasibility of serious gaming interventions. Unenjoyable games will not be played, and in the case of serious gaming, learning will not occur. Therefore, a so-called GameFlow model has been developed, which intends to guide (serious) game developers in the process of creating and evaluating enjoyment in digital (serious) games. Regarding language learning, a variety of serious games targeting specific language components exist in the market, albeit often without available assessments of enjoyment or feasibility.

OBJECTIVE

This study evaluates the enjoyment and feasibility of a tablet-based, serious story-listening game for kindergarteners, developed based on the principles of the GameFlow model. This study also preliminarily explores the possibility of using the game to foster language comprehension.

METHODS

Within the framework of a broader preventive reading intervention, 91 kindergarteners aged 5 years with a cognitive risk for dyslexia were asked to play the story game for 12 weeks, 6 days per week, either combined with a tablet-based phonics intervention or control games. The story game involved listening to and rating stories and responding to content-related questions. Game enjoyment was assessed through postintervention questionnaires, a GameFlow-based evaluation, and in-game story rating data. Feasibility was determined based on in-game general question response accuracy (QRA), reflecting the difficulty level, attrition rate, and final game exposure and training duration. Moreover, to investigate whether game enjoyment and difficulty influenced feasibility, final game exposure and training duration were predicted based on the in-game initial story ratings and initial QRA. Possible growth in language comprehension was explored by analyzing in-game QRA as a function of the game phase and baseline language skills.

RESULTS

Eventually, data from 82 participants were analyzed. The questionnaire and in-game data suggested an overall enjoyable game experience. However, the GameFlow-based evaluation implied room for game design improvement. The general QRA confirmed a well-adapted level of difficulty for the target sample. Moreover, despite the overall attrition rate of 39% (32/82), 90% (74/82) of the participants still completed 80% of the game, albeit with a large variation in training days. Higher initial QRA significantly increased game exposure (β=.35; P<.001), and lower initial story ratings significantly slackened the training duration (β=-0.16; P=.003). In-game QRA was positively predicted by game phase (β=1.44; P=.004), baseline listening comprehension (β=1.56; P=.002), and vocabulary (β=.16; P=.01), with larger QRA growth over game phases in children with lower baseline listening comprehension skills (β=-0.08; P=.04).

CONCLUSIONS

Generally, the story game seemed enjoyable and feasible. However, the GameFlow model evaluation and predictive relationships imply room for further game design improvements. Furthermore, our results cautiously suggest the potential of the game to foster language comprehension; however, future randomized controlled trials should further elucidate the impact on language comprehension.

CCi-MOBILE: A Portable Real Time Speech Processing Platform for Cochlear Implant and Hearing Research

Experimental hardware-research interfaces form a crucial role during developmental stages of any medical, signal-monitoring system as it allows researchers to test and optimize output results before perfecting the design for the actual FDA approved medical device and large-scale production. These testing platforms, intake raw signals through which performance of novel algorithms can be analyzed and modified to generate the desired data points for an optimized output, allowing the advancement of the medical device. With cochlear implants (CIs) and hearing aids (HAs) becoming a more common solution for varying degrees of hearing impairment, having modern signal processing strategies tested for such speech sensitive systems is a necessity. But the rigid design requirements of commercial CI and HA processors make it difficult to explore novel algorithms for research investigations and conducting longitudinal studies. This study presents the design, development, clinical evaluation, and applications of CCi-MOBILE, a computationally powerful signal processing testing platform built for researchers in the hearing-impaired field. The custom-made, portable research platform allows researchers to design and perform complex speech processing algorithm assessment offline and in real-time. It can be operated through user-friendly, open-source software and is compatible with implants manufactured by Cochlear Corporation. The FPGA design and hardware processing pipeline for CI stimulation is discussed followed by results from an acute study with implant users' speech intelligibility in quiet and noisy conditions. The results show a consistent level of performance compared with CI users' clinical processor, thus confirming the viability of the platform in chronic CI based studies.

Estimating health of the implanted cochlea using psychophysical strength-duration functions and electrode configuration

It is generally believed that the efficacy of cochlear implants is partly dependent on the condition of the stimulated neural population. Cochlear pathology is likely to affect the manner in which neurons respond to electrical stimulation, potentially resulting in differences in perception of electrical stimuli across cochlear implant recipients and across the electrode array in individual cochlear implant users. Several psychophysical and electrophysiological measures have been shown to predict cochlear health in animals and were used to assess conditions near individual stimulation sites in humans. In this study, we examined the relationship between psychophysical strength-duration functions and spiral ganglion neuron density in two groups of guinea pigs with cochlear implants who had minimally-overlapping cochlear health profiles. One group was implanted in a hearing ear (N = 10) and the other group was deafened by cochlear perfusion of neomycin, inoculated with an adeno-associated viral vector with an Ntf3-gene insert (AAV.Ntf3) and implanted (N = 14). Psychophysically measured strength-duration functions for both monopolar and tripolar electrode configurations were then compared for the two treatment groups. Results were also compared to their histological outcomes. Overall, there were considerable differences between the two treatment groups in terms of their psychophysical performance as well as the relation between their functional performance and histological data. Animals in the neomycin-deafened, neurotrophin-treated, and implanted group (NNI) exhibited steeper strength-duration function slopes; slopes were positively correlated with SGN density (steeper slopes in animals that had higher SGN densities). In comparison, the implanted hearing (IH) group had shallower slopes and there was no relation between slopes and spiral ganglion density. Across all animals, slopes were negatively correlated with ensemble spontaneous activity levels (shallower slopes with higher ensemble spontaneous activity levels). We hypothesize that differences in strength-duration function slopes between the two treatment groups were related to the condition of the inner hair cells, which generate spontaneous activity that could affect the across-fiber synchrony and/or the size of the population of neural elements responding to electrical stimulation. In addition, it is likely that spiral ganglion neuron peripheral processes were present in the IH group, which could affect membrane properties of the stimulated neurons. Results suggest that the two treatment groups exhibited distinct patterns of variation in conditions near the stimulating electrodes that altered detection thresholds. Overall, the results of this study suggest a complex relationship between psychophysical detection thresholds for cochlear implant stimulation and nerve survival in the implanted cochlea. This relationship seems to depend on the characteristics of the electrical stimulus, the electrode configuration, and other biological features of the implanted cochlea such as the condition of the inner hair cells and the peripheral processes.

Combination and Comparison of Sound Coding Strategies Using Cochlear Implant Simulation With Mandarin Speech

Three cochlear implant (CI) sound coding strategies were combined in the same signal processing path and compared for speech intelligibility with vocoded Mandarin sentences. The three CI coding strategies, biologically-inspired hearing aid algorithm (BioAid), envelope enhancement (EE), and fundamental frequency modulation (F0mod), were combined with the advanced combination encoder (ACE) strategy. Hence, four singular coding strategies and four combinational coding strategies were derived. Mandarin sentences with speech-shape noise were processed using these coding strategies. Speech understanding of vocoded Mandarin sentences was evaluated using short-time objective intelligibility (STOI) and subjective sentence recognition tests with normal-hearing listeners. For signal-to-noise ratios at 5 dB or above, the EE strategy had slightly higher average scores in both STOI and listening tests compared to ACE. The addition of EE to BioAid slightly increased the mean scores for BioAid+EE, which was the combination strategy with the highest scores in both objective and subjective speech intelligibility. The benefits of BioAid, F0mod, and the four combinational coding strategies were not observed in CI simulation. The findings of this study may be useful for the future design of coding strategies and related studies with Mandarin.

Ahead of maturation: Enhanced speech envelope training boosts rise time discrimination in pre-readers at cognitive risk for dyslexia

Dyslexia has frequently been related to atypical auditory temporal processing and speech perception. Results of studies emphasizing speech onset cues and reinforcing the temporal structure of the speech envelope, that is, envelope enhancement (EE), demonstrated reduced speech perception deficits in individuals with dyslexia. The use of this strategy as auditory intervention might thus reduce some of the deficits related to dyslexia. Importantly, reading-skill interventions are most effective when they are provided during kindergarten and first grade. Hence, we provided a tablet-based 12-week auditory and phonics-based intervention to pre-readers at cognitive risk for dyslexia and investigated the effect on auditory temporal processing with a rise time discrimination (RTD) task. Ninety-one pre-readers at cognitive risk for dyslexia (aged 5-6) were assigned to two groups receiving a phonics-based intervention and playing a story listening game either with (n = 31) or without (n = 31) EE or a third group playing control games and listening to non-enhanced stories (n = 29). RTD was measured directly before, directly after and 1 year after the intervention. While the groups listening to non-enhanced stories mainly improved after the intervention during first grade, the group listening to enhanced stories improved during the intervention in kindergarten and subsequently remained stable during first grade. Hence, an EE intervention improves auditory processing skills important for the development of phonological skills. This occurred before the onset of reading instruction, preceding the maturational improvement of these skills, hence potentially giving at risk children a head start when learning to read.

A Bridge over Troubled Listening: Improving Speech-in-Noise Perception by Children with Dyslexia

Developmental dyslexia is most commonly associated with phonological processing difficulties. However, children with dyslexia may experience poor speech-in-noise perception as well. Although there is an ongoing debate whether a speech perception deficit is inherent to dyslexia or acts as an aggravating risk factor compromising learning to read indirectly, improving speech perception might boost reading-related skills and reading acquisition. In the current study, we evaluated advanced speech technology as applied in auditory prostheses, to promote and eventually normalize speech perception of school-aged children with dyslexia, i.e., envelope enhancement (EE). The EE strategy automatically detects and emphasizes onset cues and consequently reinforces the temporal structure of the speech envelope. Our results confirmed speech-in-noise perception difficulties by children with dyslexia. However, we found that exaggerating temporal "landmarks" of the speech envelope (i.e., amplitude rise time and modulations)-by using EE-passively and instantaneously improved speech perception in noise for children with dyslexia. Moreover, the benefit derived from EE was large enough to completely bridge the initial gap between children with dyslexia and their typical reading peers. Taken together, the beneficial outcome of EE suggests an important contribution of the temporal structure of the envelope to speech perception in noise difficulties in dyslexia, providing an interesting foundation for future intervention studies based on auditory and speech rhythm training.

Temporal quantization deteriorates the discrimination of interaural time differences

Cochlear implants (CIs) often replace acoustic temporal fine structure by a fixed-rate pulse train. If the pulse timing is arbitrary (that is, not based on the phase information of the acoustic signal), temporal information is quantized by the pulse period. This temporal quantization is probably imperceptible with current clinical devices. However, it could result in large temporal jitter for strategies that aim to improve bilateral and bimodal CI users' perception of interaural time differences (ITDs), such as envelope enhancement. In an experiment with 16 normal-hearing listeners, it is shown that such jitter could deteriorate ITD perception for temporal quantization that corresponds to the often-used stimulation rate of 900 pulses per second (pps): the just-noticeable difference in ITD with quantization was 177 μs as compared to 129 μs without quantization. For smaller quantization step sizes, no significant deterioration of ITD perception was found. In conclusion, the binaural system can only average out the effect of temporal quantization to some extent, such that pulse timing should be well-considered. As this psychophysical procedure was somewhat unconventional, different procedural parameters were compared by simulating a number of commonly used two-down one-up adaptive procedures in Appendix B.

The effect of a coding strategy that removes temporally masked pulses on speech perception by cochlear implant users

Speech recognition in noisy environments remains a challenge for cochlear implant (CI) recipients. Unwanted charge interactions between current pulses, both within and between electrode channels, are likely to impair performance. Here we investigate the effect of reducing the number of current pulses on speech perception. This was achieved by implementing a psychoacoustic temporal-masking model where current pulses in each channel were passed through a temporal integrator to identify and remove pulses that were less likely to be perceived by the recipient. The decision criterion of the temporal integrator was varied to control the percentage of pulses removed in each condition. In experiment 1, speech in quiet was processed with a standard Continuous Interleaved Sampling (CIS) strategy and with 25, 50 and 75% of pulses removed. In experiment 2, performance was measured for speech in noise with the CIS reference and with 50 and 75% of pulses removed. Speech intelligibility in quiet revealed no significant difference between reference and test conditions. For speech in noise, results showed a significant improvement of 2.4 dB when removing 50% of pulses and performance was not significantly different between the reference and when 75% of pulses were removed. Further, by reducing the overall amount of current pulses by 25, 50, and 75% but accounting for the increase in charge necessary to compensate for the decrease in loudness, estimated average power savings of 21.15, 40.95, and 63.45%, respectively, could be possible for this set of listeners. In conclusion, removing temporally masked pulses may improve speech perception in noise and result in substantial power savings.

Effect of Noise Reduction Gain Errors on Simulated Cochlear Implant Speech Intelligibility

It has been suggested that the most important factor for obtaining high speech intelligibility in noise with cochlear implant (CI) recipients is to preserve the low-frequency amplitude modulations of speech across time and frequency by, for example, minimizing the amount of noise in the gaps between speech segments. In contrast, it has also been argued that the transient parts of the speech signal, such as speech onsets, provide the most important information for speech intelligibility. The present study investigated the relative impact of these two factors on the potential benefit of noise reduction for CI recipients by systematically introducing noise estimation errors within speech segments, speech gaps, and the transitions between them. The introduction of these noise estimation errors directly induces errors in the noise reduction gains within each of these regions. Speech intelligibility in both stationary and modulated noise was then measured using a CI simulation tested on normal-hearing listeners. The results suggest that minimizing noise in the speech gaps can improve intelligibility, at least in modulated noise. However, significantly larger improvements were obtained when both the noise in the gaps was minimized and the speech transients were preserved. These results imply that the ability to identify the boundaries between speech segments and speech gaps may be one of the most important factors for a noise reduction algorithm because knowing the boundaries makes it possible to minimize the noise in the gaps as well as enhance the low-frequency amplitude modulations of the speech.

Formant priority channel selection for an "n-of-m" sound processing strategy for cochlear implants

The Advanced Combination Encoder (ACE) signal processing strategy is used in the majority of cochlear implant (CI) sound processors manufactured by Cochlear Corporation. This "n-of-m" strategy selects "n" out of "m" available frequency channels with the highest spectral energy in each stimulation cycle. It is hypothesized that at low signal-to-noise ratio (SNR) conditions, noise-dominant frequency channels are susceptible for selection, neglecting channels containing target speech cues. In order to improve speech segregation in noise, explicit encoding of formant frequency locations within the standard channel selection framework of ACE is suggested. Two strategies using the direct formant estimation algorithms are developed within this study, FACE (formant-ACE) and VFACE (voiced-activated-formant-ACE). Speech intelligibility from eight CI users is compared across 11 acoustic conditions, including mixtures of noise and reverberation at multiple SNRs. Significant intelligibility gains were observed with VFACE over ACE in 5 dB babble noise; however, results with FACE/VFACE in all other conditions were comparable to standard ACE. An increased selection of channels associated with the second formant frequency is observed for FACE and VFACE. Both proposed methods may serve as potential supplementary channel selection techniques for the ACE sound processing strategy for cochlear implants.

Encoding speech in cochlear implants using simultaneous amplitude and rate modulation

To improve speech perception for cochlear implant (CI) users, it is essential to improve the transmission of temporal envelopes. The most common speech processors deliver temporal envelopes via the CI using fixed-rate amplitude modulated (AM) pulse trains. Psychophysical studies suggest that rate modulation (RM) and AM are perceived by a shared temporal integration mechanism, but the potential for them to constructively combine to encode temporal envelopes has yet to be explored. In this experiment, a speech processing strategy called amplitude and rate temporal modulation was developed to encode speech temporal envelopes with simultaneous AM and RM. The strategy was tested for perception of clean speech at 60 and 40 dBA, and 60 dBA speech in noise (+10 dB SNR). The amount of RM was varied and the amount of AM was held constant to determine whether the addition of RM could enhance the perception of temporal envelopes and improve speech understanding. At the lowest RM amount, speech scores were poorest for all speech conditions. For 60 dBA clean speech and speech in noise, speech scores were significantly better at the highest RM amounts, suggesting that RM combined with AM can be used to enhance perception of temporal envelopes.

Brain-inspired speech segmentation for automatic speech recognition using the speech envelope as a temporal reference

Speech segmentation is a crucial step in automatic speech recognition because additional speech analyses are performed for each framed speech segment. Conventional segmentation techniques primarily segment speech using a fixed frame size for computational simplicity. However, this approach is insufficient for capturing the quasi-regular structure of speech, which causes substantial recognition failure in noisy environments. How does the brain handle quasi-regular structured speech and maintain high recognition performance under any circumstance? Recent neurophysiological studies have suggested that the phase of neuronal oscillations in the auditory cortex contributes to accurate speech recognition by guiding speech segmentation into smaller units at different timescales. A phase-locked relationship between neuronal oscillation and the speech envelope has recently been obtained, which suggests that the speech envelope provides a foundation for multi-timescale speech segmental information. In this study, we quantitatively investigated the role of the speech envelope as a potential temporal reference to segment speech using its instantaneous phase information. We evaluated the proposed approach by the achieved information gain and recognition performance in various noisy environments. The results indicate that the proposed segmentation scheme not only extracts more information from speech but also provides greater robustness in a recognition test.

Enhancing speech envelope by integrating hair-cell adaptation into cochlear implant processing

Cochlear implants (CIs) bypass some of the mechanisms that underlie normal neural behavior as occurs in acoustic hearing. One such neural mechanism is short-term adaptation, which has been proposed to have a significant role in speech perception. Acoustically-evoked neural adaptation has been mainly attributed to the depletion of neurotransmitter in the hair-cell to auditory-nerve synapse and is therefore not fully present in CI stimulation. This study evaluated a signal processing method that integrated a physiological model of hair-cell adaptation into CI speech processing. The linear high-pass adaptation process expanded the range of rapid variations of the electrical signal generated by the clinical processing strategy. Speech perception performance with the adaptation-based processing was compared to that of the clinical strategy in seven CI users. While there was large variability across subjects, the new processing improved sentence recognition and consonant identification scores in quiet in all the tested subjects with an average improvement of 8% and 6% respectively. Consonant recognition scores in babble noise were improved at the higher signal-to-noise ratios tested (10 and 6 dB) only. Information transfer analysis of consonant features showed significant improvements for manner and place of articulation features, but not for voicing. Enhancement of within-channel envelope cues was confirmed by consonant recognition results obtained with single-channel strategies that presented the overall amplitude envelope of the signal on a single active electrode. Adaptation-inspired envelope enhancement techniques can potentially improve perception of important speech features by CI users.

Neural adaptation and behavioral measures of temporal processing and speech perception in cochlear implant recipients

The objective was to determine if one of the neural temporal features, neural adaptation, can account for the across-subject variability in behavioral measures of temporal processing and speech perception performance in cochlear implant (CI) recipients. Neural adaptation is the phenomenon in which neural responses are the strongest at the beginning of the stimulus and decline following stimulus repetition (e.g., stimulus trains). It is unclear how this temporal property of neural responses relates to psychophysical measures of temporal processing (e.g., gap detection) or speech perception. The adaptation of the electrical compound action potential (ECAP) was obtained using 1000 pulses per second (pps) biphasic pulse trains presented directly to the electrode. The adaptation of the late auditory evoked potential (LAEP) was obtained using a sequence of 1-kHz tone bursts presented acoustically, through the cochlear implant. Behavioral temporal processing was measured using the Random Gap Detection Test at the most comfortable listening level. Consonant nucleus consonant (CNC) word and AzBio sentences were also tested. The results showed that both ECAP and LAEP display adaptive patterns, with a substantial across-subject variability in the amount of adaptation. No correlations between the amount of neural adaptation and gap detection thresholds (GDTs) or speech perception scores were found. The correlations between the degree of neural adaptation and demographic factors showed that CI users having more LAEP adaptation were likely to be those implanted at a younger age than CI users with less LAEP adaptation. The results suggested that neural adaptation, at least this feature alone, cannot account for the across-subject variability in temporal processing ability in the CI users. However, the finding that the LAEP adaptive pattern was less prominent in the CI group compared to the normal hearing group may suggest the important role of normal adaptation pattern at the cortical level in speech perception.

Making use of auditory models for better mimicking of normal hearing processes with cochlear implants: the SAM coding strategy

Mimicking the human ear on the basis of auditory models has become a viable approach in many applications by now. However, only a few attempts have been made to extend the scope of physiological ear models to be employed in cochlear implants (CI). Contemporary CI systems rely on much simpler filter banks and simulate the natural signal processing of a healthy cochlea to only a very limited extent. When looking at rehabilitation outcomes, current systems seem to have reached their peak potential, which signals the need for better algorithms and/or technologies. In this paper, we present a novel sound processing strategy, SAM (Stimulation based on Auditory Modeling), that is based on neurophysiological models of the human ear and can be employed in auditory prostheses. It incorporates active cochlear filtering (basilar membrane and outer hair cells) along with the mechanoelectrical transduction of the inner hair cells, so that several psychoacoustic phenomena are accounted for inherently. Although possible, current implementation does not make use of parallel stimulation of the electrodes, which matches state-of-the-art CI hardware. This paper elaborates on SAM's signal processing and provides a computational evaluation of the strategy. Results show that aspects of normal cochlear processing that are missing in common strategies can be replicated by SAM. This is supposed to improve overall CI user performance, which we have at least partly proven in a pilot study with implantees.

A sound processor for cochlear implant using a simple dual path nonlinear model of basilar membrane

We propose a new active nonlinear model of the frequency response of the basilar membrane in biological cochlea called the simple dual path nonlinear (SDPN) model and a novel sound processing strategy for cochlear implants (CIs) based upon this model. The SDPN model was developed to utilize the advantages of the level-dependent frequency response characteristics of the basilar membrane for robust formant representation under noisy conditions. In comparison to the dual resonance nonlinear model (DRNL) which was previously proposed as an active nonlinear model of the basilar membrane, the SDPN model can reproduce similar level-dependent frequency responses with a much simpler structure and is thus better suited for incorporation into CI sound processors. By the analysis of dominant frequency component, it was confirmed that the formants of speech are more robustly represented after frequency decomposition by the nonlinear filterbank using SDPN, compared to a linear bandpass filter array which is used in conventional strategies. Acoustic simulation and hearing experiments in subjects with normal hearing showed that the proposed strategy results in better syllable recognition under speech-shaped noise compared to the conventional strategy based on fixed linear bandpass filters.

The potential of onset enhancement for increased speech intelligibility in auditory prostheses

Recent studies have shown that transient parts of a speech signal contribute most to speech intelligibility in normal-hearing listeners. In this study, the influence of enhancing the onsets of the envelope of the speech signal on speech intelligibility in noisy conditions using an eight channel cochlear implant vocoder simulation was investigated. The enhanced envelope (EE) strategy emphasizes the onsets of the speech envelope by deriving an additional peak signal at the onsets in each frequency band. A sentence recognition task in stationary speech shaped noise showed a significant speech reception threshold (SRT) improvement of 2.5 dB for the EE in comparison to the reference continuous interleaved sampling strategy and of 1.7 dB when an ideal Wiener filter was used for the onset extraction on the noisy signal. In a competitive talker condition, a significant SRT improvement of 2.6 dB was measured. A benefit was obtained in all experiments with the peak signal derived from the clean speech. Although the EE strategy is not effective in many real-life situations, the results suggest that there is potential for speech intelligibility improvement when an enhancement of the onsets of the speech envelope is included in the signal processing of auditory prostheses.

Prediction and control of neural responses to pulsatile electrical stimulation

This paper aims to predict and control the probability of firing of a neuron in response to pulsatile electrical stimulation of the type delivered by neural prostheses such as the cochlear implant, bionic eye or in deep brain stimulation. Using the cochlear implant as a model, we developed an efficient computational model that predicts the responses of auditory nerve fibers to electrical stimulation and evaluated the model's accuracy by comparing the model output with pooled responses from a group of guinea pig auditory nerve fibers. It was found that the model accurately predicted the changes in neural firing probability over time to constant and variable amplitude electrical pulse trains, including speech-derived signals, delivered at rates up to 889 pulses s(-1). A simplified version of the model that did not incorporate adaptation was used to adaptively predict, within its limitations, the pulsatile electrical stimulus required to cause a desired response from neurons up to 250 pulses s(-1). Future stimulation strategies for cochlear implants and other neural prostheses may be enhanced using similar models that account for the way that neural responses are altered by previous stimulation.

Principles of design and biological approaches for improving the selectivity of cochlear implant electrodes

The perceptual performance of cochlear implant recipients seems to have reached a plateau in recent years. This may be attributable to inadequate neural selectivity of available intracochlear electrodes, caused by current spread and electrode interactions. Attempts to improve electrode selectivity have included manipulating the number and configuration of electrodes that are stimulated at any one time, displacing perilymph from the cochlea to restrict current flow along the cochlea, and reducing the distance between electrodes and neurons. One experimental approach by which the distance between neurons and electrodes may be reduced is to use neurotrophic factors to promote the regeneration of the peripheral dendrites of auditory neurons and guide them towards intracochlear electrodes. The likely requirements of a system for regenerating auditory neurons towards the cochlear electrode include either a stable release of neurotrophin, or transient neurotrophin followed by electrical stimulation; a close proximity of electrode to osseous spiral lamina or a polymer to bridge the gap between the two; guidance signals to attract neurons towards the electrode; patterning of the electrode surface to direct dendrites to electrode contacts and a 'stop' signal to arrest regeneration once the electrode has been reached.

An improved speech processing strategy for cochlear implants based on an active nonlinear filterbank model of the biological cochlea

The purpose of this study was to improve the speech processing strategy for cochlear implants (CIs) based on a nonlinear time-varying filter model of a biological cochlea. The level-dependent frequency response characteristic of the basilar membrane is known to produce robust formant representation and speech perception in noise. A dual resonance nonlinear (DRNL) model was adopted because it is simpler than other adaptive nonlinear models of the basilar membrane and can be readily incorporated into the CI speech processor. Spectral analysis showed that formant information is more saliently represented at the output of the proposed CI speech processor compared to the conventional strategy in noisy conditions. Acoustic simulation and hearing experiments showed that the DRNL-based nonlinear strategy improves speech performance in a speech-spectrum-shaped noise.

Effects of cochlear implant processing and fundamental frequency on the intelligibility of competing sentences

Speech perception in the presence of another competing voice is one of the most challenging tasks for cochlear implant users. Several studies have shown that (1) the fundamental frequency (F0) is a useful cue for segregating competing speech sounds and (2) the F0 is better represented by the temporal fine structure than by the temporal envelope. However, current cochlear implant speech processing algorithms emphasize temporal envelope information and discard the temporal fine structure. In this study, speech recognition was measured as a function of the F0 separation of the target and competing sentence in normal-hearing and cochlear implant listeners. For the normal-hearing listeners, the combined sentences were processed through either a standard implant simulation or a new algorithm which additionally extracts a slowed-down version of the temporal fine structure (called Frequency-Amplitude-Modulation-Encoding). The results showed no benefit of increasing F0 separation for the cochlear implant or simulation groups. In contrast, the new algorithm resulted in gradual improvements with increasing F0 separation, similar to that found with unprocessed sentences. These results emphasize the importance of temporal fine structure for speech perception and demonstrate a potential remedy for difficulty in the perceptual segregation of competing speech sounds.

Cochlear implants: current status

This article reviews the current and evolving status of cochlear implants, including technology, design, candidacy, outcomes and future directions. It examines these variables in terms of their present and future impact on clinical outcomes in pediatric and adult populations.

Effects of Vowel Context on the Recognition of Initial and Medial Consonants by Cochlear Implant Users

OBJECTIVE

Scores on consonant-recognition tests are widely used as an index of speech-perception ability in cochlear implant (CI) users. The consonant stimuli in these tests are typically presented in the /alpha/ vowel context, even though consonants in conversational speech occur in many other contexts. For this reason, it would be useful to know whether vowel context has any systematic effect on consonant recognition in this population. The purpose of the present study was to compare consonant recognition for the /alpha, i/, and /u/ vowel contexts for consonants presented in both initial (Cv) and medial (vCv) positions.

DESIGN

Twenty adult CI users with one of three different implanted devices underwent consonant-confusion testing. Twelve stimulus conditions that differed according to vowel context (/alpha, i, u/), consonant position (Cv, vCv), and talker gender (male, female) were assessed in each subject.

RESULTS

Mean percent-correct consonant-recognition scores were slightly (5 to 8%) higher for the /alpha/ and /u/ vowel contexts than for the /i/ vowel context for both initial and medial consonants. This general pattern was observed for both male and female talkers, for subjects with better and poorer average consonant-recognition performance, and for subjects using low, medium, and high stimulation rates in their speech processors. In contrast to the mean data, many individual subjects demonstrated large effects of vowel context. For 10 of 20 subjects, consonant-recognition scores varied by 15% or more across vowel contexts in one or more stimulus conditions. Similar to the mean data, these differences generally reflected better performance for the /alpha/ and /u/ vowel contexts than for the /i/ vowel context. An analysis of consonant features showed that overall performance was best for the voicing feature, followed by the manner and place features, and that the place feature showed the strongest effect of vowel context. Vowel-context effects were strongest for the six consonants /d, j, n, k, m/, and /l/. For three of these consonants (/j, n, k/), the back vowels /alpha/ and /u/ produced substantially (30 to 35%) higher mean scores than the front vowel /i/. For each of the remaining three consonants, a unique pattern was observed in which a different single vowel produced substantially higher scores than the others. Several additional consonants (/s, g, w, b/, and /d/) showed strong context effects in either the initial consonant or medial consonant position. Overall, voiceless stop, nasal, and glide-liquid consonants showed the strongest effects of vowel context, whereas the voiceless fricative and voiceless affricate consonants were least affected. Consistent with the feature analysis, a qualitative assessment of phoneme errors for the six key consonants indicated that vowel-context effects stem primarily from changes in the number of place-of-articulation errors made in each context.

CONCLUSIONS

Vowel context has small but significant effects on consonant-recognition scores for the "average" CI listener, with the back vowels /alpha/ and /u/ producing better performance than the front vowel /i/. In contrast to the average results, however, the effects of vowel context are sizable in some individual subjects. This suggests that it may be beneficial to assess consonant recognition using two vowels, such as /alpha/ and /i/, which produce better and poorer performance, respectively. The present results underscore previous findings that poor transmission of spectral speech cues limits consonant-recognition performance in CI users. Spectral cue transmission may be hindered not only by poor spectral resolution in these listeners but also by the brief duration and dynamic nature of consonant place-of-articulation cues.

Temporal masking in electric hearing

Temporal masking can be defined as the detection threshold of a brief signal as a function of the signal delay in a relatively long masker. The temporal masking pattern in normal acoustic hearing reveals temporal edge enhancement in which the signal detection threshold is greater near the masker onset than in the steady-state portion. Both peripheral and central mechanisms appear to underlie temporal edge enhancement, but their relative contributions remain elusive. Cochlear implants bypass cochlear mechanical processing and stimulate the auditory nerve directly, thereby providing a unique opportunity to separate the peripheral mechanisms from the central mechanisms. Here, we systematically measured temporal masking in electric hearing by examining whether a brief signal was harder to detect at the onset than in the steady-state portion of a long masker (the "overshoot" effect). The signal and the masker were presented (1) either to the same electrode or to different electrodes, (2) at the same stimulation or different rates, and (3) in a simultaneous or an interleaved fashion. A consistent pattern of results was observed, depending on the stimulus configuration between the signal and the masker. Simultaneous stimulation at the same rate and with the same electrode produced no difference in sensitivity between the onset and the steady-state conditions, but interleaved stimulation at different rates or with different electrodes produced a significant difference. Unlike acoustic hearing, high masker levels produced an overshoot effect, and low masker levels produced an undershoot effect. Although the present results are consistent with the "on-frequency vs. off-frequency" hypothesis for the overshoot effect, results also suggest a central "same vs. different" mechanism underlying temporal masking. These results have practical implications for improving cochlear implant design.

A flexible auditory research platform using acoustic or electric stimuli for adults and young children

A user-friendly and versatile research platform for use in auditory experiments, referred to as APEX (Application for PsychoElectrical eXperiments), is described. The platform takes care of automatic stimulus presentation and collection of the subject's responses. Acoustical auditory, as well as electrical auditory experiments with CI recipients can be conducted. The platform currently supports LAURA, Nucleus CI22 and Nucleus CI24 cochlear implants. The graphical user interface for the subjects has been extended to allow for testing very young children, by embedding the psychophysical procedures in a computer game. The research platform is available free of charge.

Mandarin tone recognition in cochlear-implant subjects

This study examined tone recognition in five cochlear-implant subjects who were native speakers of Mandarin and used a Nucleus-22 device. Psychophysical experiments were conduced to measure rate discrimination in individual electrodes from the most apical to the most basal electrodes. The rate range was from 100 to 200 Hz, which corresponded to the range of variation in fundamental frequency for the tonal tokens used in this study. Speech recognition experiments were also conducted to measure tone recognition as function of the number of electrodes from a 1-electrode map to a 20-electrode map. Large individual variability was observed for both rate discrimination and tone recognition result: Average rate discrimination ranged between 0.2 and 1.2 (Weber's fraction) whereas tone recognition ranged between 30% and 70% correct. A highly significant correlation was found between rate discrimination and tone recognition with the 20-electrode map, but a non-significant correlation was observed with the 1-electrode map due to a floor effect in tone recognition. The present result supports the hypothesis that both spectral and temporal cues contribute to tone recognition. In addition, the present result shows that current cochlear-implant subjects produced significantly lower performance than acoustic simulations in normal-hearing subjects, suggesting that neither temporal nor spectral cues have been adequately and appropriately extracted and encoded in current cochlear implants. New designs are discussed to improve tone recognition in cochlear implant subjects.

Reception of Environmental Sounds Through Cochlear Implants

OBJECTIVE

The objective of this study was to measure the performance of persons with cochlear implants on a test of environmental-sound reception.

DESIGN

The reception of environmental sounds was studied using a test employing closed sets of 10 sounds in each of four different settings (General Home, Kitchen, Office, and Outside). The participants in the study were 11 subjects with cochlear implants. Identification testing was conducted under each of the four closed sets of stimuli using a one-interval, 10-alternative, forced-choice procedure. The data were summarized in terms of overall percent correct identification scores and information transfer (IT) in bits. Confusion patterns were described using a hierarchical-clustering analysis. In addition, individual performance on the environmental-sound task was related to the ability to recognize isolated words through the cochlear implant alone.

RESULTS

Levels of performance were similar across the four stimulus sets. Mean scores across subjects ranged from 45.3% correct (and IT of 1.5 bits) to 93.8% correct (and IT of 3.1 bits). Performance on the environmental-sound identification test was roughly related to NU-6 word recognition ability. Specifically, those subjects with word scores greater than 34% correct performed at levels of 80 to 94% on environmental-sound recognition, whereas subjects with word scores less than 34% had greater difficulty on the task. Results of the hierarchical clustering analysis, conducted on two groups of subjects (a high-performing [HP] group and a low-performing [LP] group), indicated that confusions were confined to three or four specific stimuli for the HP subjects and that larger clusters of confused stimuli were observed in the data of the LP group. Signals with distinct temporal-envelope characteristics were easily perceived by all subjects, and confused items tended to share similar overall durations and temporal envelopes.

CONCLUSIONS

Temporal-envelope cues appear to play a large role in the identification of environmental sounds through cochlear implants. The finer distinctions made by the HP group compared with the LP group may be related to a better ability both to resolve temporal differences and to use gross spectral cues. These findings are qualitatively consistent with patterns of confusions observed in the reception of speech segments through cochlear implants.

How cochlear implants encode speech

PURPOSE OF REVIEW

This review summarizes the history of cochlear implant signal processing and provides the rationale underlying current approaches. Present strategies are explained and recent research findings are summarized. It is suggested how these results may drive future advancements in signal processing.

RECENT FINDINGS

Substantial advances have been made in our understanding of the spectral and temporal cues necessary for cochlear implant recipients to perceive music, speech in noise, and interaural timing. It is clear that higher levels of both spectral and temporal resolution, as well as better loudness and pitch coding are necessary for higher levels of performance. These factors are highly interrelated, however, and are beneficial for differing aspects of hearing. Signal processing algorithms incorporating these findings are under active development and some are currently undergoing clinical investigation.

SUMMARY

Current implant devices, and those soon to be available, have substantial untapped potential to improve the auditory experience of their recipients. It is likely that in the near future, recent findings on pitch and loudness perception, as well as techniques to better emulate the normal functions of the cochlea will result in much higher levels of prosthetic hearing fidelity than are possible today. As the performance of these remarkable devices continues to improve, the population of hearing-impaired individuals who can benefit from implantation is likely to increase significantly.

Cochlear implants: some likely next steps

The history of cochlear implants is marked by large improvements in performance, especially over the past two decades and especially due to the development of ever-better processing strategies. Although the progress to date has been substantial, present devices still do not restore normal speech reception, even for top performers and particularly for listening to speech in competition with noise or other talkers. In addition, a wide range of outcomes persists, with some patients receiving little benefit using the same devices that support high levels of speech reception for others. The purpose of this review is to describe some likely possibilities for further improvement, including (a) combined electric and acoustic stimulation of the auditory system for patients with significant residual hearing, (b) use of bilateral implants, (c) a closer replication with implants of the processing steps in the normal cochlea, and (d) applications of knowledge about factors that are correlated with outcomes to help patients presently at the low end of the performance scale.

Effects of stimulation rates on Cantonese lexical tone perception by cochlear implant users in Hong Kong

High, moderate and low stimulation rates of 1800, 800 and 400 pulse per second (pps)/channel, respectively, were used to test the effects of stimulation rates on the discrimination and identification of Cantonese lexical tones in 11 Chinese post-lingually deafened adults with cochlear implants (CIs). The subjects were implanted with the MED-EL Combi 40+ CI system. They were randomly assigned to each of the stimulation rate conditions according to an ABC design. In both the Cantonese lexical tone perception tests, the subjects reached the highest scores in the high-stimulation-rate condition, and the lowest scores in the low-stimulation-rate condition (P < 0.01). Post hoc comparisons between different stimulation rates did not yield consistent results. This study demonstrated that the maximum stimulation rate of 1800 pps/channel could be an 'optimal' stimulation rate and an informed choice of parameter for the benefit of Cantonese-speaking CI users in lexical tone perception.

Emphasis of short-duration acoustic speech cues for cochlear implant users

A new speech-coding strategy for cochlear implant users, called the transient emphasis spectral maxima (TESM), was developed to aid perception of short-duration transient cues in speech. Speech-perception scores using the TESM strategy were compared to scores using the spectral maxima sound processor (SMSP) strategy in a group of eight adult users of the Nucleus 22 cochlear implant system. Significant improvements in mean speech-perception scores for the group were obtained on CNC open-set monosyllabic word tests in quiet (SMSP: 53.6% TESM: 61.3%, p<0.001), and on MUSL open-set sentence tests in multitalker noise (SMSP: 64.9% TESM: 70.6%, p<0.001). Significant increases were also shown for consonant scores in the word test (SMSP: 75.1% TESM: 80.6%, p<0.001) and for vowel scores in the word test (SMSP: 83.1% TESM: 85.7%, p<0.05). Analysis of consonant perception results from the CNC word tests showed that perception of nasal, stop, and fricative consonant discrimination was most improved. Information transmission analysis indicated that place of articulation was most improved, although improvements were also evident for manner of articulation. The increases in discrimination were shown to be related to improved coding of short-duration acoustic cues, particularly those of low intensity.

A concept for a research tool for experiments with cochlear implant users

APEX, an acronym for computer Application for Psycho-Electrical eXperiments, is a user friendly tool used to conduct psychophysical experiments and to investigate new speech coding algorithms with cochlear implant users. Most common psychophysical experiments can be easily programmed and all stimuli can be easily created without any knowledge of computer programing. The pulsatile stimuli are composed off-line using custom-made MATLAB (Registered trademark of The Mathworks, Inc., http://www.mathworks.com) functions and are stored on hard disk or CD ROM. These functions convert either a speech signal into a pulse sequence or generate any sequence of pulses based on the parameters specified by the experimenter. The APEX personal computer (PC) software reads a text file which specifies the experiment and the stimuli, controls the experiment, delivers the stimuli to the subject through a digital signal processor (DSP) board, collects the responses via a computer mouse or a graphics tablet, and writes the results to the same file. At present, the APEX system is implemented for the LAURA (Registered trademark of Philips Hearing Implants) cochlear implant. However, the concept-and many parts of the system-is portable to any other device. Also, psycho-acoustical experiments can be conducted by presenting the stimuli acoustically through a sound card.