Constraints on ideal binary masking for the perception of spectrally-reduced speech

Experimental Investigation of Acoustic Features to Optimize Intelligibility in Cochlear Implants

Although cochlear implants work well for people with hearing impairment in quiet conditions, it is well-known that they are not as effective in noisy environments. Noise reduction algorithms based on machine learning allied with appropriate speech features can be used to address this problem. The purpose of this study is to investigate the importance of acoustic features in such algorithms. Acoustic features are extracted from speech and noise mixtures and used in conjunction with the ideal binary mask to train a deep neural network to estimate masks for speech synthesis to produce enhanced speech. The intelligibility of this speech is objectively measured using metrics such as Short-time Objective Intelligibility (STOI), Hit Rate minus False Alarm Rate (HIT-FA) and Normalized Covariance Measure (NCM) for both simulated normal-hearing and hearing-impaired scenarios. A wide range of existing features is experimentally evaluated, including features that have not been traditionally applied in this application. The results demonstrate that frequency domain features perform best. In particular, Gammatone features performed best for normal hearing over a range of signal-to-noise ratios and noise types (STOI = 0.7826). Mel spectrogram features exhibited the best overall performance for hearing impairment (NCM = 0.7314). There is a stronger correlation between STOI and NCM than HIT-FA and NCM, suggesting that the former is a better predictor of intelligibility for hearing-impaired listeners. The results of this study may be useful in the design of adaptive intelligibility enhancement systems for cochlear implants based on both the noise level and the nature of the noise (stationary or non-stationary).

Multi-Task Learning U-Net for Single-Channel Speech Enhancement and Mask-Based Voice Activity Detection

In this paper, a multi-task learning U-shaped neural network (MTU-Net) is proposed and applied to single-channel speech enhancement (SE). The proposed MTU-based SE method estimates an ideal binary mask (IBM) or an ideal ratio mask (IRM) by extending the decoding network of a conventional U-Net to simultaneously model the speech and noise spectra as the target. The effectiveness of the proposed SE method was evaluated under both matched and mismatched noise conditions between training and testing by measuring the perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI). Consequently, the proposed SE method with IRM achieved a substantial improvement with higher average PESQ scores by 0.17, 0.52, and 0.40 than other state-of-the-art deep-learning-based methods, such as the deep recurrent neural network (DRNN), SE generative adversarial network (SEGAN), and conventional U-Net, respectively. In addition, the STOI scores of the proposed SE method are 0.07, 0.05, and 0.05 higher than those of the DRNN, SEGAN, and U-Net, respectively. Next, voice activity detection (VAD) is also proposed by using the IRM estimated by the proposed MTU-Net-based SE method, which is fundamentally an unsupervised method without any model training. Then, the performance of the proposed VAD method was compared with the performance of supervised learning-based methods using a deep neural network (DNN), a boosted DNN, and a long short-term memory (LSTM) network. Consequently, the proposed VAD methods show a slightly better performance than the three neural network-based methods under mismatched noise conditions.

The importance of processing resolution in "ideal time-frequency segregation" of masked speech and the implications for predicting speech intelligibility

Ideal time-frequency segregation (ITFS) is a signal processing technique that may be used to estimate the energetic and informational components of speech-on-speech masking. A core assumption of ITFS is that it roughly emulates the effects of energetic masking (EM) in a speech mixture. Thus, when speech identification thresholds are measured for ITFS-processed stimuli and compared to thresholds for unprocessed stimuli, the difference can be attributed to informational masking (IM). Interpreting this difference as a direct metric of IM, however, is complicated by the fine time-frequency (T-F) resolution typically used during ITFS, which may yield target "glimpses" that are too narrow/brief to be resolved by the ear in the mixture. Estimates of IM, therefore, may be inflated because the full effects of EM are not accounted for. Here, T-F resolution was varied during ITFS to determine if/how estimates of IM depend on processing resolution. Speech identification thresholds were measured for speech and noise maskers after ITFS. Reduced frequency resolution yielded poorer thresholds for both masker types. Reduced temporal resolution did so for noise maskers only. Results suggest that processing resolution strongly influences estimates of IM and implies that current approaches to predicting masked speech intelligibility should be modified to account for IM.

Prediction of Individual Cochlear Implant Recipient Speech Perception With the Output Signal to Noise Ratio Metric

OBJECTIVES

A cochlear implant (CI) implements a variety of sound processing algorithms that seek to improve speech intelligibility. Typically, only a small number of parameter combinations are evaluated with recipients but the optimal configuration may differ for individuals. The present study evaluates a novel methodology which uses the output signal to noise ratio (OSNR) to predict complete psychometric functions that relate speech recognition to signal to noise ratio for individual CI recipients.

DESIGN

Speech scores from sentence-in-noise tests in a "reference" condition were mapped to OSNR and a psychometric function was fitted. The reference variability was defined as the root mean square error between the reference scores and the fitted curve. To predict individual scores in a different condition, OSNRs in that condition were calculated and the corresponding scores were read from the reference psychometric function. In a retrospective experiment, scores were predicted for each condition and subject in three existing data sets of sentence scores. The prediction error was defined as the root mean square error between observed and predicted scores. In data set 1, sentences were mixed with 20 talker babble or speech weighted noise and presented at 65 dB sound pressure level (SPL). An adaptive test procedure was used. Sound processing was advanced combinatorial encoding (ACE, Cochlear Limited) and ACE with ideal binary mask processing, with five different threshold settings. In data set 2, sentences were mixed with speech weighted noise, street-side city noise or cocktail party noise and presented at 65 dB SPL. An adaptive test procedure was used. Sound processing was ACE and ACE with two different noise reduction schemes. In data set 3, sentences were mixed with four-talker babble at two input SNRs and presented at levels of 55-89 dB SPL. Sound processing utilised three different automatic gain control configurations.

RESULTS

For data set 1, the median of individual prediction errors across all subjects, noise types and conditions, was 12% points, slightly better than the reference variability. The OSNR prediction method was inaccurate for the specific condition with a gain threshold of +10 dB. For data set 2, the median of individual prediction errors was 17% points and the reference variability was 11% points. For data set 3, the median prediction error was 9% points and the reference variability was 7% points. A Monte Carlo simulation found that the OSNR prediction method, which used reference scores and OSNR to predict individual scores in other conditions, was significantly more accurate (p < 0.01) than simply using reference scores as predictors.

CONCLUSIONS

The results supported the hypothesis that the OSNR prediction method could accurately predict individual recipient scores for a range of algorithms and noise types, for all but one condition. The medians of the individual prediction errors for each data set were accurate within 6% points of the reference variability and compared favourably with prediction methodologies in other recent studies. Overall, the novel OSNR-based prediction method shows promise as a tool to assist researchers and clinicians in the development or fitting of CI sound processors.