Interaction Between Pitch and Timbre Perception in Normal-Hearing Listeners and Cochlear Implant Users

Reduced Channel Interaction Improves Timbre Recognition Under Vocoder Simulation of Cochlear Implant Processing

OBJECTIVE

This study aimed to investigate the influence of the number of channels and channel interaction on timbre perception in cochlear implant (CI) processing. By utilizing vocoder simulations of CI processing, the effects of different numbers of channels and channel interaction were examined to assess their impact on timbre perception, an essential aspect of music and auditory performance.

STUDY DESIGN, SETTING, AND PATIENTS

Fourteen CI recipients, with at least 1 year of CI device use, and two groups (N = 16 and N = 19) of normal hearing (NH) participants completed a timbre recognition (TR) task. NH participants were divided into two groups, with each group being tested on different aspects of the study. The first group underwent testing with varying numbers of channels (8, 12, 16, and 20) to determine an ideal number that closely reflected the TR performance of CI recipients. Subsequently, the second group of NH participants participated in the assessment of channel interaction, utilizing the identified ideal number of 20 channels, with three conditions: low interaction (54 dB/octave), medium interaction (24 dB/octave), and high interaction (12 dB/octave). Statistical analyses, including repeated-measures analysis of variance and pairwise comparisons, were conducted to examine the effects.

RESULTS

The number of channels did not demonstrate a statistically significant effect on TR in NH participants ( p > 0.05). However, it was observed that the condition with 20 channels closely resembled the TR performance of CI recipients. In contrast, channel interaction exhibited a significant effect ( p < 0.001) on TR. Both the low interaction (54 dB/octave) and high interaction (12 dB/octave) conditions differed significantly from the actual CI recipients' performance.

CONCLUSION

Timbre perception, a complex ability reliant on highly detailed spectral resolution, was not significantly influenced by the number of channels. However, channel interaction emerged as a significant factor affecting timbre perception. The differences observed under different channel interaction conditions suggest potential mechanisms, including reduced spectro-temporal resolution and degraded spectral cues. These findings highlight the importance of considering channel interaction and optimizing CI processing strategies to enhance music perception and overall auditory performance for CI recipients.

The Impact of Musical Rehabilitation on Complex Sound Perception in Cochlear Implant Users: A Systematic Review

OBJECTIVE

Musical rehabilitation has been used in clinical and nonclinical contexts to improve postimplantation auditory processing in implanted individuals. This systematic review aimed to evaluate the efficacy of music rehabilitation in controlled experimental and quasi-experimental studies on cochlear implant (CI) user speech and music perception.

DATABASES REVIEWED

PubMed/MEDLINE, EMBASE, Web of Science, PsycARTICLES, and PsycINFO databases through July 2022.

METHODS

Controlled experimental trials and prospective studies were included if they compared pretest and posttest data and excluded hearing aid-only users. Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were then used to extract data from 11 included studies with a total of 206 pediatric and adult participants. Interventions included group music therapy, melodic contour identification training, auditory-motor instruction, or structured digital music training. Studies used heterogeneous outcome measures evaluating speech and music perception. Risk of bias was assessed using the National Heart, Lung, and Blood Institute Quality Assessment Tool.

RESULTS

A total of 735 studies were screened, and 11 met the inclusion criteria. Six trials reported both speech and music outcomes, whereas five reported only music perception outcomes after the intervention relative to control. For music perception outcomes, significant findings included improvements in melodic contour identification (five studies, p < 0.05), timbre recognition (three studies, p < 0.05), and song appraisal (three studies, p < 0.05) in their respective trials. For speech prosody outcomes, only vocal emotion identification demonstrated significant improvements (two studies, p < 0.05).

CONCLUSION

Music rehabilitation improves performance on multiple measures of music perception, as well as tone-based characteristics of speech (i.e., emotional prosody). This suggests that rehabilitation may facilitate improvements in the discrimination of spectrally complex signals.

Listeners' Perception of Vocal Effects During Singing

INTRODUCTION

Effective communication is a key feature of vocal music. Singers can communicate during singing by changing their voice qualities to express emotion. Varying acceptable standards are used by performers for voice quality secondary to musical genre. Types of voice qualities that are historically perceived as abusive by some teachers of singing (ToS) and speech-language pathologists (SLPs) are vocal effects. This study investigates the perceptions of vocal effects in professional and nonprofessional listeners (NPLs).

METHODS

Participants (n = 100) completed an online survey. Participants were divided into four professional groups; Classical ToS, Contemporary ToS, SLPs, and NPLs. Participants completed an identification task to assess their ability to identify the use of a vocal effect. Secondly, participants analyzed a singer performing a vocal effect, rated their preferences towards the effect, and gave objective performance ratings using a Likert scale. Finally, participants were asked if they had concerns about the singer's voice. If the participant responded yes, they were asked who they would refer the singer to, a SLP, ToS or medical doctor (MD).

RESULTS

Statistically significant differences were observed in SLPs ability to identify the use of vocal effects compared to classical ToS (P = 0.01), contemporary ToS (P = 0.001) as well as NPLs compared to contemporary ToS (P = 0.009). NPLs were reported to have a lesser rate of concern statistically compared to professional listeners (P = .006). Statically significant differences were found when comparing performance rating scores secondary to preference for the vocal effect when comparisons were larger than one Likert rating interval. With listeners giving higher performance ratings, if they reported higher preference ratings. Finally, no significant differences were identified when comparing referral scores secondary to occupation.

CONCLUSIONS

Findings provide support for the presence of specific biases towards the use of vocal effects although no bias was found in management and care recommendations. Future research is recommended to investigate the nature of these biases.

Cochlear-implant Mandarin tone recognition with a disyllabic word corpus

Despite pitch being considered the primary cue for discriminating lexical tones, there are secondary cues such as loudness contour and duration, which may allow some cochlear implant (CI) tone discrimination even with severely degraded pitch cues. To isolate pitch cues from other cues, we developed a new disyllabic word stimulus set (Di) whose primary (pitch) and secondary (loudness) cue varied independently. This Di set consists of 270 disyllabic words, each having a distinct meaning depending on the perceived tone. Thus, listeners who hear the primary pitch cue clearly may hear a different meaning from listeners who struggle with the pitch cue and must rely on the secondary loudness contour. A lexical tone recognition experiment was conducted, which compared Di with a monosyllabic set of natural recordings. Seventeen CI users and eight normal-hearing (NH) listeners took part in the experiment. Results showed that CI users had poorer pitch cues encoding and their tone recognition performance was significantly influenced by the “missing” or “confusing” secondary cues with the Di corpus. The pitch-contour-based tone recognition is still far from satisfactory for CI users compared to NH listeners, even if some appear to integrate multiple cues to achieve high scores. This disyllabic corpus could be used to examine the performance of pitch recognition of CI users and the effectiveness of pitch cue enhancement based Mandarin tone enhancement strategies. The Di corpus is freely available online: https://github.com/BetterCI/DiTone.

Music and Speech Perception in Prelingually Deafened Young Listeners With Cochlear Implants: A Preliminary Study Using Sung Speech

PURPOSE

In the context of music and speech perception, this study aimed to assess the effect of variation in one of two auditory attributes-pitch contour and timbre-on the perception of the other in prelingually deafened young cochlear implant (CI) users, and the relationship between pitch contour perception and two cognitive functions of interest.

METHOD

Nine prelingually deafened CI users, aged 8.75-22.17 years, completed a melodic contour identification (MCI) task using stimuli of piano notes or sung speech with a fixed timbre (same word for each note) or a mixed timbre (different words for each note), a speech perception task identifying matrix-styled sentences naturally intonated or sung with a fixed pitch (same pitch for each word) or a mixed pitch (different pitches for each word), a forward digit span test indexing auditory short-term memory (STM), and the matrices section of the Kaufman Brief Intelligence Test-Second Edition indexing nonverbal IQ.

RESULTS

MCI was significantly poorer for the mixed timbre condition. Speech perception was significantly poorer for the fixed and mixed pitch conditions than for the naturally intonated condition. Auditory STM positively correlated with MCI at 2- and 3-semitone note spacings. Relative to their normal-hearing peers from a related study using the same stimuli and tasks, the CI participants showed comparable MCI at 2- or 3-semitone note spacing, and a comparable level of significant decrement in speech perception across three pitch contour conditions.

CONCLUSION

Findings suggest that prelingually deafened CI users show similar trends of normal-hearing peers for the effect of variation in pitch contour or timbre on the perception of the other, and that cognitive functions may underlie these outcomes to some extent, at least for the perception of pitch contour.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21217937.

Combining Place and Rate of Stimulation Improves Frequency Discrimination in Cochlear Implant Users

In the auditory system, frequency is represented as tonotopic and temporal response properties of the auditory nerve. While these response properties are inextricably linked in normal hearing, cochlear implants can separately excite tonotopic location and temporal synchrony using different electrodes and stimulation rates, respectively. This separation allows for the investigation of the contributions of tonotopic and temporal cues for frequency discrimination. The present study examines frequency discrimination in adult cochlear implant users as conveyed by electrode position and stimulation rate, separately and combined. The working hypothesis is that frequency discrimination is better provided by place and rate cues combined compared to either cue alone. This hypothesis was tested in two experiments. In the first experiment, frequency discrimination needed for melodic contour identification was measured for frequencies near 100, 200, and 400 Hz using frequency allocation modeled after clinical processors. In the second experiment, frequency discrimination for pitch ranking was measured for frequencies between 100 and 1600 Hz using an experimental frequency allocation designed to provide better access to place cues. The results of both experiments indicate that frequency discrimination is better with place and rate cues combined than with either cue alone. These results clarify how signal processing for cochlear implants could better encode frequency into place and rate of electrical stimulation. Further, the results provide insight into the contributions of place and rate cues for pitch.

Computer-based musical interval training program for Cochlear implant users and listeners with no known hearing loss

A musical interval is the difference in pitch between two sounds. The way that musical intervals are used in melodies relative to the tonal center of a key can strongly affect the emotion conveyed by the melody. The present study examines musical interval identification in people with no known hearing loss and in cochlear implant users. Pitch resolution varies widely among cochlear implant users with average resolution an order of magnitude worse than in normal hearing. The present study considers the effect of training on musical interval identification and tests for correlations between low-level psychophysics and higher-level musical abilities. The overarching hypothesis is that cochlear implant users are limited in their ability to identify musical intervals both by low-level access to frequency cues for pitch as well as higher-level mapping of the novel encoding of pitch that implants provide. Participants completed a 2-week, online interval identification training. The benchmark tests considered before and after interval identification training were pure tone detection thresholds, pure tone frequency discrimination, fundamental frequency discrimination, tonal and rhythm comparisons, and interval identification. The results indicate strong correlations between measures of pitch resolution with interval identification; however, only a small effect of training on interval identification was observed for the cochlear implant users. Discussion focuses on improving access to pitch cues for cochlear implant users and on improving auditory training for musical intervals.

Parameter-Specific Morphing Reveals Contributions of Timbre to the Perception of Vocal Emotions in Cochlear Implant Users

Objectives:

Research on cochlear implants (CIs) has focused on speech comprehension, with little research on perception of vocal emotions. We compared emotion perception in CI users and normal-hearing (NH) individuals, using parameter-specific voice morphing.

Design:

Twenty-five CI users and 25 NH individuals (matched for age and gender) performed fearful-angry discriminations on bisyllabic pseudoword stimuli from morph continua across all acoustic parameters (Full), or across selected parameters (F0, Timbre, or Time information), with other parameters set to a noninformative intermediate level.

Results:

Unsurprisingly, CI users as a group showed lower performance in vocal emotion perception overall. Importantly, while NH individuals used timbre and fundamental frequency (F0) information to equivalent degrees, CI users were far more efficient in using timbre (compared to F0) information for this task. Thus, under the conditions of this task, CIs were inefficient in conveying emotion based on F0 alone. There was enormous variability between CI users, with low performers responding close to guessing level. Echoing previous research, we found that better vocal emotion perception was associated with better quality of life ratings.

Conclusions:

Some CI users can utilize timbre cues remarkably well when perceiving vocal emotions.

The Impact of Pitch and Timbre Cues on Auditory Grouping and Stream Segregation

In multi-talker listening environments, the culmination of different voice streams may lead to the distortion of each source’s individual message, causing deficits in comprehension. Voice characteristics, such as pitch and timbre, are major dimensions of auditory perception and play a vital role in grouping and segregating incoming sounds based on their acoustic properties. The current study investigated how pitch and timbre cues (determined by fundamental frequency, notated as F0, and spectral slope, respectively) can affect perceptual integration and segregation of complex-tone sequences within an auditory streaming paradigm. Twenty normal-hearing listeners participated in a traditional auditory streaming experiment using two alternating sequences of harmonic tone complexes A and B with manipulating F0 and spectral slope. Grouping ranges, the F0/spectral slope ranges over which auditory grouping occurs, were measured with various F0/spectral slope differences between tones A and B. Results demonstrated that the grouping ranges were maximized in the absence of the F0/spectral slope differences between tones A and B and decreased by 2 times as their differences increased to ±1-semitone F0 and ±1-dB/octave spectral slope. In other words, increased differences in either F0 or spectral slope allowed listeners to more easily distinguish between harmonic stimuli, and thus group them together less. These findings suggest that pitch/timbre difference cues play an important role in how we perceive harmonic sounds in an auditory stream, representing our ability to group or segregate human voices in a multi-talker listening environment.

Reverberation Degrades Pitch Perception but Not Mandarin Tone and Vowel Recognition of Cochlear Implant Users

OBJECTIVES

The primary goal of this study was to investigate the effects of reverberation on Mandarin tone and vowel recognition of cochlear implant (CI) users and normal-hearing (NH) listeners. To understand the performance of Mandarin tone recognition, this study also measured participants' pitch perception and the availability of temporal envelope cues in reverberation.

DESIGN

Fifteen CI users and nine NH listeners, all Mandarin speakers, were asked to recognize Mandarin single-vowels produced in four lexical tones and rank harmonic complex tones in pitch with different reverberation times (RTs) from 0 to 1 second. Virtual acoustic techniques were used to simulate rooms with different degrees of reverberation. Vowel duration and correlation between amplitude envelope and fundamental frequency (F0) contour were analyzed for different tones as a function of the RT.

RESULTS

Vowel durations of different tones significantly increased with longer RTs. Amplitude-F0 correlation remained similar for the falling Tone 4 but greatly decreased for the other tones in reverberation. NH listeners had robust pitch-ranking, tone recognition, and vowel recognition performance as the RT increased. Reverberation significantly degraded CI users' pitch-ranking thresholds but did not significantly affect the overall scores of tone and vowel recognition with CIs. Detailed analyses of tone confusion matrices showed that CI users reduced the flat Tone-1 responses but increased the falling Tone-4 responses in reverberation, possibly due to the falling amplitude envelope of late reflections after the original vowel segment. CI users' tone recognition scores were not correlated with their pitch-ranking thresholds.

CONCLUSIONS

NH listeners can reliably recognize Mandarin tones in reverberation using salient pitch cues from spectral and temporal fine structures. However, CI users have poorer pitch perception using F0-related amplitude modulations that are reduced in reverberation. Reverberation distorts speech amplitude envelopes, which affect the distribution of tone responses but not the accuracy of tone recognition with CIs. Recognition of vowels with stationary formant trajectories is not affected by reverberation for both NH listeners and CI users, regardless of the available spectral resolution. Future studies should test how the relatively stable vowel and tone recognition may contribute to sentence recognition in reverberation of Mandarin-speaking CI users.

Comparisons in Frequency Difference Limens Between Sequential and Simultaneous Listening Conditions in Normal-Hearing Listeners

Purpose We compared frequency difference limens (FDLs) in normal-hearing listeners under two listening conditions: sequential and simultaneous. Method Eighteen adult listeners participated in three experiments. FDL was measured using a method of limits for comparison frequency. In the sequential listening condition, the tones were presented with a half-second time interval in between, but for the simultaneous listening condition, the tones were presented simultaneously. For the first experiment, one of four reference tones (125, 250, 500, or 750 Hz), which was presented to the left ear, was paired with one of four starting comparison tones (250, 500, 750, or 1000 Hz), which was presented to the right ear. The second and third experiments had the same testing conditions as the first experiment except with two- and three-tone complexes, comparison tones. The subjects were asked if the tones sounded the same or different. When a subject chose "different," the comparison frequency decreased by 10% of the frequency difference between the reference and comparison tones. FDLs were determined when the subjects chose "same" 3 times in a row. Results FDLs were significantly broader (worse) with simultaneous listening than with sequential listening for the two- and three-tone complex conditions but not for the single-tone condition. The FDLs were narrowest (best) with the three-tone complex under both listening conditions. FDLs broadened as the testing frequencies increased for the single tone and the two-tone complex. The FDLs were not broadened at frequencies > 250 Hz for the three-tone complex. Conclusion The results suggest that sequential and simultaneous frequency discriminations are mediated by different processes at different stages in the auditory pathway for complex tones, but not for pure tones.

Effect of instrument timbre on musical emotion recognition in normal-hearing listeners and cochlear implant users

The effect of instrument timbre (e.g., piano, trumpet, organ, and violin) on musical emotion recognition was tested in normal-hearing (NH) listeners and cochlear implant (CI) users. NH performance was best with the piano, and did not change when melodies were normalized with a fixed tempo. CI performance was significantly better with the piano and trumpet than with the violin and organ when both tempo and mode cues were preserved, but not for the tempo-normalized melodies. The sharper temporal onsets of piano and trumpet may enhance CI users' perception of tempo cues important for musical emotion recognition.

Vibrotactile Stimulation Based on the Fundamental Frequency Can Improve Melodic Contour Identification of Normal-Hearing Listeners With a 4-Channel Cochlear Implant Simulation

Cochlear implant (CI) users' poor speech recognition in noise and music perception may be both due to their limited access to pitch cues such as the fundamental frequency (F0). Recent studies showed that similar to residual low-frequency acoustic hearing, vibrotactile presentation of the F0 significantly improved speech recognition in noise of CI users. The present study tested whether F0-based vibrotactile stimulation can improve melodic contour identification (MCI) of normal-hearing listeners with acoustically simulated CI processing. Each melodic contour consisted of five musical notes with one of nine contour patterns (rising, falling, or flat in each half of the contour). The F0 of the middle note was 220 or 880 Hz, and the frequency intervals between adjacent notes were 1, 3, or 5 semitones. The F0 of each note was extracted in real time and transposed to a vibration frequency centered around 110 Hz at the right forearm top. MCI was tested in five experimental conditions (with a 4- or 8-channel CI simulation alone, vibrotactile stimulation alone, and 4- or 8-channel CI simulation plus vibrotactile stimulation), each after the same amount of brief training was provided. Results showed that discrimination of vibrotactile stimuli significantly improved from chance to near perfect as the vibration frequency interval increased from 0.25 to 3 semitones. The MCI performance with vibrotactile stimulation alone was similar to that with the 4-channel CI simulation alone, but was significantly worse than that with the 8-channel CI simulation alone. Significant improvement in MCI performance with the addition of vibrotactile stimulation was only found with the 4-channel CI simulation when the middle F0 was 880 Hz and when the frequency intervals were 3 or 5 semitones. The improvement in MCI performance with than without vibrotactile stimulation was significantly correlated with the baseline MCI performance with 4-channel CI simulation alone or with the MCI performance difference between vibrotactile stimulation and 8-channel CI simulation. Therefore, when the simulated or real CI performance is relatively poor, vibrotactile stimulation based on the F0 may improve MCI with acoustic CI simulations and perhaps in real CI users as well.