The Impact of Auditory Spectral Resolution on Listening Effort Revealed by Pupil Dilation

Speech-reading on the lips as a cognitive resource to understand speech in noise

In challenging acoustic scenarios, speech processing is often linked to listening effort, which can be described as the balance between cognitive demands and motivation to understand speech. In such conditions, people usually rely on several behavioral strategies to support speech understanding and reduce listening effort (e.g., speech-reading behavior). Still, it is not clear what cognitive mechanisms underlie the use of behavioral strategies for listening. We hypothesized that the cognitive and motivational dimensions of listening effort may also drive speech-reading strategies spontaneously adopted in challenging conditions. Normal-hearing adults (N = 64) performed audiovisual speech-recognition in noise, in combination with a concurrent mnemonic task with low vs. high working memory engagement to set cognitive demands. Motivation was manipulated between-subjects through fixed or performance-related monetary rewards. Speech-reading was tracked with eye-movement, and pupil dilation served as a physiological measure of listening effort, confirming manipulation effectiveness. We found that exerted listening effort intensifies speech-reading behavior, with motivation playing a key role in this behavioral adaptation to enhanced cognitive demands. These findings document the association between internal mental processes and behavioral adaptation in the speech domain.

Cortical temporal mismatch compensation in bimodal cochlear implant users: Selective attention decoding and pupillometry study

Bimodal cochlear implant (CI) users combine electrical stimulation from a CI in one ear with acoustic stimulation through either normal hearing or a hearing aid in the opposite ear. While this bimodal stimulation typically improves speech perception, the degree of improvement varies significantly and can sometimes result in interference effects. This variability is associated with the integration of electric and acoustic signals, which can be influenced by several factors, including temporal mismatch between the two sides. In previous work, we utilized cortical auditory evoked potentials (CAEPs) to estimate the temporal mismatch between the CI stimulation (CIS) side and the acoustic stimulation (AS) side, based on differences in N1 latencies when listening with the CIS alone and the AS alone. Building on this approach, the present study estimates individual temporal mismatch at cortical level through N1 latency of CAEPs and investigates the impact of compensating for this mismatch on speech perception. Behavioral and objective measures of speech perception were conducted in bimodal CI users under three bimodal listening conditions: clinical setting, a setting with compensated temporal mismatch between electric and acoustic stimulation and a setting with a large temporal mismatch of 50 ms between electric and acoustic stimulation. The behavioral measure consisted of a speech understanding test. Objective measures included pupillometry, electroencephalography (EEG) based on cortical auditory evoked potentials (CAEPs), EEG based on selective attention decoding including analysis of parietal alpha power. No significant effect of listening condition on behavioral speech understanding performance was observed, even for the condition with a large temporal mismatch of 50 ms. Similarly, pupillometry did not reveal a significant difference across listening conditions, although it was found to be related to behavioral speech understanding. N1P2 amplitude of CAEPs was greatest under the condition with compensated temporal mismatch. The phase-locking value of CAEPs, the temporal response function related to selective attention decoding, and parietal alpha power all showed a significant improvement when applying temporal mismatch compensation, compared to the condition with a substantial 50 ms temporal mismatch. However, these metrics did not exhibit significant effects when compared to the standard clinical setting condition. These findings emphasize that neural metrics are more sensitive than behavioral measures in detecting interaural mismatch effects. A significant enhancement of CAEPs N1P2 amplitude compared to clinical setting was observed. Other neural metrics showed a limited improvement with compensated listening condition, suggesting insufficient compensation solely in temporal domain.

Advance Contextual Clues Alleviate Listening Effort During Sentence Repair in Listeners With Hearing Aids

PURPOSE

When words are misperceived, listeners can rely on later context to repair an auditory perception, at the cost of increased effort. The current study examines whether the effort to repair a missing word in a sentence is alleviated when the listener has some advance knowledge of what to expect in the sentence.

METHOD

Sixteen adults with hearing aids and 17 with typical hearing heard sentences with a missing word that was followed by context sufficient to infer what the word was. They repeated the sentences with the missing words repaired. Sentences were preceded by visual text on the screen showing either "XXXX" (unprimed) or a priming word previewing the word that would be masked in the auditory signal. Along with intelligibility measures, pupillometry was used as an index of listening effort over the course of each trial to measure how priming influenced the effort needed to mentally repair a missing word.

RESULTS

When listeners were primed for the word that would need to be repaired in an upcoming sentence, listening effort was reduced, as indicated by pupil size returning more quickly toward baseline after the sentence was heard. Priming reduced the lingering cost of mental repair in both listener groups. For the group with hearing loss, priming also reduced the prevalence of errors on target words and words other than the target word in the sentence, suggesting that priming preserves the cognitive resources needed to process the whole sentence.

CONCLUSION

These results suggest that listeners with typical hearing and with hearing loss can benefit from priming (advance cueing) during speech recognition, to accurately repair speech and to process the speech less effortfully.

Salience-Dependent Disruption of Sustained Auditory Attention Can Be Inferred from Evoked Pupil Responses and Neural Tracking of Task-Irrelevant Sounds

Stimulus-driven attention allows us to react to relevant stimuli (and imminent danger!) outside our current focus of attention. But irrelevant stimuli can also disrupt attention, for example, during listening to speech. The degree to which sound captures attention is called salience, which can be estimated by existing, behaviorally validated, computational models (Huang and Elhilali, 2017). Here we examined whether neurophysiological responses to task-irrelevant sounds indicate the degree of distraction during a sustained-listening task and how much this depends on individual hearing thresholds. Forty-seven Danish-speaking adults (28/19 female/male; mean age, 60.1; SD, 15.9 years) with heterogenous hearing thresholds (PTA; mean, 25.5; SD, 18.0 db HL) listened to continuous speech while 1-s-long, task-irrelevant natural sounds (distractors) of varying computed salience were presented at unpredictable times and locations. Eye tracking and electroencephalography were used to estimate pupil response and neural tracking, respectively. The task-irrelevant sounds evoked a consistent pupil response (PR), distractor-tracking (DT), and a drop of target-tracking (ΔTT), and statistical modeling of these three measures within subjects showed that all three are enhanced for sounds with higher computed salience. Participants with larger PR showed a stronger drop in target tracking (ΔTT) and performed worse in target speech comprehension. We conclude that distraction can be inferred from neurophysiological responses to task-irrelevant stimuli. These results are a first step toward neurophysiological assessment of attention dynamics during continuous listening, with potential applications in hearing care diagnostics.

Relationships Between Subjective and Objective Measures of Listening Accuracy and Effort in an Online Speech-in-Noise Study

OBJECTIVES

Speech-in-noise performance is of paramount importance to daily function, and there exists a bewildering array of outcome measures to capture the many dimensions of this concept. The aim of the present study was to provide insight into how different speech-in-noise outcome measures relate to one another, how they behave under different test conditions, and how researchers or practitioners might go about selecting an outcome measure (or measures) depending on the context and focus of their enquiry.

DESIGN

An online speech-in-noise study was conducted using the Labvanced experimental platform. A total of 67 participants (42 who reported having normal hearing, 25 who said they had some degree of hearing loss) completed the Effort Assessment Scale (a self-reported measure of daily-life listening effort), followed by a sentence recognition task in which BKB sentences were presented in speech-shaped noise at signal to noise ratios (SNRs) of -8, -4, 0, +4, +8, and +20 dB. Participants were instructed to listen to each sentence and then repeat aloud what they heard. Responses were recorded through participants' webcams and later independently scored by 2 research assistants. Several outcome measures were used to tap into both accuracy and listening effort. Specifically, we examined: (1) objective intelligibility (percentage of keywords correctly repeated); (2) subjective intelligibility; (3) subjective listening effort; (4) subjective tendency to give up listening; and (5) verbal response time (VRT) extracted from the audio recordings. Data were analyzed using Bayesian statistical methods.

RESULTS

Hearing loss and age were associated with speech-in-noise outcomes. Specifically, we observed lower intelligibility (objective and subjective), higher subjective listening effort, and longer VRT (time to verbal response onset) in hearing-impaired compared with normal-hearing listeners, and reduced objective intelligibility and longer VRT in older compared with younger listeners. When moving from highly favorable to more adverse listening conditions, subjective listening effort was the first measure to show sensitivity to worsening SNR, followed by subjective intelligibility, objective intelligibility, subjective tendency to give up listening, and, finally, VRT. Participants, especially those with normal hearing, consistently underestimated their own performance.

CONCLUSIONS

The present findings offer useful insight into how different subjective and objective measures of listening accuracy and effort respond to variation in hearing status, age, and SNR. Although speech intelligibility remains a measure of primary importance, it is a sensitive measure only under adverse listening conditions, which may not be representative of everyday listening. Under more ecologically relevant listening conditions (generally speaking, at moderate, positive SNRs), listening effort becomes a crucial factor to consider to adequately describe the listening experience. VRT may provide a useful objective marker of listening effort, but caution is required to deal with measurement variability, differences in definition, and the potentially confounding effect of age.

Breathy Vocal Quality, Background Noise, and Hearing Loss: How Do These Adverse Conditions Affect Speech Perception by Older Adults?

OBJECTIVES

Although breathy vocal quality and hearing loss are both prevalent age-related changes, their combined impact on speech communication is poorly understood. This study investigated whether breathy vocal quality affected speech perception and listening effort by older listeners. Furthermore, the study examined how this effect was modulated by the adverse listening environment of background noise and the listener's level of hearing loss.

DESIGN

Nineteen older adults participated in the study. Their hearing ranged from near-normal to mild-moderate sensorineural hearing loss. Participants heard speech material of low-context sentences, with stimuli resynthesized to simulate original, mild-moderately breathy, and severely breathy conditions. Speech intelligibility was measured using a speech recognition in noise paradigm, with pupillometry data collected simultaneously to measure listening effort.

RESULTS

Simulated severely breathy vocal quality was found to reduce intelligibility and increase listening effort. Breathiness and background noise level independently modulated listening effort. The impact of hearing loss was not observed in this dataset, which can be due to the use of individualized signal to noise ratios and a small sample size.

CONCLUSION

Results from this study demonstrate the challenges of listening to speech with a breathy vocal quality. Theoretically, the findings highlight the importance of periodicity cues in speech perception in noise by older listeners. Breathy voice could be challenging to separate from the noise when the noise also lacks periodicity. Clinically, it suggests the need to address both listener- and talker-related factors in speech communication by older adults.

Through the Pupils' Lens: Multilingual Effort in First and Second Language Listening

OBJECTIVES

The present study aimed to examine the involvement of listening effort among multilinguals in their first (L1) and second (L2) languages in quiet and noisy listening conditions and investigate how the presence of a constraining context within sentences influences listening effort.

DESIGN

A group of 46 young adult Arabic (L1)-Hebrew (L2) multilinguals participated in a listening task. This task aimed to assess participants' perceptual performance and the effort they exert (as measured through pupillometry) while listening to single words and sentences presented in their L1 and L2, in quiet and noisy environments (signal to noise ratio = 0 dB).

RESULTS

Listening in quiet was easier than in noise, supported by both perceptual and pupillometry results. Perceptually, multilinguals performed similarly and reached ceiling levels in both languages in quiet. However, under noisy conditions, perceptual accuracy was significantly lower in L2, especially when processing sentences. Critically, pupil dilation was larger and more prolonged when listening to L2 than L1 stimuli. This difference was observed even in the quiet condition. Contextual support resulted in better perceptual performance of high-predictability sentences compared with low-predictability sentences, but only in L1 under noisy conditions. In L2, pupillometry showed increased effort when listening to high-predictability sentences compared with low-predictability sentences, but this increased effort did not lead to better understanding. In fact, in noise, speech perception was lower in high-predictability L2 sentences compared with low-predictability ones.

CONCLUSIONS

The findings underscore the importance of examining listening effort in multilingual speech processing and suggest that increased effort may be present in multilingual's L2 within clinical and educational settings.

Engaging effort improves efficiency during spoken word recognition in cochlear implant users

Word recognition is generally thought to be supported by an automatic process of lexical competition, at least in normal hearing young adults. When listening becomes challenging, either due to properties of the environment (noise) or the individual (hearing loss), the dynamics of lexical competition change and word recognition can feel effortful and fatiguing. In cochlear implant users, several dimensions of lexical competition have been identified that capture the timing of the onset of lexical competition (Wait-and-See), the degree to which competition is fully resolved (Sustained Activation), and how quickly lexical candidates are activated (Activation Rate). It is unclear, however, how these dimensions relate to listening effort. To address this question, a group of cochlear implant users (N = 79) completed a pupillometry task to index effort and a Visual World Paradigm task to index the dynamics of lexical competition as part of a larger battery of clinical and experimental tasks. Listeners who engaged more effort, as indexed by peak pupil size difference score, fell lower along the Wait-and-See dimension, suggesting that these listeners are engaging effort to be less Wait-and-See (or to begin the process of lexical competition earlier). Listeners who engaged effort earlier had better word and sentence recognition outcomes. The timing of effort was predicted by age and spectral fidelity, but no audiological or demographic factors predicted peak pupil size difference. The dissociation between the magnitude of engaged effort and the timing of effort suggests they perform different goals for spoken word recognition.

The time course of cognitive effort during disrupted speech

Listeners often find themselves in scenarios where speech is disrupted, misperceived, or otherwise difficult to recognise. In these situations, many individuals report exerting additional effort to understand speech, even when repairing speech may be difficult or impossible. This investigation aimed to characterise cognitive efforts across time during both sentence listening and a post-sentence retention interval by observing the pupillary response of participants with normal to borderline-normal hearing in response to two interrupted speech conditions: sentences interrupted by gaps of silence or bursts of noise. The pupillary response serves as a measure of the cumulative resources devoted to task completion. Both interruption conditions resulted in significantly greater levels of pupil dilation than the uninterrupted speech condition. Just prior to the end of a sentence, trials periodically interrupted by bursts of noise elicited greater pupil dilation than the silent-interrupted condition. Compared to the uninterrupted condition, both interruption conditions resulted in increased dilation after sentence end but before repetition, possibly reflecting sustained processing demands. Understanding pupil dilation as a marker of cognitive effort is important for clinicians and researchers when assessing the additional effort exerted by listeners with hearing loss who may use cochlear implants or hearing aids. Even when successful perceptual repair is unlikely, listeners may continue to exert increased effort when processing misperceived speech, which could cause them to miss upcoming speech or may contribute to heightened listening fatigue.

Evaluation of Replacement Hearing Aids in Cochlear Implant Candidates Using the Hearing in Noise Test (HINT) and Pupillometry

BACKGROUND/OBJECTIVES

Advances in cochlear implant (CI) technology have led to the expansion of the implantation criteria. As a result, more CI candidates may have greater residual hearing in one or two ears. Many of these candidates will perform better with a CI in one ear and a hearing aid (HA) in the other ear, the so-called bimodal solution. The bimodal solution often requires patients to switch to HAs that are compatible with the CI. However, this can be a challenging decision, not least because it remains unclear whether this impacts hearing performance. Our aim is to determine whether speech perception in noise remains unchanged or improves with new replacement HAs compared to original HAs in CI candidates with residual hearing.

METHODS

Fifty bilateral HA users (mean age 63.4; range 23-82) referred for CI were recruited. All participants received new replacement HAs. The new HAs were optimally fitted and verified using Real Ear Measurement (REM). Participants were tested with the Hearing in Noise Test (HINT), which aimed at determining the signal-to-noise ratio (SNR) required for a 70% correct word recognition score at a speech sound pressure level (SPL) of 65 dB. HINT testing was performed with both their original and new replacement HAs. During HINT, pupillometry was used to control for task engagement.

RESULTS

Replacing the original HAs with new replacement HAs after one month was not statistically significant with a mean change of SRT70 by -1.90 (95% CI: -4.69;0.89, p = 0.182) dB SNR.

CONCLUSIONS

New replacement HAs do not impact speech perception scores in CI candidates prior to the decision of cochlear implantation.

Pupillometry and perceived listening effort for cochlear implant users-a comparison of three speech-in-noise tests

OBJECTIVE

Measuring listening effort using pupillometry is challenging in cochlear implant (CI) users. We assess three validated speech tests (Matrix, LIST, and DIN) to identify the optimal speech material for measuring peak-pupil-dilation (PPD) in CI users as a function of signal-to-noise ratio (SNR).

DESIGN

Speech tests were administered in quiet and two noisy conditions, namely at the speech recognition threshold (0 dB re SRT), i.e. the SNR where speech intelligibility (SI) was 50%, and at a more favourable SNR of +6 dB re SRT. PPDs and subjective ratings of effort were obtained.

STUDY SAMPLE

Eighteen unilaterally implanted CI users.

RESULTS

LIST sentences revealed significantly different PPDs between +6 and 0 dB re SRT and DIN triplets between quiet and +6 dB re SRT. PPDs obtained with the Matrix test were independent of SNR and yielded large PPDs and high subjective ratings even in quiet.

CONCLUSIONS

PPD is a sensitive measure for listening effort when processing LIST sentences near 0 dB re SRT and when processing DIN triplets at more favourable listening conditions around +6 dB re SRT. PPDs obtained with the Matrix test were insensitive to SNR, likely because it is demanding for CI users even in quiet.

Acoustically Transparent Hearing Aids Increase Physiological Markers of Listening Effort

While hearing aids are beneficial in compensating for hearing loss and suppressing ambient noise, they may also introduce an unwanted processing burden to the listener's sensory and cognitive system. To investigate such adverse side effects, hearing aids may be set to a 'transparent mode', aiming to replicate natural hearing through the open ear as best as possible. Such transparent hearing aids have previously been demonstrated to exhibit a small but significant disadvantage in speech intelligibility, with less conclusive effects on self-rated listening effort. Here we aimed to reproduce these findings and expand them with neurophysiological measures of invested listening effort, including parietal alpha power and pupil size. Invested listening effort was measured across five task difficulties, ranging from nearly impossible to easy, with normal-hearing participants in both aided and unaided conditions. Results well reproduced a hearing aid disadvantage for speech intelligibility and subjective listening effort ratings. As to be expected, pupil size and parietal alpha power followed an inverted u-shape, peaking at moderate task difficulties (around SRT50). However, the transparent hearing aid increased pupil size and parietal alpha power at medium task demand (between SRT20 and SRT80). These neurophysiological effects were larger than those observed in speech intelligibility and subjective listening effort, respectively. The results gain plausibility by yielding a substantial association of individual pupil size and individual parietal alpha power. In sum, our findings suggest that key neurophysiological measures of invested listening effort are sensitive to the individual additional burden on speech intelligibility that hearing aid processing can introduce.

Comparison of Performance for Cochlear-Implant Listeners Using Audio Processing Strategies Based on Short-Time Fast Fourier Transform or Spectral Feature Extraction

OBJECTIVES

We compared sound quality and performance for a conventional cochlear-implant (CI) audio processing strategy based on short-time fast-Fourier transform (Crystalis) and an experimental strategy based on spectral feature extraction (SFE). In the latter, the more salient spectral features (acoustic events) were extracted and mapped into the CI stimulation electrodes. We hypothesized that (1) SFE would be superior to Crystalis because it can encode acoustic spectral features without the constraints imposed by the short-time fast-Fourier transform bin width, and (2) the potential benefit of SFE would be greater for CI users who have less neural cross-channel interactions.

DESIGN

To examine the first hypothesis, 6 users of Oticon Medical Digisonic SP CIs were tested in a double-blind design with the SFE and Crystalis strategies on various aspects: word recognition in quiet, speech-in-noise reception threshold (SRT), consonant discrimination in quiet, listening effort, melody contour identification (MCI), and subjective sound quality. Word recognition and SRTs were measured on the first and last day of testing (4 to 5 days apart) to assess potential learning and/or acclimatization effects. Other tests were run once between the first and last testing day. Listening effort was assessed by measuring pupil dilation. MCI involved identifying a five-tone contour among five possible contours. Sound quality was assessed subjectively using the multiple stimulus with hidden reference and anchor (MUSHRA) paradigm for sentences, music, and ambient sounds. To examine the second hypothesis, cross-channel interaction was assessed behaviorally using forward masking.

RESULTS

Word recognition was similar for the two strategies on the first day of testing and improved for both strategies on the last day of testing, with Crystalis improving significantly more. SRTs were worse with SFE than Crystalis on the first day of testing but became comparable on the last day of testing. Consonant discrimination scores were higher for Crystalis than for the SFE strategy. MCI scores and listening effort were not substantially different across strategies. Subjective sound quality scores were lower for the SFE than for the Crystalis strategy. The difference in performance with SFE and Crystalis was greater for CI users with higher channel interaction.

CONCLUSIONS

CI-user performance was similar with the SFE and Crystalis strategies. Longer acclimatization times may be required to reveal the full potential of the SFE strategy.

Repairing Misperceptions of Words Early in a Sentence is More Effortful Than Repairing Later Words, Especially for Listeners With Cochlear Implants

The process of repairing misperceptions has been identified as a contributor to effortful listening in people who use cochlear implants (CIs). The current study was designed to examine the relative cost of repairing misperceptions at earlier or later parts of a sentence that contained contextual information that could be used to infer words both predictively and retroactively. Misperceptions were enforced at specific times by replacing single words with noise. Changes in pupil dilation were analyzed to track differences in the timing and duration of effort, comparing listeners with typical hearing (TH) or with CIs. Increases in pupil dilation were time-locked to the moment of the missing word, with longer-lasting increases when the missing word was earlier in the sentence. Compared to listeners with TH, CI listeners showed elevated pupil dilation for longer periods of time after listening, suggesting a lingering effect of effort after sentence offset. When needing to mentally repair missing words, CI listeners also made more mistakes on words elsewhere in the sentence, even though these words were not masked. Changes in effort based on the position of the missing word were not evident in basic measures like peak pupil dilation and only emerged when the full-time course was analyzed, suggesting the timing analysis adds new information to our understanding of listening effort. These results demonstrate that some mistakes are more costly than others and incur different levels of mental effort to resolve the mistake, underscoring the information lost when characterizing speech perception with simple measures like percent-correct scores.

Sensory representations and pupil-indexed listening effort provide complementary contributions to multi-talker speech intelligibility

Multi-talker speech intelligibility requires successful separation of the target speech from background speech. Successful speech segregation relies on bottom-up neural coding fidelity of sensory information and top-down effortful listening. Here, we studied the interaction between temporal processing measured using Envelope Following Responses (EFRs) to amplitude modulated tones, and pupil-indexed listening effort, as it related to performance on the Quick Speech-in-Noise (QuickSIN) test in normal-hearing adults. Listening effort increased at the more difficult signal-to-noise ratios, but speech intelligibility only decreased at the hardest signal-to-noise ratio. Pupil-indexed listening effort and EFRs did not independently relate to QuickSIN performance. However, the combined effects of both EFRs and listening effort explained significant variance in QuickSIN performance. Our results suggest a synergistic interaction between sensory coding and listening effort as it relates to multi-talker speech intelligibility. These findings can inform the development of next-generation multi-dimensional approaches for testing speech intelligibility deficits in listeners with normal-hearing.

Increased listening effort and cochlear neural degeneration underlie behavioral deficits in speech perception in noise in normal hearing middle-aged adults

Middle-age is a critical period of rapid changes in brain function that presents an opportunity for early diagnostics and intervention for neurodegenerative conditions later in life. Hearing loss is one such early indicator linked to many comorbidities later in life. However, current clinical tests fail to capture hearing difficulties for ∼10% of middle-aged adults seeking help at hearing clinics. Cochlear neural degeneration (CND) could play a role in these hearing deficits, but our current understanding is limited by the lack of objective diagnostics and uncertainty regarding its perceptual consequences. Here, using a cross-species approach, we measured envelope following responses (EFRs) - neural ensemble responses to sound originating from the peripheral auditory pathway - in young and middle-aged adults with normal audiometric thresholds, and compared these responses to young and middle-aged Mongolian gerbils, where CND was histologically confirmed. We observed near identical changes in EFRs across species that were associated with CND. Perceptual effects measured as behavioral readouts showed deficits in the most challenging listening conditions and were associated with CND. Additionally, pupil-indexed listening effort increased even at moderate task difficulties where behavioral outcomes were matched. Our results reveal perceptual deficits in middle-aged adults driven by CND and increases in listening effort, which may result in increased listening fatigue and conversational disengagement.

Pupillometry reveals effects of pitch manipulation within and across words on listening effort and short-term memory

For individuals with hearing loss, even successful speech communication comes at a cost. Cochlear implants transmit degraded information, specifically for voice pitch, which demands extra and sustained listening effort. The current study hypothesized that abnormal pitch patterns contribute to the additional listening effort, even in non-tonal language native speaking normally hearing listeners. We manipulated the fundamental frequency (F0) within and across words, while participants listen and repeat (simple intelligibility task), or listen, repeat, and later recall (concurrent encoding task) the words. In both experiments, the F0 manipulations resulted in small changes in intelligibility but no difference in free recall or subjective effort ratings. Pupillary metrics were yet sensitive to these manipulations: pupil dilations were larger when words were monotonized (flat contour) or inverted (the natural contour flipped upside-down), and larger when successive words were organized into a melodic pattern. The most likely interpretation is that the natural or expected F0 contour of a word contributes to its identity and facilitate its matching and retrieval from the phonological representation stored in long-term memory. Consequently, degrading words' F0 contour can result in extra listening effort. Our results call for solutions to improve pitch saliency and naturalness in future development of cochlear implants' signal processing strategies, even for non-tonal languages.

Increased listening effort and decreased speech discrimination at high presentation sound levels in acoustic hearing listeners and cochlear implant users

The sounds we experience in our everyday communication can vary greatly in terms of level and background noise depending on the environment. Paradoxically, increasing the sound intensity may lead to worsened speech understanding, especially in noise. This is known as the "Rollover" phenomenon. There have been limited studies on rollover and how it is experienced differentially across aging groups, for those with and without hearing loss, as well as cochlear implant (CI) users. There is also mounting evidence that listening effort plays an important role in challenging listening conditions and can be directly quantified with objective measures such as pupil dilation. We found that listening effort was modulated by sound level and that rollover occurred primarily in the presence of background noise. The effect on listening effort was exacerbated by age and hearing loss in acoustic listeners, with greatest effect in older listeners with hearing loss, while there was no effect in CI users. The age- and hearing-dependent effects of rollover highlight the potential negative impact of amplification to high sound levels and therefore has implications for effective treatment of age-related hearing loss.

Reduced neural distinctiveness of speech representations in the middle-aged brain

Speech perception declines independent of hearing thresholds in middle-age, and the neurobiological reasons are unclear. In line with the age-related neural dedifferentiation hypothesis, we predicted that middle-aged adults show less distinct cortical representations of phonemes and acoustic-phonetic features relative to younger adults. In addition to an extensive audiological, auditory electrophysiological, and speech perceptual test battery, we measured electroencephalographic responses time-locked to phoneme instances (phoneme-related potential; PRP) in naturalistic, continuous speech and trained neural network classifiers to predict phonemes from these responses. Consistent with age-related neural dedifferentiation, phoneme predictions were less accurate, more uncertain, and involved a broader network for middle-aged adults compared with younger adults. Representational similarity analysis revealed that the featural relationship between phonemes was less robust in middle-age. Electrophysiological and behavioral measures revealed signatures of cochlear neural degeneration (CND) and speech perceptual deficits in middle-aged adults relative to younger adults. Consistent with prior work in animal models, signatures of CND were associated with greater cortical dedifferentiation, explaining nearly a third of the variance in PRP prediction accuracy together with measures of acoustic neural processing. Notably, even after controlling for CND signatures and acoustic processing abilities, age-group differences in PRP prediction accuracy remained. Overall, our results reveal "fuzzier" phonemic representations, suggesting that age-related cortical neural dedifferentiation can occur even in middle-age and may underlie speech perceptual challenges, despite a normal audiogram.

Predictors of Listening-Related Fatigue in Adolescents With Hearing Loss

PURPOSE

Self-reported listening-related fatigue in adolescents with hearing loss (HL) was investigated. Specifically, the extent to which listening-related fatigue is associated with school accommodations, audiologic characteristics, and listening breaks was examined.

METHOD

Participants were 144 adolescents with HL ages 12-19 years. Data were collected online via Qualtrics. The Vanderbilt Fatigue Scale-Child was used to measure listening-related fatigue. Participants also reported on their use of listening breaks and school accommodations, including an Individualized Education Program (IEP) or 504 plan, remote microphone systems, closed captioning, preferential seating, sign language interpreters, live transcriptions, and notetakers.

RESULTS

After controlling for age, HL laterality, and self-perceived listening difficulty, adolescents with an IEP or a 504 plan reported lower listening-related fatigue compared to adolescents without an IEP or a 504 plan. Adolescents who more frequently used remote microphone systems or notetakers reported higher listening-related fatigue compared to adolescents who used these accommodations less frequently, whereas increased use of a sign language interpreter was associated with decreased listening-related fatigue. Among adolescents with unilateral HL, higher age was associated with lower listening-related fatigue; no effect of age was found among adolescents with bilateral HL. Listening-related fatigue did not differ based on hearing device configuration.

CONCLUSIONS

Adolescents with HL should be considered at risk for listening-related fatigue regardless of the type of hearing devices used or the degree of HL. The individualized support provided by an IEP or 504 plan may help alleviate listening-related fatigue, especially by empowering adolescents with HL to be self-advocates in terms of their listening needs and accommodations in school. Additional research is needed to better understand the role of specific school accommodations and listening breaks in addressing listening-related fatigue.

A Comparison of Montreal Cognitive Assessment Scores among Individuals with Normal Hearing and Cochlear Implants

OBJECTIVES

The Montreal Cognitive Assessment (MoCA) is a cognitive screening tool that has 4 of 10 test items heavily dependent on auditory input, potentially leaving hearing-impaired (HI) individuals at a disadvantage. Previous work found that HI individuals scored lower than normal-hearing (NH) individuals on the MoCA, potentially attributed to the degraded auditory signals negatively impacting the ability to commit auditory information to memory. However, there is no research comparing how cochlear implant (CI) recipients perform on the MoCA relative to NH and HI individuals. This study aimed to (1) examine the effect of implementing three different hearing-adjusted scoring methods for a group of age-matched CI recipients and NH individuals, (2) determine if there is a difference between the two groups in overall scores and hearing-adjusted scores, and (3) compare scores across our CI and NH data to the published HI data for all scoring methods. We hypothesized that (1) scores for CI recipients would improve with implementation of the hearing-adjusted scoring methods over the original method, (2) CI recipients would score lower than NH participants for both original and adjusted scoring methods, and (3) the difference in scores between NH and CI listeners for both adjusted and unadjusted scores would be greater than that reported in the literature between NH and HI individuals due to the greater severity of hearing loss and relatively poor spectral resolution of CIs.

DESIGN

A total of 94 adults with CIs and 105 adults with NH were initially enrolled. After age-matching the two groups and excluding those who self-identified as NH but failed a hearing screening, a total of 75 CI participants (mean age 61.2 y) and 74 NH participants (mean age 58.8 y) were administered the MoCA. Scores were compared between the NH and CI groups, as well as to published HI data, using the original MoCA scoring method and three alternative scoring methods that excluded various auditory-dependent test items.

RESULTS

MoCA scores improved for all groups when two of the three alternative scoring methods were used, with no significant interaction between scoring method and group. Scores for CI recipients were significantly poorer than those for age-matched NH participants for all scoring methods. CI recipients scored better than the published data for HI individuals; however, the HI group was not age matched to the CI and NH groups.

CONCLUSIONS

MoCA scores are only partly affected by the potentially greater cognitive processing required to interpret degraded auditory signals. Even with the removal of the auditory-dependent items, CI recipients still did not perform as well as the age-matched NH group. Importantly, removing auditory-dependent items significantly and fundamentally alters the test, thereby reducing its sensitivity. This has important limitations for administration and interpretation of the MoCA for people with hearing loss.

Communication Under Sharply Degraded Auditory Input and the "2-Sentence" Problem

OBJECTIVES

Despite performing well in standard clinical assessments of speech perception, many cochlear implant (CI) users report experiencing significant difficulties when listening in real-world environments. We hypothesize that this disconnect may be related, in part, to the limited ecological validity of tests that are currently used clinically and in research laboratories. The challenges that arise from degraded auditory information provided by a CI, combined with the listener's finite cognitive resources, may lead to difficulties when processing speech material that is more demanding than the single words or single sentences that are used in clinical tests.

DESIGN

Here, we investigate whether speech identification performance and processing effort (indexed by pupil dilation measures) are affected when CI users or normal-hearing control subjects are asked to repeat two sentences presented sequentially instead of just one sentence.

RESULTS

Response accuracy was minimally affected in normal-hearing listeners, but CI users showed a wide range of outcomes, from no change to decrements of up to 45 percentage points. The amount of decrement was not predictable from the CI users' performance in standard clinical tests. Pupillometry measures tracked closely with task difficulty in both the CI group and the normal-hearing group, even though the latter had speech perception scores near ceiling levels for all conditions.

CONCLUSIONS

Speech identification performance is significantly degraded in many (but not all) CI users in response to input that is only slightly more challenging than standard clinical tests; specifically, when two sentences are presented sequentially before requesting a response, instead of presenting just a single sentence at a time. This potential "2-sentence problem" represents one of the simplest possible scenarios that go beyond presentation of the single words or sentences used in most clinical tests of speech perception, and it raises the possibility that even good performers in single-sentence tests may be seriously impaired by other ecologically relevant manipulations. The present findings also raise the possibility that a clinical version of a 2-sentence test may provide actionable information for counseling and rehabilitating CI users, and for people who interact with them closely.

The Effects of Light Level and Signal-to-Noise Ratio on the Task-Evoked Pupil Response in a Speech-in-Noise Task

PURPOSE

There is increasing interest in the measurement of cognitive effort during listening tasks, for both research and clinical purposes. Quantification of task-evoked pupil responses (TEPRs) is a psychophysiological method that can be used to study cognitive effort. However, light level during cognitively demanding listening tasks may affect TEPRs, complicating interpretation of listening-related changes. The objective of this study was to examine the effects of light level on TEPRs during effortful listening across a range of signal-to-noise ratios (SNRs).

METHOD

Thirty-six adults without hearing loss were asked to repeat target sentences presented in background babble noise while their pupil diameter was recorded. Light level and SNRs were manipulated in a 4 × 4 repeated-measures design. Repeated-measures analyses of variance were used to measure the effects.

RESULTS

Peak and mean dilation were typically larger in more adverse SNR conditions (except for SNR -6 dB) and smaller in higher light levels. Differences in mean and peak dilation between SNR conditions were larger in dim light than in brighter light.

CONCLUSIONS

Brighter light conditions make TEPRs less sensitive to variations in listening effort across levels of SNR. Therefore, light level must be considered and reported in detail to ensure sensitivity of TEPRs and for comparisons of findings across different studies. It is recommended that TEPR testing be conducted in relatively low light conditions, considering both background illumination and screen luminance.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.25676538.

The Costs (and Benefits?) of Effortful Listening for Older Adults: Insights from Simultaneous Electrophysiology, Pupillometry, and Memory

Although the impact of acoustic challenge on speech processing and memory increases as a person ages, older adults may engage in strategies that help them compensate for these demands. In the current preregistered study, older adults (n = 48) listened to sentences-presented in quiet or in noise-that were high constraint with either expected or unexpected endings or were low constraint with unexpected endings. Pupillometry and EEG were simultaneously recorded, and subsequent sentence recognition and word recall were measured. Like young adults in prior work, we found that noise led to increases in pupil size, delayed and reduced ERP responses, and decreased recall for unexpected words. However, in contrast to prior work in young adults where a larger pupillary response predicted a recovery of the N400 at the cost of poorer memory performance in noise, older adults did not show an associated recovery of the N400 despite decreased memory performance. Instead, we found that in quiet, increases in pupil size were associated with delays in N400 onset latencies and increased recognition memory performance. In conclusion, we found that transient variation in pupil-linked arousal predicted trade-offs between real-time lexical processing and memory that emerged at lower levels of task demand in aging. Moreover, with increased acoustic challenge, older adults still exhibited costs associated with transient increases in arousal without the corresponding benefits.

How to vocode: Using channel vocoders for cochlear-implant research

The channel vocoder has become a useful tool to understand the impact of specific forms of auditory degradation-particularly the spectral and temporal degradation that reflect cochlear-implant processing. Vocoders have many parameters that allow researchers to answer questions about cochlear-implant processing in ways that overcome some logistical complications of controlling for factors in individual cochlear implant users. However, there is such a large variety in the implementation of vocoders that the term "vocoder" is not specific enough to describe the signal processing used in these experiments. Misunderstanding vocoder parameters can result in experimental confounds or unexpected stimulus distortions. This paper highlights the signal processing parameters that should be specified when describing vocoder construction. The paper also provides guidance on how to determine vocoder parameters within perception experiments, given the experimenter's goals and research questions, to avoid common signal processing mistakes. Throughout, we will assume that experimenters are interested in vocoders with the specific goal of better understanding cochlear implants.

Cortical Zinc Signaling Is Necessary for Changes in Mouse Pupil Diameter That Are Evoked by Background Sounds with Different Contrasts

Luminance-independent changes in pupil diameter (PD) during wakefulness influence and are influenced by neuromodulatory, neuronal, and behavioral responses. However, it is unclear whether changes in neuromodulatory activity in a specific brain area are necessary for the associated changes in PD or whether some different mechanisms cause parallel fluctuations in both PD and neuromodulation. To answer this question, we simultaneously recorded PD and cortical neuronal activity in male and female mice. Namely, we measured PD and neuronal activity during adaptation to sound contrast, which is a well-described adaptation conserved in many species and brain areas. In the primary auditory cortex (A1), increases in the variability of sound level (contrast) induce a decrease in the slope of the neuronal input-output relationship, neuronal gain, which depends on cortical neuromodulatory zinc signaling. We found a previously unknown modulation of PD by changes in background sensory context: high stimulus contrast sounds evoke larger increases in evoked PD compared with low-contrast sounds. To explore whether these changes in evoked PD are controlled by cortical neuromodulatory zinc signaling, we imaged single-cell neural activity in A1, manipulated zinc signaling in the cortex, and assessed PD in the same awake mouse. We found that cortical synaptic zinc signaling is necessary for increases in PD during high-contrast background sounds compared with low-contrast sounds. This finding advances our knowledge about how cortical neuromodulatory activity affects PD changes and thus advances our understanding of the brain states, circuits, and neuromodulatory mechanisms that can be inferred from pupil size fluctuations.

Lexical effects on talker discrimination in adult cochlear implant usersa)

The lexical and phonological content of an utterance impacts the processing of talker-specific details in normal-hearing (NH) listeners. Adult cochlear implant (CI) users demonstrate difficulties in talker discrimination, particularly for same-gender talker pairs, which may alter the reliance on lexical information in talker discrimination. The current study examined the effect of lexical content on talker discrimination in 24 adult CI users. In a remote AX talker discrimination task, word pairs-produced either by the same talker (ST) or different talkers with the same (DT-SG) or mixed genders (DT-MG)-were either lexically easy (high frequency, low neighborhood density) or lexically hard (low frequency, high neighborhood density). The task was completed in quiet and multi-talker babble (MTB). Results showed an effect of lexical difficulty on talker discrimination, for same-gender talker pairs in both quiet and MTB. CI users showed greater sensitivity in quiet as well as less response bias in both quiet and MTB for lexically easy words compared to lexically hard words. These results suggest that CI users make use of lexical content in same-gender talker discrimination, providing evidence for the contribution of linguistic information to the processing of degraded talker information by adult CI users.

Pupil Dilation Reflects Perceptual Priorities During a Receptive Speech Task

OBJECTIVES

The listening demand incurred by speech perception fluctuates in normal conversation. At the acoustic-phonetic level, natural variation in pronunciation acts as speedbumps to accurate lexical selection. Any given utterance may be more or less phonetically ambiguous-a problem that must be resolved by the listener to choose the correct word. This becomes especially apparent when considering two common speech registers-clear and casual-that have characteristically different levels of phonetic ambiguity. Clear speech prioritizes intelligibility through hyperarticulation which results in less ambiguity at the phonetic level, while casual speech tends to have a more collapsed acoustic space. We hypothesized that listeners would invest greater cognitive resources while listening to casual speech to resolve the increased amount of phonetic ambiguity, as compared with clear speech. To this end, we used pupillometry as an online measure of listening effort during perception of clear and casual continuous speech in two background conditions: quiet and noise.

DESIGN

Forty-eight participants performed a probe detection task while listening to spoken, nonsensical sentences (masked and unmasked) while recording pupil size. Pupil size was modeled using growth curve analysis to capture the dynamics of the pupil response as the sentence unfolded.

RESULTS

Pupil size during listening was sensitive to the presence of noise and speech register (clear/casual). Unsurprisingly, listeners had overall larger pupil dilations during speech perception in noise, replicating earlier work. The pupil dilation pattern for clear and casual sentences was considerably more complex. Pupil dilation during clear speech trials was slightly larger than for casual speech, across quiet and noisy backgrounds.

CONCLUSIONS

We suggest that listener motivation could explain the larger pupil dilations to clearly spoken speech. We propose that, bounded by the context of this task, listeners devoted more resources to perceiving the speech signal with the greatest acoustic/phonetic fidelity. Further, we unexpectedly found systematic differences in pupil dilation preceding the onset of the spoken sentences. Together, these data demonstrate that the pupillary system is not merely reactive but also adaptive-sensitive to both task structure and listener motivation to maximize accurate perception in a limited resource system.

Input-related demands: vocoded sentences evoke different pupillometrics and subjective listening effort than sentences in speech-shaped noise

OBJECTIVES

The Framework for Effortful Listening (FUEL) suggests five input-related demands can alter listening effort: source, transmission, listener, message and context factors. We hypothesised that vocoded sentences represented a source factor degradation and sentences in speech-shaped noise represented a transmission factor degradation. We used pupillometry and a subjective scale to examine our hypothesis.

DESIGN

Participants listened to vocoded sentences and sentences in speech-shaped noise at several difficulty levels designed to produce similar word recognition abilities; they also listened to unprocessed sentences. Within-participant pupillometrics and subjective listening effort were analysed. Post-hoc analyses were performed to examine if word recognition accuracy differentially influenced pupil responses.

STUDY SAMPLES

Twenty young adults with normal hearing.

RESULTS

Baseline pupil diameter was significantly smaller, peak pupil dilation was significantly larger, peak pupil dilation latency was significantly shorter, and subjective listening effort was significantly greater for the vocoded sentences than the sentences-in-noise. Word recognition ability also affected pupillometrics, but only for the vocoded sentences.

CONCLUSIONS

Our findings suggest that source factor degradations result in greater listening effort than transmission factor degradations. Future research should address how clinical interventions tailored towards different input-related demands may lead to reduced listening effort and improve patient outcomes.

Cochlear-Implant Simulated Signal Degradation Exacerbates Listening Effort in Older Listeners

OBJECTIVES

Individuals with cochlear implants (CIs) often report that listening requires high levels of effort. Listening effort can increase with decreasing spectral resolution, which occurs when listening with a CI, and can also increase with age. What is not clear is whether these factors interact; older CI listeners potentially experience even higher listening effort with greater signal degradation than younger CI listeners. This study used pupillometry as a physiological index of listening effort to examine whether age, spectral resolution, and their interaction affect listening effort in a simulation of CI listening.

DESIGN

Fifteen younger normal-hearing listeners (ages 18 to 31 years) and 15 older normal-hearing listeners (ages 65 to 75 years) participated in this experiment; they had normal hearing thresholds from 0.25 to 4 kHz. Participants repeated sentences presented in quiet that were either unprocessed or vocoded, simulating CI listening. Stimuli frequency spectra were limited to below 4 kHz (to control for effects of age-related high-frequency hearing loss), and spectral resolution was decreased by decreasing the number of vocoder channels, with 32-, 16-, and 8-channel conditions. Behavioral speech recognition scores and pupil dilation were recorded during this task. In addition, cognitive measures of working memory and processing speed were obtained to examine if individual differences in these measures predicted changes in pupil dilation.

RESULTS

For trials where the sentence was recalled correctly, there was a significant interaction between age and spectral resolution, with significantly greater pupil dilation in the older normal-hearing listeners for the 8- and 32-channel vocoded conditions. Cognitive measures did not predict pupil dilation.

CONCLUSIONS

There was a significant interaction between age and spectral resolution, such that older listeners appear to exert relatively higher listening effort than younger listeners when the signal is highly degraded, with the largest effects observed in the eight-channel condition. The clinical implication is that older listeners may be at higher risk for increased listening effort with a CI.

Variability in Receptive Language Development Following Bilateral Cochlear Implantation

OBJECTIVES

The primary aim was to investigate the variability in language development in children aged 5-7.5 years after bilateral cochlear implantation (CI) up to the age of 2 years, and any impact of the age at implantation and additional noncognitive or anatomical disorders at implantation.

DESIGN

Data of 84 congenitally deaf children that had received simultaneous bilateral CI at the age of ≤ 24 months were included in this retrospective study. The results of language comprehension acquisition were evaluated using a standardized German language acquisition test for normal hearing preschoolers and first graders. Data on speech perception of monosyllables and sentences in quiet and noise were added.

RESULTS

In a monosyllabic test, the children achieved a median performance of 75.0 ± 12.88%. In the sentence test in quiet, the median performance was 89 ± 12.69%, but dropped to 54 ± 18.92% in noise. A simple analysis showed a significant main effect of age at implantation on monosyllabic word comprehension (p < .001), but no significant effect of comorbidities that lacked cognitive effects (p = .24). Language acquisition values correspond to the normal range of children with normal hearing. Approximately 25% of the variability in the language acquisition tests is due to the outcome of the monosyllabic speech perception test.

CONCLUSIONS

Congenitally deaf children who were fitted bilaterally in the 1st year of life can develop age-appropriate language skills by the time they start school. The high variability in the data is partly due to the age of implantation, but additional factors such as cognitive factors (e.g., working memory) are likely to influence the variability.

Effects of linguistic context and noise type on speech comprehension

Introduction

Understanding speech in background noise is an effortful endeavor. When acoustic challenges arise, linguistic context may help us fill in perceptual gaps. However, more knowledge is needed regarding how different types of background noise affect our ability to construct meaning from perceptually complex speech input. Additionally, there is limited evidence regarding whether perceptual complexity (e.g., informational masking) and linguistic complexity (e.g., occurrence of contextually incongruous words) interact during processing of speech material that is longer and more complex than a single sentence. Our first research objective was to determine whether comprehension of spoken sentence pairs is impacted by the informational masking from a speech masker. Our second objective was to identify whether there is an interaction between perceptual and linguistic complexity during speech processing.

Methods

We used multiple measures including comprehension accuracy, reaction time, and processing effort (as indicated by task-evoked pupil response), making comparisons across three different levels of linguistic complexity in two different noise conditions. Context conditions varied by final word, with each sentence pair ending with an expected exemplar (EE), within-category violation (WV), or between-category violation (BV). Forty young adults with typical hearing performed a speech comprehension in noise task over three visits. Each participant heard sentence pairs presented in either multi-talker babble or spectrally shaped steady-state noise (SSN), with the same noise condition across all three visits.

Results

We observed an effect of context but not noise on accuracy. Further, we observed an interaction of noise and context in peak pupil dilation data. Specifically, the context effect was modulated by noise type: context facilitated processing only in the more perceptually complex babble noise condition.

Discussion

These findings suggest that when perceptual complexity arises, listeners make use of the linguistic context to facilitate comprehension of speech obscured by background noise. Our results extend existing accounts of speech processing in noise by demonstrating how perceptual and linguistic complexity affect our ability to engage in higher-level processes, such as construction of meaning from speech segments that are longer than a single sentence.

Sequence effects and speech processing: cognitive load for speaker-switching within and across accents

Prior work in speech processing indicates that listening tasks with multiple speakers (as opposed to a single speaker) result in slower and less accurate processing. Notably, the trial-to-trial cognitive demands of switching between speakers or switching between accents have yet to be examined. We used pupillometry, a physiological index of cognitive load, to examine the demands of processing first (L1) and second (L2) language-accented speech when listening to sentences produced by the same speaker consecutively (no switch), a novel speaker of the same accent (within-accent switch), and a novel speaker with a different accent (across-accent switch). Inspired by research on sequential adjustments in cognitive control, we aimed to identify the cognitive demands of accommodating a novel speaker and accent by examining the trial-to-trial changes in pupil dilation during speech processing. Our results indicate that switching between speakers was more cognitively demanding than listening to the same speaker consecutively. Additionally, switching to a novel speaker with a different accent was more cognitively demanding than switching between speakers of the same accent. However, there was an asymmetry for across-accent switches, such that switching from an L1 to an L2 accent was more demanding than vice versa. Findings from the present study align with work examining multi-talker processing costs, and provide novel evidence that listeners dynamically adjust cognitive processing to accommodate speaker and accent variability. We discuss these novel findings in the context of an active control model and auditory streaming framework of speech processing.

How the eyes respond to sounds

Eye movements have been extensively studied with respect to visual stimulation. However, we live in a multisensory world, and how the eyes are driven by other senses has been explored much less. Here, we review the evidence on how audition can trigger and drive different eye responses and which cortical and subcortical neural correlates are involved. We provide an overview on how different types of sounds, from simple tones and noise bursts to spatially localized sounds and complex linguistic stimuli, influence saccades, microsaccades, smooth pursuit, pupil dilation, and eye blinks. The reviewed evidence reveals how the auditory system interacts with the oculomotor system, both behaviorally and neurally, and how this differs from visually driven eye responses. Some evidence points to multisensory interaction, and potential multisensory integration, but the underlying computational and neural mechanisms are still unclear. While there are marked differences in how the eyes respond to auditory compared to visual stimuli, many aspects of auditory-evoked eye responses remain underexplored, and we summarize the key open questions for future research.

Impact of SNR, peripheral auditory sensitivity, and central cognitive profile on the psychometric relation between pupillary response and speech performance in CI users

Despite substantial technical advances and wider clinical use, cochlear implant (CI) users continue to report high and elevated listening effort especially under challenging noisy conditions. Among all the objective measures to quantify listening effort, pupillometry is one of the most widely used and robust physiological measures. Previous studies with normally hearing (NH) and hearing-impaired (HI) listeners have shown that the relation between speech performance in noise and listening effort (as measured by peak pupil dilation) is not linear and exhibits an inverted-U shape. However, it is unclear whether the same psychometric relation exists in CI users, and whether individual differences in auditory sensitivity and central cognitive capacity affect this relation. Therefore, we recruited 17 post-lingually deaf CI adults to perform speech-in-noise tasks from 0 to 20 dB SNR with a 4 dB step size. Simultaneously, their pupillary responses and self-reported subjective effort were recorded. To characterize top-down and bottom-up individual variabilities, a spectro-temporal modulation task and a set of cognitive abilities were measured. Clinical word recognition in quiet and Quality of Life (QoL) were also collected. Results showed that at a group level, an inverted-U shape psychometric curve between task difficulty (SNR) and peak pupil dilation (PPD) was not observed. Individual shape of the psychometric curve was significantly associated with some individual factors: CI users with higher clinical word and speech-in-noise recognition showed a quadratic decrease of PPD over increasing SNRs; CI users with better non-verbal intelligence and lower QoL showed smaller average PPD. To summarize, individual differences in CI users had a significant impact on the psychometric relation between pupillary response and task difficulty, hence affecting the interpretation of pupillary response as listening effort (or engagement) at different task difficulty levels. Future research and clinical applications should further characterize the possible effects of individual factors (such as motivation or engagement) in modulating CI users' occurrence of 'tipping point' on their psychometric functions, and develop an individualized method for reliably quantifying listening effort using pupillometry.

Pupil Size Sensitivity to Listening Demand Depends on Motivational State

Motivation plays a role when a listener needs to understand speech under acoustically demanding conditions. Previous work has demonstrated pupil-linked arousal being sensitive to both listening demands and motivational state during listening. It is less clear how motivational state affects the temporal evolution of the pupil size and its relation to subsequent behavior. We used an auditory gap detection task (N = 33) to study the joint impact of listening demand and motivational state on the pupil size response and examine its temporal evolution. Task difficulty and a listener's motivational state were orthogonally manipulated through changes in gap duration and monetary reward prospect. We show that participants' performance decreased with task difficulty, but that reward prospect enhanced performance under hard listening conditions. Pupil size increased with both increased task difficulty and higher reward prospect, and this reward prospect effect was largest under difficult listening conditions. Moreover, pupil size time courses differed between detected and missed gaps, suggesting that the pupil response indicates upcoming behavior. Larger pre-gap pupil size was further associated with faster response times on a trial-by-trial within-participant level. Our results reiterate the utility of pupil size as an objective and temporally sensitive measure in audiology. However, such assessments of cognitive resource recruitment need to consider the individual's motivational state.

Pupillometry reveals differences in cognitive demands of listening to face mask-attenuated speech

Face masks offer essential protection but also interfere with speech communication. Here, audio-only sentences spoken through four types of masks were presented in noise to young adult listeners. Pupil dilation (an index of cognitive demand), intelligibility, and subjective effort and performance ratings were collected. Dilation increased in response to each mask relative to the no-mask condition and differed significantly where acoustic attenuation was most prominent. These results suggest that the acoustic impact of the mask drives not only the intelligibility of speech, but also the cognitive demands of listening. Subjective effort ratings reflected the same trends as the pupil data.

Effects of spectral smearing on speech understanding and masking release in simulated bilateral cochlear implants

Differences in spectro-temporal degradation may explain some variability in cochlear implant users' speech outcomes. The present study employs vocoder simulations on listeners with typical hearing to evaluate how differences in degree of channel interaction across ears affects spatial speech recognition. Speech recognition thresholds and spatial release from masking were measured in 16 normal-hearing subjects listening to simulated bilateral cochlear implants. 16-channel sine-vocoded speech simulated limited, broad, or mixed channel interaction, in dichotic and diotic target-masker conditions, across ears. Thresholds were highest with broad channel interaction in both ears but improved when interaction decreased in one ear and again in both ears. Masking release was apparent across conditions. Results from this simulation study on listeners with typical hearing show that channel interaction may impact speech recognition more than masking release, and may have implications for the effects of channel interaction on cochlear implant users' speech recognition outcomes.

Phonological discrimination and contrast detection in pupillometry

Introduction

The perception of phonemes is guided by both low-level acoustic cues and high-level linguistic context. However, differentiating between these two types of processing can be challenging. In this study, we explore the utility of pupillometry as a tool to investigate both low- and high-level processing of phonological stimuli, with a particular focus on its ability to capture novelty detection and cognitive processing during speech perception.

Methods

Pupillometric traces were recorded from a sample of 22 Danish-speaking adults, with self-reported normal hearing, while performing two phonological-contrast perception tasks: a nonword discrimination task, which included minimal-pair combinations specific to the Danish language, and a nonword detection task involving the detection of phonologically modified words within sentences. The study explored the perception of contrasts in both unprocessed speech and degraded speech input, processed with a vocoder.

Results

No difference in peak pupil dilation was observed when the contrast occurred between two isolated nonwords in the nonword discrimination task. For unprocessed speech, higher peak pupil dilations were measured when phonologically modified words were detected within a sentence compared to sentences without the nonwords. For vocoded speech, higher peak pupil dilation was observed for sentence stimuli, but not for the isolated nonwords, although performance decreased similarly for both tasks.

Conclusion

Our findings demonstrate the complexity of pupil dynamics in the presence of acoustic and phonological manipulation. Pupil responses seemed to reflect higher-level cognitive and lexical processing related to phonological perception rather than low-level perception of acoustic cues. However, the incorporation of multiple talkers in the stimuli, coupled with the relatively low task complexity, may have affected the pupil dilation.

The Relationship Between Cochlear Implant Speech Perception Outcomes and Electrophysiological Measures of the Electrically Evoked Compound Action Potential

OBJECTIVE

This study assessed the relationship between electrophysiological measures of the electrically evoked compound action potential (eCAP) and speech perception scores measured in quiet and in noise in postlingually deafened adult cochlear implant (CI) users. It tested the hypothesis that how well the auditory nerve (AN) responds to electrical stimulation is important for speech perception with a CI in challenging listening conditions.

DESIGN

Study participants included 24 postlingually deafened adult CI users. All participants used Cochlear Nucleus CIs in their test ears. In each participant, eCAPs were measured at multiple electrode locations in response to single-pulse, paired-pulse, and pulse-train stimuli. Independent variables included six metrics calculated from the eCAP recordings: the electrode-neuron interface (ENI) index, the neural adaptation (NA) ratio, NA speed, the adaptation recovery (AR) ratio, AR speed, and the amplitude modulation (AM) ratio. The ENI index quantified the effectiveness of the CI electrodes in stimulating the targeted AN fibers. The NA ratio indicated the amount of NA at the AN caused by a train of constant-amplitude pulses. NA speed was defined as the speed/rate of NA. The AR ratio estimated the amount of recovery from NA at a fixed time point after the cessation of pulse-train stimulation. AR speed referred to the speed of recovery from NA caused by previous pulse-train stimulation. The AM ratio provided a measure of AN sensitivity to AM cues. Participants' speech perception scores were measured using Consonant-Nucleus-Consonant (CNC) word lists and AzBio sentences presented in quiet, as well as in noise at signal-to-noise ratios (SNRs) of +10 and +5 dB. Predictive models were created for each speech measure to identify eCAP metrics with meaningful predictive power.

RESULTS

The ENI index and AR speed individually explained at least 10% of the variance in most of the speech perception scores measured in this study, while the NA ratio, NA speed, the AR ratio, and the AM ratio did not. The ENI index was identified as the only eCAP metric that had unique predictive power for each of the speech test results. The amount of variance in speech perception scores (both CNC words and AzBio sentences) explained by the eCAP metrics increased with increased difficulty under the listening condition. Over half of the variance in speech perception scores measured in +5 dB SNR noise (both CNC words and AzBio sentences) was explained by a model with only three eCAP metrics: the ENI index, NA speed, and AR speed.

CONCLUSIONS

Of the six electrophysiological measures assessed in this study, the ENI index is the most informative predictor for speech perception performance in CI users. In agreement with the tested hypothesis, the response characteristics of the AN to electrical stimulation are more important for speech perception with a CI in noise than they are in quiet.

The impact of speech type on listening effort and intelligibility for native and non-native listeners

Listeners are routinely exposed to many different types of speech, including artificially-enhanced and synthetic speech, styles which deviate to a greater or lesser extent from naturally-spoken exemplars. While the impact of differing speech types on intelligibility is well-studied, it is less clear how such types affect cognitive processing demands, and in particular whether those speech forms with the greatest intelligibility in noise have a commensurately lower listening effort. The current study measured intelligibility, self-reported listening effort, and a pupillometry-based measure of cognitive load for four distinct types of speech: (i) plain i.e. natural unmodified speech; (ii) Lombard speech, a naturally-enhanced form which occurs when speaking in the presence of noise; (iii) artificially-enhanced speech which involves spectral shaping and dynamic range compression; and (iv) speech synthesized from text. In the first experiment a cohort of 26 native listeners responded to the four speech types in three levels of speech-shaped noise. In a second experiment, 31 non-native listeners underwent the same procedure at more favorable signal-to-noise ratios, chosen since second language listening in noise has a more detrimental effect on intelligibility than listening in a first language. For both native and non-native listeners, artificially-enhanced speech was the most intelligible and led to the lowest subjective effort ratings, while the reverse was true for synthetic speech. However, pupil data suggested that Lombard speech elicited the lowest processing demands overall. These outcomes indicate that the relationship between intelligibility and cognitive processing demands is not a simple inverse, but is mediated by speech type. The findings of the current study motivate the search for speech modification algorithms that are optimized for both intelligibility and listening effort.

Sensory representations and pupil-indexed listening effort provide complementary contributions to multi-talker speech intelligibility

Optimal speech perception in noise requires successful separation of the target speech stream from multiple competing background speech streams. The ability to segregate these competing speech streams depends on the fidelity of bottom-up neural representations of sensory information in the auditory system and top-down influences of effortful listening. Here, we use objective neurophysiological measures of bottom-up temporal processing using envelope-following responses (EFRs) to amplitude modulated tones and investigate their interactions with pupil-indexed listening effort, as it relates to performance on the Quick speech in noise (QuickSIN) test in young adult listeners with clinically normal hearing thresholds. We developed an approach using ear-canal electrodes and adjusting electrode montages for modulation rate ranges, which extended the rage of reliable EFR measurements as high as 1024Hz. Pupillary responses revealed changes in listening effort at the two most difficult signal-to-noise ratios (SNR), but behavioral deficits at the hardest SNR only. Neither pupil-indexed listening effort nor the slope of the EFR decay function independently related to QuickSIN performance. However, a linear model using the combination of EFRs and pupil metrics significantly explained variance in QuickSIN performance. These results suggest a synergistic interaction between bottom-up sensory coding and top-down measures of listening effort as it relates to speech perception in noise. These findings can inform the development of next-generation tests for hearing deficits in listeners with normal-hearing thresholds that incorporates a multi-dimensional approach to understanding speech intelligibility deficits.

Eye Movements Decrease during Effortful Speech Listening

Hearing impairment affects many older adults but is often diagnosed decades after speech comprehension in noisy situations has become effortful. Accurate assessment of listening effort may thus help diagnose hearing impairment earlier. However, pupillometry-the most used approach to assess listening effort-has limitations that hinder its use in practice. The current study explores a novel way to assess listening effort through eye movements. Building on cognitive and neurophysiological work, we examine the hypothesis that eye movements decrease when speech listening becomes challenging. In three experiments with human participants from both sexes, we demonstrate, consistent with this hypothesis, that fixation duration increases and spatial gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (simple sentences, naturalistic stories). In contrast, pupillometry was less sensitive to speech masking during story listening, suggesting pupillometric measures may not be as effective for the assessments of listening effort in naturalistic speech-listening paradigms. Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in the brain regions that support the regulation of eye movements, such as frontal eye field and superior colliculus, are modulated when listening is effortful.SIGNIFICANCE STATEMENT Assessment of listening effort is critical for early diagnosis of age-related hearing loss. Pupillometry is most used but has several disadvantages. The current study explores a novel way to assess listening effort through eye movements. We examine the hypothesis that eye movements decrease when speech listening becomes effortful. We demonstrate, consistent with this hypothesis, that fixation duration increases and gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (sentences, naturalistic stories). Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in brain regions that support the regulation of eye movements are modulated when listening is effortful.

Effects of spectral degradation on gated word recognition

Although much is known about how normal-hearing listeners process spoken words under ideal listening conditions, little is known about how a degraded signal, such as speech transmitted via cochlear implants, affects the word recognition process. In this study, gated word recognition performance was measured with the goal of describing the time course of word identification by using a noise-band vocoder simulation. The results of this study demonstrate that spectral degradations can impact the temporal aspects of speech processing. These results also provide insights into the potential advantages of enhancing spectral resolution in the processing of spoken words.

Use of the first-acquired language modulates pupil size in the processing of island constraint violations

Introduction

Traditional studies of the population called "heritage speakers" (HS) have treated this group as distinct from other bilingual populations, e.g., simultaneous or late bilinguals (LB), focusing on group differences in the competencies of the first-acquired language or "heritage language". While several explanations have been proposed for such differences (e.g., incomplete acquisition, attrition, differential processing mechanisms), few have taken into consideration the individual variation that must occur, due to the fluctuation of factors such as exposure and use that characterize all bilinguals. In addition, few studies have used implicit measures, e.g., psychophysiological methods (ERPs; Eye-tracking), that can circumvent confounding variables such as resorting to conscious metalinguistic knowledge.

Methodology

This study uses pupillometry, a method that has only recently been used in psycholinguistic studies of bilingualism, to investigate pupillary responses to three syntactic island constructions in two groups of Spanish/English bilinguals: heritage speakers and late bilinguals. Data were analyzed using generalized additive mixed effects models (GAMMs) and two models were created and compared to one another: one with group (LB/HS) and the other with groups collapsed and current and historical use of Spanish as continuous variables.

Results

Results show that group-based models generally yield conflicting results while models collapsing groups and having usage as a predictor yield consistent ones. In particular, current use predicts sensitivity to L1 ungrammaticality across both HS and LB populations. We conclude that individual variation, as measured by use, is a critical factor tha must be taken into account in the description of the language competencies and processing of heritage and late bilinguals alike.

Listening efficiency in adult cochlear-implant users compared with normally-hearing controls at ecologically relevant signal-to-noise ratios

Introduction

Due to having to work with an impoverished auditory signal, cochlear-implant (CI) users may experience reduced speech intelligibility and/or increased listening effort in real-world listening situations, compared to their normally-hearing (NH) peers. These two challenges to perception may be usefully integrated in a measure of listening efficiency: conceptually, the amount of accuracy achieved for a certain amount of effort expended.

Methods

We describe a novel approach to quantifying listening efficiency based on the rate of evidence accumulation toward a correct response in a linear ballistic accumulator (LBA) model of choice decision-making. Estimation of this objective measure within a hierarchical Bayesian framework confers further benefits, including full quantification of uncertainty in parameter estimates. We applied this approach to examine the speech-in-noise performance of a group of 24 CI users (M age: 60.3, range: 20-84 years) and a group of 25 approximately age-matched NH controls (M age: 55.8, range: 20-79 years). In a laboratory experiment, participants listened to reverberant target sentences in cafeteria noise at ecologically relevant signal-to-noise ratios (SNRs) of +20, +10, and +4 dB SNR. Individual differences in cognition and self-reported listening experiences were also characterised by means of cognitive tests and hearing questionnaires.

Results

At the group level, the CI group showed much lower listening efficiency than the NH group, even in favourable acoustic conditions. At the individual level, within the CI group (but not the NH group), higher listening efficiency was associated with better cognition (i.e., working-memory and linguistic-closure) and with more positive self-reported listening experiences, both in the laboratory and in daily life.

Discussion

We argue that listening efficiency, measured using the approach described here, is: (i) conceptually well-motivated, in that it is theoretically impervious to differences in how individuals approach the speed-accuracy trade-off that is inherent to all perceptual decision making; and (ii) of practical utility, in that it is sensitive to differences in task demand, and to differences between groups, even when speech intelligibility remains at or near ceiling level. Further research is needed to explore the sensitivity and practical utility of this metric across diverse listening situations.

Short Implicit Voice Training Affects Listening Effort During a Voice Cue Sensitivity Task With Vocoder-Degraded Speech

OBJECTIVES

Understanding speech in real life can be challenging and effortful, such as in multiple-talker listening conditions. Fundamental frequency ( fo ) and vocal-tract length ( vtl ) voice cues can help listeners segregate between talkers, enhancing speech perception in adverse listening conditions. Previous research showed lower sensitivity to fo and vtl voice cues when speech signal was degraded, such as in cochlear implant hearing and vocoder-listening compared to normal hearing, likely contributing to difficulties in understanding speech in adverse listening. Nevertheless, when multiple talkers are present, familiarity with a talker's voice, via training or exposure, could provide a speech intelligibility benefit. In this study, the objective was to assess how an implicit short-term voice training could affect perceptual discrimination of voice cues ( fo+vtl ), measured in sensitivity and listening effort, with or without vocoder degradations.

DESIGN

Voice training was provided via listening to a recording of a book segment for approximately 30 min, and answering text-related questions, to ensure engagement. Just-noticeable differences (JNDs) for fo+vtl were measured with an odd-one-out task implemented as a 3-alternative forced-choice adaptive paradigm, while simultaneously collecting pupil data. The reference voice either belonged to the trained voice or an untrained voice. Effects of voice training (trained and untrained voice), vocoding (non-vocoded and vocoded), and item variability (fixed or variable consonant-vowel triplets presented across three items) on voice cue sensitivity ( fo+vtl JNDs) and listening effort (pupillometry measurements) were analyzed.

RESULTS

Results showed that voice training did not have a significant effect on voice cue discrimination. As expected, fo+vtl JNDs were significantly larger for vocoded conditions than for non-vocoded conditions and with variable item presentations than fixed item presentations. Generalized additive mixed models analysis of pupil dilation over the time course of stimulus presentation showed that pupil dilation was significantly larger during fo+vtl discrimination while listening to untrained voices compared to trained voices, but only for vocoder-degraded speech. Peak pupil dilation was significantly larger for vocoded conditions compared to non-vocoded conditions and variable items increased the pupil baseline relative to fixed items, which could suggest a higher anticipated task difficulty.

CONCLUSIONS

In this study, even though short voice training did not lead to improved sensitivity to small fo+vtl voice cue differences at the discrimination threshold level, voice training still resulted in reduced listening effort for discrimination among vocoded voice cues.

Speech recognition in noise task among children and young-adults: a pupillometry study

Introduction

Children experience unique challenges when listening to speech in noisy environments. The present study used pupillometry, an established method for quantifying listening and cognitive effort, to detect temporal changes in pupil dilation during a speech-recognition-in-noise task among school-aged children and young adults.

Methods

Thirty school-aged children and 31 young adults listened to sentences amidst four-talker babble noise in two signal-to-noise ratios (SNR) conditions: high accuracy condition (+10 dB and  + 6 dB, for children and adults, respectively) and low accuracy condition (+5 dB and + 2 dB, for children and adults, respectively). They were asked to repeat the sentences while pupil size was measured continuously during the task.

Results

During the auditory processing phase, both groups displayed pupil dilation; however, adults exhibited greater dilation than children, particularly in the low accuracy condition. In the second phase (retention), only children demonstrated increased pupil dilation, whereas adults consistently exhibited a decrease in pupil size. Additionally, the children's group showed increased pupil dilation during the response phase.

Discussion

Although adults and school-aged children produce similar behavioural scores, group differences in dilation patterns point that their underlying auditory processing differs. A second peak of pupil dilation among the children suggests that their cognitive effort during speech recognition in noise lasts longer than in adults, continuing past the first auditory processing peak dilation. These findings support effortful listening among children and highlight the need to identify and alleviate listening difficulties in school-aged children, to provide proper intervention strategies.

Neural α Oscillations and Pupil Size Differentially Index Cognitive Demand under Competing Audiovisual Task Conditions

Cognitive demand is thought to modulate two often used, but rarely combined, measures: pupil size and neural α (8-12 Hz) oscillatory power. However, it is unclear whether these two measures capture cognitive demand in a similar way under complex audiovisual-task conditions. Here we recorded pupil size and neural α power (using electroencephalography), while human participants of both sexes concurrently performed a visual multiple object-tracking task and an auditory gap detection task. Difficulties of the two tasks were manipulated independent of each other. Participants' performance decreased in accuracy and speed with increasing cognitive demand. Pupil size increased with increasing difficulty for both the auditory and the visual task. In contrast, α power showed diverging neural dynamics: parietal α power decreased with increasing difficulty in the visual task, but not with increasing difficulty in the auditory task. Furthermore, independent of task difficulty, within-participant trial-by-trial fluctuations in pupil size were negatively correlated with α power. Difficulty-induced changes in pupil size and α power, however, did not correlate, which is consistent with their different cognitive-demand sensitivities. Overall, the current study demonstrates that the dynamics of the neurophysiological indices of cognitive demand and associated effort are multifaceted and potentially modality-dependent under complex audiovisual-task conditions.SIGNIFICANCE STATEMENT Pupil size and oscillatory α power are associated with cognitive demand and effort, but their relative sensitivity under complex audiovisual-task conditions is unclear, as is the extent to which they share underlying mechanisms. Using an audiovisual dual-task paradigm, we show that pupil size increases with increasing cognitive demands for both audition and vision. In contrast, changes in oscillatory α power depend on the respective task demands: parietal α power decreases with visual demand but not with auditory task demand. Hence, pupil size and α power show different sensitivity to cognitive demands, perhaps suggesting partly different underlying neural mechanisms.