Impact of Background Noise and Sentence Complexity on Processing Demands during Sentence Comprehension

Saccades and Blinks Index Cognitive Demand during Auditory Noncanonical Sentence Comprehension

Noncanonical sentence structures pose comprehension challenges because they require increased cognitive demand. Prosody may partially alleviate this cognitive load. These findings largely stem from behavioral studies, yet physiological measures may reveal additional insights into how cognition is deployed to parse sentences. Pupillometry has been at the forefront of investigations into physiological measures of cognitive demand during auditory sentence comprehension. This study offers an alternative approach by examining whether eye-tracking measures, including blinks and saccades, index cognitive demand during auditory noncanonical sentence comprehension and whether these metrics are sensitive to reductions in cognitive load associated with typical prosodic cues. We further investigated how eye-tracking patterns differ across correct and incorrect responses, as a function of time, and how each related to behavioral measures of cognition. Canonical and noncanonical sentence comprehension was measured in 30 younger adults using an auditory sentence-picture matching task. We also assessed participants' attention and working memory. Blinking and saccades both differentiate noncanonical sentences from canonical sentences. Saccades further distinguish noncanonical structures from each other. Participants made more saccades on incorrect than correct trials. The number of saccades also related to working memory, regardless of syntax. However, neither eye-tracking metric was sensitive to the changes in cognitive demand that was behaviorally observed in response to typical prosodic cues. Overall, these findings suggest that eye-tracking indices, particularly saccades, reflect cognitive demand during auditory noncanonical sentence comprehension when visual information is present, offering greater insights into the strategies and neural resources participants use to parse auditory sentences.

Language processing in emergencies recruits both language and default mode networks

Effective language processing in emergencies is crucial for professionals, including firefighters, soldiers, and doctors. Substantial research has been undertaken on language processing in silence, with several studies indicating the impact of noise on language processing in non-emergencies. However, it remains unclear about the neural mechanisms involved in language processing during emergencies, especially the role of the language network (LN) and the default mode network (DMN) in such contexts. In this study, we adopted functional near-infrared spectroscopy (fNIRS) to investigate brain activities of 60 participants who were performing language processing tasks in simulated high-emergency and low-emergency scenarios. Compared to the resting state, the reading task demonstrated reduced activation in bilateral superior and middle frontal gyri (SFG/MFG), components of the DMN, alongside enhanced activation in Broca's area, left temporal lobe and left inferior parietal lobule (IPL) - key LN regions. Furthermore, the activity of the left MFG was positively correlated with the level of stress experienced by participants during the task. Additionally, a positive correlation was identified between language processing performance and activation of the left MFG, exclusively in the participants exposed to high-emergency scenarios. These results support the view that language processing during emergency relies on both the LN and DMN. The current study deepens our understanding of the neural mechanisms that underlie language processing in complex, real-life emergency scenarios.

Advance Contextual Clues Alleviate Listening Effort During Sentence Repair in Listeners With Hearing Aids

PURPOSE

When words are misperceived, listeners can rely on later context to repair an auditory perception, at the cost of increased effort. The current study examines whether the effort to repair a missing word in a sentence is alleviated when the listener has some advance knowledge of what to expect in the sentence.

METHOD

Sixteen adults with hearing aids and 17 with typical hearing heard sentences with a missing word that was followed by context sufficient to infer what the word was. They repeated the sentences with the missing words repaired. Sentences were preceded by visual text on the screen showing either "XXXX" (unprimed) or a priming word previewing the word that would be masked in the auditory signal. Along with intelligibility measures, pupillometry was used as an index of listening effort over the course of each trial to measure how priming influenced the effort needed to mentally repair a missing word.

RESULTS

When listeners were primed for the word that would need to be repaired in an upcoming sentence, listening effort was reduced, as indicated by pupil size returning more quickly toward baseline after the sentence was heard. Priming reduced the lingering cost of mental repair in both listener groups. For the group with hearing loss, priming also reduced the prevalence of errors on target words and words other than the target word in the sentence, suggesting that priming preserves the cognitive resources needed to process the whole sentence.

CONCLUSION

These results suggest that listeners with typical hearing and with hearing loss can benefit from priming (advance cueing) during speech recognition, to accurately repair speech and to process the speech less effortfully.

Through the Pupils' Lens: Multilingual Effort in First and Second Language Listening

OBJECTIVES

The present study aimed to examine the involvement of listening effort among multilinguals in their first (L1) and second (L2) languages in quiet and noisy listening conditions and investigate how the presence of a constraining context within sentences influences listening effort.

DESIGN

A group of 46 young adult Arabic (L1)-Hebrew (L2) multilinguals participated in a listening task. This task aimed to assess participants' perceptual performance and the effort they exert (as measured through pupillometry) while listening to single words and sentences presented in their L1 and L2, in quiet and noisy environments (signal to noise ratio = 0 dB).

RESULTS

Listening in quiet was easier than in noise, supported by both perceptual and pupillometry results. Perceptually, multilinguals performed similarly and reached ceiling levels in both languages in quiet. However, under noisy conditions, perceptual accuracy was significantly lower in L2, especially when processing sentences. Critically, pupil dilation was larger and more prolonged when listening to L2 than L1 stimuli. This difference was observed even in the quiet condition. Contextual support resulted in better perceptual performance of high-predictability sentences compared with low-predictability sentences, but only in L1 under noisy conditions. In L2, pupillometry showed increased effort when listening to high-predictability sentences compared with low-predictability sentences, but this increased effort did not lead to better understanding. In fact, in noise, speech perception was lower in high-predictability L2 sentences compared with low-predictability ones.

CONCLUSIONS

The findings underscore the importance of examining listening effort in multilingual speech processing and suggest that increased effort may be present in multilingual's L2 within clinical and educational settings.

The time course of cognitive effort during disrupted speech

Listeners often find themselves in scenarios where speech is disrupted, misperceived, or otherwise difficult to recognise. In these situations, many individuals report exerting additional effort to understand speech, even when repairing speech may be difficult or impossible. This investigation aimed to characterise cognitive efforts across time during both sentence listening and a post-sentence retention interval by observing the pupillary response of participants with normal to borderline-normal hearing in response to two interrupted speech conditions: sentences interrupted by gaps of silence or bursts of noise. The pupillary response serves as a measure of the cumulative resources devoted to task completion. Both interruption conditions resulted in significantly greater levels of pupil dilation than the uninterrupted speech condition. Just prior to the end of a sentence, trials periodically interrupted by bursts of noise elicited greater pupil dilation than the silent-interrupted condition. Compared to the uninterrupted condition, both interruption conditions resulted in increased dilation after sentence end but before repetition, possibly reflecting sustained processing demands. Understanding pupil dilation as a marker of cognitive effort is important for clinicians and researchers when assessing the additional effort exerted by listeners with hearing loss who may use cochlear implants or hearing aids. Even when successful perceptual repair is unlikely, listeners may continue to exert increased effort when processing misperceived speech, which could cause them to miss upcoming speech or may contribute to heightened listening fatigue.

Acoustically Transparent Hearing Aids Increase Physiological Markers of Listening Effort

While hearing aids are beneficial in compensating for hearing loss and suppressing ambient noise, they may also introduce an unwanted processing burden to the listener's sensory and cognitive system. To investigate such adverse side effects, hearing aids may be set to a 'transparent mode', aiming to replicate natural hearing through the open ear as best as possible. Such transparent hearing aids have previously been demonstrated to exhibit a small but significant disadvantage in speech intelligibility, with less conclusive effects on self-rated listening effort. Here we aimed to reproduce these findings and expand them with neurophysiological measures of invested listening effort, including parietal alpha power and pupil size. Invested listening effort was measured across five task difficulties, ranging from nearly impossible to easy, with normal-hearing participants in both aided and unaided conditions. Results well reproduced a hearing aid disadvantage for speech intelligibility and subjective listening effort ratings. As to be expected, pupil size and parietal alpha power followed an inverted u-shape, peaking at moderate task difficulties (around SRT50). However, the transparent hearing aid increased pupil size and parietal alpha power at medium task demand (between SRT20 and SRT80). These neurophysiological effects were larger than those observed in speech intelligibility and subjective listening effort, respectively. The results gain plausibility by yielding a substantial association of individual pupil size and individual parietal alpha power. In sum, our findings suggest that key neurophysiological measures of invested listening effort are sensitive to the individual additional burden on speech intelligibility that hearing aid processing can introduce.

Pupillometry is sensitive to speech masking during story listening: A commentary on the critical role of modeling temporal trends

An increase in pupil size is an important index of listening effort, for example, when listening to speech masked by noise. Specifically, the pupil dilates as the signal-to-noise ratio decreases. A growing body of work aims to assess listening effort under naturalistic conditions using continuous speech, such as spoken stories. However, a recent study found that pupil size was sensitive to speech masking only when listening to sentences but not under naturalistic conditions when listening to stories. The pupil typically constricts with increasing time on task during an experimental block or session, and it may be necessary to account for this temporal trend in experimental design and data analysis in paradigms using longer, continuous stimuli. In the current work, we re-analyze the previously published pupil data, taking into account a problematic constraint of randomization and time-on-task, and use the data to outline methodological solutions for accounting for temporal trends in physiological data using linear mixed models. The results show that, in contrast to the previous work, pupil size is indeed sensitive to speech masking even during continuous story listening. Furthermore, accounting for the temporal trend allowed modeling the dynamic changes in the speech masking effect on pupil size over time as the continuous story unfolded. After demonstrating the importance of accounting for temporal trends in the analysis of empirical data, we provide simulations, methodological considerations, and user recommendations for the analysis of temporal trends in experimental data using linear mixed models.

Pupillometry reveals resting state alpha power correlates with individual differences in adult auditory language comprehension

Although individual differences in adult language processing are well-documented, the neural basis of this variability remains largely unexplored. The current study addressed this gap in the literature by examining the relationship between resting state alpha activity and individual differences in auditory language comprehension. Alpha oscillations modulate cortical excitability, facilitating efficient information processing in the brain. While resting state alpha oscillations have been tied to individual differences in cognitive performance, their association with auditory language comprehension is less clear. Participants in the study were 80 healthy adults with a mean age of 25.8 years (SD = 7.2 years). Resting state alpha activity was acquired using electroencephalography while participants looked at a benign stimulus for 3 min. Participants then completed a language comprehension task that involved listening to 'syntactically simple' subject-relative clause sentences and 'syntactically complex' object-relative clause sentences. Pupillometry measured real-time processing demand changes, with larger pupil dilation indicating increased processing loads. Replicating past research, comprehending object relative clauses, compared to subject relative clauses, was associated with lower accuracy, slower reaction times, and larger pupil dilation. Resting state alpha power was found to be positively correlated with the pupillometry data. That is, participants with higher resting state alpha activity evidenced larger dilation during sentence comprehension. This effect was more pronounced for the 'complex' object sentences compared to the 'simple' subject sentences. These findings suggest the brain's capacity to generate a robust resting alpha rhythm contributes to variability in processing demands associated with auditory language comprehension, especially when faced with challenging syntactic structures. More generally, the study demonstrates that the intrinsic functional architecture of the brain likely influences individual differences in language comprehension.

Auditory Challenges and Listening Effort in School-Age Children With Autism: Insights From Pupillary Dynamics During Speech-in-Noise Perception

PURPOSE

This study aimed to investigate challenges in speech-in-noise (SiN) processing faced by school-age children with autism spectrum conditions (ASCs) and their impact on listening effort.

METHOD

Participants, including 23 Mandarin-speaking children with ASCs and 19 age-matched neurotypical (NT) peers, underwent sentence recognition tests in both quiet and noisy conditions, with a speech-shaped steady-state noise masker presented at 0-dB signal-to-noise ratio in the noisy condition. Recognition accuracy rates and task-evoked pupil responses were compared to assess behavioral performance and listening effort during auditory tasks.

RESULTS

No main effect of group was found on accuracy rates. Instead, significant effects emerged for autistic trait scores, listening conditions, and their interaction, indicating that higher trait scores were associated with poorer performance in noise. Pupillometric data revealed significantly larger and earlier peak dilations, along with more varied pupillary dynamics in the ASC group relative to the NT group, especially under noisy conditions. Importantly, the ASC group's peak dilation in quiet mirrored that of the NT group in noise. However, the ASC group consistently exhibited reduced mean dilations than the NT group.

CONCLUSIONS

Pupillary responses suggest a different resource allocation pattern in ASCs: An initial sharper and larger dilation may signal an intense, narrowed resource allocation, likely linked to heightened arousal, engagement, and cognitive load, whereas a subsequent faster tail-off may indicate a greater decrease in resource availability and engagement, or a quicker release of arousal and cognitive load. The presence of noise further accentuates this pattern. This highlights the unique SiN processing challenges children with ASCs may face, underscoring the importance of a nuanced, individual-centric approach for interventions and support.

Pupil Size and Eye Movements Differently Index Effort in Both Younger and Older Adults

The assessment of mental effort is increasingly relevant in neurocognitive and life span domains. Pupillometry, the measure of the pupil size, is often used to assess effort but has disadvantages. Analysis of eye movements may provide an alternative, but research has been limited to easy and difficult task demands in younger adults. An effort measure must be sensitive to the whole effort profile, including "giving up" effort investment, and capture effort in different age groups. The current study comprised three experiments in which younger (n = 66) and older (n = 44) adults listened to speech masked by background babble at different signal-to-noise ratios associated with easy, difficult, and impossible speech comprehension. We expected individuals to invest little effort for easy and impossible speech (giving up) but to exert effort for difficult speech. Indeed, pupil size was largest for difficult but lower for easy and impossible speech. In contrast, gaze dispersion decreased with increasing speech masking in both age groups. Critically, gaze dispersion during difficult speech returned to levels similar to easy speech after sentence offset, when acoustic stimulation was similar across conditions, whereas gaze dispersion during impossible speech continued to be reduced. These findings show that a reduction in eye movements is not a byproduct of acoustic factors, but instead suggest that neurocognitive processes, different from arousal-related systems regulating the pupil size, drive reduced eye movements during high task demands. The current data thus show that effort in one sensory domain (audition) differentially impacts distinct functional properties in another sensory domain (vision).

Associations Between Auditory Working Memory, Self-Perceived Listening Effort, and Hearing Difficulty in Adults With Mild Traumatic Brain Injury

OBJECTIVES

Mild traumatic brain injury (TBI) can have persistent effects in the auditory domain (e.g., difficulty listening in noise), despite individuals having normal pure-tone auditory sensitivity. Individuals with a history of mild TBI often perceive hearing difficulty and greater listening effort in complex listening situations. The purpose of the present study was to examine self-perceived hearing difficulty, listening effort, and performance on an auditory processing test battery in adults with a history of mild TBI compared with a control group.

DESIGN

Twenty adults ages 20 to 53 years old participated divided into a mild TBI (n = 10) and control group (n = 10). Perceived hearing difficulties were measured using the Adult Auditory Processing Scale and the Hearing Handicap Inventory for Adults. Listening effort was measured using the National Aeronautics and Space Administration-Task Load Index. Listening effort ratings were obtained at baseline, after each auditory processing test, and at the completion of the test battery. The auditory processing test battery included (1) dichotic word recognition, (2) the 500-Hz masking level difference, (3) the Listening in Spatialized Noise-Sentences test, and (4) the Word Auditory Recognition and Recall Measure (WARRM).

RESULTS

Results indicated that individuals with a history of mild TBI perceived significantly greater degrees of hearing difficulty and listening effort than the control group. There were no significant group differences on two of the auditory processing tasks (dichotic word recognition or Listening in Spatialized Noise-Sentences). The mild TBI group exhibited significantly poorer performance on the 500-Hz MLD and the WARRM, a measure of auditory working memory, than the control group. Greater degrees of self-perceived hearing difficulty were significantly associated with greater listening effort and poorer auditory working memory. Greater listening effort was also significantly associated with poorer auditory working memory.

CONCLUSIONS

Results demonstrate that adults with a history of mild TBI may experience subjective hearing difficulty and listening effort when listening in challenging acoustic environments. Poorer auditory working memory on the WARRM task was observed for the adults with mild TBI and was associated with greater hearing difficulty and listening effort. Taken together, the present study suggests that conventional clinical audiometric battery alone may not provide enough information about auditory processing deficits in individuals with a history of mild TBI. The results support the use of a multifaceted battery of auditory processing tasks and subjective measures when evaluating individuals with a history of mild TBI.

Effects of linguistic context and noise type on speech comprehension

Introduction

Understanding speech in background noise is an effortful endeavor. When acoustic challenges arise, linguistic context may help us fill in perceptual gaps. However, more knowledge is needed regarding how different types of background noise affect our ability to construct meaning from perceptually complex speech input. Additionally, there is limited evidence regarding whether perceptual complexity (e.g., informational masking) and linguistic complexity (e.g., occurrence of contextually incongruous words) interact during processing of speech material that is longer and more complex than a single sentence. Our first research objective was to determine whether comprehension of spoken sentence pairs is impacted by the informational masking from a speech masker. Our second objective was to identify whether there is an interaction between perceptual and linguistic complexity during speech processing.

Methods

We used multiple measures including comprehension accuracy, reaction time, and processing effort (as indicated by task-evoked pupil response), making comparisons across three different levels of linguistic complexity in two different noise conditions. Context conditions varied by final word, with each sentence pair ending with an expected exemplar (EE), within-category violation (WV), or between-category violation (BV). Forty young adults with typical hearing performed a speech comprehension in noise task over three visits. Each participant heard sentence pairs presented in either multi-talker babble or spectrally shaped steady-state noise (SSN), with the same noise condition across all three visits.

Results

We observed an effect of context but not noise on accuracy. Further, we observed an interaction of noise and context in peak pupil dilation data. Specifically, the context effect was modulated by noise type: context facilitated processing only in the more perceptually complex babble noise condition.

Discussion

These findings suggest that when perceptual complexity arises, listeners make use of the linguistic context to facilitate comprehension of speech obscured by background noise. Our results extend existing accounts of speech processing in noise by demonstrating how perceptual and linguistic complexity affect our ability to engage in higher-level processes, such as construction of meaning from speech segments that are longer than a single sentence.

How the eyes respond to sounds

Eye movements have been extensively studied with respect to visual stimulation. However, we live in a multisensory world, and how the eyes are driven by other senses has been explored much less. Here, we review the evidence on how audition can trigger and drive different eye responses and which cortical and subcortical neural correlates are involved. We provide an overview on how different types of sounds, from simple tones and noise bursts to spatially localized sounds and complex linguistic stimuli, influence saccades, microsaccades, smooth pursuit, pupil dilation, and eye blinks. The reviewed evidence reveals how the auditory system interacts with the oculomotor system, both behaviorally and neurally, and how this differs from visually driven eye responses. Some evidence points to multisensory interaction, and potential multisensory integration, but the underlying computational and neural mechanisms are still unclear. While there are marked differences in how the eyes respond to auditory compared to visual stimuli, many aspects of auditory-evoked eye responses remain underexplored, and we summarize the key open questions for future research.

Divergent effects of listening demands and evaluative threat on listening effort in online and laboratory settings

Objective

Listening effort (LE) varies as a function of listening demands, motivation and resource availability, among other things. Motivation is posited to have a greater influence on listening effort under high, compared to low, listening demands.

Methods

To test this prediction, we manipulated the listening demands of a speech recognition task using tone vocoders to create moderate and high listening demand conditions. We manipulated motivation using evaluative threat, i.e., informing participants that they must reach a particular "score" for their results to be usable. Resource availability was assessed by means of working memory span and included as a fixed effects predictor. Outcome measures were indices of LE, including reaction times (RTs), self-rated work and self-rated tiredness, in addition to task performance (correct response rates). Given the recent popularity of online studies, we also wanted to examine the effect of experimental context (online vs. laboratory) on the efficacy of manipulations of listening demands and motivation. We carried out two highly similar experiments with two groups of 37 young adults, a laboratory experiment and an online experiment. To make listening demands comparable between the two studies, vocoder settings had to differ. All results were analysed using linear mixed models.

Results

Results showed that under laboratory conditions, listening demands affected all outcomes, with significantly lower correct response rates, slower RTs and greater self-rated work with higher listening demands. In the online study, listening demands only affected RTs. In addition, motivation affected self-rated work. Resource availability was only a significant predictor for RTs in the online study.

Discussion

These results show that the influence of motivation and listening demands on LE depends on the type of outcome measures used and the experimental context. It may also depend on the exact vocoder settings. A controlled laboratory settings and/or particular vocoder settings may be necessary to observe all expected effects of listening demands and motivation.

Combining Cardiovascular and Pupil Features Using k-Nearest Neighbor Classifiers to Assess Task Demand, Social Context, and Sentence Accuracy During Listening

In daily life, both acoustic factors and social context can affect listening effort investment. In laboratory settings, information about listening effort has been deduced from pupil and cardiovascular responses independently. The extent to which these measures can jointly predict listening-related factors is unknown. Here we combined pupil and cardiovascular features to predict acoustic and contextual aspects of speech perception. Data were collected from 29 adults (mean  =  64.6 years, SD  =  9.2) with hearing loss. Participants performed a speech perception task at two individualized signal-to-noise ratios (corresponding to 50% and 80% of sentences correct) and in two social contexts (the presence and absence of two observers). Seven features were extracted per trial: baseline pupil size, peak pupil dilation, mean pupil dilation, interbeat interval, blood volume pulse amplitude, pre-ejection period and pulse arrival time. These features were used to train k-nearest neighbor classifiers to predict task demand, social context and sentence accuracy. The k-fold cross validation on the group-level data revealed above-chance classification accuracies: task demand, 64.4%; social context, 78.3%; and sentence accuracy, 55.1%. However, classification accuracies diminished when the classifiers were trained and tested on data from different participants. Individually trained classifiers (one per participant) performed better than group-level classifiers: 71.7% (SD  =  10.2) for task demand, 88.0% (SD  =  7.5) for social context, and 60.0% (SD  =  13.1) for sentence accuracy. We demonstrated that classifiers trained on group-level physiological data to predict aspects of speech perception generalized poorly to novel participants. Individually calibrated classifiers hold more promise for future applications.

Phonological discrimination and contrast detection in pupillometry

Introduction

The perception of phonemes is guided by both low-level acoustic cues and high-level linguistic context. However, differentiating between these two types of processing can be challenging. In this study, we explore the utility of pupillometry as a tool to investigate both low- and high-level processing of phonological stimuli, with a particular focus on its ability to capture novelty detection and cognitive processing during speech perception.

Methods

Pupillometric traces were recorded from a sample of 22 Danish-speaking adults, with self-reported normal hearing, while performing two phonological-contrast perception tasks: a nonword discrimination task, which included minimal-pair combinations specific to the Danish language, and a nonword detection task involving the detection of phonologically modified words within sentences. The study explored the perception of contrasts in both unprocessed speech and degraded speech input, processed with a vocoder.

Results

No difference in peak pupil dilation was observed when the contrast occurred between two isolated nonwords in the nonword discrimination task. For unprocessed speech, higher peak pupil dilations were measured when phonologically modified words were detected within a sentence compared to sentences without the nonwords. For vocoded speech, higher peak pupil dilation was observed for sentence stimuli, but not for the isolated nonwords, although performance decreased similarly for both tasks.

Conclusion

Our findings demonstrate the complexity of pupil dynamics in the presence of acoustic and phonological manipulation. Pupil responses seemed to reflect higher-level cognitive and lexical processing related to phonological perception rather than low-level perception of acoustic cues. However, the incorporation of multiple talkers in the stimuli, coupled with the relatively low task complexity, may have affected the pupil dilation.

Auditory, Lexical, and Multitasking Demands Interactively Impact Listening Effort

PURPOSE

This study examined the extent to which acoustic, linguistic, and cognitive task demands interactively impact listening effort.

METHOD

Using a dual-task paradigm, on each trial, participants were instructed to perform either a single task or two tasks. In the primary word recognition task, participants repeated Northwestern University Auditory Test No. 6 words presented in speech-shaped noise at either an easier or a harder signal-to-noise ratio (SNR). The words varied in how commonly they occur in the English language (lexical frequency). In the secondary visual task, participants were instructed to press a specific key as soon as a number appeared on screen (simpler task) or one of two keys to indicate whether the visualized number was even or odd (more complex task).

RESULTS

Manipulation checks revealed that key assumptions of the dual-task design were met. A significant three-way interaction was observed, such that the expected effect of SNR on effort was only observable for words with lower lexical frequency and only when multitasking demands were relatively simpler.

CONCLUSIONS

This work reveals that variability across speech stimuli can influence the sensitivity of the dual-task paradigm for detecting changes in listening effort. In line with previous work, the results of this study also suggest that higher cognitive demands may limit the ability to detect expected effects of SNR on measures of effort. With implications for real-world listening, these findings highlight that even relatively minor changes in lexical and multitasking demands can alter the effort devoted to listening in noise.

The impact of speech type on listening effort and intelligibility for native and non-native listeners

Listeners are routinely exposed to many different types of speech, including artificially-enhanced and synthetic speech, styles which deviate to a greater or lesser extent from naturally-spoken exemplars. While the impact of differing speech types on intelligibility is well-studied, it is less clear how such types affect cognitive processing demands, and in particular whether those speech forms with the greatest intelligibility in noise have a commensurately lower listening effort. The current study measured intelligibility, self-reported listening effort, and a pupillometry-based measure of cognitive load for four distinct types of speech: (i) plain i.e. natural unmodified speech; (ii) Lombard speech, a naturally-enhanced form which occurs when speaking in the presence of noise; (iii) artificially-enhanced speech which involves spectral shaping and dynamic range compression; and (iv) speech synthesized from text. In the first experiment a cohort of 26 native listeners responded to the four speech types in three levels of speech-shaped noise. In a second experiment, 31 non-native listeners underwent the same procedure at more favorable signal-to-noise ratios, chosen since second language listening in noise has a more detrimental effect on intelligibility than listening in a first language. For both native and non-native listeners, artificially-enhanced speech was the most intelligible and led to the lowest subjective effort ratings, while the reverse was true for synthetic speech. However, pupil data suggested that Lombard speech elicited the lowest processing demands overall. These outcomes indicate that the relationship between intelligibility and cognitive processing demands is not a simple inverse, but is mediated by speech type. The findings of the current study motivate the search for speech modification algorithms that are optimized for both intelligibility and listening effort.

Eye Movements Decrease during Effortful Speech Listening

Hearing impairment affects many older adults but is often diagnosed decades after speech comprehension in noisy situations has become effortful. Accurate assessment of listening effort may thus help diagnose hearing impairment earlier. However, pupillometry-the most used approach to assess listening effort-has limitations that hinder its use in practice. The current study explores a novel way to assess listening effort through eye movements. Building on cognitive and neurophysiological work, we examine the hypothesis that eye movements decrease when speech listening becomes challenging. In three experiments with human participants from both sexes, we demonstrate, consistent with this hypothesis, that fixation duration increases and spatial gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (simple sentences, naturalistic stories). In contrast, pupillometry was less sensitive to speech masking during story listening, suggesting pupillometric measures may not be as effective for the assessments of listening effort in naturalistic speech-listening paradigms. Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in the brain regions that support the regulation of eye movements, such as frontal eye field and superior colliculus, are modulated when listening is effortful.SIGNIFICANCE STATEMENT Assessment of listening effort is critical for early diagnosis of age-related hearing loss. Pupillometry is most used but has several disadvantages. The current study explores a novel way to assess listening effort through eye movements. We examine the hypothesis that eye movements decrease when speech listening becomes effortful. We demonstrate, consistent with this hypothesis, that fixation duration increases and gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (sentences, naturalistic stories). Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in brain regions that support the regulation of eye movements are modulated when listening is effortful.

Neural α Oscillations and Pupil Size Differentially Index Cognitive Demand under Competing Audiovisual Task Conditions

Cognitive demand is thought to modulate two often used, but rarely combined, measures: pupil size and neural α (8-12 Hz) oscillatory power. However, it is unclear whether these two measures capture cognitive demand in a similar way under complex audiovisual-task conditions. Here we recorded pupil size and neural α power (using electroencephalography), while human participants of both sexes concurrently performed a visual multiple object-tracking task and an auditory gap detection task. Difficulties of the two tasks were manipulated independent of each other. Participants' performance decreased in accuracy and speed with increasing cognitive demand. Pupil size increased with increasing difficulty for both the auditory and the visual task. In contrast, α power showed diverging neural dynamics: parietal α power decreased with increasing difficulty in the visual task, but not with increasing difficulty in the auditory task. Furthermore, independent of task difficulty, within-participant trial-by-trial fluctuations in pupil size were negatively correlated with α power. Difficulty-induced changes in pupil size and α power, however, did not correlate, which is consistent with their different cognitive-demand sensitivities. Overall, the current study demonstrates that the dynamics of the neurophysiological indices of cognitive demand and associated effort are multifaceted and potentially modality-dependent under complex audiovisual-task conditions.SIGNIFICANCE STATEMENT Pupil size and oscillatory α power are associated with cognitive demand and effort, but their relative sensitivity under complex audiovisual-task conditions is unclear, as is the extent to which they share underlying mechanisms. Using an audiovisual dual-task paradigm, we show that pupil size increases with increasing cognitive demands for both audition and vision. In contrast, changes in oscillatory α power depend on the respective task demands: parietal α power decreases with visual demand but not with auditory task demand. Hence, pupil size and α power show different sensitivity to cognitive demands, perhaps suggesting partly different underlying neural mechanisms.

Time Scales and Moments of Listening Effort Revealed in Pupillometry

This article offers a collection of observations that highlight the value of time course data in pupillometry and points out ways in which these observations create deeper understanding of listening effort. The main message is that listening effort should be considered on a moment-to-moment basis rather than as a singular amount. A review of various studies and the reanalysis of data reveal distinct signatures of effort before a stimulus, during a stimulus, in the moments after a stimulus, and changes over whole experimental testing sessions. Collectively these observations motivate questions that extend beyond the "amount" of effort, toward understanding how long the effort lasts, and how precisely someone can allocate effort at specific points in time or reduce effort at other times. Apparent disagreements between studies are reconsidered as informative lessons about stimulus selection and the nature of pupil dilation as a reflection of decision making rather than the difficulty of sensory encoding.

EEG and pupillometric signatures of working memory overload

Understanding the physiological correlates of cognitive overload has implications for gauging the limits of human cognition, developing novel methods to define cognitive overload, and mitigating the negative outcomes associated with overload. Most previous psychophysiological studies manipulated verbal working memory load in a narrow range (an average load of 5 items). It is unclear, however, how the nervous system responds to a working memory load exceeding typical capacity limits. The objective of the current study was to characterize the central and autonomic nervous system changes associated with memory overload, by means of combined recording of electroencephalogram (EEG) and pupillometry. Eighty-six participants were presented with a digit span task involving the serial auditory presentation of items. Each trial consisted of sequences of either 5, 9, or 13 digits, each separated by 2 s. Both theta activity and pupil size, after the initial rise, expressed a pattern of a short plateau and a decrease with reaching the state of memory overload, indicating that pupil size and theta possibly have similar neural mechanisms. Based on the described above triphasic pattern of pupil size temporal dynamics, we concluded that cognitive overload causes physiological systems to reset, and release effort. Although memory capacity limits were exceeded and effort was released (as indicated by pupil dilation), alpha continued to decrease with increasing memory load. These results suggest that associating alpha with the focus of attention and distractor suppression is not warranted.

Pupil Dilation Response to Prosody and Syntax During Auditory Sentence Processing

Pupil dilation response has been shown to reflect different levels of sentence processing during prosodic and syntactic processing in language comprehension. Our pupillometry experiment aimed to investigate whether pupil diameter was sensitive to the auditory sentence processing involved in comprehending congruent and incongruent statements. Twenty-one participants were presented with 300 auditory stimuli consisting of syntactically and/or prosodically congruent and incongruent sentences in Turkish. The pupillary response results were significant only for syntactically incongruent sentences and for sentences that were both syntactically and prosodically incongruent. This indicates that prosody had no significant effect on its own. Based on the hypothesis that prosodic and syntactic processing require cognitive sensitivity for auditory sentence comprehension, we expected an increase in pupil diameter for both processes. However, our findings are consistent with the previous assumptions that pupil size increases during syntactic manipulation, but our findings showed that prosodic processing does not increase pupil size, contrary to previous studies.

The Involvement of Listening Effort in Explaining Bilingual Listening Under Adverse Listening Conditions

The current review examines listening effort to uncover how it is implicated in bilingual performance under adverse listening conditions. Various measures of listening effort, including physiological, behavioral, and subjective measures, have been employed to examine listening effort in bilingual children and adults. Adverse listening conditions, stemming from environmental factors, as well as factors related to the speaker or listener, have been examined. The existing literature, although relatively limited to date, points to increased listening effort among bilinguals in their nondominant second language (L2) compared to their dominant first language (L1) and relative to monolinguals. Interestingly, increased effort is often observed even when speech intelligibility remains unaffected. These findings emphasize the importance of considering listening effort alongside speech intelligibility. Building upon the insights gained from the current review, we propose that various factors may modulate the observed effects. These include the particular measure selected to examine listening effort, the characteristics of the adverse condition, as well as factors related to the particular linguistic background of the bilingual speaker. Critically, further research is needed to better understand the impact of these factors on listening effort. The review outlines avenues for future research that would promote a comprehensive understanding of listening effort in bilingual individuals.

Self-reported listening effort in adults with and without hearing loss: the Danish version of the Effort Assessment Scale (D-EAS)

PURPOSE

Hearing-impaired individuals experience higher levels of listening effort in challenging situations, affecting their communication in daily life. The purpose of this study was to linguistically and culturally adapt the Effort Assessment Scale (EAS) into Danish (D-EAS) and to investigate its reliability and validity in normal-hearing and hearing-impaired listeners.

METHODS

The translation and cross-cultural adaptation of the EAS aligns with recommendations to adapt hearing-related questionnaires for different languages and cultures. Participants were 157 listeners (85 females) aged 20-86 years (M_age = 62.5, SD = 16.8), with (non-hearing aid and hearing aid users) and without hearing loss.

RESULTS

Reliability analysis showed good internal consistency for the six items in the D-EAS (Cronbach's α= 0.93). Exploratory and confirmatory factor analyses showed that D-EAS is a one-dimensional instrument. Significant differences were observed across items and overall scores between normal hearing (NH) and hearing loss groups.

CONCLUSIONS

The D-EAS reliably estimates self-perception of listening effort in adults with and without hearing loss and is sensitive to the impact of hearing loss. Thus, the D-EAS can provide hearing care professionals and hearing researchers with valuable insights into people's self-perception of listening effort to help guide clinical and other rehabilitation decisions.Implications for RehabilitationThe Effort Assessment Scale (EAS) into Danish (D-EAS) is a reliable tool to estimate self-perception of listening effort in hearing-impaired adults.The D-EAS could be a helpful tool providing insights about aspects of hearing disability that is not commonly addressed with the traditional hearing assessments.The D-EAS can be incorporated in the hearing rehabilitation process as a tool for evaluating self-perception of listening effort in daily life situations.The D-EAS is easy to administer and requires a short time to answer, allowing its use by clinicians and hearing researchers in different settings.

The Effect of Breathy Vocal Quality on Speech Intelligibility and Listening Effort in Background Noise

Speech perception is challenging under adverse conditions. However, there is limited evidence regarding how multiple adverse conditions affect speech perception. The present study investigated two conditions that are frequently encountered in real-life communication: background noise and breathy vocal quality. The study first examined the effects of background noise and breathiness on speech perception as measured by intelligibility. Secondly, the study tested the hypothesis that both noise and breathiness affect listening effort, as indicated by linear and nonlinear changes in pupil dilation. Low-context sentences were resynthesized to create three levels of breathiness (original, mild-moderate, and severe). The sentences were presented in a fluctuating nonspeech noise with two signal-to-noise ratios (SNRs) of -5 dB (favorable) and -9 dB (adverse) SNR. Speech intelligibility and pupil dilation data were collected from young listeners with normal hearing thresholds. The results demonstrated that a breathy vocal quality presented in noise negatively affected speech intelligibility, with the degree of breathiness playing a critical role. Listening effort, as measured by the magnitude of pupil dilation, showed significant effects with both severe and mild-moderate breathy voices that were independent of noise level. The findings contributed to the literature by demonstrating the impact of vocal quality on the perception of speech in noise. They also highlighted the complex dynamics between overall task demand and processing resources in understanding the combined impact of multiple adverse conditions.

Interaural speech asymmetry predicts bilateral speech intelligibility but not listening effort in adults with bilateral cochlear implants

Introduction

Bilateral cochlear implants (BiCIs) can facilitate improved speech intelligibility in noise and sound localization abilities compared to a unilateral implant in individuals with bilateral severe to profound hearing loss. Still, many individuals with BiCIs do not benefit from binaural hearing to the same extent that normal hearing (NH) listeners do. For example, binaural redundancy, a speech intelligibility benefit derived from having access to duplicate copies of a signal, is highly variable among BiCI users. Additionally, patients with hearing loss commonly report elevated listening effort compared to NH listeners. There is some evidence to suggest that BiCIs may reduce listening effort compared to a unilateral CI, but the limited existing literature has not shown this consistently. Critically, no studies to date have investigated this question using pupillometry to quantify listening effort, where large pupil sizes indicate high effort and small pupil sizes indicate low effort. Thus, the present study aimed to build on existing literature by investigating the potential benefits of BiCIs for both speech intelligibility and listening effort.

Methods

Twelve BiCI adults were tested in three listening conditions: Better Ear, Poorer Ear, and Bilateral. Stimuli were IEEE sentences presented from a loudspeaker at 0° azimuth in quiet. Participants were asked to repeat back the sentences, and responses were scored by an experimenter while changes in pupil dilation were measured.

Results

On average, participants demonstrated similar speech intelligibility in the Better Ear and Bilateral conditions, and significantly worse speech intelligibility in the Poorer Ear condition. Despite similar speech intelligibility in the Better Ear and Bilateral conditions, pupil dilation was significantly larger in the Bilateral condition.

Discussion

These results suggest that the BiCI users tested in this study did not demonstrate binaural redundancy in quiet. The large interaural speech asymmetries demonstrated by participants may have precluded them from obtaining binaural redundancy, as shown by the inverse relationship between the two variables. Further, participants did not obtain a release from effort when listening with two ears versus their better ear only. Instead, results indicate that bilateral listening elicited increased effort compared to better ear listening, which may be due to poor integration of asymmetric inputs.

Baseline pupil size encodes task-related information and modulates the task-evoked response in a speech-in-noise task

Pupillometry data are commonly reported relative to a baseline value recorded in a controlled pre-task condition. In this study, the influence of the experimental design and the preparatory processing related to task difficulty on the baseline pupil size was investigated during a speech intelligibility in noise paradigm. Furthermore, the relationship between the baseline pupil size and the temporal dynamics of the pupil response was assessed. The analysis revealed strong effects of block presentation order, within-block sentence order and task difficulty on the baseline values. An interaction between signal-to-noise ratio and block order was found, indicating that baseline values reflect listener expectations arising from the order in which the different blocks were presented. Furthermore, the baseline pupil size was found to affect the slope, delay and curvature of the pupillary response as well as the peak pupil dilation. This suggests that baseline correction might be sufficient when reporting pupillometry results in terms of mean pupil dilation only, but not when a more complex characterization of the temporal dynamics of the response is considered. By clarifying which factors affect baseline pupil size and how baseline values interact with the task-evoked response, the results from the present study can contribute to a better interpretation of the pupillary response as a marker of cognitive processing.

Effortful Listening Despite Correct Responses: The Cost of Mental Repair in Sentence Recognition by Listeners With Cochlear Implants

PURPOSE

Speech recognition percent correct scores fail to capture the effort of mentally repairing the perception of speech that was initially misheard. This study measured the effort of listening to stimuli specifically designed to elicit mental repair in adults who use cochlear implants (CIs).

METHOD

CI listeners heard and repeated sentences in which specific words were distorted or masked by noise but recovered based on later context: a signature of mental repair. Changes in pupil dilation were tracked as an index of effort and time-locked with specific landmarks during perception.

RESULTS

Effort significantly increases when a listener needs to repair a misperceived word, even if the verbal response is ultimately correct. Mental repair of words in a sentence was accompanied by greater prevalence of errors elsewhere in the same sentence, suggesting that effort spreads to consume resources across time. The cost of mental repair in CI listeners was essentially the same as that observed in listeners with normal hearing in previous work.

CONCLUSIONS

Listening effort as tracked by pupil dilation is better explained by the mental repair and reconstruction of words rather than the appearance of correct or incorrect perception. Linguistic coherence drives effort more heavily than the mere presence of mistakes, highlighting the importance of testing materials that do not constrain coherence by design.

Musical experience partially counteracts temporal speech processing deficits in putative mild cognitive impairment

Mild cognitive impairment (MCI) commonly results in more rapid cognitive and behavioral declines than typical aging. Individuals with MCI can exhibit impaired receptive speech abilities that may reflect neurophysiological changes in auditory-sensory processing prior to usual cognitive deficits. Benefits from current interventions targeting communication difficulties in MCI are limited. Yet, neuroplasticity associated with musical experience has been implicated in improving neural representations of speech and offsetting age-related declines in perception. Here, we asked whether these experience-dependent effects of musical experience might extend to aberrant aging and offer some degree of cognitive protection against MCI. During a vowel categorization task, we recorded single-channel electroencephalograms (EEGs) in older adults with putative MCI to evaluate speech encoding across subcortical and cortical levels of the auditory system. Critically, listeners varied in their duration of formal musical experience (0-21 years). Musical experience sharpened temporal precision in auditory cortical responses, suggesting that musical experience produces more efficient processing of acoustic features by counteracting age-related neural delays. Additionally, robustness of brainstem responses predicted the severity of cognitive decline, suggesting that early speech representations are sensitive to preclinical stages of cognitive impairment. Our results extend prior studies by demonstrating positive benefits of musical experience in older adults with emergent cognitive impairments.

Hearing Aid Noise Reduction Lowers the Sustained Listening Effort During Continuous Speech in Noise-A Combined Pupillometry and EEG Study

OBJECTIVES

The investigation of auditory cognitive processes recently moved from strictly controlled, trial-based paradigms toward the presentation of continuous speech. This also allows the investigation of listening effort on larger time scales (i.e., sustained listening effort). Here, we investigated the modulation of sustained listening effort by a noise reduction algorithm as applied in hearing aids in a listening scenario with noisy continuous speech. The investigated directional noise reduction algorithm mainly suppresses noise from the background.

DESIGN

We recorded the pupil size and the EEG in 22 participants with hearing loss who listened to audio news clips in the presence of background multi-talker babble noise. We estimated how noise reduction (off, on) and signal-to-noise ratio (SNR; +3 dB, +8 dB) affect pupil size and the power in the parietal EEG alpha band (i.e., parietal alpha power) as well as the behavioral performance.

RESULTS

Our results show that noise reduction reduces pupil size, while there was no significant effect of the SNR. It is important to note that we found interactions of SNR and noise reduction, which suggested that noise reduction reduces pupil size predominantly under the lower SNR. Parietal alpha power showed a similar yet nonsignificant pattern, with increased power under easier conditions. In line with the participants' reports that one of the two presented talkers was more intelligible, we found a reduced pupil size, increased parietal alpha power, and better performance when people listened to the more intelligible talker.

CONCLUSIONS

We show that the modulation of sustained listening effort (e.g., by hearing aid noise reduction) as indicated by pupil size and parietal alpha power can be studied under more ecologically valid conditions. Mainly concluded from pupil size, we demonstrate that hearing aid noise reduction lowers sustained listening effort. Our study approximates to real-world listening scenarios and evaluates the benefit of the signal processing as can be found in a modern hearing aid.

A comparison of simultaneously-obtained measures of listening effort: pupil dilation, verbal response time and self-rating

OBJECTIVE

The aim of this study was to assess to what extent simultaneously-obtained measures of listening effort (task-evoked pupil dilation, verbal response time [RT], and self-rating) could be sensitive to auditory and cognitive manipulations in a speech perception task. The study also aimed to explore the possible relationship between RT and pupil dilation.

DESIGN

A within-group design was adopted. All participants were administered the Matrix Sentence Test in 12 conditions (signal-to-noise ratios [SNR] of -3, -6, -9 dB; attentional resources focussed vs divided; spatial priors present vs absent).

STUDY SAMPLE

Twenty-four normal-hearing adults, 20-41 years old (M = 23.5), were recruited in the study.

RESULTS

A significant effect of the SNR was found for all measures. However, pupil dilation discriminated only partially between the SNRs. Neither of the cognitive manipulations were effective in modulating the measures. No relationship emerged between pupil dilation, RT and self-ratings.

CONCLUSIONS

RT, pupil dilation, and self-ratings can be obtained simultaneously when administering speech perception tasks, even though some limitations remain related to the absence of a retention period after the listening phase. The sensitivity of the three measures to changes in the auditory environment differs. RTs and self-ratings proved most sensitive to changes in SNR.

Cognitive and Physiological Measures of Listening Effort During Degraded Speech Perception: Relating Dual-Task and Pupillometry Paradigms

Purpose Listening effort is quickly becoming an important metric for assessing speech perception in less-than-ideal situations. However, the relationship between the construct of listening effort and the measures used to assess it remains unclear. We compared two measures of listening effort: a cognitive dual task and a physiological pupillometry task. We sought to investigate the relationship between these measures of effort and whether engaging effort impacts speech accuracy. Method In Experiment 1, 30 participants completed a dual task and a pupillometry task that were carefully matched in stimuli and design. The dual task consisted of a spoken word recognition task and a visual match-to-sample task. In the pupillometry task, pupil size was monitored while participants completed a spoken word recognition task. Both tasks presented words at three levels of listening difficulty (unmodified, eight-channel vocoding, and four-channel vocoding) and provided response feedback on every trial. We refined the pupillometry task in Experiment 2 (n = 31); crucially, participants no longer received response feedback. Finally, we ran a new group of subjects on both tasks in Experiment 3 (n = 30). Results In Experiment 1, accuracy in the visual task decreased with increased signal degradation in the dual task, but pupil size was sensitive to accuracy and not vocoding condition. After removing feedback in Experiment 2, changes in pupil size were predicted by listening condition, suggesting the task was now sensitive to engaged effort. Both tasks were sensitive to listening difficulty in Experiment 3, but there was no relationship between the tasks and neither task predicted speech accuracy. Conclusions Consistent with previous work, we found little evidence for a relationship between different measures of listening effort. We also found no evidence that effort predicts speech accuracy, suggesting that engaging more effort does not lead to improved speech recognition. Cognitive and physiological measures of listening effort are likely sensitive to different aspects of the construct of listening effort. Supplemental Material https://doi.org/10.23641/asha.16455900.

Contrast Is in the Eye of the Beholder: Infelicitous Beat Gesture Increases Cognitive Load During Online Spoken Discourse Comprehension

We investigated how two cues to contrast-beat gesture and contrastive pitch accenting-affect comprehenders' cognitive load during processing of spoken referring expressions. In two visual-world experiments, we orthogonally manipulated the presence of these cues and their felicity, or fit, with the local (sentence-level) referential context in critical referring expressions while comprehenders' task-evoked pupillary responses (TEPRs) were examined. In Experiment 1, beat gesture and contrastive accenting always matched the referential context of filler referring expressions and were therefore relatively felicitous on the global (experiment) level, whereas in Experiment 2, beat gesture and contrastive accenting never fit the referential context of filler referring expressions and were therefore infelicitous on the global level. The results revealed that both beat gesture and contrastive accenting increased comprehenders' cognitive load. For beat gesture, this increase in cognitive load was driven by both local and global infelicity. For contrastive accenting, this increase in cognitive load was unaffected when cues were globally felicitous but exacerbated when cues were globally infelicitous. Together, these results suggest that comprehenders' cognitive resources are taxed by processing infelicitous use of beat gesture and contrastive accenting to convey contrast on both the local and global levels.

Listening Effort Is Not the Same as Speech Intelligibility Score

Listening effort is a valuable and important notion to measure because it is among the primary complaints of people with hearing loss. It is tempting and intuitive to accept speech intelligibility scores as a proxy for listening effort, but this link is likely oversimplified and lacks actionable explanatory power. This study was conducted to explain the mechanisms of listening effort that are not captured by intelligibility scores, using sentence-repetition tasks where specific kinds of mistakes were prospectively planned or analyzed retrospectively. Effort measured as changes in pupil size among 20 listeners with normal hearing and 19 listeners with cochlear implants. Experiment 1 demonstrates that mental correction of misperceived words increases effort even when responses are correct. Experiment 2 shows that for incorrect responses, listening effort is not a function of the proportion of words correct but is rather driven by the types of errors, position of errors within a sentence, and the need to resolve ambiguity, reflecting how easily the listener can make sense of a perception. A simple taxonomy of error types is provided that is both intuitive and consistent with data from these two experiments. The diversity of errors in these experiments implies that speech perception tasks can be designed prospectively to elicit the mistakes that are more closely linked with effort. Although mental corrective action and number of mistakes can scale together in many experiments, it is possible to dissociate them to advance toward a more explanatory (rather than correlational) account of listening effort.

Three New Outcome Measures That Tap Into Cognitive Processes Required for Real-Life Communication

To increase the ecological validity of outcomes from laboratory evaluations of hearing and hearing devices, it is desirable to introduce more realistic outcome measures in the laboratory. This article presents and discusses three outcome measures that have been designed to go beyond traditional speech-in-noise measures to better reflect realistic everyday challenges. The outcome measures reviewed are: the Sentence-final Word Identification and Recall (SWIR) test that measures working memory performance while listening to speech in noise at ceiling performance; a neural tracking method that produces a quantitative measure of selective speech attention in noise; and pupillometry that measures changes in pupil dilation to assess listening effort while listening to speech in noise. According to evaluation data, the SWIR test provides a sensitive measure in situations where speech perception performance might be unaffected. Similarly, pupil dilation has also shown sensitivity in situations where traditional speech-in-noise measures are insensitive. Changes in working memory capacity and effort mobilization were found at positive signal-to-noise ratios (SNR), that is, at SNRs that might reflect everyday situations. Using stimulus reconstruction, it has been demonstrated that neural tracking is a robust method at determining to what degree a listener is attending to a specific talker in a typical cocktail party situation. Using both established and commercially available noise reduction schemes, data have further shown that all three measures are sensitive to variation in SNR. In summary, the new outcome measures seem suitable for testing hearing and hearing devices under more realistic and demanding everyday conditions than traditional speech-in-noise tests.

Perceptual Quality of Audio-Visual Content with Common Video and Audio Degradations

Audio-visual quality assessment remains as a complex research field. A great effort is being made to understand how visual and auditory domains are integrated and processed by humans. In this work, we analyzed and compared the results of three psychophisical experiments that collected quality and content scores given by a pool of subjects. The experiments include diverse content audio-visual material, e.g., Sports, TV Commercials, Interviews, Music, Documentaries and Cartoons, impaired with several visual (bitrate compression, packet-loss, and frame-freezing) and auditory (background noise, echo, clip, chop) distortions. Each experiment explores a particular domain. In Experiment 1, the video component was degraded with visual artifacts, meanwhile, the audio component did not suffer any type of degradation. In Experiment 2, the audio component was degraded while the video component remained untouched. Finally, in Experiment 3 both audio and video components were degraded. As expected, results confirmed a dominance of the visual component in the overall audio-visual quality. However, a detailed analysis showed that, for certain types of audio distortions, the audio component played a more important role in the construction of the overall perceived quality.

Pupil Dilation Is Sensitive to Semantic Ambiguity and Acoustic Degradation

Speech comprehension is challenged by background noise, acoustic interference, and linguistic factors, such as the presence of words with more than one meaning (homonyms and homophones). Previous work suggests that homophony in spoken language increases cognitive demand. Here, we measured pupil dilation—a physiological index of cognitive demand—while listeners heard high-ambiguity sentences, containing words with more than one meaning, or well-matched low-ambiguity sentences without ambiguous words. This semantic-ambiguity manipulation was crossed with an acoustic manipulation in two experiments. In Experiment 1, sentences were masked with 30-talker babble at 0 and +6 dB signal-to-noise ratio (SNR), and in Experiment 2, sentences were heard with or without a pink noise masker at –2 dB SNR. Speech comprehension was measured by asking listeners to judge the semantic relatedness of a visual probe word to the previous sentence. In both experiments, comprehension was lower for high- than for low-ambiguity sentences when SNRs were low. Pupils dilated more when sentences included ambiguous words, even when no noise was added (Experiment 2). Pupil also dilated more when SNRs were low. The effect of masking was larger than the effect of ambiguity for performance and pupil responses. This work demonstrates that the presence of homophones, a condition that is ubiquitous in natural language, increases cognitive demand and reduces intelligibility of speech heard with a noisy background.

The Principle of Least Effort and Comprehension of Spoken Sentences by Younger and Older Adults

There is considerable evidence that listeners' understanding of a spoken sentence need not always follow from a full analysis of the words and syntax of the utterance. Rather, listeners may instead conduct a superficial analysis, sampling some words and using presumed plausibility to arrive at an understanding of the sentence meaning. Because this latter strategy occurs more often for sentences with complex syntax that place a heavier processing burden on the listener than sentences with simpler syntax, shallow processing may represent a resource conserving strategy reflected in reduced processing effort. This factor may be even more important for older adults who as a group are known to have more limited working memory resources. In the present experiment, 40 older adults (M age = 75.5 years) and 20 younger adults (M age = 20.7) were tested for comprehension of plausible and implausible sentences with a simpler subject-relative embedded clause structure or a more complex object-relative embedded clause structure. Dilation of the pupil of the eye was recorded as an index of processing effort. Results confirmed greater comprehension accuracy for plausible than implausible sentences, and for sentences with simpler than more complex syntax, with both effects amplified for the older adults. Analysis of peak pupil dilations for implausible sentences revealed a complex three-way interaction between age, syntactic complexity, and plausibility. Results are discussed in terms of models of sentence comprehension, and pupillometry as an index of intentional task engagement.

Dimensions of self-reported listening effort and fatigue on a digits-in-noise task, and association with baseline pupil size and performance accuracy

OBJECTIVE

Pupillometry is sensitive to cognitive resource allocation and has been used as a potential measure of listening-related effort and fatigue. We investigated associations between peak pupil diameter, pre-stimulus pupil diameter, performance on a listening task, and the dimensionality of self-reported outcomes (task-related listening effort and fatigue).

DESIGN

Pupillometry was recorded while participants performed a speech-in-noise task. Participants rated their experience of listening effort and fatigue using the NASA-Task Load Index (NASA-TLX) and the Visual Analogue Scale of Fatigue (VAS-F), respectively. The dimensionality of the NASA-TLX and the VAS-F was investigated using factor analysis.

STUDY SAMPLE

82 participants with either normal hearing or aided hearing impairment (age range: 55-85 years old, 43 male).

RESULTS

Hierarchal linear regression analyses suggested that pre-stimulus pupil diameter predicts a dimension of self-reported fatigue, which we interpreted as tiredness/drowsiness, and listening task performance when controlling for hearing level and age: Larger pre-stimulus pupil diameter was associated with less tiredness/drowsiness and better task performance.

CONCLUSION

Pre-stimulus pupil diameter is a potential index of listening fatigue associated with speech processing in challenging listening conditions. To our knowledge, this is the first investigation of the associations between pre-stimulus pupil diameter and self-reported ratings of listening effort and fatigue.

Investigating the Influences of Task Demand and Reward on Cardiac Pre-Ejection Period Reactivity During a Speech-in-Noise Task

OBJECTIVES

Effort investment during listening varies as a function of task demand and motivation. Several studies have manipulated both these factors to elicit and measure changes in effort associated with listening. The cardiac pre-ejection period (PEP) is a relatively novel measure in the field of cognitive hearing science. This measure, which reflects sympathetic nervous system activity on the heart, has previously been implemented during a tone discrimination task but not during a speech-in-noise task. Therefore, the primary goal of this study was to explore the influences of signal to noise ratio (SNR) and monetary reward level on PEP reactivity during a speech-in-noise task.

DESIGN

Thirty-two participants with normal hearing (mean age = 22.22 years, SD = 3.03) were recruited at VU University Medical Center. Participants completed a Dutch speech-in-noise test with a single-interfering-talker masking noise. Six fixed SNRs, selected to span the entire psychometric performance curve, were presented in a block-wise fashion. Participants could earn a low (€0.20) or high (€5.00) reward by obtaining a score of ≥70% of words correct in each block. The authors analyzed PEP reactivity: the change in PEP measured during the task, relative to the baseline during rest. Two separate methods of PEP analysis were used, one including data from the whole task block and the other including data obtained during presentation of the target sentences only. After each block, participants rated their effort investment, performance, tendency to give up, and the perceived difficulty of the task. They also completed the need for recovery questionnaire and the reading span test, which are indices of additional factors (fatigue and working memory capacity, respectively) that are known to influence listening effort.

RESULTS

Average sentence perception scores ranged from 2.73 to 91.62%, revealing a significant effect of SNR. In addition, an improvement in performance was elicited by the high, compared to the low reward level. A linear relationship between SNR and PEP reactivity was demonstrated: at the lower SNRs PEP reactivity was the most negative, indicating greater effort investment compared to the higher SNRs. The target stimuli method of PEP analysis was more sensitive to this effect than the block-wise method. Contrary to expectations, no significant impact of reward on PEP reactivity was found in the present dataset. Also, there was no physiological evidence that participants were disengaged, even when performance was poor. A significant correlation between need for recovery scores and average PEP reactivity was demonstrated, indicating that a lower need for recovery was associated with less effort investment.

CONCLUSIONS

This study successfully implemented the measurement of PEP during a standard speech-in-noise test and included two distinct methods of PEP analysis. The results revealed for the first time that PEP reactivity varies linearly with task demand during a speech-in-noise task, although the effect size was small. No effect of reward on PEP was demonstrated. Finally, participants with a higher need for recovery score invested more effort, as shown by average PEP reactivity, than those with a lower need for recovery score.

Absorption and Enjoyment During Listening to Acoustically Masked Stories

Comprehension of speech masked by background sound requires increased cognitive processing, which makes listening effortful. Research in hearing has focused on such challenging listening experiences, in part because they are thought to contribute to social withdrawal in people with hearing impairment. Research has focused less on positive listening experiences, such as enjoyment, despite their potential importance in motivating effortful listening. Moreover, the artificial speech materials-such as disconnected, brief sentences-commonly used to investigate speech intelligibility and listening effort may be ill-suited to capture positive experiences when listening is challenging. Here, we investigate how listening to naturalistic spoken stories under acoustic challenges influences the quality of listening experiences. We assess absorption (the feeling of being immersed/engaged in a story), enjoyment, and listening effort and show that (a) story absorption and enjoyment are only minimally affected by moderate speech masking although listening effort increases, (b) thematic knowledge increases absorption and enjoyment and reduces listening effort when listening to a story presented in multitalker babble, and (c) absorption and enjoyment increase and effort decreases over time as individuals listen to several stories successively in multitalker babble. Our research indicates that naturalistic, spoken stories can reveal several concurrent listening experiences and that expertise in a topic can increase engagement and reduce effort. Our work also demonstrates that, although listening effort may increase with speech masking, listeners may still find the experience both absorbing and enjoyable.

Pupillometry as an Objective Measure of Sustained Attention in Young and Older Listeners

The ability to sustain attention on a task-relevant sound source while avoiding distraction from concurrent sounds is fundamental to listening in crowded environments. We aimed to (a) devise an experimental paradigm with which this aspect of listening can be isolated and (b) evaluate the applicability of pupillometry as an objective measure of sustained attention in young and older populations. We designed a paradigm that continuously measured behavioral responses and pupillometry during 25-s trials. Stimuli contained a number of concurrent, spectrally distinct tone streams. On each trial, participants detected gaps in one of the streams while resisting distraction from the others. Behavior demonstrated increasing difficulty with time-on-task and with number/proximity of distractor streams. In young listeners (N = 20; aged 18 to 35 years), pupil diameter (on the group and individual level) was dynamically modulated by instantaneous task difficulty: Periods where behavioral performance revealed a strain on sustained attention were accompanied by increased pupil diameter. Only trials on which participants performed successfully were included in the pupillometry analysis so that the observed effects reflect task demands as opposed to failure to attend. In line with existing reports, we observed global changes to pupil dynamics in the older group (N = 19; aged 63 to 79 years) including decreased pupil diameter, limited dilation range, and reduced temporal variability. However, despite these changes, older listeners showed similar effects of attentive tracking to those observed in the young listeners. Overall, our results demonstrate that pupillometry can be a reliable and time-sensitive measure of attentive tracking over long durations in both young and (with caveats) older listeners.

Effects of acoustic and semantic cues on listening effort during native and non-native speech perception

Relative to native listeners, non-native listeners who are immersed in a second language environment experience increased listening effort and a reduced ability to successfully perform an additional task while listening. Previous research demonstrated that listeners can exploit a variety of intelligibility-enhancing cues to cope with adverse listening conditions. However, little is known about the implications of those speech perception strategies for listening effort. The current research aims to investigate by means of pupillometry how listening effort is modulated in native and non-native listeners by the availability of semantic context and acoustic enhancements during the comprehension of spoken sentences. For this purpose, semantic plausibility and speaking style were manipulated both separately and in combination during a speech perception task in noise. The signal to noise ratio was individually adjusted for each participant in order to target 50% intelligibility level. Behavioural results indicated that native and non-native listeners were equally able to fruitfully exploit both semantic and acoustic cues to aid their comprehension. Pupil data indicated that listening effort was reduced for both groups of listeners when acoustic enhancements were available, while the presence of a plausible semantic context did not lead to a reduction in listening effort.

Measures of Listening Effort Are Multidimensional

OBJECTIVES

Listening effort can be defined as the cognitive resources required to perform a listening task. The literature on listening effort is as confusing as it is voluminous: measures of listening effort rarely correlate with each other and sometimes result in contradictory findings. Here, we directly compared simultaneously recorded multimodal measures of listening effort. After establishing the reliability of the measures, we investigated validity by quantifying correlations between measures and then grouping-related measures through factor analysis.

DESIGN

One hundred and sixteen participants with audiometric thresholds ranging from normal to severe hearing loss took part in the study (age range: 55 to 85 years old, 50.3% male). We simultaneously measured pupil size, electroencephalographic alpha power, skin conductance, and self-report listening effort. One self-report measure of fatigue was also included. The signal to noise ratio (SNR) was adjusted at 71% criterion performance using sequences of 3 digits. The main listening task involved correct recall of a random digit from a sequence of six presented at a SNR where performance was around 82 to 93%. Test-retest reliability of the measures was established by retesting 30 participants 7 days after the initial session.

RESULTS

With the exception of skin conductance and the self-report measure of fatigue, interclass correlation coefficients (ICC) revealed good test-retest reliability (minimum ICC: 0.71). Weak or nonsignificant correlations were identified between measures. Factor analysis, using only the reliable measures, revealed four underlying dimensions: factor 1 included SNR, hearing level, baseline alpha power, and performance accuracy; factor 2 included pupillometry; factor 3 included alpha power (during speech presentation and during retention); factor 4 included self-reported listening effort and baseline alpha power.

CONCLUSIONS

The good ICC suggests that poor test reliability is not the reason for the lack of correlation between measures. We have demonstrated that measures traditionally used as indicators of listening effort tap into multiple underlying dimensions. We therefore propose that there is no "gold standard" measure of listening effort and that different measures of listening effort should not be used interchangeably. When choosing method(s) to measure listening effort, the nature of the task and aspects of increased listening demands that are of interest should be taken into account. The findings of this study provide a framework for understanding and interpreting listening effort measures.

Anticipatory Baseline Pupil Diameter Is Sensitive to Differences in Hearing Thresholds

Task-evoked changes in pupil dilation have long been used as a physiological index of cognitive effort. Unlike this response, that is measured during or after an experimental trial, the baseline pupil dilation (BPD) is a measure taken prior to an experimental trial. As such, it is considered to reflect an individual’s arousal level in anticipation of an experimental trial. We report data for 68 participants, ages 18 to 89, whose hearing acuity ranged from normal hearing to a moderate hearing loss, tested over a series 160 trials on an auditory sentence comprehension task. Results showed that BPDs progressively declined over the course of the experimental trials, with participants with poorer pure tone detection thresholds showing a steeper rate of decline than those with better thresholds. Data showed this slope difference to be due to participants with poorer hearing having larger BPDs than those with better hearing at the start of the experiment, but with their BPDs approaching that of the better hearing participants by the end of the 160 trials. A finding of increasing response accuracy over trials was seen as inconsistent with a fatigue or reduced task engagement account of the diminishing BPDs. Rather, the present results imply BPD as reflecting a heightened arousal level in poorer-hearing participants in anticipation of a task that demands accurate speech perception, a concern that dissipates over trials with task success. These data taken with others suggest that the baseline pupillary response may not reflect a single construct.

Autonomic Nervous System Correlates of Speech Categorization Revealed Through Pupillometry

Human perception requires the many-to-one mapping between continuous sensory elements and discrete categorical representations. This grouping operation underlies the phenomenon of categorical perception (CP)-the experience of perceiving discrete categories rather than gradual variations in signal input. Speech perception requires CP because acoustic cues do not share constant relations with perceptual-phonetic representations. Beyond facilitating perception of unmasked speech, we reasoned CP might also aid the extraction of target speech percepts from interfering sound sources (i.e., noise) by generating additional perceptual constancy and reducing listening effort. Specifically, we investigated how noise interference impacts cognitive load and perceptual identification of unambiguous (i.e., categorical) vs. ambiguous stimuli. Listeners classified a speech vowel continuum (/u/-/a/) at various signal-to-noise ratios (SNRs [unmasked, 0 and -5 dB]). Continuous recordings of pupil dilation measured processing effort, with larger, later dilations reflecting increased listening demand. Critical comparisons were between time-locked changes in eye data in response to unambiguous (i.e., continuum endpoints) tokens vs. ambiguous tokens (i.e., continuum midpoint). Unmasked speech elicited faster responses and sharper psychometric functions, which steadily declined in noise. Noise increased pupil dilation across stimulus conditions, but not straightforwardly. Noise-masked speech modulated peak pupil size (i.e., [0 and -5 dB] > unmasked). In contrast, peak dilation latency varied with both token and SNR. Interestingly, categorical tokens elicited earlier pupil dilation relative to ambiguous tokens. Our pupillary data suggest CP reconstructs auditory percepts under challenging listening conditions through interactions between stimulus salience and listeners' internalized effort and/or arousal.

Psychobiological Responses Reveal Audiovisual Noise Differentially Challenges Speech Recognition

OBJECTIVES

In noisy environments, listeners benefit from both hearing and seeing a talker, demonstrating audiovisual (AV) cues enhance speech-in-noise (SIN) recognition. Here, we examined the relative contribution of auditory and visual cues to SIN perception and the strategies used by listeners to decipher speech in noise interference(s).

DESIGN

Normal-hearing listeners (n = 22) performed an open-set speech recognition task while viewing audiovisual TIMIT sentences presented under different combinations of signal degradation including visual (AVn), audio (AnV), or multimodal (AnVn) noise. Acoustic and visual noises were matched in physical signal-to-noise ratio. Eyetracking monitored participants' gaze to different parts of a talker's face during SIN perception.

RESULTS

As expected, behavioral performance for clean sentence recognition was better for A-only and AV compared to V-only speech. Similarly, with noise in the auditory channel (AnV and AnVn speech), performance was aided by the addition of visual cues of the talker regardless of whether the visual channel contained noise, confirming a multimodal benefit to SIN recognition. The addition of visual noise (AVn) obscuring the talker's face had little effect on speech recognition by itself. Listeners' eye gaze fixations were biased toward the eyes (decreased at the mouth) whenever the auditory channel was compromised. Fixating on the eyes was negatively associated with SIN recognition performance. Eye gazes on the mouth versus eyes of the face also depended on the gender of the talker.

CONCLUSIONS

Collectively, results suggest listeners (1) depend heavily on the auditory over visual channel when seeing and hearing speech and (2) alter their visual strategy from viewing the mouth to viewing the eyes of a talker with signal degradations, which negatively affects speech perception.

Bilingual aphasia: Assessing cross-linguistic asymmetries and bilingual advantage in sentence comprehension deficits

People with aphasia frequently have difficulties understanding semantically reversible sentences presented in derived word order. This impairment may be related to the inconsistent processing of morphological information, as well as to difficulties inhibiting the inverse interpretation of the sentence. Studies on bilingual aphasia may contribute to our understanding of these issues by shedding light on i) differences in processing of morphology across languages; ii) enhanced control mechanisms. We studied early Basque-Spanish bilingual speakers with aphasia and monolingual Spanish speakers with aphasia, as well as unimpaired individuals. Using comparable sets of materials across languages, we combined behavioural and eye-tracking methods. Results indicate that i) at the group level, bilingual speakers perform better in Spanish than in Basque, particularly in sentences with Theme-Agent argument order. Individual case analysis shows a pattern of weak dissociation across languages in several participants; ii) bilingual people with aphasia do not outperform monolingual people with aphasia in comprehension accuracy, although gaze data suggests that bilingual speakers exhibit higher inhibition and monitoring abilities.