Autonomic Nervous System Responses During Perception of Masked Speech may Reflect Constructs other than Subjective Listening Effort

Incorporating Virtual Reality Agents During a Dichotic Speech Reception Task: Insights From the Heart

OBJECTIVES

Listening effort is moderated by not only task difficulty, but also success importance. In real communication scenarios, success importance varies based upon the social context. However, in the laboratory, it can be challenging to manipulate social context without compromising experimental control. Outside of hearing sciences, studies have applied virtual reality (VR) to incorporate social context in a controlled and repeatable manner. Several of these studies have demonstrated that social manipulations in VR can reliably elicit changes in cardiovascular measures. Here, we investigated the effect of adding VR agents to a speech reception task, while measuring from the cardiovascular system.

DESIGN

Twenty-eight, normally hearing participants undertook a dichotic speech reception task. Sentences in stationary noise were presented dichotically, that is, different sentences presented simultaneously to each ear. Participants were tasked to either repeat one of the sentences (single-sentence condition) or both of the sentences (dual-sentence condition). The task was conducted under two VR conditions: (1) in the presence of agents, who provided sporadic performance feedback and (2) in the presence of nonagent controls, without any feedback given. Alongside task performance, we quantified changes in cardiovascular measures, relative to pretask baselines: heart rate variability, pre-ejection period, heart rate, and blood pressure. After each condition, participants rated their subjective effort, difficulty, performance, and engagement.

RESULTS

Performance and the subjective perception of performance were lower, while subjective effort and difficulty were higher, in the dual-sentence condition, compared with the single-sentence condition. Heart rate was the only cardiovascular measure that was sensitive to the experimental manipulations. Contrary to our expectations, heart rate increased in the nonagent control conditions, compared with the agent conditions. An exploratory analysis revealed heart rate fluctuations within a trial: heart rate was higher during the first 6 sec of the trial (reflecting the presentence masking noise and the sentence presentation) in the dual-sentence condition, compared with the single-sentence condition.

CONCLUSIONS

This study was the first to incorporate VR agents who provided performance feedback during a dichotic speech reception task. Our results suggest that the VR agents did not increase success importance, which could be attributed to a lack of realism of the agents. We also demonstrated that the cardiovascular response to experimental manipulations may differ depending on the data window selected for analysis.

The time course of cognitive effort during disrupted speech

Listeners often find themselves in scenarios where speech is disrupted, misperceived, or otherwise difficult to recognise. In these situations, many individuals report exerting additional effort to understand speech, even when repairing speech may be difficult or impossible. This investigation aimed to characterise cognitive efforts across time during both sentence listening and a post-sentence retention interval by observing the pupillary response of participants with normal to borderline-normal hearing in response to two interrupted speech conditions: sentences interrupted by gaps of silence or bursts of noise. The pupillary response serves as a measure of the cumulative resources devoted to task completion. Both interruption conditions resulted in significantly greater levels of pupil dilation than the uninterrupted speech condition. Just prior to the end of a sentence, trials periodically interrupted by bursts of noise elicited greater pupil dilation than the silent-interrupted condition. Compared to the uninterrupted condition, both interruption conditions resulted in increased dilation after sentence end but before repetition, possibly reflecting sustained processing demands. Understanding pupil dilation as a marker of cognitive effort is important for clinicians and researchers when assessing the additional effort exerted by listeners with hearing loss who may use cochlear implants or hearing aids. Even when successful perceptual repair is unlikely, listeners may continue to exert increased effort when processing misperceived speech, which could cause them to miss upcoming speech or may contribute to heightened listening fatigue.

The Neurophysiological Costs of Learning in a Noisy Classroom: An Ecological Virtual Reality Study

Many real-life situations can be extremely noisy, which makes it difficult to understand what people say. Here, we introduce a novel audiovisual virtual reality experimental platform to study the behavioral and neurophysiological consequences of background noise on processing continuous speech in highly realistic environments. We focus on a context where the ability to understand speech is particularly important: the classroom. Participants (n = 32) experienced sitting in a virtual reality classroom and were told to pay attention to a virtual teacher giving a lecture. Trials were either quiet or contained background construction noise, emitted from outside the classroom window. Two realistic types of noise were used: continuous drilling and intermittent air hammers. Alongside behavioral outcomes, we measured several neurophysiological metrics, including neural activity (EEG), eye-gaze and skin conductance (galvanic skin response). Our results confirm the detrimental effect of background noise. Construction noise, and particularly intermittent noise, was associated with reduced behavioral performance, reduced neural tracking of the teacher's speech and an increase in skin conductance, although it did not have a significant effect on alpha-band oscillations or eye-gaze patterns. These results demonstrate the neurophysiological costs of learning in noisy environments and emphasize the role of temporal dynamics in speech-in-noise perception. The finding that intermittent noise was more disruptive than continuous noise supports a "habituation" rather than "glimpsing" hypothesis of speech-in-noise processing. These results also underscore the importance of increasing the ecologically relevance of neuroscientific research and considering acoustic, temporal, and semantic features of realistic stimuli as well as the cognitive demands of real-life environments.

Electromyographic correlates of effortful listening in the vestigial auriculomotor system

Recently, electromyographic (EMG) signals of auricular muscles have been shown to be an indicator of spatial auditory attention in humans, based on a vestigial pinna-orienting system. Because spatial auditory attention in a competing speaker task is closely related to the more generalized concept of attentional effort in listening, the current study investigated the possibility that the EMG activity of auricular muscles could also reflect correlates of effortful listening in general. Twenty participants were recruited. EMG signals from the left and right superior and posterior auricular muscles (SAM, PAM) were recorded while participants attended a target podcast in a competing speaker paradigm. Three different conditions, each more difficult and requiring a higher amount of effortful listening, were generated by varying the number and pitch of distractor streams, as well as the signal-to-noise ratio. All audio streams were either presented from a loudspeaker placed in front of the participants (0°), or in the back (180°). Overall, averaged PAM activity was not affected by different levels of effortful listening, but was significantly larger when stimuli were presented from the back, as opposed to the front. Averaged SAM activity, however, was significantly larger in the most difficult condition, which required the largest amount of effort, compared to the easier conditions, but was not affected by stimulus direction. We interpret the increased SAM activity to be the response of the vestigial pinna-orienting system to an effortful stream segregation task.

Pupillometry and perceived listening effort for cochlear implant users-a comparison of three speech-in-noise tests

OBJECTIVE

Measuring listening effort using pupillometry is challenging in cochlear implant (CI) users. We assess three validated speech tests (Matrix, LIST, and DIN) to identify the optimal speech material for measuring peak-pupil-dilation (PPD) in CI users as a function of signal-to-noise ratio (SNR).

DESIGN

Speech tests were administered in quiet and two noisy conditions, namely at the speech recognition threshold (0 dB re SRT), i.e. the SNR where speech intelligibility (SI) was 50%, and at a more favourable SNR of +6 dB re SRT. PPDs and subjective ratings of effort were obtained.

STUDY SAMPLE

Eighteen unilaterally implanted CI users.

RESULTS

LIST sentences revealed significantly different PPDs between +6 and 0 dB re SRT and DIN triplets between quiet and +6 dB re SRT. PPDs obtained with the Matrix test were independent of SNR and yielded large PPDs and high subjective ratings even in quiet.

CONCLUSIONS

PPD is a sensitive measure for listening effort when processing LIST sentences near 0 dB re SRT and when processing DIN triplets at more favourable listening conditions around +6 dB re SRT. PPDs obtained with the Matrix test were insensitive to SNR, likely because it is demanding for CI users even in quiet.

Listening Effort Measured With Pupillometry in Cochlear Implant Users Depends on Sound Level, But Not on the Signal to Noise Ratio When Using the Matrix Test

OBJECTIVES

We investigated whether listening effort is dependent on task difficulty for cochlear implant (CI) users when using the Matrix speech-in-noise test. To this end, we measured peak pupil dilation (PPD) at a wide range of signal to noise ratios (SNR) by systematically changing the noise level at a constant speech level, and vice versa.

DESIGN

A group of mostly elderly CI users performed the Dutch/Flemish Matrix test in quiet and in multitalker babble at different SNRs. SNRs were set relative to the speech-recognition threshold (SRT), namely at SRT, and 5 and 10 dB above SRT (0 dB, +5 dB, and +10 dB re SRT). The latter 2 conditions were obtained by either varying speech level (at a fixed noise level of 60 dBA) or by varying noise level (with a fixed speech level). We compared these PPDs with those of a group of typical hearing (TH) listeners. In addition, listening effort was assessed with subjective ratings on a Likert scale.

RESULTS

PPD for the CI group did not significantly depend on SNR, whereas SNR significantly affected PPDs for TH listeners. Subjective effort ratings depended significantly on SNR for both groups. For CI users, PPDs were significantly larger, and effort was rated higher when speech was varied, and noise was fixed for CI users. By contrast, for TH listeners effort ratings were significantly higher and performance scores lower when noise was varied, and speech was fixed.

CONCLUSIONS

The lack of a significant effect of varying SNR on PPD suggests that the Matrix test may not be a feasible speech test for measuring listening effort with pupillometric measures for CI users. A rating test appeared more promising in this population, corroborating earlier reports that subjective measures may reflect different dimensions of listening effort than pupil dilation. Establishing the SNR by varying speech or noise level can have subtle, but significant effects on measures of listening effort, and these effects can differ between TH listeners and CI users.

Empatica E4 Assessment of Child Physiological Measures of Listening Effort During Remote and In-Person Communication

PURPOSE

Telepractice is a growing service model that delivers aural rehabilitation to deaf and hard-of hearing children via telecommunications technology. Despite known benefits of telepractice, this delivery approach may increase patients' listening effort (LE) characterized as an allocation of cognitive resources toward an auditory task. The study tested techniques for collecting physiological measures of LE in normal-hearing (NH) children during remote (referred to as tele-) and in-person communication using the wearable Empatica E4 wristband.

METHOD

Participants were 10 children (age range: 9-12 years old) who came to two tele- and two in-person weekly sessions, order counterbalanced. During each session, the children heard a short passage read by the clinical provider, completed an auditory passage comprehension task, and self-rated their effort as a part of the larger study. Measures of electrodermal activity and blood volume pulse amplitude were collected from the child E4 wristband.

RESULTS

No differences in child subjective, physiological measures of LE or passage comprehension scores were found between in-person sessions and telesessions. However, an effect of treatment duration on subjective and physiological measures of LE was identified. Children self-reported a significant increase in LE over time. However, their physiological measures demonstrated a trend indicating a decrease in LE. A significant association between subjective measures and the passage comprehension task was found suggesting that those children who reported more effort demonstrated a higher proportion of correct responses.

CONCLUSIONS

The study demonstrated the feasibility of collection of physiological measures of LE in NH children during remote and in-person communication using the E4 wristband. The results suggest that measures of LE are multidimensional and may reflect different sources of, or cognitive responses to, increased listening demand.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.27122064.

A Multimodal Approach to Measuring Listening Effort: A Systematic Review on the Effects of Auditory Task Demand on Physiological Measures and Their Relationship

OBJECTIVES

Listening effort involves the mental effort required to perceive an auditory stimulus, for example in noisy environments. Prolonged increased listening effort, for example due to impaired hearing ability, may increase risk of health complications. It is therefore important to identify valid and sensitive measures of listening effort. Physiological measures have been shown to be sensitive to auditory task demand manipulations and are considered to reflect changes in listening effort. Such measures include pupil dilation, alpha power, skin conductance level, and heart rate variability. The aim of the current systematic review was to provide an overview of studies to listening effort that used multiple physiological measures. The two main questions were: (1) what is the effect of changes in auditory task demand on simultaneously acquired physiological measures from various modalities? and (2) what is the relationship between the responses in these physiological measures?

DESIGN

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, relevant articles were sought in PubMed, PsycInfo, and Web of Science and by examining the references of included articles. Search iterations with different combinations of psychophysiological measures were performed in conjunction with listening effort-related search terms. Quality was assessed using the Appraisal Tool for Cross-Sectional Studies.

RESULTS

A total of 297 articles were identified from three databases, of which 27 were included. One additional article was identified from reference lists. Of the total 28 included articles, 16 included an analysis regarding the relationship between the physiological measures. The overall quality of the included studies was reasonable.

CONCLUSIONS

The included studies showed that most of the physiological measures either show no effect to auditory task demand manipulations or a consistent effect in the expected direction. For example, pupil dilation increased, pre-ejection period decreased, and skin conductance level increased with increasing auditory task demand. Most of the relationships between the responses of these physiological measures were nonsignificant or weak. The physiological measures varied in their sensitivity to auditory task demand manipulations. One of the identified knowledge gaps was that the included studies mostly used tasks with high-performance levels, resulting in an underrepresentation of the physiological changes at lower performance levels. This makes it difficult to capture how the physiological responses behave across the full psychometric curve. Our results support the Framework for Understanding Effortful Listening and the need for a multimodal approach to listening effort. We furthermore discuss focus points for future studies.

Effects of hearing acuity on psychophysiological responses to effortful speech perception

In recent studies, psychophysiological measures have been used as markers of listening effort, but there is limited research on the effect of hearing loss on such measures. The aim of the current study was to investigate the effect of hearing acuity on physiological responses and subjective measures acquired during different levels of listening demand, and to investigate the relationship between these measures. A total of 125 participants (37 males and 88 females, age range 37-72 years, pure-tone average hearing thresholds at the best ear between -5.0 to 68.8 dB HL and asymmetry between ears between 0.0 and 87.5 dB) completed a listening task. A speech reception threshold (SRT) test was used with target sentences spoken by a female voice masked by male speech. Listening demand was manipulated using three levels of intelligibility: 20 % correct speech recognition, 50 %, and 80 % (IL20 %/IL50 %/IL80 %, respectively). During the task, peak pupil dilation (PPD), heart rate (HR), pre-ejection period (PEP), respiratory sinus arrhythmia (RSA), and skin conductance level (SCL) were measured. For each condition, subjective ratings of effort, performance, difficulty, and tendency to give up were also collected. Linear mixed effects models tested the effect of intelligibility level, hearing acuity, hearing asymmetry, and tinnitus complaints on the physiological reactivity (compared to baseline) and subjective measures. PPD and PEP reactivity showed a non-monotonic relationship with intelligibility level, but no such effects were found for HR, RSA, or SCL reactivity. Participants with worse hearing acuity had lower PPD at all intelligibility levels and showed lower PEP baseline levels. Additionally, PPD and SCL reactivity were lower for participants who reported suffering from tinnitus complaints. For IL80 %, but not IL50 % or IL20 %, participants with worse hearing acuity rated their listening effort to be relatively high compared to participants with better hearing. The reactivity of the different physiological measures were not or only weakly correlated with each other. Together, the results suggest that hearing acuity may be associated with altered sympathetic nervous system (re)activity. Research using psychophysiological measures as markers of listening effort to study the effect of hearing acuity on such measures are best served by the use of the PPD and PEP.

Influence of Speech Rate on Auditory Recognition in Cochlear Implant Users

INTRODUCTION

This study aimed to verify the influence of speech stimulus presentation and speed on auditory recognition in cochlear implant (CI) users with poorer performance.

METHODS

The cross-sectional observational study applied auditory speech perception tests to fifteen adults, using three different ways of presenting the stimulus, in the absence of competitive noise: monitored live voice (MLV); recorded speech at typical speed (RSTS); recorded speech at slow speed (RSSS). The scores were assessed using the Percent Sentence Recognition Index (PSRI). The data were inferentially analysed using the Friedman and Wilcoxon tests with a 95% confidence interval and 5% significance level (p < 0.05).

RESULTS

The mean age was 41.1 years, the mean duration of CI use was 11.4 years, and the mean hearing threshold was 29.7 ± 5.9 dBHL. Test performance, as determined by the PSRI, was MLV = 42.4 ± 17.9%; RSTS = 20.3 ± 14.3%; RSSS = 40.6 ± 20.7%. There was a significant difference identified for RSTS compared to MLV and RSSS.

CONCLUSION

The way the stimulus is presented and the speed at which it is presented enable greater auditory speech recognition in CI users, thus favouring comprehension when the tests are applied in the MLV and RSSS modalities.

Combining Cardiovascular and Pupil Features Using k-Nearest Neighbor Classifiers to Assess Task Demand, Social Context, and Sentence Accuracy During Listening

In daily life, both acoustic factors and social context can affect listening effort investment. In laboratory settings, information about listening effort has been deduced from pupil and cardiovascular responses independently. The extent to which these measures can jointly predict listening-related factors is unknown. Here we combined pupil and cardiovascular features to predict acoustic and contextual aspects of speech perception. Data were collected from 29 adults (mean  =  64.6 years, SD  =  9.2) with hearing loss. Participants performed a speech perception task at two individualized signal-to-noise ratios (corresponding to 50% and 80% of sentences correct) and in two social contexts (the presence and absence of two observers). Seven features were extracted per trial: baseline pupil size, peak pupil dilation, mean pupil dilation, interbeat interval, blood volume pulse amplitude, pre-ejection period and pulse arrival time. These features were used to train k-nearest neighbor classifiers to predict task demand, social context and sentence accuracy. The k-fold cross validation on the group-level data revealed above-chance classification accuracies: task demand, 64.4%; social context, 78.3%; and sentence accuracy, 55.1%. However, classification accuracies diminished when the classifiers were trained and tested on data from different participants. Individually trained classifiers (one per participant) performed better than group-level classifiers: 71.7% (SD  =  10.2) for task demand, 88.0% (SD  =  7.5) for social context, and 60.0% (SD  =  13.1) for sentence accuracy. We demonstrated that classifiers trained on group-level physiological data to predict aspects of speech perception generalized poorly to novel participants. Individually calibrated classifiers hold more promise for future applications.

Listening Effort in Tinnitus: A Pilot Study Employing a Light EEG Headset and Skin Conductance Assessment during the Listening to a Continuous Speech Stimulus under Different SNR Conditions

Background noise elicits listening effort. What else is tinnitus if not an endogenous background noise? From such reasoning, we hypothesized the occurrence of increased listening effort in tinnitus patients during listening tasks. Such a hypothesis was tested by investigating some indices of listening effort through electroencephalographic and skin conductance, particularly parietal and frontal alpha and electrodermal activity (EDA). Furthermore, tinnitus distress questionnaires (THI and TQ12-I) were employed. Parietal alpha values were positively correlated to TQ12-I scores, and both were negatively correlated to EDA; Pre-stimulus frontal alpha correlated with the THI score in our pilot study; finally, results showed a general trend of increased frontal alpha activity in the tinnitus group in comparison to the control group. Parietal alpha during the listening to stimuli, positively correlated to the TQ12-I, appears to reflect a higher listening effort in tinnitus patients and the perception of tinnitus symptoms. The negative correlation between both listening effort (parietal alpha) and tinnitus symptoms perception (TQ12-I scores) with EDA levels could be explained by a less responsive sympathetic nervous system to prepare the body to expend increased energy during the "fight or flight" response, due to pauperization of energy from tinnitus perception.

Brightening the Study of Listening Effort with Functional Near-Infrared Spectroscopy: A Scoping Review

Listening effort is a long-standing area of interest in auditory cognitive neuroscience. Prior research has used multiple techniques to shed light on the neurophysiological mechanisms underlying listening during challenging conditions. Functional near-infrared spectroscopy (fNIRS) is growing in popularity as a tool for cognitive neuroscience research, and its recent advances offer many potential advantages over other neuroimaging modalities for research related to listening effort. This review introduces the basic science of fNIRS and its uses for auditory cognitive neuroscience. We also discuss its application in recently published studies on listening effort and consider future opportunities for studying effortful listening with fNIRS. After reading this article, the learner will know how fNIRS works and summarize its uses for listening effort research. The learner will also be able to apply this knowledge toward generation of future research in this area.

Effortful Listening Despite Correct Responses: The Cost of Mental Repair in Sentence Recognition by Listeners With Cochlear Implants

PURPOSE

Speech recognition percent correct scores fail to capture the effort of mentally repairing the perception of speech that was initially misheard. This study measured the effort of listening to stimuli specifically designed to elicit mental repair in adults who use cochlear implants (CIs).

METHOD

CI listeners heard and repeated sentences in which specific words were distorted or masked by noise but recovered based on later context: a signature of mental repair. Changes in pupil dilation were tracked as an index of effort and time-locked with specific landmarks during perception.

RESULTS

Effort significantly increases when a listener needs to repair a misperceived word, even if the verbal response is ultimately correct. Mental repair of words in a sentence was accompanied by greater prevalence of errors elsewhere in the same sentence, suggesting that effort spreads to consume resources across time. The cost of mental repair in CI listeners was essentially the same as that observed in listeners with normal hearing in previous work.

CONCLUSIONS

Listening effort as tracked by pupil dilation is better explained by the mental repair and reconstruction of words rather than the appearance of correct or incorrect perception. Linguistic coherence drives effort more heavily than the mere presence of mistakes, highlighting the importance of testing materials that do not constrain coherence by design.

Adding noise is a confounded nuisance

A wide variety of research and clinical assessments involve presenting speech stimuli in the presence of some kind of noise. Here, I selectively review two theoretical perspectives and discuss ways in which these perspectives may help researchers understand the consequences for listeners of adding noise to a speech signal. I argue that adding noise changes more about the listening task than merely making the signal more difficult to perceive. To fully understand the effects of an added noise on speech perception, we must consider not just how much the noise affects task difficulty, but also how it affects all of the systems involved in understanding speech: increasing message uncertainty, modifying attentional demand, altering affective response, and changing motivation to perform the task.

Interdisciplinary Approaches to the Study of Listening Effort in Young Children with Cochlear Implants

Very early bilateral implantation is thought to significantly reduce the attentional effort required to acquire spoken language, and consequently offer a profound improvement in quality of life. Despite the early intervention, however, auditory and communicative outcomes in children with cochlear implants remain poorer than in hearing children. The distorted auditory input via the cochlear implants requires more auditory attention resulting in increased listening effort and fatigue. Listening effort and fatigue may critically affect attention to speech, and in turn language processing, which may help to explain the variation in language and communication abilities. However, measuring attention to speech and listening effort is demanding in infants and very young children. Three objective techniques for measuring listening effort are presented in this paper that may address the challenges of testing very young and/or uncooperative children with cochlear implants: pupillometry, electroencephalography, and functional near-infrared spectroscopy. We review the studies of listening effort that used these techniques in paediatric populations with hearing loss, and discuss potential benefits of the systematic evaluation of listening effort in these populations.

Psychophysiological responses to potentially annoying heating, ventilation, and air conditioning noise during mentally demanding work

Exposure to noise-or unwanted sound-is considered a major public health issue in the United States and internationally. Previous work has shown that even acute noise exposure can influence physiological response in humans and that individuals differ markedly in their susceptibility to noise. Recent research also suggests that specific acoustic properties of noise may have distinct effects on human physiological response. Much of the existing research on physiological response to noise consists of laboratory studies using very simple acoustic stimuli-like white noise or tone bursts-or field studies of longer-term workplace noise exposure that may neglect acoustic properties of the noise entirely. By using laboratory exposure to realistic heating, ventilation, and air conditioning (HVAC) noise, the current study explores the interaction between acoustic properties of annoying noise and individual response to working in occupational noise. This study assessed autonomic response to two acoustically distinct noises while participants performed cognitively demanding work. Results showed that the two HVAC noises affected physiological arousal in different ways. Individual differences in physiological response to noise as a function of noise sensitivity were also observed. Further research is necessary to link specific acoustic characteristics with differential physiological responses in humans.

Understanding degraded speech leads to perceptual gating of a brainstem reflex in human listeners

The ability to navigate "cocktail party" situations by focusing on sounds of interest over irrelevant, background sounds is often considered in terms of cortical mechanisms. However, subcortical circuits such as the pathway underlying the medial olivocochlear (MOC) reflex modulate the activity of the inner ear itself, supporting the extraction of salient features from auditory scene prior to any cortical processing. To understand the contribution of auditory subcortical nuclei and the cochlea in complex listening tasks, we made physiological recordings along the auditory pathway while listeners engaged in detecting non(sense) words in lists of words. Both naturally spoken and intrinsically noisy, vocoded speech-filtering that mimics processing by a cochlear implant (CI)-significantly activated the MOC reflex, but this was not the case for speech in background noise, which more engaged midbrain and cortical resources. A model of the initial stages of auditory processing reproduced specific effects of each form of speech degradation, providing a rationale for goal-directed gating of the MOC reflex based on enhancing the representation of the energy envelope of the acoustic waveform. Our data reveal the coexistence of 2 strategies in the auditory system that may facilitate speech understanding in situations where the signal is either intrinsically degraded or masked by extrinsic acoustic energy. Whereas intrinsically degraded streams recruit the MOC reflex to improve representation of speech cues peripherally, extrinsically masked streams rely more on higher auditory centres to denoise signals.

Listening Effort Is Not the Same as Speech Intelligibility Score

Listening effort is a valuable and important notion to measure because it is among the primary complaints of people with hearing loss. It is tempting and intuitive to accept speech intelligibility scores as a proxy for listening effort, but this link is likely oversimplified and lacks actionable explanatory power. This study was conducted to explain the mechanisms of listening effort that are not captured by intelligibility scores, using sentence-repetition tasks where specific kinds of mistakes were prospectively planned or analyzed retrospectively. Effort measured as changes in pupil size among 20 listeners with normal hearing and 19 listeners with cochlear implants. Experiment 1 demonstrates that mental correction of misperceived words increases effort even when responses are correct. Experiment 2 shows that for incorrect responses, listening effort is not a function of the proportion of words correct but is rather driven by the types of errors, position of errors within a sentence, and the need to resolve ambiguity, reflecting how easily the listener can make sense of a perception. A simple taxonomy of error types is provided that is both intuitive and consistent with data from these two experiments. The diversity of errors in these experiments implies that speech perception tasks can be designed prospectively to elicit the mistakes that are more closely linked with effort. Although mental corrective action and number of mistakes can scale together in many experiments, it is possible to dissociate them to advance toward a more explanatory (rather than correlational) account of listening effort.

Slower Speaking Rate Reduces Listening Effort Among Listeners With Cochlear Implants

OBJECTIVES

Slowed speaking rate was examined for its effects on speech intelligibility, its interaction with the benefit of contextual cues, and the impact of these factors on listening effort in adults with cochlear implants.

DESIGN

Participants (n = 21 cochlear implant users) heard high- and low-context sentences that were played at the original speaking rate, as well as a slowed (1.4× duration) speaking rate, using uniform pitch-synchronous time warping. In addition to intelligibility measures, changes in pupil dilation were measured as a time-varying index of processing load or listening effort. Slope of pupil size recovery to baseline after the sentence was used as an index of resolution of perceptual ambiguity.

RESULTS

Speech intelligibility was better for high-context compared to low-context sentences and slightly better for slower compared to original-rate speech. Speech rate did not affect magnitude and latency of peak pupil dilation relative to sentence offset. However, baseline pupil size recovered more substantially for slower-rate sentences, suggesting easier processing in the moment after the sentence was over. The effect of slowing speech rate was comparable to changing a sentence from low context to high context. The effect of context on pupil dilation was not observed until after the sentence was over, and one of two analyses suggested that context had greater beneficial effects on listening effort when the speaking rate was slower. These patterns maintained even at perfect sentence intelligibility, suggesting that correct speech repetition does not guarantee efficient or effortless processing. With slower speaking rates, there was less variability in pupil dilation slopes following the sentence, implying mitigation of some of the difficulties shown by individual listeners who would otherwise demonstrate prolonged effort after a sentence is heard.

CONCLUSIONS

Slowed speaking rate provides release from listening effort when hearing an utterance, particularly relieving effort that would have lingered after a sentence is over. Context arguably provides even more release from listening effort when speaking rate is slower. The pattern of prolonged pupil dilation for faster speech is consistent with increased need to mentally correct errors, although that exact interpretation cannot be verified with intelligibility data alone or with pupil data alone. A pattern of needing to dwell on a sentence to disambiguate misperceptions likely contributes to difficulty in running conversation where there are few opportunities to pause and resolve recently heard utterances.

Listener characteristics differentially affect self-reported and physiological measures of effort associated with two challenging listening conditions

Listeners vary in their ability to understand speech in adverse conditions. Differences in both cognitive and linguistic capacities play a role, but increasing evidence suggests that such factors may contribute differentially depending on the listening challenge. Here, we used multilevel modeling to evaluate contributions of individual differences in age, hearing thresholds, vocabulary, selective attention, working memory capacity, personality traits, and noise sensitivity to variability in measures of comprehension and listening effort in two listening conditions. A total of 35 participants completed a battery of cognitive and linguistic tests as well as a spoken story comprehension task using (1) native-accented English speech masked by speech-shaped noise and (2) nonnative accented English speech without masking. Masker levels were adjusted individually to ensure each participant would show (close to) equivalent word recognition performance across the two conditions. Dependent measures included comprehension tests results, self-rated effort, and electrodermal, cardiovascular, and facial electromyographic measures associated with listening effort. Results showed varied patterns of responsivity across different dependent measures as well as across listening conditions. In particular, results suggested that working memory capacity may play a greater role in the comprehension of nonnative accented speech than noise-masked speech, while hearing acuity and personality may have a stronger influence on physiological responses affected by demands of understanding speech in noise. Furthermore, electrodermal measures may be more strongly affected by affective response to noise-related interference while cardiovascular responses may be more strongly affected by demands on working memory and lexical access.

NeuroDante: Poetry Mentally Engages More Experts but Moves More Non-Experts, and for Both the Cerebral Approach Tendency Goes Hand in Hand with the Cerebral Effort

Neuroaesthetics, the science studying the biological underpinnings of aesthetic experience, recently extended its area of investigation to literary art; this was the humus where neurocognitive poetics blossomed. Divina Commedia represents one of the most important, famous and studied poems worldwide. Poetry stimuli are characterized by elements (meter and rhyme) promoting the processing fluency, a core aspect of neuroaesthetics theories. In addition, given the evidence of different neurophysiological reactions between experts and non-experts in response to artistic stimuli, the aim of the present study was to investigate, in poetry, a different neurophysiological cognitive and emotional reaction between Literature (L) and Non-Literature (NL) students. A further aim was to investigate whether neurophysiological underpinnings would support explanation of behavioral data. Investigation methods employed: self-report assessments (recognition, appreciation, content recall) and neurophysiological indexes (approach/withdrawal (AW), cerebral effort (CE) and galvanic skin response (GSR)). The main behavioral results, according to fluency theories in aesthetics, suggested in the NL but not in the L group that the appreciation/liking went hand by hand with the self-declared recognition and with the content recall. The main neurophysiological results were: (i) higher galvanic skin response in NL, whilst higher CE values in L; (ii) a positive correlation between AW and CE indexes in both groups. The present results extended previous evidence relative to figurative art also to auditory poetry stimuli, suggesting an emotional attenuation “expertise-specific” showed by experts, but increased cognitive processing in response to the stimuli.

Listening effort: Are we measuring cognition or affect, or both?

Listening effort is increasingly recognized as a factor in communication, particularly for and with nonnative speakers, for the elderly, for individuals with hearing impairment and/or for those working in noise. However, as highlighted by McGarrigle et al., International Journal of Audiology, 2014, 53, 433-445, the term "listening effort" encompasses a wide variety of concepts, including the engagement and control of multiple possibly distinct neural systems for information processing, and the affective response to the expenditure of those resources in a given context. Thus, experimental or clinical methods intended to objectively quantify listening effort may ultimately reflect a complex interaction between the operations of one or more of those information processing systems, and/or the affective and motivational response to the demand on those systems. Here we examine theoretical, behavioral, and psychophysiological factors related to resolving the question of what we are measuring, and why, when we measure "listening effort." This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Theory and Methods Psychology > Attention Psychology > Emotion and Motivation.

Moderate Reverberation Does Not Increase Subjective Fatigue, Subjective Listening Effort, or Behavioral Listening Effort in School-Aged Children

Background noise and reverberation levels in typical classrooms have negative effects on speech recognition, but their effects on listening effort and fatigue are less well understood. Based on the Framework for Understanding Effortful Listening, noise and reverberation would be expected to increase both listening effort and fatigue. However, previous investigations of the effects of reverberation for adults have resulted in mixed findings. Some discrepancies in the literature might be accounted for by methodological differences; behavioral and subjective indices of listening effort do not often align in adults. The effects of sustained listening on self-reported fatigue in school-aged children are also not well understood. The purposes of this project were to (1) evaluate the effects of noise and reverberation on listening effort in school-aged children using behavioral and subjective measures, (2) compare subjective and behavioral indices of listening effort, and (3) evaluate the effects of reverberation on self-reported fatigue. Twenty typically developing children (10-17 years old) participated. Participants completed dual-task testing in two rooms that varied in terms of reverberation, an audiometric sound booth and a moderately reverberant room. In each room, testing was completed in quiet and in two levels of background noise. Participants provided subjective ratings of listening effort after completing the dual-task in each listening condition. Subjective ratings of fatigue were completed before and after testing in each level of reverberation. Results revealed background noise, not reverberation, increased behavioral and subjective listening effort. Subjective ratings of perceived performance, ease of listening, and desire to control the listening situation revealed a similar pattern of results as word recognition performance, making them poor candidates for providing an indication of behavioral listening effort. However, ratings of time perception were moderately correlated with behavioral listening effort. Finally, sustained listening for approximately 25 min increased self-reported fatigue, although changes in fatigue were comparable in low and moderately reverberant environments. In total, these data offer no evidence that a moderate level of reverberation increases listening effort or fatigue, but the data do support the reduction of background noise in classrooms.

Effect of Auditory Task Type on Physiological and Subjective Measures of Listening Effort in Individuals With Normal Hearing

Purpose Listening effort has traditionally been measured using subjective rating scales and behavioral measures. Recent physiological measures of listening effort have utilized pupil dilation. Using a combination of physiological and subjective measures of listening effort, this study aimed to identify differences in listening effort during 2 auditory tasks: sentence recognition and word recognition. Method Pupil dilation and subjective ratings of listening effort were obtained for auditory tasks utilizing AzBio sentences recognition and Northwestern University Auditory Test No. 6 words recognition, across 3 listening situations: in quiet, at +6 dB signal-to-noise ratio, and at 0 dB signal-to-noise ratio. Task accuracy was recorded for each of the 6 conditions, as well as peak pupil dilation and a subjective rating of listening effort. Results A significant impact of listening situation (quiet vs. noise) and task type (sentence recognition vs. word recognition) on both physiological and subjective measures was found. There was a significant interaction between listening situation and task type, suggesting that contextual cues may only be beneficial when audibility is uncompromised. The current study found no correlation between the physiological and subjective measures, possibly suggesting that these measures analyze different aspects of cognitive effort in a listening task.

Pupillometry Reveals That Context Benefit in Speech Perception Can Be Disrupted by Later-Occurring Sounds, Especially in Listeners With Cochlear Implants

Contextual cues can be used to improve speech recognition, especially for people with hearing impairment. However, previous work has suggested that when the auditory signal is degraded, context might be used more slowly than when the signal is clear. This potentially puts the hearing-impaired listener in a dilemma of continuing to process the last sentence when the next sentence has already begun. This study measured the time course of the benefit of context using pupillary responses to high- and low-context sentences that were followed by silence or various auditory distractors (babble noise, ignored digits, or attended digits). Participants were listeners with cochlear implants or normal hearing using a 12-channel noise vocoder. Context-related differences in pupil dilation were greater for normal hearing than for cochlear implant listeners, even when scaled for differences in pupil reactivity. The benefit of context was systematically reduced for both groups by the presence of the later-occurring sounds, including virtually complete negation when sentences were followed by another attended utterance. These results challenge how we interpret the benefit of context in experiments that present just one utterance at a time. If a listener uses context to “repair” part of a sentence, and later-occurring auditory stimuli interfere with that repair process, the benefit of context might not survive outside the idealized laboratory or clinical environment. Elevated listening effort in hearing-impaired listeners might therefore result not just from poor auditory encoding but also inefficient use of context and prolonged processing of misperceived utterances competing with perception of incoming speech.

Emotionally conditioning the target-speech voice enhances recognition of the target speech under "cocktail-party" listening conditions

Under a noisy "cocktail-party" listening condition with multiple people talking, listeners can use various perceptual/cognitive unmasking cues to improve recognition of the target speech against informational speech-on-speech masking. One potential unmasking cue is the emotion expressed in a speech voice, by means of certain acoustical features. However, it was unclear whether emotionally conditioning a target-speech voice that has none of the typical acoustical features of emotions (i.e., an emotionally neutral voice) can be used by listeners for enhancing target-speech recognition under speech-on-speech masking conditions. In this study we examined the recognition of target speech against a two-talker speech masker both before and after the emotionally neutral target voice was paired with a loud female screaming sound that has a marked negative emotional valence. The results showed that recognition of the target speech (especially the first keyword in a target sentence) was significantly improved by emotionally conditioning the target speaker's voice. Moreover, the emotional unmasking effect was independent of the unmasking effect of the perceived spatial separation between the target speech and the masker. Also, (skin conductance) electrodermal responses became stronger after emotional learning when the target speech and masker were perceptually co-located, suggesting an increase of listening efforts when the target speech was informationally masked. These results indicate that emotionally conditioning the target speaker's voice does not change the acoustical parameters of the target-speech stimuli, but the emotionally conditioned vocal features can be used as cues for unmasking target speech.

Psychophysiological measurement of affective responses during speech perception

When people make decisions about listening, such as whether to continue attending to a particular conversation or whether to wear their hearing aids to a particular restaurant, they do so on the basis of more than just their estimated performance. Recent research has highlighted the vital role of more subjective qualities such as effort, motivation, and fatigue. Here, we argue that the importance of these factors is largely mediated by a listener's emotional response to the listening challenge, and suggest that emotional responses to communication challenges may provide a crucial link between day-to-day communication stress and long-term health. We start by introducing some basic concepts from the study of emotion and affect. We then develop a conceptual framework to guide future research on this topic through examination of a variety of autonomic and peripheral physiological responses that have been employed to investigate both cognitive and affective phenomena related to challenging communication. We conclude by suggesting the need for further investigation of the links between communication difficulties, emotional response, and long-term health, and make some recommendations intended to guide future research on affective psychophysiology in speech communication.

Best Practices and Advice for Using Pupillometry to Measure Listening Effort: An Introduction for Those Who Want to Get Started

Within the field of hearing science, pupillometry is a widely used method for quantifying listening effort. Its use in research is growing exponentially, and many labs are (considering) applying pupillometry for the first time. Hence, there is a growing need for a methods paper on pupillometry covering topics spanning from experiment logistics and timing to data cleaning and what parameters to analyze. This article contains the basic information and considerations needed to plan, set up, and interpret a pupillometry experiment, as well as commentary about how to interpret the response. Included are practicalities like minimal system requirements for recording a pupil response and specifications for peripheral, equipment, experiment logistics and constraints, and different kinds of data processing. Additional details include participant inclusion and exclusion criteria and some methodological considerations that might not be necessary in other auditory experiments. We discuss what data should be recorded and how to monitor the data quality during recording in order to minimize artifacts. Data processing and analysis are considered as well. Finally, we share insights from the collective experience of the authors and discuss some of the challenges that still lie ahead.