Is Listening in Noise Worth It? The Neurobiology of Speech Recognition in Challenging Listening Conditions

Reduced Semantic Context and Signal-to-Noise Ratio Increase Listening Effort As Measured Using Functional Near-Infrared Spectroscopy

OBJECTIVES

Understanding speech-in-noise can be highly effortful. Decreasing the signal-to-noise ratio (SNR) of speech increases listening effort, but it is relatively unclear if decreasing the level of semantic context does as well. The current study used functional near-infrared spectroscopy to evaluate two primary hypotheses: (1) listening effort (operationalized as oxygenation of the left lateral PFC) increases as the SNR decreases and (2) listening effort increases as context decreases.

DESIGN

Twenty-eight younger adults with normal hearing completed the Revised Speech Perception in Noise Test, in which they listened to sentences and reported the final word. These sentences either had an easy SNR (+4 dB) or a hard SNR (-2 dB), and were either low in semantic context (e.g., "Tom could have thought about the sport") or high in context (e.g., "She had to vacuum the rug"). PFC oxygenation was measured throughout using functional near-infrared spectroscopy.

RESULTS

Accuracy on the Revised Speech Perception in Noise Test was worse when the SNR was hard than when it was easy, and worse for sentences low in semantic context than high in context. Similarly, oxygenation across the entire PFC (including the left lateral PFC) was greater when the SNR was hard, and left lateral PFC oxygenation was greater when context was low.

CONCLUSIONS

These results suggest that activation of the left lateral PFC (interpreted here as reflecting listening effort) increases to compensate for acoustic and linguistic challenges. This may reflect the increased engagement of domain-general and domain-specific processes subserved by the dorsolateral prefrontal cortex (e.g., cognitive control) and inferior frontal gyrus (e.g., predicting the sensory consequences of articulatory gestures), respectively.

Semantic Predictability Facilitates Comprehension of Degraded Speech in a Graded Manner

Previous studies have shown that at moderate levels of spectral degradation, semantic predictability facilitates language comprehension. It is argued that when speech is degraded, listeners have narrowed expectations about the sentence endings; i.e., semantic prediction may be limited to only most highly predictable sentence completions. The main objectives of this study were to (i) examine whether listeners form narrowed expectations or whether they form predictions across a wide range of probable sentence endings, (ii) assess whether the facilitatory effect of semantic predictability is modulated by perceptual adaptation to degraded speech, and (iii) use and establish a sensitive metric for the measurement of language comprehension. For this, we created 360 German Subject-Verb-Object sentences that varied in semantic predictability of a sentence-final target word in a graded manner (high, medium, and low) and levels of spectral degradation (1, 4, 6, and 8 channels noise-vocoding). These sentences were presented auditorily to two groups: One group (n =48) performed a listening task in an unpredictable channel context in which the degraded speech levels were randomized, while the other group (n =50) performed the task in a predictable channel context in which the degraded speech levels were blocked. The results showed that at 4 channels noise-vocoding, response accuracy was higher in high-predictability sentences than in the medium-predictability sentences, which in turn was higher than in the low-predictability sentences. This suggests that, in contrast to the narrowed expectations view, comprehension of moderately degraded speech, ranging from low- to high- including medium-predictability sentences, is facilitated in a graded manner; listeners probabilistically preactivate upcoming words from a wide range of semantic space, not limiting only to highly probable sentence endings. Additionally, in both channel contexts, we did not observe learning effects; i.e., response accuracy did not increase over the course of experiment, and response accuracy was higher in the predictable than in the unpredictable channel context. We speculate from these observations that when there is no trial-by-trial variation of the levels of speech degradation, listeners adapt to speech quality at a long timescale; however, when there is a trial-by-trial variation of the high-level semantic feature (e.g., sentence predictability), listeners do not adapt to low-level perceptual property (e.g., speech quality) at a short timescale.

Listening Effort Is Not the Same as Speech Intelligibility Score

Listening effort is a valuable and important notion to measure because it is among the primary complaints of people with hearing loss. It is tempting and intuitive to accept speech intelligibility scores as a proxy for listening effort, but this link is likely oversimplified and lacks actionable explanatory power. This study was conducted to explain the mechanisms of listening effort that are not captured by intelligibility scores, using sentence-repetition tasks where specific kinds of mistakes were prospectively planned or analyzed retrospectively. Effort measured as changes in pupil size among 20 listeners with normal hearing and 19 listeners with cochlear implants. Experiment 1 demonstrates that mental correction of misperceived words increases effort even when responses are correct. Experiment 2 shows that for incorrect responses, listening effort is not a function of the proportion of words correct but is rather driven by the types of errors, position of errors within a sentence, and the need to resolve ambiguity, reflecting how easily the listener can make sense of a perception. A simple taxonomy of error types is provided that is both intuitive and consistent with data from these two experiments. The diversity of errors in these experiments implies that speech perception tasks can be designed prospectively to elicit the mistakes that are more closely linked with effort. Although mental corrective action and number of mistakes can scale together in many experiments, it is possible to dissociate them to advance toward a more explanatory (rather than correlational) account of listening effort.

Age-related hearing loss and cognitive decline: MRI and cellular evidence

Extensive evidence supports the association between age-related hearing loss (ARHL) and cognitive decline. It is, however, unknown whether a causal relationship exists between these two, or whether they both result from shared mechanisms. This paper intends to study this relationship through a comprehensive review of MRI findings as well as evidence of cellular alterations. Our review of structural MRI studies demonstrates that ARHL is independently linked to accelerated atrophy of total and regional brain volumes and reduced white matter integrity. Resting-state and task-based fMRI studies on ARHL also show changes in spontaneous neural activity and brain functional connectivity; and alterations in brain areas supporting auditory, language, cognitive, and affective processing independent of age, respectively. Although MRI findings support a causal relationship between ARHL and cognitive decline, the contribution of potential shared mechanisms should also be considered. In this regard, the review of cellular evidence indicates their role as possible common mechanisms underlying both age-related changes in hearing and cognition. Considering existing evidence, no single hypothesis can explain the link between ARHL and cognitive decline, and the contribution of both causal (i.e., the sensory hypothesis) and shared (i.e., the common cause hypothesis) mechanisms is expected.

Neural correlates of subjective comprehension of noise-vocoded speech

Under an acoustically degraded condition, the degree of speech comprehension fluctuates within individuals. Understanding the relationship between such fluctuations in comprehension and neural responses might reveal perceptual processing for distorted speech. In this study we investigated the cerebral activity associated with the degree of subjective comprehension of noise-vocoded speech sounds (NVSS) using functional magnetic resonance imaging. Our results indicate that higher comprehension of NVSS sentences was associated with greater activation in the right superior temporal cortex, and that activity in the left inferior frontal gyrus (Broca's area) was increased when a listener recognized words in a sentence they did not fully comprehend. In addition, results of laterality analysis demonstrated that recognition of words in an NVSS sentence led to less lateralized responses in the temporal cortex, though a left-lateralization was observed when no words were recognized. The data suggest that variation in comprehension within individuals can be associated with changes in lateralization in the temporal auditory cortex.

Pupil Dilation Is Sensitive to Semantic Ambiguity and Acoustic Degradation

Speech comprehension is challenged by background noise, acoustic interference, and linguistic factors, such as the presence of words with more than one meaning (homonyms and homophones). Previous work suggests that homophony in spoken language increases cognitive demand. Here, we measured pupil dilation—a physiological index of cognitive demand—while listeners heard high-ambiguity sentences, containing words with more than one meaning, or well-matched low-ambiguity sentences without ambiguous words. This semantic-ambiguity manipulation was crossed with an acoustic manipulation in two experiments. In Experiment 1, sentences were masked with 30-talker babble at 0 and +6 dB signal-to-noise ratio (SNR), and in Experiment 2, sentences were heard with or without a pink noise masker at –2 dB SNR. Speech comprehension was measured by asking listeners to judge the semantic relatedness of a visual probe word to the previous sentence. In both experiments, comprehension was lower for high- than for low-ambiguity sentences when SNRs were low. Pupils dilated more when sentences included ambiguous words, even when no noise was added (Experiment 2). Pupil also dilated more when SNRs were low. The effect of masking was larger than the effect of ambiguity for performance and pupil responses. This work demonstrates that the presence of homophones, a condition that is ubiquitous in natural language, increases cognitive demand and reduces intelligibility of speech heard with a noisy background.

The Principle of Least Effort and Comprehension of Spoken Sentences by Younger and Older Adults

There is considerable evidence that listeners' understanding of a spoken sentence need not always follow from a full analysis of the words and syntax of the utterance. Rather, listeners may instead conduct a superficial analysis, sampling some words and using presumed plausibility to arrive at an understanding of the sentence meaning. Because this latter strategy occurs more often for sentences with complex syntax that place a heavier processing burden on the listener than sentences with simpler syntax, shallow processing may represent a resource conserving strategy reflected in reduced processing effort. This factor may be even more important for older adults who as a group are known to have more limited working memory resources. In the present experiment, 40 older adults (M age = 75.5 years) and 20 younger adults (M age = 20.7) were tested for comprehension of plausible and implausible sentences with a simpler subject-relative embedded clause structure or a more complex object-relative embedded clause structure. Dilation of the pupil of the eye was recorded as an index of processing effort. Results confirmed greater comprehension accuracy for plausible than implausible sentences, and for sentences with simpler than more complex syntax, with both effects amplified for the older adults. Analysis of peak pupil dilations for implausible sentences revealed a complex three-way interaction between age, syntactic complexity, and plausibility. Results are discussed in terms of models of sentence comprehension, and pupillometry as an index of intentional task engagement.

Disentangling listening effort and memory load beyond behavioural evidence: Pupillary response to listening effort during a concurrent memory task

Recent research has demonstrated that pupillometry is a robust measure for quantifying listening effort. However, pupillary responses in listening situations where multiple cognitive functions are engaged and sustained over a period of time remain hard to interpret. This limits our conceptualisation and understanding of listening effort in realistic situations, because rarely in everyday life are people challenged by one task at a time. Therefore, the purpose of this experiment was to reveal the dynamics of listening effort in a sustained listening condition using a word repeat and recall task. Words were presented in quiet and speech-shaped noise at different signal-to-noise ratios (SNR): 0dB, 7dB, 14dB and quiet. Participants were presented with lists of 10 words, and required to repeat each word after its presentation. At the end of the list, participants either recalled as many words as possible or moved on to the next list. Simultaneously, their pupil dilation was recorded throughout the whole experiment. When only word repeating was required, peak pupil dilation (PPD) was bigger in 0dB versus other conditions; whereas when recall was required, PPD showed no difference among SNR levels and PPD in 0dB was smaller than repeat-only condition. Baseline pupil diameter and PPD followed different variation patterns across the 10 serial positions within a block for conditions requiring recall: baseline pupil diameter built up progressively and plateaued in the later positions (but shot up when listeners were recalling the previously heard words from memory); PPD decreased at a pace quicker than in repeat-only condition. The current findings demonstrate that additional cognitive load during a speech intelligibility task could disturb the well-established relation between pupillary response and listening effort. Both the magnitude and temporal pattern of task-evoked pupillary response differ greatly in complex listening conditions, urging for more listening effort studies in complex and realistic listening situations.

Difficulties with Speech-in-Noise Perception Related to Fundamental Grouping Processes in Auditory Cortex

In our everyday lives, we are often required to follow a conversation when background noise is present ("speech-in-noise" [SPIN] perception). SPIN perception varies widely-and people who are worse at SPIN perception are also worse at fundamental auditory grouping, as assessed by figure-ground tasks. Here, we examined the cortical processes that link difficulties with SPIN perception to difficulties with figure-ground perception using functional magnetic resonance imaging. We found strong evidence that the earliest stages of the auditory cortical hierarchy (left core and belt areas) are similarly disinhibited when SPIN and figure-ground tasks are more difficult (i.e., at target-to-masker ratios corresponding to 60% rather than 90% performance)-consistent with increased cortical gain at lower levels of the auditory hierarchy. Overall, our results reveal a common neural substrate for these basic (figure-ground) and naturally relevant (SPIN) tasks-which provides a common computational basis for the link between SPIN perception and fundamental auditory grouping.

Simultaneous auditory agnosia: Systematic description of a new type of auditory segregation deficit following a right hemisphere lesion

We investigated auditory processing in a young patient who experienced a single embolus causing an infarct in the right middle cerebral artery territory. This led to damage to auditory cortex including planum temporale that spared medial Heschl's gyrus, and included damage to the posterior insula and inferior parietal lobule. She reported chronic difficulties with segregating speech from noise and segregating elements of music. Clinical tests showed no evidence for abnormal cochlear function. Follow-up tests confirmed difficulties with auditory segregation in her left ear that spanned multiple domains, including words-in-noise and music streaming. Testing with a stochastic figure-ground task-a way of estimating generic acoustic foreground and background segregation-demonstrated that this was also abnormal. This is the first demonstration of an acquired deficit in the segregation of complex acoustic patterns due to cortical damage, which we argue is a causal explanation for the symptomatic deficits in the segregation of speech and music. These symptoms are analogous to the visual symptom of simultaneous agnosia. Consistent with functional imaging studies on normal listeners, the work implicates non-primary auditory cortex. Further, the work demonstrates a (partial) lateralisation of the necessary anatomical substrate for segregation that has not been previously highlighted.

Acute Alcohol Intake Deteriorates Hearing Thresholds and Speech Perception in Noise

OBJECTIVES

The hearing process involves complex peripheral and central auditory pathways and could be influenced by various situations or medications. To date, there is very little known about the effects of alcohol on the auditory performances. The purpose of the present study was to evaluate how acute alcohol administration affects various aspects of hearing performance in human subjects, from the auditory perceptive threshold to the speech-in-noise task, which is cognitively demanding.

METHODS

A total of 43 healthy volunteers were recruited, and each of the participants received calculated amounts of alcohol according to their body weight and sex with a targeted blood alcohol content level of 0.05% using the Widmark formula. Hearing was tested in alcohol-free conditions (no alcohol intake within the previous 24 h) and acute alcohol conditions. A test battery composed of pure-tone audiometry, speech reception threshold (SRT), word recognition score (WRS), distortion product otoacoustic emission (DPOAE), gaps-in-noise (GIN) test, and Korean matrix sentence test (testing speech perception in noise) was performed in the 2 conditions.

RESULTS

Acute alcohol intake elevated pure-tone hearing thresholds and SRT but did not affect WRS. Both otoacoustic emissions recorded with DPOAE and the temporal resolution measured with the GIN test were not influenced by alcohol intake. The hearing performance in a noisy environment in both easy (-2 dB signal-to-noise ratio [SNR]) and difficult (-8 dB SNR) conditions was decreased by alcohol.

CONCLUSIONS

Acute alcohol elevated auditory perceptive thresholds and affected performance in complex and difficult auditory tasks rather than simple tasks.

How Can Hearing Loss Cause Dementia?

Epidemiological studies identify midlife hearing loss as an independent risk factor for dementia, estimated to account for 9% of cases. We evaluate candidate brain bases for this relationship. These bases include a common pathology affecting the ascending auditory pathway and multimodal cortex, depletion of cognitive reserve due to an impoverished listening environment, and the occupation of cognitive resources when listening in difficult conditions. We also put forward an alternate mechanism, drawing on new insights into the role of the medial temporal lobe in auditory cognition. In particular, we consider how aberrant activity in the service of auditory pattern analysis, working memory, and object processing may interact with dementia pathology in people with hearing loss. We highlight how the effect of hearing interventions on dementia depends on the specific mechanism and suggest avenues for work at the molecular, neuronal, and systems levels to pin this down.

Absorption and Enjoyment During Listening to Acoustically Masked Stories

Comprehension of speech masked by background sound requires increased cognitive processing, which makes listening effortful. Research in hearing has focused on such challenging listening experiences, in part because they are thought to contribute to social withdrawal in people with hearing impairment. Research has focused less on positive listening experiences, such as enjoyment, despite their potential importance in motivating effortful listening. Moreover, the artificial speech materials-such as disconnected, brief sentences-commonly used to investigate speech intelligibility and listening effort may be ill-suited to capture positive experiences when listening is challenging. Here, we investigate how listening to naturalistic spoken stories under acoustic challenges influences the quality of listening experiences. We assess absorption (the feeling of being immersed/engaged in a story), enjoyment, and listening effort and show that (a) story absorption and enjoyment are only minimally affected by moderate speech masking although listening effort increases, (b) thematic knowledge increases absorption and enjoyment and reduces listening effort when listening to a story presented in multitalker babble, and (c) absorption and enjoyment increase and effort decreases over time as individuals listen to several stories successively in multitalker babble. Our research indicates that naturalistic, spoken stories can reveal several concurrent listening experiences and that expertise in a topic can increase engagement and reduce effort. Our work also demonstrates that, although listening effort may increase with speech masking, listeners may still find the experience both absorbing and enjoyable.

The Effects of Age-Related Hearing Loss on the Brain and Cognitive Function

Age-related hearing loss (ARHL) is a common problem for older adults, leading to communication difficulties, isolation, and cognitive decline. Recently, hearing loss has been identified as potentially the most modifiable risk factor for dementia. Listening in challenging situations, or when the auditory system is damaged, strains cortical resources, and this may change how the brain responds to cognitively demanding situations more generally. We review the effects of ARHL on brain areas involved in speech perception, from the auditory cortex, through attentional networks, to the motor system. We explore current perspectives on the possible causal relationship between hearing loss, neural reorganisation, and cognitive impairment. Through this synthesis we aim to inspire innovative research and novel interventions for alleviating hearing loss and cognitive decline.

Decoding of Envelope vs. Fundamental Frequency During Complex Auditory Stream Segregation

Hearing-in-noise perception is a challenging task that is critical to human function, but how the brain accomplishes it is not well understood. A candidate mechanism proposes that the neural representation of an attended auditory stream is enhanced relative to background sound via a combination of bottom-up and top-down mechanisms. To date, few studies have compared neural representation and its task-related enhancement across frequency bands that carry different auditory information, such as a sound's amplitude envelope (i.e., syllabic rate or rhythm; 1-9 Hz), and the fundamental frequency of periodic stimuli (i.e., pitch; >40 Hz). Furthermore, hearing-in-noise in the real world is frequently both messier and richer than the majority of tasks used in its study. In the present study, we use continuous sound excerpts that simultaneously offer predictive, visual, and spatial cues to help listeners separate the target from four acoustically similar simultaneously presented sound streams. We show that while both lower and higher frequency information about the entire sound stream is represented in the brain's response, the to-be-attended sound stream is strongly enhanced only in the slower, lower frequency sound representations. These results are consistent with the hypothesis that attended sound representations are strengthened progressively at higher level, later processing stages, and that the interaction of multiple brain systems can aid in this process. Our findings contribute to our understanding of auditory stream separation in difficult, naturalistic listening conditions and demonstrate that pitch and envelope information can be decoded from single-channel EEG data.

Cingulo-opercular adaptive control for younger and older adults during a challenging gap detection task

Cingulo-opercular activity is hypothesized to reflect an adaptive control function that optimizes task performance through adjustments in attention and behavior, and outcome monitoring. While auditory perceptual task performance appears to benefit from elevated activity in cingulo-opercular regions of frontal cortex before stimuli are presented, this association appears reduced for older adults compared to younger adults. However, adaptive control function may be limited by difficult task conditions for older adults. An fMRI study was used to characterize adaptive control differences while 15 younger (average age = 24 years) and 15 older adults (average age = 68 years) performed a gap detection in noise task designed to limit age-related differences. During the fMRI study, participants listened to a noise recording and indicated with a button-press whether it contained a gap. Stimuli were presented between sparse fMRI scans (TR = 8.6 s) and BOLD measurements were collected during separate listening and behavioral response intervals. Age-related performance differences were limited by presenting gaps in noise with durations calibrated at or above each participant's detection threshold. Cingulo-opercular BOLD increased significantly throughout listening and behavioral response intervals, relative to a resting baseline. Correct behavioral responses were significantly more likely on trials with elevated pre-stimulus cingulo-opercular BOLD, consistent with an adaptive control framework. Cingulo-opercular adaptive control estimates appeared higher for participants with better gap sensitivity and lower response bias, irrespective of age, which suggests that this mechanism can benefit performance across the lifespan under conditions that limit age-related performance differences.

In a Concurrent Memory and Auditory Perception Task, the Pupil Dilation Response Is More Sensitive to Memory Load Than to Auditory Stimulus Characteristics

OBJECTIVES

Speech understanding may be cognitively demanding, but it can be enhanced when semantically related text cues precede auditory sentences. The present study aimed to determine whether (a) providing text cues reduces pupil dilation, a measure of cognitive load, during listening to sentences, (b) repeating the sentences aloud affects recall accuracy and pupil dilation during recall of cue words, and (c) semantic relatedness between cues and sentences affects recall accuracy and pupil dilation during recall of cue words.

DESIGN

Sentence repetition following text cues and recall of the text cues were tested. Twenty-six participants (mean age, 22 years) with normal hearing listened to masked sentences. On each trial, a set of four-word cues was presented visually as text preceding the auditory presentation of a sentence whose meaning was either related or unrelated to the cues. On each trial, participants first read the cue words, then listened to a sentence. Following this they spoke aloud either the cue words or the sentence, according to instruction, and finally on all trials orally recalled the cues. Peak pupil dilation was measured throughout listening and recall on each trial. Additionally, participants completed a test measuring the ability to perceive degraded verbal text information and three working memory tests (a reading span test, a size-comparison span test, and a test of memory updating).

RESULTS

Cue words that were semantically related to the sentence facilitated sentence repetition but did not reduce pupil dilation. Recall was poorer and there were more intrusion errors when the cue words were related to the sentences. Recall was also poorer when sentences were repeated aloud. Both behavioral effects were associated with greater pupil dilation. Larger reading span capacity and smaller size-comparison span were associated with larger peak pupil dilation during listening. Furthermore, larger reading span and greater memory updating ability were both associated with better cue recall overall.

CONCLUSIONS

Although sentence-related word cues facilitate sentence repetition, our results indicate that they do not reduce cognitive load during listening in noise with a concurrent memory load. As expected, higher working memory capacity was associated with better recall of the cues. Unexpectedly, however, semantic relatedness with the sentence reduced word cue recall accuracy and increased intrusion errors, suggesting an effect of semantic confusion. Further, speaking the sentence aloud also reduced word cue recall accuracy, probably due to articulatory suppression. Importantly, imposing a memory load during listening to sentences resulted in the absence of formerly established strong effects of speech intelligibility on the pupil dilation response. This nullified intelligibility effect demonstrates that the pupil dilation response to a cognitive (memory) task can completely overshadow the effect of perceptual factors on the pupil dilation response. This highlights the importance of taking cognitive task load into account during auditory testing.

Listening effort: Are we measuring cognition or affect, or both?

Listening effort is increasingly recognized as a factor in communication, particularly for and with nonnative speakers, for the elderly, for individuals with hearing impairment and/or for those working in noise. However, as highlighted by McGarrigle et al., International Journal of Audiology, 2014, 53, 433-445, the term "listening effort" encompasses a wide variety of concepts, including the engagement and control of multiple possibly distinct neural systems for information processing, and the affective response to the expenditure of those resources in a given context. Thus, experimental or clinical methods intended to objectively quantify listening effort may ultimately reflect a complex interaction between the operations of one or more of those information processing systems, and/or the affective and motivational response to the demand on those systems. Here we examine theoretical, behavioral, and psychophysiological factors related to resolving the question of what we are measuring, and why, when we measure "listening effort." This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Theory and Methods Psychology > Attention Psychology > Emotion and Motivation.

Listening Difficulties in Children With Fetal Alcohol Spectrum Disorders: More Than a Problem of Audibility

Purpose Data from standardized caregiver questionnaires indicate that children with fetal alcohol spectrum disorders (FASDs) frequently exhibit atypical auditory behaviors, including reduced responsivity to spoken stimuli. Another body of evidence suggests that prenatal alcohol exposure may result in auditory dysfunction involving loss of audibility (i.e., hearing loss) and/or impaired processing of clearly audible, "suprathreshold" sounds necessary for sound-in-noise listening. Yet, the nexus between atypical auditory behavior and underlying auditory dysfunction in children with FASDs remains largely unexplored. Method To investigate atypical auditory behaviors in FASDs and explore their potential physiological bases, we examined clinical data from 325 children diagnosed with FASDs at the University of Washington using the FASD 4-Digit Diagnostic Code. Atypical behaviors reported on the "auditory filtering" domain of the Short Sensory Profile were assessed to document their prevalence across FASD diagnoses and explore their relationship to reported hearing loss and/or central nervous system measures of cognition, attention, and language function that may indicate suprathreshold processing deficits. Results Atypical auditory behavior was reported among 80% of children with FASDs, a prevalence that did not vary by FASD diagnostic severity or hearing status but was positively correlated with attention-deficit/hyperactivity disorder. In contrast, hearing loss was documented in the clinical records of 40% of children with fetal alcohol syndrome (FAS; a diagnosis on the fetal alcohol spectrum characterized by central nervous system dysfunction, facial dysmorphia, and growth deficiency), 16-fold more prevalent than for those with less severe FASDs (2.4%). Reported hearing loss was significantly associated with physical features characteristic of FAS. Conclusion Children with FAS but not other FASDs may be at a particular risk for hearing loss. However, listening difficulties in the absence of hearing loss-presumably related to suprathreshold processing deficits-are prevalent across the entire fetal alcohol spectrum. The nature and impact of both listening difficulties and hearing loss in FASDs warrant further investigation.

Poorer Speech Reception Threshold in Noise Is Associated With Lower Brain Volume in Auditory and Cognitive Processing Regions

Purpose Hearing loss is associated with changes in brain volume in regions supporting auditory and cognitive processing. The purpose of this study was to determine whether there is a systematic association between hearing ability and brain volume in cross-sectional data from a large nonclinical cohort of middle-aged adults available from the UK Biobank Resource ( http://www.ukbiobank.ac.uk ). Method We performed a set of regression analyses to determine the association between speech reception threshold in noise (SRTn) and global brain volume as well as predefined regions of interest (ROIs) based on T1-weighted structural images, controlling for hearing-related comorbidities and cognition as well as demographic factors. In a 2nd set of analyses, we additionally controlled for hearing aid (HA) use. We predicted statistically significant associations globally and in ROIs including auditory and cognitive processing regions, possibly modulated by HA use. Results Whole-brain gray matter volume was significantly lower for individuals with poorer SRTn. Furthermore, the volume of 9 predicted ROIs including both auditory and cognitive processing regions was lower for individuals with poorer SRTn. The greatest percentage difference (-0.57%) in ROI volume relating to a 1 SD worsening of SRTn was found in the left superior temporal gyrus. HA use did not substantially modulate the pattern of association between brain volume and SRTn. Conclusions In a large middle-aged nonclinical population, poorer hearing ability is associated with lower brain volume globally as well as in cortical and subcortical regions involved in auditory and cognitive processing, but there was no conclusive evidence that this effect is moderated by HA use. This pattern of results supports the notion that poor hearing leads to reduced volume in brain regions recruited during speech understanding under challenging conditions. These findings should be tested in future longitudinal, experimental studies. Supplemental Material https://doi.org/10.23641/asha.7949357.

Listening Effort: How the Cognitive Consequences of Acoustic Challenge Are Reflected in Brain and Behavior

Everyday conversation frequently includes challenges to the clarity of the acoustic speech signal, including hearing impairment, background noise, and foreign accents. Although an obvious problem is the increased risk of making word identification errors, extracting meaning from a degraded acoustic signal is also cognitively demanding, which contributes to increased listening effort. The concepts of cognitive demand and listening effort are critical in understanding the challenges listeners face in comprehension, which are not fully predicted by audiometric measures. In this article, the authors review converging behavioral, pupillometric, and neuroimaging evidence that understanding acoustically degraded speech requires additional cognitive support and that this cognitive load can interfere with other operations such as language processing and memory for what has been heard. Behaviorally, acoustic challenge is associated with increased errors in speech understanding, poorer performance on concurrent secondary tasks, more difficulty processing linguistically complex sentences, and reduced memory for verbal material. Measures of pupil dilation support the challenge associated with processing a degraded acoustic signal, indirectly reflecting an increase in neural activity. Finally, functional brain imaging reveals that the neural resources required to understand degraded speech extend beyond traditional perisylvian language networks, most commonly including regions of prefrontal cortex, premotor cortex, and the cingulo-opercular network. Far from being exclusively an auditory problem, acoustic degradation presents listeners with a systems-level challenge that requires the allocation of executive cognitive resources. An important point is that a number of dissociable processes can be engaged to understand degraded speech, including verbal working memory and attention-based performance monitoring. The specific resources required likely differ as a function of the acoustic, linguistic, and cognitive demands of the task, as well as individual differences in listeners' abilities. A greater appreciation of cognitive contributions to processing degraded speech is critical in understanding individual differences in comprehension ability, variability in the efficacy of assistive devices, and guiding rehabilitation approaches to reducing listening effort and facilitating communication.

Neural Switch Asymmetry in Feature-Based Auditory Attention Tasks

Active listening involves dynamically switching attention between competing talkers and is essential to following conversations in everyday environments. Previous investigations in human listeners have examined the neural mechanisms that support switching auditory attention within the acoustic featural cues of pitch and auditory space. Here, we explored the cortical circuitry underlying endogenous switching of auditory attention between pitch and spatial cues necessary to discern target from masker words. Because these tasks are of unequal difficulty, we expected an asymmetry in behavioral switch costs for hard-to-easy versus easy-to-hard switches, mirroring prior evidence from vision-based cognitive task-switching paradigms. We investigated the neural correlates of this behavioral switch asymmetry and associated cognitive control operations in the present auditory paradigm. Behaviorally, we observed no switch-cost asymmetry, i.e., no performance difference for switching from the more difficult attend-pitch to the easier attend-space condition (P→S) versus switching from easy-to-hard (S→P). However, left lateral prefrontal cortex activity, correlated with improved performance, was observed during a silent gap period when listeners switched attention from P→S, relative to switching within pitch cues. No such differential activity was seen for the analogous easy-to-hard switch. We hypothesize that this neural switch asymmetry reflects proactive cognitive control mechanisms that successfully reconfigured neurally-specified task parameters and resolved competition from other such "task sets," thereby obviating the expected behavioral switch-cost asymmetry. The neural switch activity observed was generally consistent with that seen in cognitive paradigms, suggesting that established cognitive models of attention switching may be productively applied to better understand similar processes in audition.

Functional Near-Infrared Spectroscopy as a Measure of Listening Effort in Older Adults Who Use Hearing Aids

Listening effort may be reduced when hearing aids improve access to the acoustic signal. However, this possibility is difficult to evaluate because many neuroimaging methods used to measure listening effort are incompatible with hearing aid use. Functional near-infrared spectroscopy (fNIRS), which can be used to measure the concentration of oxygen in the prefrontal cortex (PFC), appears to be well-suited to this application. The first aim of this study was to establish whether fNIRS could measure cognitive effort during listening in older adults who use hearing aids. The second aim was to use fNIRS to determine if listening effort, a form of cognitive effort, differed depending on whether or not hearing aids were used when listening to sound presented at 35 dB SL (flat gain). Sixteen older adults who were experienced hearing aid users completed an auditory n-back task and a visual n-back task; both tasks were completed with and without hearing aids. We found that PFC oxygenation increased with n-back working memory demand in both modalities, supporting the use of fNIRS to measure cognitive effort during listening in this population. PFC oxygenation was weakly and nonsignificantly correlated with self-reported listening effort and reaction time, respectively, suggesting that PFC oxygenation assesses a dimension of listening effort that differs from these other measures. Furthermore, the extent to which hearing aids reduced PFC oxygenation in the left lateral PFC was positively correlated with age and pure-tone average thresholds. The implications of these findings as well as future directions are discussed.

Functional near-infrared spectroscopy as a tool for assessing speech and spoken language processing in pediatric and adult cochlear implant users

Much of what is known about the course of auditory learning in following cochlear implantation is based on behavioral indicators that users are able to perceive sound. Both prelingually deafened children and postlingually deafened adults who receive cochlear implants display highly variable speech and language processing outcomes, although the basis for this is poorly understood. To date, measuring neural activity within the auditory cortex of implant recipients of all ages has been challenging, primarily because the use of traditional neuroimaging techniques is limited by the implant itself. Functional near-infrared spectroscopy (fNIRS) is an imaging technology that works with implant users of all ages because it is non-invasive, compatible with implant devices, and not subject to electrical artifacts. Thus, fNIRS can provide insight into processing factors that contribute to variations in spoken language outcomes in implant users, both children and adults. There are important considerations to be made when using fNIRS, particularly with children, to maximize the signal-to-noise ratio and to best identify and interpret cortical responses. This review considers these issues, recent data, and future directions for using fNIRS as a tool to understand spoken language processing in children and adults who hear through a cochlear implant.

Psychophysiological measurement of affective responses during speech perception

When people make decisions about listening, such as whether to continue attending to a particular conversation or whether to wear their hearing aids to a particular restaurant, they do so on the basis of more than just their estimated performance. Recent research has highlighted the vital role of more subjective qualities such as effort, motivation, and fatigue. Here, we argue that the importance of these factors is largely mediated by a listener's emotional response to the listening challenge, and suggest that emotional responses to communication challenges may provide a crucial link between day-to-day communication stress and long-term health. We start by introducing some basic concepts from the study of emotion and affect. We then develop a conceptual framework to guide future research on this topic through examination of a variety of autonomic and peripheral physiological responses that have been employed to investigate both cognitive and affective phenomena related to challenging communication. We conclude by suggesting the need for further investigation of the links between communication difficulties, emotional response, and long-term health, and make some recommendations intended to guide future research on affective psychophysiology in speech communication.

Effects of Second Language Proficiency and Linguistic Uncertainty on Recognition of Speech in Native and Nonnative Competing Speech

PURPOSE

The purpose of this study was to investigate the effects of 2nd language proficiency and linguistic uncertainty on performance and listening effort in mixed language contexts.

METHOD

Thirteen native speakers of Dutch with varying degrees of fluency in English listened to and repeated sentences produced in both Dutch and English and presented in the presence of single-talker competing speech in both Dutch and English. Target and masker language combinations were presented in both blocked and mixed (unpredictable) conditions. In the blocked condition, in each block of trials the target-masker language combination remained constant, and the listeners were informed of both prior to beginning the block. In the mixed condition, target and masker language varied randomly from trial to trial. All listeners participated in all conditions. Performance was assessed in terms of speech reception thresholds, whereas listening effort was quantified in terms of pupil dilation.

RESULTS

Performance (speech reception thresholds) and listening effort (pupil dilation) were both affected by 2nd language proficiency (English test score) and target and masker language: Performance was better in blocked as compared to mixed conditions, with Dutch as compared to English targets, and with English as compared to Dutch maskers. English proficiency was correlated with listening performance. Listeners also exhibited greater peak pupil dilation in mixed as compared to blocked conditions for trials with Dutch maskers, whereas pupil dilation during preparation for speaking was higher for English targets as compared to Dutch ones in almost all conditions.

CONCLUSIONS

Both listener's proficiency in a 2nd language and uncertainty about the target language on a given trial play a significant role in how bilingual listeners attend to speech in the presence of competing speech in different languages, but precise effects also depend on which language is serving as target and which as masker.

Facilitation of motor excitability during listening to spoken sentences is not modulated by noise or semantic coherence

Comprehending speech can be particularly challenging in a noisy environment and in the absence of semantic context. It has been proposed that the articulatory motor system would be recruited especially in difficult listening conditions. However, it remains unknown how signal-to-noise ratio (SNR) and semantic context affect the recruitment of the articulatory motor system when listening to continuous speech. The aim of the present study was to address the hypothesis that involvement of the articulatory motor cortex increases when the intelligibility and clarity of the spoken sentences decreases, because of noise and the lack of semantic context. We applied Transcranial Magnetic Stimulation (TMS) to the lip and hand representations in the primary motor cortex and measured motor evoked potentials from the lip and hand muscles, respectively, to evaluate motor excitability when young adults listened to sentences. In Experiment 1, we found that the excitability of the lip motor cortex was facilitated during listening to both semantically anomalous and coherent sentences in noise relative to non-speech baselines, but neither SNR nor semantic context modulated the facilitation. In Experiment 2, we replicated these findings and found no difference in the excitability of the lip motor cortex between sentences in noise and clear sentences without noise. Thus, our results show that the articulatory motor cortex is involved in speech processing even in optimal and ecologically valid listening conditions and that its involvement is not modulated by the intelligibility and clarity of speech.

Orthographic influence on spoken word identification: Behavioral and fMRI evidence

The current study investigated behavioral and neuroimaging evidence for orthographic influences on auditory word identification. To assess such influences, the proportion of similar sounding words (i.e. phonological neighbors) that were also spelled similarly (i.e., orthographic neighbors) was computed for each auditorily presented word as the Orthographic-to-Phonological Overlap Ratio (OPOR). Speech intelligibility was manipulated by presenting monosyllabic words in multi-talker babble at two signal-to-noise ratios: + 3 and + 10 dB SNR. Identification rates were lower for high overlap words in the challenging + 3 dB SNR condition. In addition, BOLD contrast increased with OPOR at the more difficult SNR, and decreased with OPOR under more favorable SNR conditions. Both voxel-based and region of interest analyses demonstrated robust effects of OPOR in several cingulo-opercular regions. However, contrary to prior theoretical accounts, no task-related activity was observed in posterior regions associated with phonological or orthographic processing. We suggest that, when processing is difficult, orthographic-to-phonological feature overlap increases the availability of competing responses, which then requires additional support from domain general performance systems in order to produce a single response.

A Taxonomy of Fatigue Concepts and Their Relation to Hearing Loss

Fatigue is common in individuals with a variety of chronic health conditions and can have significant negative effects on quality of life. Although limited in scope, recent work suggests persons with hearing loss may be at increased risk for fatigue, in part due to effortful listening that is exacerbated by their hearing impairment. However, the mechanisms responsible for hearing loss-related fatigue, and the efficacy of audiologic interventions for reducing fatigue, remain unclear. To improve our understanding of hearing loss-related fatigue, as a field it is important to develop a common conceptual understanding of this construct. In this article, the broader fatigue literature is reviewed to identify and describe core constructs, consequences, and methods for assessing fatigue and related constructs. Finally, the current knowledge linking hearing loss and fatigue is described and may be summarized as follows: Hearing impairment may increase the risk of subjective fatigue and vigor deficits; adults with hearing loss require more time to recover from fatigue after work and have more work absences; sustained, effortful, listening can be fatiguing; optimal methods for eliciting and measuring fatigue in persons with hearing loss remain unclear and may vary with listening condition; and amplification may minimize decrements in cognitive processing speed during sustained effortful listening. Future research is needed to develop reliable measurement methods to quantify hearing loss-related fatigue, explore factors responsible for modulating fatigue in people with hearing loss, and identify and evaluate potential interventions for reducing hearing loss-related fatigue.

Listening in Naturalistic Scenes: What Can Functional Near-Infrared Spectroscopy and Intersubject Correlation Analysis Tell Us About the Underlying Brain Activity?

Listening to speech in the noisy conditions of everyday life can be effortful, reflecting the increased cognitive workload involved in extracting meaning from a degraded acoustic signal. Studying the underlying neural processes has the potential to provide mechanistic insight into why listening is effortful under certain conditions. In a move toward studying listening effort under ecologically relevant conditions, we used the silent and flexible neuroimaging technique functional near-infrared spectroscopy (fNIRS) to examine brain activity during attentive listening to speech in naturalistic scenes. Thirty normally hearing participants listened to a series of narratives continuously varying in acoustic difficulty while undergoing fNIRS imaging. Participants then listened to another set of closely matched narratives and rated perceived effort and intelligibility for each scene. As expected, self-reported effort generally increased with worsening signal-to-noise ratio. After controlling for better-ear signal-to-noise ratio, perceived effort was greater in scenes that contained competing speech than in those that did not, potentially reflecting an additional cognitive cost of overcoming informational masking. We analyzed the fNIRS data using intersubject correlation, a data-driven approach suitable for analyzing data collected under naturalistic conditions. Significant intersubject correlation was seen in the bilateral auditory cortices and in a range of channels across the prefrontal cortex. The involvement of prefrontal regions is consistent with the notion that higher order cognitive processes are engaged during attentive listening to speech in complex real-world conditions. However, further research is needed to elucidate the relationship between perceived listening effort and activity in these extended cortical networks.

Best Practices and Advice for Using Pupillometry to Measure Listening Effort: An Introduction for Those Who Want to Get Started

Within the field of hearing science, pupillometry is a widely used method for quantifying listening effort. Its use in research is growing exponentially, and many labs are (considering) applying pupillometry for the first time. Hence, there is a growing need for a methods paper on pupillometry covering topics spanning from experiment logistics and timing to data cleaning and what parameters to analyze. This article contains the basic information and considerations needed to plan, set up, and interpret a pupillometry experiment, as well as commentary about how to interpret the response. Included are practicalities like minimal system requirements for recording a pupil response and specifications for peripheral, equipment, experiment logistics and constraints, and different kinds of data processing. Additional details include participant inclusion and exclusion criteria and some methodological considerations that might not be necessary in other auditory experiments. We discuss what data should be recorded and how to monitor the data quality during recording in order to minimize artifacts. Data processing and analysis are considered as well. Finally, we share insights from the collective experience of the authors and discuss some of the challenges that still lie ahead.

A Novel Communication Value Task Demonstrates Evidence of Response Bias in Cases with Presbyacusis

Decision-making about the expected value of an experience or behavior can explain hearing health behaviors in older adults with hearing loss. Forty-four middle-aged to older adults (68.45 ± 7.73 years) performed a task in which they were asked to decide whether information from a surgeon or an administrative assistant would be important to their health in hypothetical communication scenarios across visual signal-to-noise ratios (SNR). Participants also could choose to view the briefly presented sentences multiple times. The number of these effortful attempts to read the stimuli served as a measure of demand for information to make a health importance decision. Participants with poorer high frequency hearing more frequently decided that information was important to their health compared to participants with better high frequency hearing. This appeared to reflect a response bias because participants with high frequency hearing loss demonstrated shorter response latencies when they rated the sentences as important to their health. However, elevated high frequency hearing thresholds did not predict demand for information to make a health importance decision. The results highlight the utility of a performance-based measure to characterize effort and expected value from performing tasks in older adults with hearing loss.

Investigating the role of temporal lobe activation in speech perception accuracy with normal hearing adults: An event-related fNIRS study

Functional near infrared spectroscopy (fNIRS) is a safe, non-invasive, relatively quiet imaging technique that is tolerant of movement artifact making it uniquely ideal for the assessment of hearing mechanisms. Previous research demonstrates the capacity for fNIRS to detect cortical changes to varying speech intelligibility, revealing a positive relationship between cortical activation amplitude and speech perception score. In the present study, we use an event-related design to investigate the hemodynamic response in the temporal lobe across different listening conditions. We presented participants with a speech recognition task using sentences in quiet, sentences in noise, and vocoded sentences. Hemodynamic responses were examined across conditions and then compared when speech perception was accurate compared to when speech perception was inaccurate in the context of noisy speech. Repeated measures, two-way ANOVAs revealed that the speech in noise condition (-2.8dB signal-to-noise ratio/SNR) demonstrated significantly greater activation than the easier listening conditions on multiple channels bilaterally. Further analyses comparing correct recognition trials to incorrect recognition trials (during the presentation phase of the trial) revealed that activation was significantly greater during correct trials. Lastly, during the repetition phase of the trial, where participants correctly repeated the sentence, the hemodynamic response demonstrated significantly higher deoxyhemoglobin than oxyhemoglobin, indicating a difference between the effects of perception and production on the cortical response. Using fNIRS, the present study adds meaningful evidence to the body of knowledge that describes the brain/behavior relationship related to speech perception.

Theta- and alpha-power enhancements in the electroencephalogram as an auditory delayed match-to-sample task becomes impossibly difficult

Recent studies have related enhancements of theta- (∼4-8 Hz) and alpha-power (∼8-13 Hz) to listening effort based on parallels between enhancement and task difficulty. In contrast, nonauditory works demonstrate that, although increases in difficulty are initially accompanied by increases in effort, effort decreases when a task becomes so difficult as to exceed one's ability. Given the latter, we examined whether theta- and alpha-power enhancements thought to reflect effortful listening show a quadratic trend across levels of listening difficulty from impossible to easy. Listeners (n = 14) performed an auditory delayed match-to-sample task with frequency-modulated tonal sweeps under impossible, difficult (at ∼70.7% correct threshold), and easy (well above threshold) conditions. Frontal midline theta-power and posterior alpha-power enhancements were observed during the retention interval, with greatest enhancement in the difficult condition. Independent component-based analyses of data suggest that theta-power enhancements stemmed from medial frontal sources at or near the anterior cingulate cortex, whereas alpha-power effects stemmed from occipital cortices. Results support the notion that theta- and alpha-power enhancements reflect effortful cognitive processes during listening, related to auditory working memory and the inhibition of task-irrelevant cortical processing regions, respectively. Theta- and alpha-power dynamics can be used to characterize the cognitive processes that make up effortful listening, including qualitatively different types of listening effort.

Toward a taxonomic model of attention in effortful listening

In recent years, there has been increasing interest in studying listening effort. Research on listening effort intersects with the development of active theories of speech perception and contributes to the broader endeavor of understanding speech perception within the context of neuroscientific theories of perception, attention, and effort. Due to the multidisciplinary nature of the problem, researchers vary widely in their precise conceptualization of the catch-all term listening effort. Very recent consensus work stresses the relationship between listening effort and the allocation of cognitive resources, providing a conceptual link to current cognitive neuropsychological theories associating effort with the allocation of selective attention. By linking listening effort to attentional effort, we enable the application of a taxonomy of external and internal attention to the characterization of effortful listening. More specifically, we use a vectorial model to decompose the demand causing listening effort into its mutually orthogonal external and internal components and map the relationship between demanded and exerted effort by means of a resource-limiting term that can represent the influence of motivation as well as vigilance and arousal. Due to its quantitative nature and easy graphical interpretation, this model can be applied to a broad range of problems dealing with listening effort. As such, we conclude that the model provides a good starting point for further research on effortful listening within a more differentiated neuropsychological framework.

Cingulo-opercular activity affects incidental memory encoding for speech in noise

Correctly understood speech in difficult listening conditions is often difficult to remember. A long-standing hypothesis for this observation is that the engagement of cognitive resources to aid speech understanding can limit resources available for memory encoding. This hypothesis is consistent with evidence that speech presented in difficult conditions typically elicits greater activity throughout cingulo-opercular regions of frontal cortex that are proposed to optimize task performance through adaptive control of behavior and tonic attention. However, successful memory encoding of items for delayed recognition memory tasks is consistently associated with increased cingulo-opercular activity when perceptual difficulty is minimized. The current study used a delayed recognition memory task to test competing predictions that memory encoding for words is enhanced or limited by the engagement of cingulo-opercular activity during challenging listening conditions. An fMRI experiment was conducted with twenty healthy adult participants who performed a word identification in noise task that was immediately followed by a delayed recognition memory task. Consistent with previous findings, word identification trials in the poorer signal-to-noise ratio condition were associated with increased cingulo-opercular activity and poorer recognition memory scores on average. However, cingulo-opercular activity decreased for correctly identified words in noise that were not recognized in the delayed memory test. These results suggest that memory encoding in difficult listening conditions is poorer when elevated cingulo-opercular activity is not sustained. Although increased attention to speech when presented in difficult conditions may detract from more active forms of memory maintenance (e.g., sub-vocal rehearsal), we conclude that task performance monitoring and/or elevated tonic attention supports incidental memory encoding in challenging listening conditions.

Cognitive persistence: Development and validation of a novel measure from the Wisconsin Card Sorting Test

The Wisconsin Card Sorting Test (WCST) has long been used as a neuropsychological assessment of executive function abilities, in particular, cognitive flexibility or "set-shifting". Recent advances in scoring the task have helped to isolate specific WCST performance metrics that index set-shifting abilities and have improved our understanding of how prefrontal and parietal cortex contribute to set-shifting. We present evidence that the ability to overcome task difficulty to achieve a goal, or "cognitive persistence", is another important prefrontal function that is characterized by the WCST and that can be differentiated from efficient set-shifting. This novel measure of cognitive persistence was developed using the WCST-64 in an adult lifespan sample of 230 participants. The measure was validated using individual variation in cingulo-opercular cortex function in a sub-sample of older adults who had completed a challenging speech recognition in noise fMRI task. Specifically, older adults with higher cognitive persistence were more likely to demonstrate word recognition benefit from cingulo-opercular activity. The WCST-derived cognitive persistence measure can be used to disentangle neural processes involved in set-shifting from those involved in persistence.