Communication Under Sharply Degraded Auditory Input and the "2-Sentence" Problem

Listening Effort Is Difficult to Detect in a Person's Voice: Implications for Audiology Evaluations and Conversation Partners

PURPOSE

Listening can be effortful for a variety of reasons, including when a person misperceives a word in a sentence and then mentally repairs it using later context. The current study explored whether an external observer (in the role of a tester/clinician) could detect that effort by hearing the listener's voice as they repeat the sentence.

METHOD

Stimuli were audio recordings of 13 adults with cochlear implants repeating sentences that were either intact or with a masked word that could be inferred/repaired using context (the latter of which were previously documented to elicit greater effort). Participants (n = 171, including 28 audiologists) used a continuous visual analog scale to judge whether the talker heard one type of stimulus or the other. Participants were also surveyed for experiences related to detecting effort or confusion in a talker's voice.

RESULTS

Participant judges were unable to discern when the CI users were forced to effortfully infer words from context when repeating a sentence. Ratings indicated a general bias toward assuming the listener heard the original sentence correctly without any need for repair. Acoustic properties of the CI users' voices (hypothesized higher voice pitch and delayed verbal reaction time for stimuli involving repair) did not reliably correlate with ratings of uncertainty. There were also no statistically detectable advantages for audiologists or for people who reported experience or skill in discerning uncertainty in a talker's voice.

CONCLUSIONS

Despite clear evidence that mental repair incurs extra effort, the process of mental repair gives no reliably perceptible signature in a talker's voice, even for audiologists and others who profess to have experience and skill in conversing with people who have hearing loss. Listening effort is at risk of going unnoticed by conversation partners and by audiologists who might underestimate a patient's effort when listening to speech.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.28688012.

Repairing Misperceptions of Words Early in a Sentence is More Effortful Than Repairing Later Words, Especially for Listeners With Cochlear Implants

The process of repairing misperceptions has been identified as a contributor to effortful listening in people who use cochlear implants (CIs). The current study was designed to examine the relative cost of repairing misperceptions at earlier or later parts of a sentence that contained contextual information that could be used to infer words both predictively and retroactively. Misperceptions were enforced at specific times by replacing single words with noise. Changes in pupil dilation were analyzed to track differences in the timing and duration of effort, comparing listeners with typical hearing (TH) or with CIs. Increases in pupil dilation were time-locked to the moment of the missing word, with longer-lasting increases when the missing word was earlier in the sentence. Compared to listeners with TH, CI listeners showed elevated pupil dilation for longer periods of time after listening, suggesting a lingering effect of effort after sentence offset. When needing to mentally repair missing words, CI listeners also made more mistakes on words elsewhere in the sentence, even though these words were not masked. Changes in effort based on the position of the missing word were not evident in basic measures like peak pupil dilation and only emerged when the full-time course was analyzed, suggesting the timing analysis adds new information to our understanding of listening effort. These results demonstrate that some mistakes are more costly than others and incur different levels of mental effort to resolve the mistake, underscoring the information lost when characterizing speech perception with simple measures like percent-correct scores.

The Effort of Repairing a Misperceived Word Can Impair Perception of Following Words, Especially for Listeners With Cochlear Implants

OBJECTIVES

In clinical and laboratory settings, speech recognition is typically assessed in a way that cannot distinguish accurate auditory perception from misperception that was mentally repaired or inferred from context. Previous work showed that the process of repairing misperceptions elicits greater listening effort, and that this elevated effort lingers well after the sentence is heard. That result suggests that cognitive repair strategies might appear successful when testing a single utterance but fail for everyday continuous conversational speech. The present study tested the hypothesis that the effort of repairing misperceptions has the consequence of carrying over to interfere with perception of later words after the sentence.

DESIGN

Stimuli were open-set coherent sentences that were presented intact or with a word early in the sentence replaced with noise, forcing the listener to use later context to mentally repair the missing word. Sentences were immediately followed by digit triplets, which served to probe carryover effort from the sentence. Control conditions allowed for the comparison to intact sentences that did not demand mental repair, as well as to listening conditions that removed the need to attend to the post-sentence stimuli, or removed the post-sentence digits altogether. Intelligibility scores for the sentences and digits were accompanied by time-series measurements of pupil dilation to assess cognitive load during the task, as well as subjective rating of effort. Participants included adults with cochlear implants (CIs), as well as an age-matched group and a younger group of listeners with typical hearing for comparison.

RESULTS

For the CI group, needing to repair a missing word during a sentence resulted in more errors on the digits after the sentence, especially when the repair process did not result in a coherent sensible perception. Sentences that needed repair also contained more errors on the words that were unmasked. All groups showed substantial increase of pupil dilation when sentences required repair, even when the repair was successful. Younger typical hearing listeners showed clear differences in moment-to-moment allocation of effort in the different conditions, while the other groups did not.

CONCLUSIONS

For CI listeners, the effort of needing to repair misperceptions in a sentence can last long enough to interfere with words that follow the sentence. This pattern could pose a serious problem for regular communication but would go overlooked in typical testing with single utterances, where a listener has a chance to repair misperceptions before responding. Carryover effort was not predictable by basic intelligibility scores, but can be revealed in behavioral data when sentences are followed immediately by extra probe words such as digits.