Speech-reading: cognitive predictors and displayed emotion

Phonological store and speechreading performance of Chinese students with hearing impairment

This study presents three experiments to examine the role of the phonological store component of working memory in the speechreading performance of students with hearing impairment (HI) in China. In Experiment 1, 86 high school students with HI completed an immediate serial recall task with four lists of monosyllabic words that differed in phonological and visual similarities. In Experiment 2 and Experiment 3, 40 participants divided into high or low phonological store capacity (PS) and 40 participants divided into high or low visual phonological story capacity (VPS) completed a speechreading test at the word, phrase and sentence levels. Results revealed that (1) immediate serial recall showed effects of phonological and visual similarity and their interaction; (2) there was no significant effect of phonological store capacities on speechreading; and (3) there was a significant effect of visual phonological store capacities on accuracy but not speed of speechreading. These findings point to a general phonological store system for visual orthographic coding and phonological coding that students with HI engage in speechreading in Chinese. It provides evidence for the contention that the visual-based coding has a more direct impact on speechreading performance of Chinese students with HI than the speech-based coding.

Speaker Variations Influence Speechreading Speed for Dynamic Faces

We investigated the influence of task-irrelevant speaker variations on speechreading performance. In three experiments with video digitised faces presented either in dynamic, static-sequential, or static mode, participants performed speeded classifications on vowel utterances (German vowels /u/ and /i/). A Garner interference paradigm was used, in which speaker identity was task-irrelevant but could be either correlated, constant, or orthogonal to the vowel uttered. Reaction times for facial speech classifications were slowed by task-irrelevant speaker variations for dynamic stimuli. The results are discussed with reference to distributed models of face perception (Haxby et al, 2000 Trends in Cognitive Sciences4 223–233) and the relevance of both dynamic information and speaker characteristics for speechreading.

Speechreading Skill and Visual Movement Sensitivity are Related in Deaf Speechreaders

Individual speechreading abilities have been linked with a range of cognitive and language-processing factors. The role of specifically visual abilities in relation to the processing of visible speech is less studied. Here we report that the detection of coherent visible motion in random-dot kinematogram displays is related to speechreading skill in deaf, but not in hearing, speechreaders. A control task requiring the detection of visual form showed no such relationship. Additionally, people born deaf were better speechreaders than hearing people on a new test of silent speechreading.

Comparison of informational vs. energetic masking effects on speechreading performance

The effects of two types of auditory distracters (steady-state noise vs. four-talker babble) on visual-only speechreading accuracy were tested against a baseline (silence) in 23 participants with above-average speechreading ability. Their task was to speechread high frequency Swedish words. They were asked to rate their own performance and effort, and report how distracting each type of auditory distracter was. Only four-talker babble impeded speechreading accuracy. This suggests competition for phonological processing, since the four-talker babble demands phonological processing, which is also required for the speechreading task. Better accuracy was associated with lower self-rated effort in silence; no other correlations were found.

A resource of validated affective and neutral sentences to assess identification of emotion in spoken language after a brain injury

PRIMARY OBJECTIVE

The ability to identify emotions in spoken language is an essential component of communication and could be disrupted in persons with brain injury. Current tools to assess this function show important shortcomings. The aim is to present a set of validated and linguistically equated lexical sentences that can be used to separate the impact of lexical content and prosody on the processing of emotion in speech in persons with brain injury.

METHODS AND PROCEDURES

Using six-point Likert scales, a set of 125 sentences, carefully matched for linguistic variables, were rated by a group of young adults (n = 48) on their suitability to represent a particular emotion (anger, fear, happiness and sadness) in their lexical content.

MAIN OUTCOMES AND RESULTS

The findings identified a set of 50 sentences that were reliably associated with one particular emotion only or no emotion at all (neutral). Using less stringent criteria, 94 sentences were also found to be good representatives for these affective categories.

CONCLUSIONS

The findings generated a robust set of validated lexical stimuli necessary to reliably identify the specific contributions of verbal and prosodic information on difficulties in identifying emotions in speech with persons with brain injury.

Lipreading, processing speed, and working memory in younger and older adults

PURPOSE

To examine several cognitive and perceptual abilities--including working memory (WM), information processing speed (PS), perceptual closure, and perceptual disembedding skill--as factors contributing to individual differences in lipreading performance and to examine how patterns in predictor variables change across age groups.

METHOD

Forty-three younger adults (mean age = 20.8 years, SD = 2.4) and 38 older adults (mean age = 76.8 years, SD = 5.6) completed tasks measuring lipreading ability, verbal WM, spatial WM (SWM), PS, and perceptual abilities.

RESULTS

Younger adults demonstrated superior lipreading ability and perceptual skills compared with older adults. In addition, younger participants exhibited longer WM spans and faster PS than did the older participants. SWM and PS accounted for a significant proportion of the variance in lipreading ability in both younger and older adults, and the pattern of predictor variables remained consistent over age groups.

CONCLUSIONS

These findings suggest that the large individual variability in lipreading ability can be explained, in part, by individual differences in SWM and PS. Furthermore, as both of these abilities are known to decline with age, the findings suggest that age-related impairments in either or both of these abilities may account for the poorer lipreading ability of older compared with younger adults.

Visual phonemic ambiguity and speechreading

PURPOSE

To study the role of visual perception of phonemes in visual perception of sentences and words among normal-hearing individuals.

METHOD

Twenty-four normal-hearing adults identified consonants, words, and sentences, spoken by either a human or a synthetic talker. The synthetic talker was programmed with identical parameters within phoneme groups, hypothetically resulting in simplified articulation. Proportions of correctly identified phonemes per participant, condition, and task, as well as sensitivity to single consonants and clusters of consonants, were measured. Groups of mutually exclusive consonants were used for sensitivity analyses and hierarchical cluster analyses.

RESULTS

Consonant identification performance did not differ as a function of talker, nor did average sensitivity to single consonants. The bilabial and labiodental clusters were most readily identified and cohesive for both talkers. Word and sentence identification was better for the human talker than the synthetic talker. The participants were more sensitive to the clusters of the least visible consonants with the human talker than with the synthetic talker.

CONCLUSIONS

It is suggested that ability to distinguish between clusters of the least visually distinct phonemes is important in speechreading. Specifically, it reduces the number of candidates, and thereby facilitates lexical identification.

Motivation and appraisal in perception of poorly specified speech

Normal-hearing students (n = 72) performed sentence, consonant, and word identification in either A (auditory), V (visual), or AV (audiovisual) modality. The auditory signal had difficult speech-to-noise relations. Talker (human vs. synthetic), topic (no cue vs. cue-words), and emotion (no cue vs. facially displayed vs. cue-words) were varied within groups. After the first block, effects of modality, face, topic, and emotion on initial appraisal and motivation were assessed. After the entire session, effects of modality on longer-term appraisal and motivation were assessed. The results from both assessments showed that V identification was more positively appraised than A identification. Correlations were tentatively interpreted such that evaluation of self-rated performance possibly depends on subjective standard and is reflected on motivation (if below subjective standard, AV group), or on appraisal (if above subjective standard, A group). Suggestions for further research are presented.

Bottom-up driven speechreading in a speechreading expert: the case of AA (JK023)

OBJECTIVE

This case study tested the threshold hypothesis (Rönnberg et al., 1998), which states that superior speechreading skill is possible only if high-order cognitive functions, such as capacious verbal working memory, enable efficient strategies.

DESIGN

In a case study, a speechreading expert (AA) was tested on a number of speechreading and cognitive tasks and compared with control groups (z scores). Sentence-based speechreading tests, a word-decoding test, and a phoneme identification task were used to assess speechreading skill at different analytical levels. The cognitive test battery used included tasks of working memory (e.g., reading span), inference-making, phonological processing (e.g., rhyme-judgment), and central-executive functions (verbal fluency, Stroop task).

RESULTS

Contrary to previous cases of extreme speechreading skill, AA excels on both low-order (phoneme identification: z = +2.83) and high-order (sentence-based: z = +8.12 and word-decoding: z = +4.21) speechreading tasks. AA does not display superior verbal inference-making ability (sentence-completion task: z = -0.36). Neither does he possess a superior working memory (reading span: z = +0.80). However, AA outperforms the controls on two measures of executive retrieval functions, the semantic (z = +3.77) and phonological verbal fluency tasks (z = +3.55).

CONCLUSIONS

The performance profile is inconsistent with the threshold hypothesis. Extreme speechreading accuracy can be obtained in ways other than via well-developed high-order cognitive functions. It is suggested that AA's extreme speechreading skill, which capitalizes on low-order functions in combination with efficient central executive functions, is due to early onset of hearing impairment.

Effects of displayed emotion on attitude and impression formation in visual speech-reading

In two experiments on visual speech-reading, with a total of 132 normal-hearing participants, the effects of displayed emotion and task specificity on speech-reading performance, on attitude toward the task and on person impression were explored, as well as associations between speech-reading performance, attitude, and person impression. The results show that displayed emotion increased speech-reading performance and attitude ratings, and that the actor was perceived as more extraverted both when displaying emotions, and when his speech was high in specificity The main conclusion was that displayed emotion enhances speech-reading performance by providing information that is useful to the speech-reader.