Patients with hippocampal amnesia successfully integrate gesture and speech

Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury

PURPOSE

Real-world communication is situated in rich multimodal contexts, containing speech and gesture. Speakers often convey unique information in gesture that is not present in the speech signal (e.g., saying "He searched for a new recipe" while making a typing gesture). We examine the narrative retellings of participants with and without moderate-severe traumatic brain injury across three timepoints over two online Zoom sessions to investigate whether people with TBI can integrate information from co-occurring speech and gesture and if information from gesture persists across delays.

METHODS

60 participants with TBI and 60 non-injured peers watched videos of a narrator telling four short stories. On key details, the narrator produced complementary gestures that conveyed unique information. Participants retold the stories at three timepoints: immediately after, 20-min later, and one-week later. We examined the words participants used when retelling these key details, coding them as a Speech Match (e.g., "He searched for a new recipe"), a Gesture Match (e.g., "He searched for a new recipe online), or Other ("He looked for a new recipe"). We also examined whether participants produced representative gestures themselves when retelling these details.

RESULTS

Despite recalling fewer story details, participants with TBI were as likely as non-injured peers to report information from gesture in their narrative retellings. All participants were more likely to report information from gesture and produce representative gestures themselves one-week later compared to immediately after hearing the story.

CONCLUSION

We demonstrated that speech-gesture integration is intact after TBI in narrative retellings. This finding has exciting implications for the utility of gesture to support comprehension and memory after TBI and expands our understanding of naturalistic multimodal language processing in this population.

Evidence of Audience Design in Amnesia: Adaptation in Gesture but Not Speech

Speakers design communication for their audience, providing more information in both speech and gesture when their listener is naïve to the topic. We test whether the hippocampal declarative memory system contributes to multimodal audience design. The hippocampus, while traditionally linked to episodic and relational memory, has also been linked to the ability to imagine the mental states of others and use language flexibly. We examined the speech and gesture use of four patients with hippocampal amnesia when describing how to complete everyday tasks (e.g., how to tie a shoe) to an imagined child listener and an adult listener. Although patients with amnesia did not increase their total number of words and instructional steps for the child listener, they did produce representational gestures at significantly higher rates for the imagined child compared to the adult listener. They also gestured at similar frequencies to neurotypical peers, suggesting that hand gesture can be a meaningful communicative resource, even in the case of severe declarative memory impairment. We discuss the contributions of multiple memory systems to multimodal audience design and the potential of gesture to act as a window into the social cognitive processes of individuals with neurologic disorders.

Fractionating the episodic buffer

The episodic buffer is a putative component of working memory proposed to account for several short-term memory functions, including unexpectedly preserved immediate prose recall by amnesic patients. Over the course of time, this component has increasingly become associated with binding functions. Considering recent findings regarding the performance of both memory-impaired and healthy individuals on the range of tasks purported to require the contribution of the episodic buffer, we suggest that it should be fractionated into two functional systems. One is a schematic store instantiated in brain areas responsible for conceptual and schema representations, which is likely to be hippocampus-independent, and preserved in the face of amnesia. In contrast, short-term maintenance of novel associative binding is likely to require the contribution of the hippocampus and may therefore not be functionally dissociable from long-term memory.

The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders

When people talk, they gesture. Gesture is a fundamental component of language that contributes meaningful and unique information to a spoken message and reflects the speaker's underlying knowledge and experiences. Theoretical perspectives of speech and gesture propose that they share a common conceptual origin and have a tightly integrated relationship, overlapping in time, meaning, and function to enrich the communicative context. We review a robust literature from the field of psychology documenting the benefits of gesture for communication for both speakers and listeners, as well as its important cognitive functions for organizing spoken language, and facilitating problem-solving, learning, and memory. Despite this evidence, gesture has been relatively understudied in populations with neurogenic communication disorders. While few studies have examined the rehabilitative potential of gesture in these populations, others have ignored gesture entirely or even discouraged its use. We review the literature characterizing gesture production and its role in intervention for people with aphasia, as well as describe the much sparser literature on gesture in cognitive communication disorders including right hemisphere damage, traumatic brain injury, and Alzheimer's disease. The neuroanatomical and behavioral profiles of these patient populations provide a unique opportunity to test theories of the relationship of speech and gesture and advance our understanding of their neural correlates. This review highlights several gaps in the field of communication disorders which may serve as a bridge for applying the psychological literature of gesture to the study of language disorders. Such future work would benefit from considering theoretical perspectives of gesture and using more rigorous and quantitative empirical methods in its approaches. We discuss implications for leveraging gesture to explore its untapped potential in understanding and rehabilitating neurogenic communication disorders.

The role of motor context in the beneficial effects of hand gesture on memory

Viewing co-speech hand gestures with spoken phrases enhances memory for phrases, as compared to when the phrases are presented without gesture. Prior work investigating the mechanism underlying the effect of gesture on memory has implicated engagement of the motor system; when the hands are engaged in an unrelated motor task when viewing gesture, the beneficial effect of gesture is absent. However, one alternative interpretation of these findings is that the beneficial effect of gesture disappears due to mismatched contexts at encoding and retrieval: The hands are engaged during either encoding or retrieval, but not during both stages. Here we examined whether matching the motor context at encoding and retrieval plays a role in the beneficial effect of gesture on memory during a phrase recall task. Participants were presented with phrases that were viewed with and without gesture. Participants were assigned to one of four conditions that determined whether they would complete an unrelated motor task at (1) encoding only, (2) retrieval only, (3) both encoding and retrieval, or (4) neither. During stages in which they were not completing a motor task, participants' hands were in their laps. We found that gesture enhanced memory for phrases both when participants engaged in an unrelated motor task at encoding and retrieval and when they did not complete the motor task during either stage. Furthermore, phrases observed with gesture were more likely to be paraphrased than to be recalled literally. Together, these findings demonstrate that gesture can enhance memory even when the motor system is engaged in another task, as long as that same task is performed at retrieval.

Gesture height reflects common ground status even in patients with amnesia

When we communicate, we alter our language and gesture based on the mutually shared knowledge - common ground - that we have with our listener. How memory supports these alterations remain unclear. We asked healthy adults and patients with hippocampal amnesia to engage in a referential communication task. Previous work suggests that common ground can be encoded by distinct memory systems; Amnesic patients show normal learning and referential label use as common ground increases, but inconsistently mark these labels with definite determiners (e.g., the vs. a windmill). Which memory systems support the ability to mark common ground via hand gesture? We found that gestures of both healthy participants and amnesic patients reflected common ground status. Both groups produced high gestures when common ground was lacking, and were less likely to do so as common ground increased. These findings suggest that gesture can reflect common ground status during conversation, potentially via non-declarative memory.