Perceived Conventionality in Co-speech Gestures Involves the Fronto-Temporal Language Network

Language and gesture neural correlates: A meta-analysis of functional magnetic resonance imaging studies

BACKGROUND

Humans often use co-speech gestures to promote effective communication. Attention has been paid to the cortical areas engaged in the processing of co-speech gestures.

AIMS

To investigate the neural network underpinned in the processing of co-speech gestures and to observe whether there is a relationship between areas involved in language and gesture processing.

METHODS & PROCEDURES

We planned to include studies with neurotypical and/or stroke participants who underwent a bimodal task (i.e., processing of co-speech gestures with relative speech) and a unimodal task (i.e., speech or gesture alone) during a functional magnetic resonance imaging (fMRI) session. After a database search, abstract and full-text screening were conducted. Qualitative and quantitative data were extracted, and a meta-analysis was performed with the software GingerALE 3.0.2, performing contrast analyses of uni- and bimodal tasks.

MAIN CONTRIBUTION

The database search produced 1024 records. After the screening process, 27 studies were included in the review. Data from 15 studies were quantitatively analysed through meta-analysis. Meta-analysis found three clusters with a significant activation of the left middle frontal gyrus and inferior frontal gyrus, and bilateral middle occipital gyrus and inferior temporal gyrus.

CONCLUSIONS

There is a close link at the neural level for the semantic processing of auditory and visual information during communication. These findings encourage the integration of the use of co-speech gestures during aphasia treatment as a strategy to foster the possibility to communicate effectively for people with aphasia.

WHAT THIS PAPER ADDS

What is already known on this subject Gestures are an integral part of human communication, and they may have a relationship at neural level with speech processing. What this paper adds to the existing knowledge During processing of bi- and unimodal communication, areas related to semantic processing and multimodal processing are activated, suggesting that there is a close link between co-speech gestures and spoken language at a neural level. What are the potential or actual clinical implications of this work? Knowledge of the functions related to gesture and speech processing neural networks will allow for the adoption of model-based neurorehabilitation programs to foster recovery from aphasia by strengthening the specific functions of these brain networks.

Conventionality determines the time course of indirect replies comprehension: An ERP study

Indirect language comprehension requires decoding both the literal meaning and the intended meaning of an utterance, in which pragmatic inference is involved. This study tests the role of conventionality in the time course of indirect reply processing by comparing conventional and non-conventional indirect replies with direct reply, respectively. We constructed discourses which consist of a context and a dialogue with one question (e.g., May I buy a necklace for you) and one reply (e.g., I really have too many). The reply utterance was segmented into three phrases and presented orderly for EEG recording, e.g., with the subject as the first phrase (e.g., I), the adverbial as the second phrase (e.g., really), and the predicate as the third phrase (e.g., have too many). Our results showed that for conventional indirect replies, the second phrase elicited a larger anterior negativity, and the third phrase elicited a larger anterior N400 compared with those in direct replies. By contrast, for the non-conventional indirect reply, only the third phrase elicited a larger late negativity than the direct replies. These findings suggest that conventionality determines the time course of the pragmatic inferences for the most relevant interpretation during indirect replies comprehension.

Neural-level associations of non-verbal pragmatic comprehension in young Finnish autistic adults

This video-based study examines the pragmatic non-verbal comprehension skills and corresponding neural-level findings in young Finnish autistic adults, and controls. Items from the Assessment Battery of Communication (ABaCo) were chosen to evaluate the comprehension of non-verbal communication. Inter-subject correlation (ISC) analysis of the functional magnetic resonance imaging data was used to reveal the synchrony of brain activation across participants during the viewing of pragmatically complex scenes of ABaCo videos. The results showed a significant difference between the ISC maps of the autistic and control groups in tasks involving the comprehension of non-verbal communication, thereby revealing several brain regions where correlation of brain activity was greater within the control group. The results suggest a possible weaker modulation of brain states in response to the pragmatic non-verbal communicative situations in autistic participants. Although there was no difference between the groups in behavioural responses to ABaCo items, there was more variability in the accuracy of the responses in the autistic group. Furthermore, mean answering and reaction times correlated with the severity of autistic traits. The results indicate that even if young autistic adults may have learned to use compensatory resources in their communicative-pragmatic comprehension, pragmatic processing in naturalistic situations still requires additional effort.

Data-Driven Approach to the Analysis of Real-Time FMRI Neurofeedback Data: Disorder-Specific Brain Synchrony in PTSD

Brain-computer interfaces (BCIs) can be used in real-time fMRI neurofeedback (rtfMRI NF) investigations to provide feedback on brain activity to enable voluntary regulation of the blood-oxygen-level dependent (BOLD) signal from localized brain regions. However, the temporal pattern of successful self-regulation is dynamic and complex. In particular, the general linear model (GLM) assumes fixed temporal model functions and misses other dynamics. We propose a novel data-driven analyses approach for rtfMRI NF using intersubject covariance (ISC) analysis. The potential of ISC was examined in a reanalysis of data from 21 healthy individuals and nine patients with post-traumatic stress-disorder (PTSD) performing up-regulation of the anterior cingulate cortex (ACC). ISC in the PTSD group differed from healthy controls in a network including the right inferior frontal gyrus (IFG). In both cohorts, ISC decreased throughout the experiment indicating the development of individual regulation strategies. ISC analyses are a promising approach to reveal novel information on the mechanisms involved in voluntary self-regulation of brain signals and thus extend the results from GLM-based methods. ISC enables a novel set of research questions that can guide future neurofeedback and neuroimaging investigations.

Judgmental perception of co-speech gestures in MDD

Cognitive bias in depression may increase sensitivity to judgmental appraisal of communicative cues. Nonverbal communication encompassing co-speech gestures is crucial for social functioning and is perceived differentially by men and women, however, little is known about the effect of depression on the perception of appraisal. We investigate if a cognitive bias influences the perception of appraisal and judgement of nonverbal communication in major depressive disorder (MDD). During watching videos of speakers retelling a story and gesticulating, 22 patients with MDD and 22 matched healthy controls pressed a button when they perceived the speaker as appraising in a positive or negative way. The speakers were presented in four different conditions (with and without speech and with natural speaker or as stick-figures) to evaluate context effects. Inter-subject covariance (ISC) of the button-press time series measured consistency across the groups of the response pattern depending on the factors diagnosis and gender. Significant effects emerged for the factors diagnosis (p = .002), gender (p = .007), and their interaction (p < .001). The female healthy controls perceived the gestures more consistently appraising than male controls, the female patients, and male patients whereas the latter three groups did not differ. Further, the ISC measure for consistency correlated negatively with depression severity. The natural speaker video without audio speech yielded the highest responses consistency. Indeed co-speech gestures may drive these ISC effects because number of gestures but not facial shrugs correlated with ISC amplitude. During co-speech gestures, a cognitive bias led to disturbed perception of appraisal in MDD for females. Social communication is critical for functional outcomes in mental disorders; thus perception of gestural communication is important in rehabilitation.

The Role of Iconic Gestures in Speech Comprehension: An Overview of Various Methodologies

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process.

Visuo-Kinetic Signs Are Inherently Metonymic: How Embodied Metonymy Motivates Forms, Functions, and Schematic Patterns in Gesture

This paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of gestural abstraction and enaction. Introducing visuo-kinetic signs as an umbrella term for co-speech gestures and signed languages, the paper shows how a frame-based approach to gesture may integrate different cognitive/functional linguistic and semiotic accounts of metonymy (e.g., experiential domains, frame metonymy, contiguity, and pragmatic inferencing). The guiding assumption is that gestures metonymically profile deeply embodied, routinized aspects of familiar scenes, that is, the motivating context of frames. The discussion shows how gestures may evoke frame structures exhibiting varying degrees of groundedness, complexity, and schematicity: basic physical action and object frames; more complex frames; and highly abstract, complex frame structures. It thereby provides gestural evidence for the idea that metonymy is more basic and more directly experientially grounded than metaphor and thus often feeds into correlated metaphoric processes. Furthermore, the paper offers some initial insights into how metonymy also seems to induce the emergence of schematic patterns in gesture which may result from action-based and discourse-driven processes of habituation and conventionalization. It exemplifies how these forces may engender grammaticalization of a basic physical action into a gestural marker that shows strong metonymic form reduction, decreased transitivity, and interacting pragmatic functions. Finally, addressing basic metonymic operations in signed lexemes elucidates certain similarities regarding sign constitution in gesture and sign. English and German multimodal discourse data as well as German Sign Language (DGS) are drawn upon to illustrate the theoretical points of the paper. Overall, this paper presents a unified account of metonymy's role in underpinning forms, functions, and patterns in visuo-kinetic signs.