Language, Gesture, and Emotional Communication: An Embodied View of Social Interaction

Talking body: the effect of body and voice anthropomorphism on perception of social agents

Introduction

In human-agent interaction, trust is often measured using human-trust constructs such as competence, benevolence, and integrity, however, it is unclear whether technology-trust constructs such as functionality, helpfulness, and reliability are more suitable. There is also evidence that perception of "humanness" measured through anthropomorphism varies based on the characteristics of the agent, but dimensions of anthropomorphism are not highlighted in empirical studies.

Methods

In order to study how different embodiments and qualities of speech of agents influence type of trust and dimensions of anthropomorphism in perception of the agent, we conducted an experiment using two agent "bodies", a speaker and robot, employing four levels of "humanness of voice", and measured perception of the agent using human-trust, technology-trust, and Godspeed series questionnaires.

Results

We found that the agents elicit both human and technology conceptions of trust with no significant difference, that differences in body and voice of an agent have no significant impact on trust, even though body and voice are both independently significant in anthropomorphism perception.

Discussion

Interestingly, the results indicate that voice may be a stronger characteristic in influencing the perception of agents (not relating to trust) than physical appearance or body. We discuss the implications of our findings for research on human-agent interaction and highlight future research areas.

The success elements of humor use in workplace leadership: A proposed framework with cognitive and emotional competencies

This qualitative study aims to investigate the competencies and effectiveness of humor use in workplace leadership. By exploring the elements underlying successful and unsuccessful humor use, this research offers insights into the competencies required for leaders to leverage humor effectively. Adopting a qualitative inductive approach, fifteen individual semi-structured interviews were conducted, generating a dataset of 51 critical incidents of humor use. Reflexive thematic analysis was employed to identify key themes, resulting in the identification of five central elements: Reading the context, Intention and motivation, Judgement and decision, Skillful delivery, and Understanding reactions. These elements provide a comprehensive framework for understanding humor use in the context of workplace leadership, emphasizing the importance of cognitive and emotional intelligence / competencies. The study proposes a theoretical framework based on these findings, providing the foundation of a new paradigm for understanding and measuring humor use. This research contributes to a deeper understanding of the competencies and complexities involved in using humor as a leadership tool and provides practical implications for leaders aiming to enhance their leadership effectiveness through humor.

Regionally specific cortical lateralization of abstract and concrete verb processing: Magnetic mismatch negativity study

The neural underpinnings of processing concrete and abstract semantics remain poorly understood. Previous fMRI studies have shown that multimodal and amodal neural networks respond differentially to different semantic types; importantly, abstract semantics activates more left-lateralized networks, as opposed to more bilateral activity for concrete words. Due to the lack of temporal resolution, these fMRI results do not allow to easily separate language- and task-specific brain responses and to disentangle early processing stages from later post-comprehension phenomena. To tackle this, we used magnetoencephalography (MEG), a time-resolved neuroimaging technique, in combination with a task-free oddball mismatch negativity (MMN) paradigm, an established approach to tracking early automatic activation of word-specific memory traces in the brain. We recorded the magnetic MMN responses in 30 healthy adults to auditorily presented abstract and concrete action verbs to assess lateralization of word-specific lexico-semantic processing in a set of neocortical areas. We found that MMN responses to these stimuli showed different lateralization patterns of activity in the upper limb motor area (BA4) and parts of Broca's area (BA45/BA47) within ∼100-350 ms after the word disambiguation point. Importantly, the greater leftward response lateralization for abstract semantics was due to the lesser involvement of the right-hemispheric homologues, not increased left-hemispheric activity. These findings suggest differential region-specific involvement of bilateral sensorimotor systems already in the early automatic stages of processing abstract and concrete action semantics.

Multi-modal analysis of infant cry types characterization: Acoustics, body language and brain signals

BACKGROUND

Infant crying is the first attempt babies use to communicate during their initial months of life. A misunderstanding of the cry message can compromise infant care and future neurodevelopmental process.

METHODS

An exploratory study collecting multimodal data (i.e., crying, electroencephalography (EEG), near-infrared spectroscopy (NIRS), facial expressions, and body movements) from 38 healthy full-term newborns was conducted. Cry types were defined based on different conditions (i.e., hunger, sleepiness, fussiness, need to burp, and distress). Statistical analysis, Machine Learning (ML), and Deep Learning (DL) techniques were used to identify relevant features for cry type classification and to evaluate a robust DL algorithm named Acoustic MultiStage Interpreter (AMSI).

RESULTS

Significant differences were found across cry types based on acoustics, EEG, NIRS, facial expressions, and body movements. Acoustics and body language were identified as the most relevant ML features to support the cause of crying. The DL AMSI algorithm achieved an accuracy rate of 92%.

CONCLUSIONS

This study set a precedent for cry analysis research by highlighting the complexity of newborn cry expression and strengthening the potential use of infant cry analysis as an objective, reliable, accessible, and non-invasive tool for cry interpretation, improving the infant-parent relationship and ensuring family well-being.

Exploring the Emotional Functions of Co-Speech Hand Gesture in Language and Communication

Research over the past four decades has built a convincing case that co-speech hand gestures play a powerful role in human cognition . However, this recent focus on the cognitive function of gesture has, to a large extent, overlooked its emotional role-a role that was once central to research on bodily expression. In the present review, we first give a brief summary of the wealth of research demonstrating the cognitive function of co-speech gestures in language acquisition, learning, and thinking. Building on this foundation, we revisit the emotional function of gesture across a wide range of communicative contexts, from clinical to artistic to educational, and spanning diverse fields, from cognitive neuroscience to linguistics to affective science. Bridging the cognitive and emotional functions of gesture highlights promising avenues of research that have varied practical and theoretical implications for human-machine interactions, therapeutic interventions, language evolution, embodied cognition, and more.

Symbolic meanings of ordinary city streets and their trees

Symbolic meaning is one of a number of modes of humans' relationships with physical settings. Although symbolic meaning is qualitative and ambiguous, it is an encompassing mode of interaction: symbolic meanings assemble feelings, urges, and abstract concepts; they shape people's understanding of the world and motivate their purposes, attitudes, and actions. Early literature in environmental psychology acknowledged symbolic meaning's promise, but in recent decades it has been inadequately studied; theoretical and methodological research has been needed. This paper advances the understanding and use of symbolic meaning by, first, presenting a theory which posits that in ordinary environmental settings symbolic meanings emerge from interaction between the perceptible qualities of environmental features and people's psychological predisposition to respond to them. The paper then demonstrates methods which use the theory to objectively guide the identification of symbolic meanings in the case of ordinary urban streets and their trees. Although symbolic interpretation is intuitive and subjective, in this study it is guided by objective empirical knowledge and theoretical frameworks from human sciences. A combination of methods is applied, making conclusions answerable to diverse types of underlying data. One method was in firsthand observation of present-day streets; interpretations were accepted which linked objects' perceptible qualities with people's known dispositions to respond to them. A second method was interpretation in conventional street features' documented historical evolution. Interpretations were accepted which linked objects' perceptible qualities with people's known disposition to respond, and with symbols' known cultural tendency over time to specialize, differentiate, and evolve into coherent systems consistent with social norms. The results confirm that ordinary streets and their trees form a coherent system of symbols. Their meanings are social and sociomoral; they are guides to and affirmations of humane social life; they deserve to be prioritized in design agendas alongside tangible performance measures. It is concluded that symbolic meanings are present in ordinary urban settings, that their presence can be explained theoretically, and that their interpretation can be objectively guided.

Embodying Language through Gestures: Residuals of Motor Memories Modulate Motor Cortex Excitability during Abstract Words Comprehension

There is a debate about whether abstract semantics could be represented in a motor domain as concrete language. A contextual association with a motor schema (action or gesture) seems crucial to highlighting the motor system involvement. The present study with transcranial magnetic stimulation aimed to assess motor cortex excitability changes during abstract word comprehension after conditioning word reading to a gesture execution with congruent or incongruent meaning. Twelve healthy volunteers were engaged in a lexical-decision task responding to abstract words or meaningless verbal stimuli. Motor cortex (M1) excitability was measured at different after-stimulus intervals (100, 250, or 500 ms) before and after an associative-learning training where the execution of the gesture followed word processing. Results showed a significant post-training decrease in hand motor evoked potentials at an early processing stage (100 ms) in correspondence to words congruent with the gestures presented during the training. We hypothesized that traces of individual semantic memory, combined with training effects, induced M1 inhibition due to the redundancy of evoked motor representation. No modulation of cortical excitability was found for meaningless or incongruent words. We discuss data considering the possible implications in research to understand the neural basis of language development and language rehabilitation protocols.

Video meeting signals: Experimental evidence for a technique to improve the experience of video conferencing

We found evidence from two experiments that a simple set of gestural techniques can improve the experience of online meetings. Video conferencing technology has practical benefits, but psychological costs. It has allowed industry, education and social interactions to continue in some form during the covid-19 lockdowns. But it has left many users feeling fatigued and socially isolated, perhaps because the limitations of video conferencing disrupt users' ability to coordinate interactions and foster social affiliation. Video Meeting Signals (VMS™) is a simple technique that uses gestures to overcome some of these limitations. First, we carried out a randomised controlled trial with over 100 students, in which half underwent a short training session in VMS. All participants rated their subjective experience of two weekly seminars, and transcripts were objectively coded for the valence of language used. Compared to controls, students with VMS training rated their personal experience, their feelings toward their seminar group, and their perceived learning outcomes as significantly higher. Also, they were more likely to use positive language and less likely to use negative language. A second, larger experiment replicated the first, and added a condition where groups were given a version of the VMS training but taught to use emoji response buttons rather than gestures to signal the same information. The emoji-trained groups did not experience the same improvement as the VMS groups. By exploiting the specific benefits of gestural communication, VMS has great potential to overcome the psychological problems of group video meetings.

Interpretation of Expressions through Hand Signs Using Deep Learning Techniques

It is a challenging task to interpret sign language automatically, as it comprises high-level vision features to accurately understand and interpret the meaning of the signer or vice versa. In the current study, we automatically distinguish hand signs and classify seven basic gestures representing symbolic emotions or expressions like happy, sad, neutral, disgust, scared, anger, and surprise. Convolutional Neural Network is a famous method for classifications using vision-based deep learning; here in the current study, proposed transfer learning using a well-known architecture of VGG16 to speed up the convergence and improve accuracy by using pre-trained weights. We obtained a high accuracy of 99.98% of the proposed architecture with a minimal and low-quality data set of 455 images collected by 65 individuals for seven hand gesture classes. Further, compared the performance of VGG16 architecture with two different optimizers, SGD, and Adam, along with some more architectures of Alex Net, LeNet05, and ResNet50.

Altered Processing of Social Emotions in Individuals With Autistic Traits

Social impairment is a defining phenotypic feature of autism. The present study investigated whether individuals with autistic traits exhibit altered perceptions of social emotions. Two groups of participants (High-AQ and Low-AQ) were recruited based on their scores on the autism-spectrum quotient (AQ). Their behavioral responses and event-related potentials (ERPs) elicited by social and non-social stimuli with positive, negative, and neutral emotional valence were compared in two experiments. In Experiment 1, participants were instructed to view social-emotional and non-social emotional pictures. In Experiment 2, participants were instructed to listen to social-emotional and non-social emotional audio recordings. More negative emotional reactions and smaller amplitudes of late ERP components (the late positive potential in Experiment 1 and the late negative component in Experiment 2) were found in the High-AQ group than in the Low-AQ group in response to the social-negative stimuli. In addition, amplitudes of these late ERP components in both experiments elicited in response to social-negative stimuli were correlated with the AQ scores of the High-AQ group. These results suggest that individuals with autistic traits have altered emotional processing of social-negative emotions.

Autonomic system tuning during gesture observation and reproduction

Gestural communication allows providing information about thoughts and feelings, characterizing face-to-face interactions, also during non-verbal exchanges. In the present study, the autonomic responses and peripheral synchronization mechanisms of two individuals (encoder and decoder) were recorded simultaneously, through the use of biofeedback in hyperscanning, during two different experimental phases consisting in the observation (watching videos of gestures) and reproduction of positive and negative different types of gestures (affective, social and informative) supported by linguistic contexts. Therefore, the main aim of this study was focused on the analysis of simultaneous individuals' peripheral mechanisms during the performing of complex joint action, consisting of the observation (watching videos) and the reproduction of positive and negative social, affective, and informative gestures each supported by a linguistic script. Single-subject and inter-subject correlation analyses were conducted to observe individuals' autonomic responses and physiological synchronization. Single-subject results revealed an increase in emotional arousal, indicated by an increase in electrodermal activity (skin conductance level - SCL and response - SCR), during both the observation (watching videos) and reproduction of negative social and affective gestures contextualized by a linguistic context. Moreover, an increase of emotional engagement, expressed by an increase in heart rate (HR) activity, emerged in the encoder compare to the decoder during gestures reproduction (simulation of gestures). Inter-subject correlation results showed the presence of mirroring mechanisms, indicated by an increase in SCL, SCR, and HR synchronization, during the linguistic contexts and gesture observation (watching videos). Furthermore, an increase in SCL and SCR synchronization emerged during the observation (watching videos) and reproduction of negative social and affective gestures. Therefore, the present study allowed to obtain information on the mirroring mechanisms and physiological synchronization underlying the linguistic and gesture system during non-verbal interaction.

Research on Discrete Semantics in Continuous Hand Joint Movement Based on Perception and Expression

Continuous movements of the hand contain discrete expressions of meaning, forming a variety of semantic gestures. For example, it is generally considered that the bending of the finger includes three semantic states of bending, half bending, and straightening. However, there is still no research on the number of semantic states that can be conveyed by each movement primitive of the hand, especially the interval of each semantic state and the representative movement angle. To clarify these issues, we conducted experiments of perception and expression. Experiments 1 and 2 focused on perceivable semantic levels and boundaries of different motion primitive units from the perspective of visual semantic perception. Experiment 3 verified and optimized the segmentation results obtained above and further determined the typical motion values of each semantic state. Furthermore, in Experiment 4, the empirical application of the above semantic state segmentation was illustrated by using Leap Motion as an example. We ended up with the discrete gesture semantic expression space both in the real world and Leap Motion Digital World, containing the clearly defined number of semantic states of each hand motion primitive unit and boundaries and typical motion angle values of each state. Construction of this quantitative semantic expression will play a role in guiding and advancing research in the fields of gesture coding, gesture recognition, and gesture design.

A New Neurorehabilitative Postsurgery Intervention for Facial Palsy Based on Smile Observation and Hand-Mouth Motor Synergies

Objective

To perform a preliminary test of a new rehabilitation treatment (FIT-SAT), based on mirror mechanisms, for gracile muscles after smile surgery.

Method

A pre- and postsurgery longitudinal design was adopted to study the efficacy of FIT-SAT. Four patients with bilateral facial nerve paralysis (Moebius syndrome) were included. They underwent two surgeries with free muscle transfers, one year apart from each other. The side of the face first operated on was rehabilitated with the traditional treatment, while the second side was rehabilitated with FIT-SAT. The FIT-SAT treatment includes video clips of an actor performing a unilateral or a bilateral smile to be imitated (FIT condition). In addition to this, while smiling, the participants close their hand in order to exploit the overlapped cortical motor representation of the hand and the mouth, which may facilitate the synergistic activity of the two effectors during the early phases of recruitment of the transplanted muscles (SAT). The treatment was also aimed at avoiding undesired movements such as teeth grinding. Discussion. Results support FIT-SAT as a viable alternative for smile rehabilitation after free muscle transfer. We propose that the treatment potentiates the effect of smile observation by activating the same neural structures responsible for the execution of the smile and therefore by facilitating its production. Closing of the hand induces cortical recruitment of hand motor neurons, recruiting the transplanted muscles, and reducing the risk of associating other unwanted movements such as teeth clenching to the smile movements.

Review and Challenges of Technologies for Real-Time Human Behavior Monitoring

A person's behavior significantly influences their health and well-being. It also contributes to the social environment in which humans interact, with cascading impacts to the health and behaviors of others. During social interactions, our understanding and awareness of vital nonverbal messages expressing beliefs, emotions, and intentions can be obstructed by a variety of factors including greatly flawed self-awareness. For these reasons, human behavior is a very important topic to study using the most advanced technology. Moreover, technology offers a breakthrough opportunity to improve people's social awareness and self-awareness through machine-enhanced recognition and interpretation of human behaviors. This paper reviews (1) the social psychology theories that have established the framework to study human behaviors and their manifestations during social interactions and (2) the technologies that have contributed to the monitoring of human behaviors. State-of-the-art in sensors, signal features, and computational models are categorized, summarized, and evaluated from a comprehensive transdisciplinary perspective. This review focuses on assessing technologies most suitable for real-time monitoring while highlighting their challenges and opportunities in near-future applications. Although social behavior monitoring has been highly reported in psychology and engineering literature, this paper uniquely aims to serve as a disciplinary convergence bridge and a guide for engineers capable of bringing new technologies to bear against the current challenges in real-time human behavior monitoring.

Modality-specific dysfunctional neural processing of social-abstract and non-social-concrete information in schizophrenia

Schizophrenia is characterized by marked communication dysfunctions encompassing potential impairments in the processing of social-abstract and non-social-concrete information, especially in everyday situations where multiple modalities are present in the form of speech and gesture. To date, the neurobiological basis of these deficits remains elusive. In a functional magnetic resonance imaging (fMRI) study, 17 patients with schizophrenia or schizoaffective disorder, and 18 matched controls watched videos of an actor speaking, gesturing (unimodal), and both speaking and gesturing (bimodal) about social or non-social events in a naturalistic way. Participants were asked to judge whether each video contains person-related (social) or object-related (non-social) information. When processing social-abstract content, patients showed reduced activation in the medial prefrontal cortex (mPFC) only in the gesture but not in the speech condition. For non-social-concrete content, remarkably, reduced neural activation for patients in the left postcentral gyrus and the right insula was observed only in the speech condition. Moreover, in the bimodal conditions, patients displayed improved task performance and comparable activation to controls in both social and non-social content. To conclude, patients with schizophrenia displayed modality-specific aberrant neural processing of social and non-social information, which is not present for the bimodal conditions. This finding provides novel insights into dysfunctional multimodal communication in schizophrenia, and may have potential therapeutic implications.

Differential mirror neuron system (MNS) activation during action observation with and without social-emotional components in autism: a meta-analysis of neuroimaging studies

Background

Impaired imitation has been found to be an important factor contributing to social communication deficits in individuals with autism spectrum disorder (ASD). It has been hypothesized that the neural correlate of imitation, the mirror neuron system (MNS), is dysfunctional in ASD, resulting in imitation impairment as one of the key behavioral manifestations in ASD. Previous MNS studies produced inconsistent results, leaving the debate of whether “broken” mirror neurons in ASD are unresolved.

Methods

This meta-analysis aimed to explore the differences in MNS activation patterns between typically developing (TD) and ASD individuals when they observe biological motions with or without social-emotional components. Effect size signed differential mapping (ES-SDM) was adopted to synthesize the available fMRI data.

Results

ES-SDM analysis revealed hyperactivation in the right inferior frontal gyrus and left supplementary motor area in ASD during observation of biological motions. Subgroup analysis of experiments involving the observation of stimuli with or without emotional component revealed hyperactivation in the left inferior parietal lobule and left supplementary motor during action observation without emotional components, whereas hyperactivation of the right inferior frontal gyrus was found during action observation with emotional components in ASD. Subgroup analyses of age showed hyperactivation of the bilateral inferior frontal gyrus in ASD adolescents, while hyperactivation in the right inferior frontal gyrus was noted in ASD adults. Meta-regression within ASD individuals indicated that the right cerebellum crus I activation increased with age, while the left inferior temporal gyrus activation decreased with age.

Limitations

This meta-analysis is limited in its generalization of the findings to individuals with ASD by the restricted age range, heterogeneous study sample, and the large within-group variation in MNS activation patterns during object observation. Furthermore, we only included action observation studies which might limit the generalization of our results to the imitation deficits in ASD. In addition, the relatively small sample size for individual studies might also potentially overestimate the effect sizes.

Conclusion

The MNS is impaired in ASD. The abnormal activation patterns were found to be modulated by the nature of stimuli and age, which might explain the contradictory results from earlier studies on the “broken mirror neuron” debate.