Synchronization of Nonverbal Behaviors in Detecting Mediated and Non-mediated Deception

Mining Bodily Cues to Deception

A significant body of research has investigated potential correlates of deception and bodily behavior. The vast majority of these studies consider discrete, subjectively coded bodily movements such as specific hand or head gestures. Such studies fail to consider quantitative aspects of body movement such as the precise movement direction, magnitude and timing. In this paper, we employ an innovative data mining approach to systematically study bodily correlates of deception. We re-analyze motion capture data from a previously published deception study, and experiment with different data coding options. We report how deception detection rates are affected by variables such as body part, the coding of the pose and movement, the length of the observation, and the amount of measurement noise. Our results demonstrate the feasibility of a data mining approach, with detection rates above 65%, significantly outperforming human judgement (52.80%). Owing to the systematic analysis, our analyses allow for an understanding of the importance of various coding factor. Moreover, we can reconcile seemingly discrepant findings in previous research. Our approach highlights the merits of data-driven research to support the validation and development of deception theory.

Nonverbal Synchrony in Technology-Mediated Interviews: A Cross-Cultural Study

Technology-mediated communication has changed the way we interact. Since the onset of the COVID-19 pandemic in March 2020, this trend became even more pronounced. Media interviews are no exception. Yet, studies on nonverbal behaviors, especially nonverbal synchrony during such mediated settings, have been scarce. To fill the research gap, this study investigated synchronized patterns between interview hosts' and guests' facial emotional displays and upper body movement during mediated interviews recorded in the countries in Western (mainly the US, with the addition of the UK) and Eastern cultures (Japan). The interviews were categorized into information- or entertainment-driven interviews, depending on the social attributes of the guest. The time series of the valence in facial displays and upper body movement was automatedly measured using FaceReader and Motion Energy Analysis software, respectively, which was analyzed in terms of simultaneous movements, a primary component of synchrony. As predicted, facial synchrony was more prevalent in information-driven interviews, supporting the motivational and strategic account of synchrony. In addition, female-hosted interviews had a higher degree of synchrony, especially in information-driven interviews. Similar patterns were seen in movement synchrony, although not significant. This study is the first evidence of synchrony in technology-mediated interviews in which a host and a guest appear on split-screen to inform or entertain audiences. However, no cultural differences in synchrony were observed. Situational demands in front of the interactants and the goal-driven nature of communication seemed to play a more prominent role than cultural differences in nonverbal synchrony.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10919-022-00416-3.

Video-based tracking approach for nonverbal synchrony: A comparison of Motion Energy Analysis and OpenPose

The present study investigated the comparability of two video-based tracking techniques, namely, Motion Energy Analysis and OpenPose, in the context of nonverbal synchrony research. Participants aged 23-69 years held a 15-min unstructured conversation with a same-gender partner of the same generation. Each participant's movements were quantified with two algorithms, which were separately submitted to a wavelet transform to assess synchrony between the partners. Wavelet coherence and information in the relative phase were used to measure the intensities and patterns of synchrony, respectively. As a benchmark of the test results, gender effects and correlation with the Big Five personality traits were examined. Irrespective of the tracking technique used, the results consistently illustrated gender effects: female dyads showed more synchrony than male dyads. Among the Big Five personality traits, only Extraversion was significantly associated with synchrony in either tracking technique. The gender effect remained significant in both the tracking techniques even when controlling for personality traits, indicating that both individual (i.e., personality) and social (i.e., gender) factors contribute to synchrony. The use of video-tracking techniques in the research on synchrony for future studies was also discussed.

Who's Leading This Dance?: Theorizing Automatic and Strategic Synchrony in Human-Exoskeleton Interactions

Wearable robots are an emerging form of technology that allow organizations to combine the strength, precision, and performance of machines with the flexibility, intelligence, and problem-solving abilities of human wearers. Active exoskeletons are a type of wearable robot that gives wearers the ability to effortlessly lift up to 200 lbs., as well as perform other types of physically demanding tasks that would be too strenuous for most humans. Synchronization between exoskeleton suits and wearers is one of the most challenging requirements to operate these technologies effectively. In this conceptual paper, we extend interpersonal adaption theory (IAT) to the exoskeleton context and explicate (a) the antecedents that are most likely to shape synchrony in human-exoskeleton interactions, (b) automatic and strategic synchrony as adaptive behaviors in human-exoskeleton interactions, and (c) outcome variables that are especially important in these processes. Lastly, we offer a discussion of key methodological challenges for measuring synchrony in human-exoskeleton interactions and offer a future research agenda for this important area.

Remote unaffiliated presurgical psychosocial evaluations: a qualitative assessment of the attitudes of ASMBS members

BACKGROUND

A psychosocial evaluation is an important component of the preoperative assessment process for people seeking metabolic and bariatric surgery (MBS), and is required for accreditation of MBS programs. Recently, independent companies without affiliations with MBS programs have been marketing remotely administered, unaffiliated psychosocial evaluations for MBS (RUS), and American Society for Metabolic and Bariatric Surgery (ASMBS) members have raised concerns about these evaluations.

OBJECTIVES

To explore ASMBS members' beliefs about RUS.

SETTING

Online survey.

METHODS

We developed a survey to evaluate ASMBS members' opinions, experiences, and/or concerns about in-person and RUS psychosocial evaluations for MBS.

RESULTS

In total, 635 ASMBS members responded to the online survey and 156 responded to an open-ended question on RUS. Responses were coded based on a manual developed for this study, yielding themes of concerns about the quality of RUS, lack of ongoing relationships in RUS, and conditions under which/reasons why RUS evaluations could be acceptable.

CONCLUSION

Respondents expressed both interest in and concerns about RUS in pre-MBS psychosocial evaluations. Use of RUS has the potential to improve access to MBS by providing a convenient and efficient means of completing the psychosocial evaluation. Conversely, respondents expressed concerns about the background and training of RUS providers, the quality of the reports, and the limited relationships between the RUS provider and both the MBS patient and the MBS team. We discuss the clinical and research implications of response themes, particularly for patients in rural areas or those who have other barriers to care.

A liar and a copycat: nonverbal coordination increases with lie difficulty

Studies of the nonverbal correlates of deception tend to examine liars' behaviours as independent from the behaviour of the interviewer, ignoring joint action. To address this gap, experiment 1 examined the effect of telling a truth and easy, difficult and very difficult lies on nonverbal coordination. Nonverbal coordination was measured automatically by applying a dynamic time warping algorithm to motion-capture data. In experiment 2, interviewees also received instructions that influenced the attention they paid to either the nonverbal or verbal behaviour of the interviewer. Results from both experiments found that interviewer-interviewee nonverbal coordination increased with lie difficulty. This increase was not influenced by the degree to which interviewees paid attention to their nonverbal behaviour, nor by the degree of interviewer's suspicion. Our findings are consistent with the broader proposition that people rely on automated processes such as mimicry when under cognitive load.

Clusters of nonverbal behavior differentiate truths and lies about future malicious intent in checkpoint screening interviews

Recent research has shown that nonverbal behavior (NVB) assessed across multiple channels can differentiate truthtellers from liars. No study, however, has examined whether or not multiple NVBs can differentiate truths from lies about intent regarding future malicious behavior, or across multiple cultural/ethnic groups. We address this gap by examining truths and lies about intent to commit a malicious act in the future in brief, checkpoint-type security screening interviews. Data from four NVB channels producing twenty-one observable NVBs were coded and analyzed using different analytic strategies. Clusters of NVB were found to differentiate truthtellers from liars at statistically significant levels, and substantially beyond the ability of human observers. The findings showed that clusters of NVB can differentiate truthtellers from liars even in brief, checkpoint-type interviews.

Conversing with a devil's advocate: Interpersonal coordination in deception and disagreement

This study investigates the presence of dynamical patterns of interpersonal coordination in extended deceptive conversations across multimodal channels of behavior. Using a novel "devil's advocate" paradigm, we experimentally elicited deception and truth across topics in which conversational partners either agreed or disagreed, and where one partner was surreptitiously asked to argue an opinion opposite of what he or she really believed. We focus on interpersonal coordination as an emergent behavioral signal that captures interdependencies between conversational partners, both as the coupling of head movements over the span of milliseconds, measured via a windowed lagged cross correlation (WLCC) technique, and more global temporal dependencies across speech rate, using cross recurrence quantification analysis (CRQA). Moreover, we considered how interpersonal coordination might be shaped by strategic, adaptive conversational goals associated with deception. We found that deceptive conversations displayed more structured speech rate and higher head movement coordination, the latter with a peak in deceptive disagreement conversations. Together the results allow us to posit an adaptive account, whereby interpersonal coordination is not beholden to any single functional explanation, but can strategically adapt to diverse conversational demands.

Is interactional dissynchrony a clue to deception? Insights from automated analysis of nonverbal visual cues

Detecting deception in interpersonal dialog is challenging since deceivers take advantage of the give-and-take of interaction to adapt to any sign of skepticism in an interlocutor's verbal and nonverbal feedback. Human detection accuracy is poor, often with no better than chance performance. In this investigation, we consider whether automated methods can produce better results and if emphasizing the possible disruption in interactional synchrony can signal whether an interactant is truthful or deceptive. We propose a data-driven and unobtrusive framework using visual cues that consists of face tracking, head movement detection, facial expression recognition, and interactional synchrony estimation. Analysis were conducted on 242 video samples from an experiment in which deceivers and truth-tellers interacted with professional interviewers either face-to-face or through computer mediation. Results revealed that the framework is able to automatically track head movements and expressions of both interlocutors to extract normalized meaningful synchrony features and to learn classification models for deception recognition. Further experiments show that these features reliably capture interactional synchrony and efficiently discriminate deception from truth.