Automated Video Analysis of Non-verbal Communication in a Medical Setting

Bio-behavioral synchrony is a potential mechanism for mate selection in humans

The decision with whom to form a romantic bond is of great importance, yet the biological or behavioral mechanisms underlying this selective process in humans are largely unknown. Classic evolutionary theories of mate selection emphasize immediate and static features such as physical appearance and fertility. However, they do not explain how initial attraction temporally unfolds during an interaction, nor account for mutual physiological or behavioral adaptations that take place when two people become attracted. Instead, recent theories on social bonding emphasize the importance of co-regulation during social interactions (i.e., the social coordination of physiology and behavior between partners), and predict that co-regulation plays a role in bonding with others. In a speed-date experiment of forty-six heterosexual dates, we recorded the naturally occurring patterns of electrodermal activity and behavioral motion in men and women, and calculated their co-regulation during the date. We demonstrate that co-regulation of behavior and physiology is associated with the date outcome: when a man and a woman synchronize their electrodermal activity and dynamically tune their behavior to one another, they are more likely to be romantically and sexually attracted to one another. This study supports the hypothesis that co-regulation of sympathetic and behavioral rhythms between a man and a woman serves as a mechanism that promotes attraction.

Liking as a balance between synchronization, complexity and novelty

Synchronization has been identified as a key aspect in social bonding. While synchronization could be maximized by increasing the predictability of an interaction, such predictability is in tension with individuals’ level of interest, which is tied to the interaction’s complexity and novelty. In this study, we tested the interplay between synchronization and interest. We asked 104 female dyads to play the Mirror Game, in which they had to move their hands as coordinately as possible, and then report how much they liked each other. Utilizing information theory and video processing tools, we found that a combination of movement synchronization and complexity explained liking almost two times better than movement synchronization alone. Moreover, we found that people initiated novel and challenging interactions, even though they paid a price—being less synchronized. Examining the interactions’ dynamics, we found that people who liked each other moved in a more synchronized, complex, and novel manner during most of the interaction. This suggests that in addition to synchronization, maintaining interest may be critical for positive social bonding. Thus, we propose a new framework in which balancing synchronization and interest, rather than merely maximizing synchronization, optimizes the interaction quality.

Prediction of Communication Effectiveness During Media Skills Training Using Commercial Automatic Non-verbal Recognition Systems

It is well recognised that social signals play an important role in communication effectiveness. Observation of videos to understand non-verbal behaviour is time-consuming and limits the potential to incorporate detailed and accurate feedback of this behaviour in practical applications such as communication skills training or performance evaluation. The aim of the current research is twofold: (1) to investigate whether off-the-shelf emotion recognition technology can detect social signals in media interviews and (2) to identify which combinations of social signals are most promising for evaluating trainees' performance in a media interview. To investigate this, non-verbal signals were automatically recognised from practice on-camera media interviews conducted within a media training setting with a sample size of 34. Automated non-verbal signal detection consists of multimodal features including facial expression, hand gestures, vocal behaviour and 'honest' signals. The on-camera interviews were categorised into effective and poor communication exemplars based on communication skills ratings provided by trainers and neutral observers which served as a ground truth. A correlation-based feature selection method was used to select signals associated with performance. To assess the accuracy of the selected features, a number of machine learning classification techniques were used. Naive Bayes analysis produced the best results with an F-measure of 0.76 and prediction accuracy of 78%. Results revealed that a combination of body movements, hand movements and facial expression are relevant for establishing communication effectiveness in the context of media interviews. The results of the current study have implications for the automatic evaluation of media interviews with a number of potential application areas including enhancing communication training including current media skills training.

Automatic Classification of Screen Gaze and Dialogue in Doctor-Patient-Computer Interactions: Computational Ethnography Algorithm Development and Validation

Background

The study of doctor-patient-computer interactions is a key research area for examining doctor-patient relationships; however, studying these interactions is costly and obtrusive as researchers usually set up complex mechanisms or intrude on consultations to collect, then manually analyze the data.

Objective

We aimed to facilitate human-computer and human-human interaction research in clinics by providing a computational ethnography tool: an unobtrusive automatic classifier of screen gaze and dialogue combinations in doctor-patient-computer interactions.

Methods

The classifier’s input is video taken by doctors using their computers' internal camera and microphone. By estimating the key points of the doctor's face and the presence of voice activity, we estimate the type of interaction that is taking place. The classification output of each video segment is 1 of 4 interaction classes: (1) screen gaze and dialogue, wherein the doctor is gazing at the computer screen while conversing with the patient; (2) dialogue, wherein the doctor is gazing away from the computer screen while conversing with the patient; (3) screen gaze, wherein the doctor is gazing at the computer screen without conversing with the patient; and (4) other, wherein no screen gaze or dialogue are detected. We evaluated the classifier using 30 minutes of video provided by 5 doctors simulating consultations in their clinics both in semi- and fully inclusive layouts.

Results

The classifier achieved an overall accuracy of 0.83, a performance similar to that of a human coder. Similar to the human coder, the classifier was more accurate in fully inclusive layouts than in semi-inclusive layouts.

Conclusions

The proposed classifier can be used by researchers, care providers, designers, medical educators, and others who are interested in exploring and answering questions related to screen gaze and dialogue in doctor-patient-computer interactions.

Clinician-Patient Movement Synchrony Mediates Social Group Effects on Interpersonal Trust and Perceived Pain

Pain is an unfortunate consequence of many medical procedures, which in some patients becomes chronic and debilitating. Among the factors affecting medical pain, clinician-patient (C-P) similarity and nonverbal communication are particularly important for pain diagnosis and treatment. Participants (N = 66) were randomly assigned to clinician and patient roles and were grouped into C-P dyads. Clinicians administered painful stimuli to patients as an analogue of a painful medical procedure. We manipulated the perceived C-P similarity of each dyad using groups ostensibly based on shared beliefs and values, and each patient was tested twice: Once with a same group clinician (concordant, CC) and once with a clinician from the other group (discordant, DC). Movement synchrony was calculated as a marker of nonverbal communication. We tested whether movement synchrony mediated the effects of group concordance on patients' pain and trust in the clinician. Movement synchrony was higher in CC than DC dyads. Higher movement synchrony predicted reduced pain and increased trust in the clinician. Movement synchrony also formally mediated the group concordance effects on pain and trust. These findings increase our understanding of the role of nonverbal C-P communication on pain and related outcomes. Interpersonal synchrony may be associated with better pain outcomes, independent of the specific treatment provided. PERSPECTIVE: This article demonstrates that movement synchrony in C-P interactions is an unobtrusive measure related to their relationship quality, trust toward the clinician, and pain. These findings suggest that interpersonal synchrony may be associated with better patient outcomes, independent of the specific treatment provided.

Beyond Convenience: Patients' Perceptions of Physician Interactional Skills and Compassion via Telemedicine

Objectives

To understand the interpersonal and communication behaviors that are perceived positively by patients in a video encounter and whether patient-centered relationships can be established virtually.

Patients and Methods

A qualitative analysis of patient visit feedback was performed to build consensus around exemplary interpersonal and communication practices during a virtual urgent care visit. Voluntarily submitted patient comments associated with a 5-star review after a visit were randomly selected from more than 49,000 comments in an 11-month period, from January 1, 2016, through November 30, 2016. Researchers used a consensus-based, widely used health care communications framework as a sensitizing scaffold to develop a preliminary set of codes.

Results

More than 30% of the comments coded were classified as Building Rapport. The next most frequently assigned code was Shares Information/Provides Guidance. Among codable comments, the third most frequently assigned code was Elicits Information. Provided Treatment accounted for only 2% of comments.

Conclusion

These results suggest that patients who are satisfied with telemedicine encounters appreciate their relational experiences with the clinician and overall user experience, including access and convenience. Highly satisfied patients who interacted with providers on this platform commented on key aspects of medical communication, particularly skills that demonstrate patient-centered relationship building. This supports the notion that clinician-patient relationships can be established in a video-first model, without a previous in-person encounter, and that positive ratings do not seem to be focused solely on prescription receipt.

Associations between movement synchrony and outcome in patients with social anxiety disorder: Evidence for treatment specific effects

Background: Studies with heterogeneous samples in naturalistic treatment settings suggest that movement synchrony (MS) between therapists and patients correlates with therapeutic success. In this study, we examined a homogeneous sample of patients with social anxiety disorder and investigated whether MS in sessions 3 and 8 would be associated with therapy outcome and therapeutic alliance, and whether these associations depend on the therapeutic approach. Methods: The patients (N = 267) were treated with either manual-guided cognitive behavior therapy (CBT), manual-guided psychodynamic therapy (PDT), or naturalistic CBT. The Helping Alliance Questionnaire (HAQ), the Inventory of Interpersonal Problems (IIP) and the Beck-Depression-Inventory (BDI) were used as measures. Body motions were coded with motion energy analysis. MS was quantified using time series analysis methods. Results: MS was observed more frequently in both CBT conditions than in PDT. In both CBT groups, more synchrony was predictive of lower IIP scores at the end of therapy. If the patient lead synchrony more often than the therapist, higher IIP and BDI scores were observed at the end of treatment. PDT showed the largest effect size for the synchrony-alliance-association. Conclusion: Movement synchrony and therapeutic success are associated. The effect of therapeutic approach and leading/following are relevant for this association.

Dramatic action: A theater-based paradigm for analyzing human interactions

Existing approaches to describe social interactions consider emotional states or use ad-hoc descriptors for microanalysis of interactions. Such descriptors are different in each context thereby limiting comparisons, and can also mix facets of meaning such as emotional states, short term tactics and long-term goals. To develop a systematic set of concepts for second-by-second social interactions, we suggest a complementary approach based on practices employed in theater. Theater uses the concept of dramatic action, the effort that one makes to change the psychological state of another. Unlike states (e.g. emotions), dramatic actions aim to change states; unlike long-term goals or motivations, dramatic actions can last seconds. We defined a set of 22 basic dramatic action verbs using a lexical approach, such as ‘to threaten’–the effort to incite fear, and ‘to encourage’–the effort to inspire hope or confidence. We developed a set of visual cartoon stimuli for these basic dramatic actions, and find that people can reliably and reproducibly assign dramatic action verbs to these stimuli. We show that each dramatic action can be carried out with different emotions, indicating that the two constructs are distinct. We characterized a principal valence axis of dramatic actions. Finally, we re-analyzed three widely-used interaction coding systems in terms of dramatic actions, to suggest that dramatic actions might serve as a common vocabulary across research contexts. This study thus operationalizes and tests dramatic action as a potentially useful concept for research on social interaction, and in particular on influence tactics.

Communication of health in experimentally sick men and women: A pilot study

The way people communicate their ill-health and the factors involved in ill-health communication remain poorly known. In the present study, we tested how men and women communicate their sickness and assessed whether sickness-related variables (i.e., body temperature, immune response, subjective sickness symptoms) predicted communicative behaviors. Twenty-two participants were filmed during experimentally induced sickness, triggered by lipopolysaccharide administration (2ng/kg body weight), and after placebo administration, in presence of female care providers. Two trained raters scored participants' communicative behaviors (verbal complaints, moaning and sighs/deep breaths). The physiological and subjective sickness responses were similar in both sexes. Participants were more likely to moan and complain when sick, although the frequency of these behaviors remained low and no clear sex differences was observed. Nevertheless, frequency of sighs/deep breaths was increased amongst sick men but not in women. Sickness-related variables did not predict sigh/deep breath frequency. In this setting, sick men appear to display a lower threshold of expressing their malaise as compared to similarly sick women.

A reduced-dimensionality approach to uncovering dyadic modes of body motion in conversations

Face-to-face conversations are central to human communication and a fascinating example of joint action. Beyond verbal content, one of the primary ways in which information is conveyed in conversations is body language. Body motion in natural conversations has been difficult to study precisely due to the large number of coordinates at play. There is need for fresh approaches to analyze and understand the data, in order to ask whether dyads show basic building blocks of coupled motion. Here we present a method for analyzing body motion during joint action using depth-sensing cameras, and use it to analyze a sample of scientific conversations. Our method consists of three steps: defining modes of body motion of individual participants, defining dyadic modes made of combinations of these individual modes, and lastly defining motion motifs as dyadic modes that occur significantly more often than expected given the single-person motion statistics. As a proof-of-concept, we analyze the motion of 12 dyads of scientists measured using two Microsoft Kinect cameras. In our sample, we find that out of many possible modes, only two were motion motifs: synchronized parallel torso motion in which the participants swayed from side to side in sync, and still segments where neither person moved. We find evidence of dyad individuality in the use of motion modes. For a randomly selected subset of 5 dyads, this individuality was maintained for at least 6 months. The present approach to simplify complex motion data and to define motion motifs may be used to understand other joint tasks and interactions. The analysis tools developed here and the motion dataset are publicly available.