Performance-driven facial animation: basic research on human judgments of emotional state in facial avatars

Virtual environments to study emotional responses to clinical communication: A scoping review

OBJECTIVE

This scoping review explores the potential for virtual environments (VE) to evaluate emotional outcomes in clinical communication research. Authors representing multiple disciplines use review results to propose potential research opportunities and considerations.

METHODS

We utilized a structured framework for scoping reviews. We searched four literature databases for relevant articles. We applied multidisciplinary perspectives to synthesize relevant potential opportunities for emotion-focused communications research using VE.

RESULTS

Twenty-one articles met inclusion criteria. They applied different methodological approaches, including a range of VE technologies and diverse emotional outcome measures, such as psychophysiological arousal, emotional valence, or empathy. Major research topics included use of virtual reality to provoke and measure emotional responses, train clinicians in communication skills, and increase clinician empathy.

CONCLUSION

Researchers may leverage VE technologies to ethically and systematically examine how characteristics of clinical interactions, environments, and communication impact emotional reactions and responses among patients and clinicians. Variability exists in how VE technologies are employed and reported in published literature, and this may limit the internal and external validity of the research. However, virtual reality can provide a low-cost, low-risk, experimentally controlled, and ecologically valid approach for studying clinician-patient communication.

PRACTICE IMPLICATIONS

Future research should leverage psychophysiological measures to further examine emotional responses during clinical communication scenarios and clearly report virtual environment characteristics to support evaluation of study conclusions, study replicability, and meta-analyses.

Virtual eye region: development of a realistic model to convey emotion

The human eyes and their surrounding features are capable of conveying an array of emotional and social information through expressions. Producing virtual human eyes which are able to communicate these complex mental states continues to be a challenging research topic in computer graphics (CG) as subtle inaccuracies can be the difference between realistic and uncanny. With the recent emergence of virtual customer service agents, the demand for expressive virtual eyes is increasing. One essential question that remains to be answered is: Can virtual human eyes effectively transmit emotion? Through a combination of 3D scanning and manual hand modelling techniques, we developed an efficient pipeline to realise a virtual model of the human eye area that displays real-world characteristics. From this model eye expression renders of the six basic emotions, anger, disgust, fear, happiness, sadness and surprise were generated (Ekman et al., 1969). The perceptual quality of the model was evaluated by showing respondents from two age groups the six eye expressions renders and corresponding real-world photos. Respondents then judged which of the six emotions best described each image. Our findings indicate a clear relationship between the recognition levels for both photographic and virtual stimuli plus a significant level of emotional perception was found for the virtual eye expressions of sadness and anger. This research of human cognition and CG is a starting point for investigating the use of artificial human eye expressions as an effective research tool in the perceptual community.

Virtual Antenatal Encounter and Standardized Simulation Assessment (VANESSA): Pilot Study

Background

Prenatal counseling at the limits of newborn viability involves sensitive interactions between neonatal providers and families. Empathetic discussions are currently learned through practice in times of high stress. Decision aids may help improve provider communication but have not been universally adopted. Virtual standardized patients are increasingly recognized as a modality for education, but prenatal counseling simulations have not been described. To be valuable as a tool, a virtual patient would need to accurately portray emotions and elicit a realistic response from the provider.

Objective

To determine if neonatal providers can accurately identify a standardized virtual prenatal patient’s emotional states and examine the frequency of empathic responses to statements made by the patient.

Methods

A panel of Neonatologists, Simulation Specialists, and Ethicists developed a dialogue and identified empathic responses. Virtual Antenatal Encounter and Standardized Simulation Assessment (VANESSA), a screen-based simulation of a woman at 23 weeks gestation, was capable of displaying anger, fear, sadness, and happiness through animations. Twenty-four neonatal providers, including a subgroup with an ethics interest, were asked to identify VANESSA’s emotions 28 times, respond to statements, and answer open-ended questions. The emotions were displayed in different formats: without dialogue, with text dialogue, and with audio dialogue. Participants completed a post-encounter survey describing demographics and experience. Data were reported using descriptive statistics. Qualitative data from open ended questions (eg, “What would you do?”) were examined using thematic analysis.

Results

Half of our participants had over 10 years of clinical experience. Most participants reported using medical research (18/23, 78%) and mortality calculators (17/23, 74%). Only the ethics-interested subgroup (10/23, 43%) listed counseling literature (7/10, 70%). Of 672 attempts, participants accurately identified VANESSA’s emotions 77.8% (523/672) of the time, and most (14/23, 61%) reported that they were confident in identifying these emotions. The ethics interest group was more likely to choose empathic responses (P=.002). Participants rated VANESSA as easy to use (22/23, 96%) and reported that she had realistic dialogue (15/23, 65%).

Conclusions

This pilot study shows that a prenatal counseling simulation is feasible and can yield useful data on prenatal counseling communication. Our participants showed a high rate of emotion recognition and empathy in their responses.

Is it the real deal? Perception of virtual characters versus humans: an affective cognitive neuroscience perspective

Recent developments in neuroimaging research support the increased use of naturalistic stimulus material such as film, avatars, or androids. These stimuli allow for a better understanding of how the brain processes information in complex situations while maintaining experimental control. While avatars and androids are well suited to study human cognition, they should not be equated to human stimuli. For example, the uncanny valley hypothesis theorizes that artificial agents with high human-likeness may evoke feelings of eeriness in the human observer. Here we review if, when, and how the perception of human-like avatars and androids differs from the perception of humans and consider how this influences their utilization as stimulus material in social and affective neuroimaging studies. First, we discuss how the appearance of virtual characters affects perception. When stimuli are morphed across categories from non-human to human, the most ambiguous stimuli, rather than the most human-like stimuli, show prolonged classification times and increased eeriness. Human-like to human stimuli show a positive linear relationship with familiarity. Secondly, we show that expressions of emotions in human-like avatars can be perceived similarly to human emotions, with corresponding behavioral, physiological and neuronal activations, with exception of physical dissimilarities. Subsequently, we consider if and when one perceives differences in action representation by artificial agents versus humans. Motor resonance and predictive coding models may account for empirical findings, such as an interference effect on action for observed human-like, natural moving characters. However, the expansion of these models to explain more complex behavior, such as empathy, still needs to be investigated in more detail. Finally, we broaden our outlook to social interaction, where virtual reality stimuli can be utilized to imitate complex social situations.

Psychometric challenges and proposed solutions when scoring facial emotion expression codes

Coding of facial emotion expressions is increasingly performed by automated emotion expression scoring software; however, there is limited discussion on how best to score the resulting codes. We present a discussion of facial emotion expression theories and a review of contemporary emotion expression coding methodology. We highlight methodological challenges pertinent to scoring software-coded facial emotion expression codes and present important psychometric research questions centered on comparing competing scoring procedures of these codes. Then, on the basis of a time series data set collected to assess individual differences in facial emotion expression ability, we derive, apply, and evaluate several statistical procedures, including four scoring methods and four data treatments, to score software-coded emotion expression data. These scoring procedures are illustrated to inform analysis decisions pertaining to the scoring and data treatment of other emotion expression questions and under different experimental circumstances. Overall, we found applying loess smoothing and controlling for baseline facial emotion expression and facial plasticity are recommended methods of data treatment. When scoring facial emotion expression ability, maximum score is preferred. Finally, we discuss the scoring methods and data treatments in the larger context of emotion expression research.

Enhanced avatar design using cognitive map-based simulation

With the advent of the Internet era and the maturation of electronic commerce, strategic avatar design has become an important way of keeping up with market changes and customer tastes. In this study, we propose a new approach to an adaptive avatar design that uses cognitive map (CM) as a what-if simulation vehicle. The main virtue of the new design is its ability to change specific avatar design features with objective consideration of the subsequent effects upon other design features, thereby enhancing user satisfaction. Statistical analyses of focus group interview results with a group of experts majoring in avatars and CM showed that our proposed approach could be used to effectively analyze avatar design in an adaptive and practical manner when the market situation is changing.