Naturalistic Emotion Decoding From Facial Action Sets

An Artificial Intelligence Model for Sensing Affective Valence and Arousal from Facial Images

Artificial intelligence (AI) models can sense subjective affective states from facial images. Although recent psychological studies have indicated that dimensional affective states of valence and arousal are systematically associated with facial expressions, no AI models have been developed to estimate these affective states from facial images based on empirical data. We developed a recurrent neural network-based AI model to estimate subjective valence and arousal states from facial images. We trained our model using a database containing participant valence/arousal states and facial images. Leave-one-out cross-validation supported the validity of the model for predicting subjective valence and arousal states. We further validated the effectiveness of the model by analyzing a dataset containing participant valence/arousal ratings and facial videos. The model predicted second-by-second valence and arousal states, with prediction performance comparable to that of FaceReader, a commercial AI model that estimates dimensional affective states based on a different approach. We constructed a graphical user interface to show real-time affective valence and arousal states by analyzing facial video data. Our model is the first distributable AI model for sensing affective valence and arousal from facial images/videos to be developed based on an empirical database; we anticipate that it will have many practical uses, such as in mental health monitoring and marketing research.

Sensing emotional valence and arousal dynamics through automated facial action unit analysis

Information about the concordance between dynamic emotional experiences and objective signals is practically useful. Previous studies have shown that valence dynamics can be estimated by recording electrical activity from the muscles in the brows and cheeks. However, whether facial actions based on video data and analyzed without electrodes can be used for sensing emotion dynamics remains unknown. We investigated this issue by recording video of participants' faces and obtaining dynamic valence and arousal ratings while they observed emotional films. Action units (AUs) 04 (i.e., brow lowering) and 12 (i.e., lip-corner pulling), detected through an automated analysis of the video data, were negatively and positively correlated with dynamic ratings of subjective valence, respectively. Several other AUs were also correlated with dynamic valence or arousal ratings. Random forest regression modeling, interpreted using the SHapley Additive exPlanation tool, revealed non-linear associations between the AUs and dynamic ratings of valence or arousal. These results suggest that an automated analysis of facial expression video data can be used to estimate dynamic emotional states, which could be applied in various fields including mental health diagnosis, security monitoring, and education.

Automated facial expression analysis of participants self-criticising via the two-chair technique: exploring facial behavioral markers of self-criticism

Introduction

As self-rating scales are prone to many measurement distortions, there is a growing call for more objective measures based on physiological or behavioural indicators. Self-criticism is one of the major transdiagnostic factor of all mental disorders therefore it is important to be able to distinguish what are the characteristic facial features of self-criticizing. To the best of our knowledge, there has been no automated facial emotion expression analysis of participants self-criticising via the two-chair technique. The aim of this study was to detect which action units of facial expressions were significantly more often present in participants performing self-criticism using the two-chair technique. The broader goal was to contribute to the scientific knowledge on objective behavioural descriptions of self-criticism and to provide an additional diagnostic means to the existing self-rating scales by exploring facial behavioral markers of self-criticism.

Methods

The non-clinical sample consisted of 80 participants (20 men and 60 women) aged 19 years to 57 years (M = 23.86; SD = 5.98). In the analysis we used iMotions's Affectiva AFFDEX module (Version 8.1) to classify the participants' actions units from the self-criticising videos. For the statistical analysis we used a multilevel model to account for the repeated-measures design.

Results

Based on the significant results the self-critical facial expression may therefore comprise the following action units: Dimpler, Lip Press, Eye Closure, Jaw Drop, and Outer Brow Raise, which are related to contempt, fear, and embarrassment or shame; and Eye Closure and Eye Widen (in rapid sequence Blink), which are a sign that highly negative stimuli are being emotionally processed.

Discussion

The research study need to be further analysed using clinical samples to compare the results.

The Borderline Bias in Explicit Emotion Interpretation

Atypical emotion interpretation has been widely reported in individuals with borderline personality disorder (iBPD); however, empirical studies reported mixed results so far. We suggest that discrepancies in observations of emotion interpretation by iBPD can be explained by biases related to their fear of rejection and abandonment, i.e., the three moral emotions of anger, disgust, and contempt. In this study, we hypothesized that iBPD would show a higher tendency to correctly interpret these three displays of social rejection and attribute more negative valence. A total of 28 inpatient iBPDs and 28 healthy controls were asked to judge static and dynamic facial expressions in terms of emotions, valence, and self-reported arousal evoked by the observed faces. Our results partially confirmed our expectations. The iBPD correctly interpreted the three unambiguous moral emotions. Contempt, a complex emotion with a difficulty in recognizing facial expressions, was recognized better by iBPD than by healthy controls. All negative emotions were judged more negatively by iBPD than by controls, but no difference was observed in the neutral or positive emotion. Alexithymia and anxiety trait and state levels were controlled in all analyses.

Contextual effects on smile perception and recognition memory

Most past research has focused on the role played by social context information in emotion classification, such as whether a display is perceived as belonging to one emotion category or another. The current study aims to investigate whether the effect of context extends to the interpretation of emotion displays, i.e. smiles that could be judged either as posed or spontaneous readouts of underlying positive emotion. A between-subjects design (N = 93) was used to investigate the perception and recall of posed smiles, presented together with a happy or polite social context scenario. Results showed that smiles seen in a happy context were judged as more spontaneous than the same smiles presented in a polite context. Also, smiles were misremembered as having more of the physical attributes (i.e., Duchenne marker) associated with spontaneous enjoyment when they appeared in the happy than polite context condition. Together, these findings indicate that social context information is routinely encoded during emotion perception, thereby shaping the interpretation and recognition memory of facial expressions.

Semantics Based on the Physical Characteristics of Facial Expressions Used to Produce Japanese Vowels

Previous studies have reported that verbal sounds are associated-non-arbitrarily-with specific meanings (e.g., sound symbolism and onomatopoeia), including visual forms of information such as facial expressions; however, it remains unclear how mouth shapes used to utter each vowel create our semantic impressions. We asked 81 Japanese participants to evaluate mouth shapes associated with five Japanese vowels by using 10 five-item semantic differential scales. The results reveal that the physical characteristics of the facial expressions (mouth shapes) induced specific evaluations. For example, the mouth shape made to voice the vowel "a" was the one with the biggest, widest, and highest facial components compared to other mouth shapes, and people perceived words containing that vowel sound as bigger. The mouth shapes used to pronounce the vowel "i" were perceived as more likable than the other four vowels. These findings indicate that the mouth shapes producing vowels imply specific meanings. Our study provides clues about the meaning of verbal sounds and what the facial expressions in communication represent to the perceiver.

Dynamic Facial Expression of Emotion and Observer Inference

Research on facial emotion expression has mostly focused on emotion recognition, assuming that a small number of discrete emotions is elicited and expressed via prototypical facial muscle configurations as captured in still photographs. These are expected to be recognized by observers, presumably via template matching. In contrast, appraisal theories of emotion propose a more dynamic approach, suggesting that specific elements of facial expressions are directly produced by the result of certain appraisals and predicting the facial patterns to be expected for certain appraisal configurations. This approach has recently been extended to emotion perception, claiming that observers first infer individual appraisals and only then make categorical emotion judgments based on the estimated appraisal patterns, using inference rules. Here, we report two related studies to empirically investigate the facial action unit configurations that are used by actors to convey specific emotions in short affect bursts and to examine to what extent observers can infer a person's emotions from the predicted facial expression configurations. The results show that (1) professional actors use many of the predicted facial action unit patterns to enact systematically specified appraisal outcomes in a realistic scenario setting, and (2) naïve observers infer the respective emotions based on highly similar facial movement configurations with a degree of accuracy comparable to earlier research findings. Based on estimates of underlying appraisal criteria for the different emotions we conclude that the patterns of facial action units identified in this research correspond largely to prior predictions and encourage further research on appraisal-driven expression and inference.