Mask wearing affects emotion perception

How Brow Rotation Affects Emotional Expression Utilizing Artificial Intelligence

BACKGROUND

It is well known that brow position affects emotional expression. However, there is little literature on how and to what degree this change in emotional expression happens. Previous studies on this topic have utilized manual rating; this method of study remains small and labor intensive. Our objective is to correlate manual brow rotations with emotional outcomes using artificial intelligence to objectively determine how specific brow manipulations affected human expression.

METHODS

We included 53 brow-lift patients in this study. Pre-operative patients' brows were rotated to - 20, - 10, +10, and +20 degrees in respect to the central axis of their existing brow using PIXLR, a cloud-based set of image editing tools and utilities. These images were analyzed using FaceReader, a validated software package that uses computer vision technology for facial expression recognition. The primary facial emotion and intensity of facial action units (0 = no action unit detected to 4 = most intense action unit detected) generated by the software were recorded.

RESULTS

265 total images [5 images (pre-operative, - 20 degree brow rotation, - 10, +10, and +20) per patient] were analyzed using FaceReader. The primary emotion detected in the majority of images was neutral. The percentage of disgust in patients' expressions, as detected by FaceReader, increased with increased positive brow rotation (1.76% disgust detected at - 20 degrees, 2.09% at - 10 degrees, 2.65% at neutral, 2.61% at +10 degrees, and 2.95% at +20 degrees). In contrast, the percentage of sadness in patients' expressions decreased with increased positive brow rotation (29.92% sadness detected at - 20 degrees, 21.5% at - 10 degrees, 11.42% at neutral, 15.75% at +10 degrees, and 12.86% at +20 degrees). Our facial action unit analysis corresponded with primary emotion analysis. The intensity of the inner brow raiser decreased with increased positive brow rotation 8.54% at - 20 degrees, 4.21% at - 10 degrees, 1.48% at neutral, 0.84% at +10 degrees, and 0.76% at +20 degrees). The intensity of the outer brow raiser increased with increased positive brow rotation (0.97% at - 20 degrees, 0.45% at - 10 degrees, 1.12% at neutral, 5.45% at +10 degrees, and 11.19% at +20 degrees).

CONCLUSION

We demonstrated that increasing the degree of brow rotation correlated positively with the percentage of disgust and inversely with the percentage of sadness detected by FaceReader. This study demonstrated how different manipulated brow positions affected emotional outcomes using artificial intelligence. Physicians can use these findings to better understand how brow-lifts can affect the perceived emotion of their patients.

LEVEL OF EVIDENCE III

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

The effect of masks on the emotion perception of a facial crowd

The present study investigated the effect of facial masks on people's ability to perceive emotions in crowds. We presented faces with the bottom halves occluded by masks or full faces without occlusion. In two sequentially presented crowds, we varied the number of faces, emotional valence, and intensity of facial expressions, examining the impact of masks on the perception of crowd emotion. Participants reported which of the two crowds they would avoid based on the crowds' average emotions. The participants' ability to judge the average emotion of a crowd, especially a crowd expressing happiness, was impaired when the crowd wore masks. For faces covered by masks, crowd emotion judgments were more negatively biased than those without masks. However, participants could still distinguish the emotional intensities of a crowd wearing masks above chance. Additionally, participants responded more quickly to a crowd with more people without compromising accuracy, despite the perceptual challenges imposed by facial masks. Our results suggest that under ambiguous social situations in which individuals' emotions are partially hidden by masks, a large group may provide stronger social cues than a small group, thereby promoting communication and regulating social behaviors.

COVID-19 masks: A barrier to facial and vocal information

With the COVID-19 pandemic, we have become used to wearing masks and have experienced how masks seem to impair emotion and speech recognition. While several studies have focused on facial emotion recognition by adding images of masks on photographs of emotional faces, we have created a video database with actors really wearing masks to test its effect in more ecological conditions. After validating the emotions displayed by the actors, we found that surgical mask impaired happiness and sadness recognition but not neutrality. Moreover, for happiness, this effect was specific to the mask and not to covering the lower part of the face, possibly due to a cognitive bias associated with the surgical mask. We also created videos with speech and tested the effect of mask on emotion and speech recognition when displayed in auditory, visual, or audiovisual modalities. In visual and audiovisual modalities, mask impaired happiness and sadness but improved neutrality recognition. Mask impaired the recognition of bilabial syllables regardless of modality. In addition, it altered speech recognition only in the audiovisual modality for participants above 70 years old. Overall, COVID-19 masks mainly impair emotion recognition, except for older participants for whom it also impacts speech recognition, probably because they rely more on visual information to compensate age-related hearing loss.

How facial masks alter the interaction of gaze direction, head orientation, and emotion recognition

The COVID-19 pandemic has altered the way we interact with each other: mandatory mask-wearing obscures facial information that is crucial for emotion recognition. Whereas the influence of wearing a mask on emotion recognition has been repeatedly investigated, little is known about the impact on interaction effects among emotional signals and other social signals. Therefore, the current study sought to explore how gaze direction, head orientation, and emotional expression interact with respect to emotion perception, and how these interactions are altered by wearing a face mask. In two online experiments, we presented face stimuli from the Radboud Faces Database displaying different facial expressions (anger, fear, happiness, neutral, and sadness), gaze directions (−13°, 0°, and 13°), and head orientations (−45°, 0°, and 45°) – either without (Experiment 1) or with mask (Experiment 2). Participants categorized the displayed emotional expressions. Not surprisingly, masks impaired emotion recognition. Surprisingly, without the mask, emotion recognition was unaffected by averted head orientations and only slightly affected by gaze direction. The mask strongly interfered with this ability. The mask increased the influence of head orientation and gaze direction, in particular for the emotions that were poorly recognized with mask. The results suggest that in case of uncertainty due to ambiguity or absence of signals, we seem to unconsciously factor in extraneous information.