Measuring facial expression of emotion

Day-to-day dynamics of facial emotion expressions in posttraumatic stress disorder

Facial expressions are an essential component of emotions that may reveal mechanisms maintaining posttraumatic stress disorder (PTSD). However, most research on emotions in PTSD has relied on self-reports, which only capture subjective affect. The few studies on outward emotion expressions have been hampered by methodological limitations, including low ecological validity and failure to capture the dynamic nature of emotions and symptoms. Our study addresses these limitations with an approach that has not been applied to psychopathology: person-specific models of day-to-day facial emotion expression and PTSD symptom dynamics. We studied a sample of World Trade Center responders (N = 112) with elevated PTSD pathology who recorded a daily video diary and self-reported symptoms for 90 days (8953 videos altogether). Facial expressions were detected from video recordings with a facial emotion recognition model. In data-driven, idiographic network models, most participants (80 %) had at least one, reliable expression-symptom link. Six expression-symptom dynamics were significant for >10 % of the sample. Each of these dynamics had statistically meaningful heterogeneity, with some people's symptoms related to over-expressivity and others to under-expressivity. Our results provide the foundation for a more complete understanding of emotions in PTSD that not only includes subjective feelings but also outward emotion expressions.

Facial expression recognition through muscle synergies and estimation of facial keypoint displacements through a skin-musculoskeletal model using facial sEMG signals

The development of facial expression recognition (FER) and facial expression generation (FEG) systems is essential to enhance human-robot interactions (HRI). The facial action coding system is widely used in FER and FEG tasks, as it offers a framework to relate the action of facial muscles and the resulting facial motions to the execution of facial expressions. However, most FER and FEG studies are based on measuring and analyzing facial motions, leaving the facial muscle component relatively unexplored. This study introduces a novel framework using surface electromyography (sEMG) signals from facial muscles to recognize facial expressions and estimate the displacement of facial keypoints during the execution of the expressions. For the facial expression recognition task, we studied the coordination patterns of seven muscles, expressed as three muscle synergies extracted through non-negative matrix factorization, during the execution of six basic facial expressions. Muscle synergies are groups of muscles that show coordinated patterns of activity, as measured by their sEMG signals, and are hypothesized to form the building blocks of human motor control. We then trained two classifiers for the facial expressions based on extracted features from the sEMG signals and the synergy activation coefficients of the extracted muscle synergies, respectively. The accuracy of both classifiers outperformed other systems that use sEMG to classify facial expressions, although the synergy-based classifier performed marginally worse than the sEMG-based one (classification accuracy: synergy-based 97.4%, sEMG-based 99.2%). However, the extracted muscle synergies revealed common coordination patterns between different facial expressions, allowing a low-dimensional quantitative visualization of the muscle control strategies involved in human facial expression generation. We also developed a skin-musculoskeletal model enhanced by linear regression (SMSM-LRM) to estimate the displacement of facial keypoints during the execution of a facial expression based on sEMG signals. Our proposed approach achieved a relatively high fidelity in estimating these displacements (NRMSE 0.067). We propose that the identified muscle synergies could be used in combination with the SMSM-LRM model to generate motor commands and trajectories for desired facial displacements, potentially enabling the generation of more natural facial expressions in social robotics and virtual reality.

"Sentio ergo est": Unmasking the psychological realities of emotional misperception

Perception is an important aspect of our personal lives, interpersonal interactions and professional activities and performance. A large body of psychological research has been dedicated to exploring how perception happens, whether and when it involves conscious awareness and what are the physiological correlates, such as skin-conductance and heart-rate responses, that occur when we perceive particularly emotional elicitors. A more recent and less explored question in psychological science is how and when misperception happens, and what are the physiological characteristics of the misperception of emotion. Therefore, in the current study, for the first time in relevant research, we recruited participants using trial-contour power calculations for false-positive responses, such as incorrectly reporting that a brief backward masked face was presented and thoroughly explored these responses. We reported that false-positive responses for backward masked emotional faces were characterised by pre-trial arousal, and post-trial arousal increases, high confidence ratings, and corresponding to stimulus-type misperception valence and arousal participant ratings. These outcomes were most pronounced for false-positive responses for fearful faces. Based on these findings, we discussed the possibility of a mechanism for partial self-encapsulated emotional-experiential apperception and the possibility of a fear primacy socio-emotional response module during combined visual ambiguity and high psychophysiological arousal.

Linking Affect Dynamics and Well-Being: A Novel Methodological Approach for Mental Health

Emotions are dynamic processes; their variability relates to psychological well-being and psychopathology. Affective alterations have been linked to mental diseases like depression, although little is known about how similar patterns occur in healthy individuals. This study investigates the psychophysiological correlations of emotional processing in healthy subjects, specifically exploring the relationship between depressive traits, cognitive distortions, and facial electromyographic (f-EMG) responses during affective transitions. A cohort of 44 healthy participants underwent f-EMG recording while viewing emotional images from the International Affective Picture System (IAPS). Self-report measures included the Beck Depression Inventory (BDI) and the Cognitive Distortion Scale (CDS). Higher BDI scores were associated with increased EMG activity in the corrugator muscle during transitions between positive and negative emotional states. Cognitive distortions such as Catastrophizing, All-or-Nothing Thinking, and Minimization showed significant positive correlations with EMG activity, indicating that individuals with higher levels of these distortions experienced greater facial muscle activation during emotional transitions. This study's results indicate that there is a bidirectional correlation between depressed features and cognitive distortions and alterations in facial emotional processing, even in healthy subjects. Facial EMG in the context of dynamic affective transitions has the potential to be used as a non-invasive method for detecting abnormal emotional reactions at an early stage. This might help in identifying individuals who are at risk of developing depression and guide therapies to prevent its advancement.

Autism spectrum disorder detection using facial images: A performance comparison of pretrained convolutional neural networks

Autism spectrum disorder (ASD) is a complex psychological syndrome characterized by persistent difficulties in social interaction, restricted behaviours, speech, and nonverbal communication. The impacts of this disorder and the severity of symptoms vary from person to person. In most cases, symptoms of ASD appear at the age of 2 to 5 and continue throughout adolescence and into adulthood. While this disorder cannot be cured completely, studies have shown that early detection of this syndrome can assist in maintaining the behavioural and psychological development of children. Experts are currently studying various machine learning methods, particularly convolutional neural networks, to expedite the screening process. Convolutional neural networks are considered promising frameworks for the diagnosis of ASD. This study employs different pre-trained convolutional neural networks such as ResNet34, ResNet50, AlexNet, MobileNetV2, VGG16, and VGG19 to diagnose ASD and compared their performance. Transfer learning was applied to every model included in the study to achieve higher results than the initial models. The proposed ResNet50 model achieved the highest accuracy, 92%, compared to other transfer learning models. The proposed method also outperformed the state-of-the-art models in terms of accuracy and computational cost.

The Feel of Speech: Multisystem and Polymodal Somatosensation in Speech Production

PURPOSE

The oral structures such as the tongue and lips have remarkable somatosensory capacities, but understanding the roles of somatosensation in speech production requires a more comprehensive knowledge of somatosensation in the speech production system in its entirety, including the respiratory, laryngeal, and supralaryngeal subsystems. This review was conducted to summarize the system-wide somatosensory information available for speech production.

METHOD

The search was conducted with PubMed/Medline and Google Scholar for articles published until November 2023. Numerous search terms were used in conducting the review, which covered the topics of psychophysics, basic and clinical behavioral research, neuroanatomy, and neuroscience.

RESULTS AND CONCLUSIONS

The current understanding of speech somatosensation rests primarily on the two pillars of psychophysics and neuroscience. The confluence of polymodal afferent streams supports the development, maintenance, and refinement of speech production. Receptors are both canonical and noncanonical, with the latter occurring especially in the muscles innervated by the facial nerve. Somatosensory representation in the cortex is disproportionately large and provides for sensory interactions. Speech somatosensory function is robust over the lifespan, with possible declines in advanced aging. The understanding of somatosensation in speech disorders is largely disconnected from research and theory on speech production. A speech somatoscape is proposed as the generalized, system-wide sensation of speech production, with implications for speech development, speech motor control, and speech disorders.

Automating the analysis of facial emotion expression dynamics: A computational framework and application in psychotic disorders

Facial emotion expressions play a central role in interpersonal interactions; these displays are used to predict and influence the behavior of others. Despite their importance, quantifying and analyzing the dynamics of brief facial emotion expressions remains an understudied methodological challenge. Here, we present a method that leverages machine learning and network modeling to assess the dynamics of facial expressions. Using video recordings of clinical interviews, we demonstrate the utility of this approach in a sample of 96 people diagnosed with psychotic disorders and 116 never-psychotic adults. Participants diagnosed with schizophrenia tended to move from neutral expressions to uncommon expressions (e.g., fear, surprise), whereas participants diagnosed with other psychoses (e.g., mood disorders with psychosis) moved toward expressions of sadness. This method has broad applications to the study of normal and altered expressions of emotion and can be integrated with telemedicine to improve psychiatric assessment and treatment.

Optomyography-based sensing of facial expression derived arousal and valence in adults with depression

Background

Continuous assessment of affective behaviors could improve the diagnosis, assessment and monitoring of chronic mental health and neurological conditions such as depression. However, there are no technologies well suited to this, limiting potential clinical applications.

Aim

To test if we could replicate previous evidence of hypo reactivity to emotional salient material using an entirely new sensing technique called optomyography which is well suited to remote monitoring.

Methods

Thirty-eight depressed and 37 controls (≥18, ≤40 years) who met a research diagnosis of depression and an age-matched non-depressed control group. Changes in facial muscle activity over the brow (corrugator supercilli) and cheek (zygomaticus major) were measured whilst volunteers watched videos varying in emotional salience.

Results

Across all participants, videos rated as subjectively positive were associated with activation of muscles in the cheek relative to videos rated as neutral or negative. Videos rated as subjectively negative were associated with brow activation relative to videos judged as neutral or positive. Self-reported arousal was associated with a step increase in facial muscle activation across the brow and cheek. Group differences were significantly reduced activation in facial muscles during videos considered subjectively negative or rated as high arousal in depressed volunteers compared with controls.

Conclusion

We demonstrate for the first time that it is possible to detect facial expression hypo-reactivity in adults with depression in response to emotional content using glasses-based optomyography sensing. It is hoped these results may encourage the use of optomyography-based sensing to track facial expressions in the real-world, outside of a specialized testing environment.

An Analysis of Body Language of Patients Using Artificial Intelligence

In recent decades, epidemic and pandemic illnesses have grown prevalent and are a regular source of concern throughout the world. The extent to which the globe has been affected by the COVID-19 epidemic is well documented. Smart technology is now widely used in medical applications, with the automated detection of status and feelings becoming a significant study area. As a result, a variety of studies have begun to focus on the automated detection of symptoms in individuals infected with a pandemic or epidemic disease by studying their body language. The recognition and interpretation of arm and leg motions, facial recognition, and body postures is still a developing field, and there is a dearth of comprehensive studies that might aid in illness diagnosis utilizing artificial intelligence techniques and technologies. This literature review is a meta review of past papers that utilized AI for body language classification through full-body tracking or facial expressions detection for various tasks such as fall detection and COVID-19 detection, it looks at different methods proposed by each paper, their significance and their results.

Effects of diagnostic regions on facial emotion recognition: The moving window technique

With regard to facial emotion recognition, previous studies found that specific facial regions were attended more in order to identify certain emotions. We investigated whether a preferential search for emotion-specific diagnostic regions could contribute toward the accurate recognition of facial emotions. Twenty-three neurotypical adults performed an emotion recognition task using six basic emotions: anger, disgust, fear, happiness, sadness, and surprise. The participants’ exploration patterns for the faces were measured using the Moving Window Technique (MWT). This technique presented a small window on a blurred face, and the participants explored the face stimuli through a mouse-controlled window in order to recognize the emotions on the face. Our results revealed that when the participants explored the diagnostic regions for each emotion more frequently, the correct recognition of the emotions occurred at a faster rate. To the best of our knowledge, this current study is the first to present evidence that an exploration of emotion-specific diagnostic regions can predict the reaction time of accurate emotion recognition among neurotypical adults. Such findings can be further applied in the evaluation and/or training (regarding emotion recognition functions) of both typically and atypically developing children with emotion recognition difficulties.

Body Language Analysis in Healthcare: An Overview

Given the current COVID-19 pandemic, medical research today focuses on epidemic diseases. Innovative technology is incorporated in most medical applications, emphasizing the automatic recognition of physical and emotional states. Most research is concerned with the automatic identification of symptoms displayed by patients through analyzing their body language. The development of technologies for recognizing and interpreting arm and leg gestures, facial features, and body postures is still in its early stage. More extensive research is needed using artificial intelligence (AI) techniques in disease detection. This paper presents a comprehensive survey of the research performed on body language processing. Upon defining and explaining the different types of body language, we justify the use of automatic recognition and its application in healthcare. We briefly describe the automatic recognition framework using AI to recognize various body language elements and discuss automatic gesture recognition approaches that help better identify the external symptoms of epidemic and pandemic diseases. From this study, we found that since there are studies that have proven that the body has a language called body language, it has proven that language can be analyzed and understood by machine learning (ML). Since diseases also show clear and different symptoms in the body, the body language here will be affected and have special features related to a particular disease. From this examination, we discovered that it is possible to specialize the features and language changes of each disease in the body. Hence, ML can understand and detect diseases such as pandemic and epidemic diseases and others.

Measuring the Candidates' Emotions in Political Debates Based on Facial Expression Recognition Techniques

This article presents the analysis of the main Spanish political candidates for the elections to be held on April 2019. The analysis focuses on the Facial Expression Analysis (FEA), a technique widely used in neuromarketing research. It allows to identify the micro-expressions that are very brief, involuntary. They are signals of hidden emotions that cannot be controlled voluntarily. The video with the final interventions of every candidate has been post-processed using the classification algorithms given by the iMotions's AFFDEX platform. We have then analyzed these data. Firstly, we have identified and compare the basic emotions showed by each politician. Second, we have associated the basic emotions with specific moments of the candidate's speech, identifying the topics they address and relating them directly to the expressed emotion. Third, we have analyzed whether the differences shown by each candidate in every emotion are statistically significant. In this sense, we have applied the non-parametric chi-squared goodness-of-fit test. We have also considered the ANOVA analysis in order to test whether, on average, there are differences between the candidates. Finally, we have checked if there is consistency between the results provided by different surveys from the main media in Spain regarding the evaluation of the debate and those obtained in our empirical analysis. A predominance of negative emotions has been observed. Some inconsistencies were found between the emotion expressed in the facial expression and the verbal content of the message. Also, evidences got from statistical analysis confirm that the differences observed between the various candidates with respect to the basic emotions, on average, are statistically significant. In this sense, this article provides a methodological contribution to the analysis of the public figures' communication, which could help politicians to improve the effectiveness of their messages identifying and evaluating the intensity of the expressed emotions.

Exploring the Emotional Experience During Instant Messaging Among Young Adults: An Experimental Study Incorporating Physiological Correlates of Arousal

Instant messaging (IM) is a highly diffused form of communication among younger populations, yet little is known about the emotional experience during IM. The present study aimed to investigate the emotional experience during IM by drawing on the Circumplex Model of Affect and measuring heart rate and electrodermal activity as indicators of arousal in addition to self-reported perceived emotional valence. Using an experimental design, we manipulated message latency (i.e., response after 1 min versus 7 min) and message valence (positive versus negative response). Based on data collected from 65 young adults (50% male; M_age = 23.28, SD = 3.75), we observed arousal as participants’ electrodermal activity levels increased from the time a fictitious peer started typing a response to the receipt of that response, especially in the delayed condition. Electrodermal activity levels also increased in both the positive and the negative message conditions. No changes were observed for heart rate. Participants’ self-report perceived emotional valence revealed that positive messages were evaluated as more pleasant and the peer as more available, while no difference in the self-report was found for message latency. These findings shed light on the emotional experience during IM by adding valuable insights on the physiological processes underlying the anticipation of social reward, but only during delayed IM exchange that can be observed in Human–Computer-Interaction.

How and Why Could Smiling Influence Physical Health? A Conceptual Review

Smiling has been a topic of interest to psychologists for decades, with a myriad of studies tying this behavior to well-being. Despite this, we know surprisingly little about the nature of the connections between smiling and physical health. We review the literature connecting both naturally occurring smiles and experimentally manipulated smiles to physical health and health-relevant outcomes. This work is discussed in the context of existing affect and health-relevant theoretical models that help explain the connection between smiling and physical health including the facial feedback hypothesis, the undoing hypothesis, the generalized unsafety theory of stress, and polyvagal theory. We also describe a number of plausible pathways, some new and relatively untested, through which smiling may influence physical health such as trait or state positive affect, social relationships, stress buffering, and the oculocardiac reflex. Finally, we provide a discussion of possible future directions, including the importance of cultural variation and replication. Although this field is still in its infancy, the findings from both naturally occurring smile studies and experimentally manipulated smile studies consistently suggest that smiling may have a number of health-relevant benefits including beneficially impacting our physiology during acute stress, improved stress recovery, and reduced illness over time.

What's in a face: Automatic facial coding of untrained study participants compared to standardized inventories

Automatic facial coding (AFC) is a novel research tool to automatically analyze emotional facial expressions. AFC can classify emotional expressions with high accuracy in standardized picture inventories of intensively posed and prototypical expressions. However, classification of facial expressions of untrained study participants is more error prone. This discrepancy requires a direct comparison between these two sources of facial expressions. To this end, 70 untrained participants were asked to express joy, anger, surprise, sadness, disgust, and fear in a typical laboratory setting. Recorded videos were scored with a well-established AFC software (FaceReader, Noldus Information Technology). These were compared with AFC measures of standardized pictures from 70 trained actors (i.e., standardized inventories). We report the probability estimates of specific emotion categories and, in addition, Action Unit (AU) profiles for each emotion. Based on this, we used a novel machine learning approach to determine the relevant AUs for each emotion, separately for both datasets. First, misclassification was more frequent for some emotions of untrained participants. Second, AU intensities were generally lower in pictures of untrained participants compared to standardized pictures for all emotions. Third, although profiles of relevant AU overlapped substantially across the two data sets, there were also substantial differences in their AU profiles. This research provides evidence that the application of AFC is not limited to standardized facial expression inventories but can also be used to code facial expressions of untrained participants in a typical laboratory setting.

Validation of the P1vital® Faces Set for Use as Stimuli in Tests of Facial Emotion Recognition

BACKGROUND

Negative bias in facial emotion recognition is a well-established concept in mental disorders such as depression. However, existing face sets of emotion recognition tests may be of limited use in international research, which could benefit from more contemporary and diverse alternatives. Here, we developed and provide initial validation for the P1vital® Affective Faces set (PAFs) as a contemporary alternative to the widely-used Pictures of Facial Affect (PoFA).

METHODS

The PAFs was constructed of 133 color photographs of facial expressions of ethnically-diverse trained actors and compared with the PoFA, comprised of 110 black and white photographs of facial expressions of generally Caucasian actors. Sixty-one recruits were asked to classify faces from both sets over six emotions (happy, sad, fear, anger, disgust, surprise) varying in intensity in 10% increments from 0 to 100%.

RESULTS

Participants were significantly more accurate in identifying correct emotions viewing faces from the PAFs. In both sets, participants identified happy faces more accurately than fearful faces, were least likely to misclassify facial expressions as happy and most likely to misclassify all emotions at low intensity as neutral. Accuracy in identifying facial expressions improved with increasing emotion intensity for both sets, reaching peaks at 60 and 80% intensity for the PAFs and PoFA, respectively. The study was limited by small sizes and age-range of participants and ethnic diversity of actors.

CONCLUSIONS

The PAFs successfully depicted a range of emotional expressions with improved performance over the PoFA and may be used as a contemporary set in facial expression recognition tests.

COVID-19 and psychiatric disorders: The impact of face masks in emotion recognition face masks and emotion recognition in psychiatry

Since the outbreak of the COVID-19 pandemic, reading facial expressions has become more complex due to face masks covering the lower part of people's faces. A history of psychiatric illness has been associated with higher rates of complications, hospitalization, and mortality due to COVID-19. Psychiatric patients have well-documented difficulties reading emotions from facial expressions; accordingly, this study assesses how using face masks, such as those worn for preventing COVID-19 transmission, impacts the emotion recognition skills of patients with psychiatric disorders. To this end, the current study asked patients with bipolar disorder, major depressive disorder, schizophrenia, and healthy individuals to identify facial emotions on face images with and without facial masks. Results demonstrate that the emotion recognition skills of all participants were negatively influenced by face masks. Moreover, the main insight of the study is that the impairment is crucially significant when patients with major depressive disorder and schizophrenia had to identify happiness at a low-intensity level. These findings have important implications for satisfactory social relationships and well-being. If emotions with positive valence are hardly understood by specific psychiatric patients, there is an even greater requirement for doctor-patient interactions in public primary care.

Lie to my face: An electromyography approach to the study of deceptive behavior

BACKGROUND

Deception is present in all walks of life, from social interactions to matters of homeland security. Nevertheless, reliable indicators of deceptive behavior in real-life scenarios remain elusive.

METHODS

By integrating electrophysiological and communicative approaches, we demonstrate a new and objective detection approach to identify participant-specific indicators of deceptive behavior in an interactive scenario of a two-person deception task. We recorded participants' facial muscle activity using novel dry screen-printed electrode arrays and applied machine-learning algorithms to identify lies based on brief facial responses.

RESULTS

With an average accuracy of 73%, we identified two groups of participants: Those who revealed their lies by activating their cheek muscles and those who activated their eyebrows. We found that the participants lied more often with time, with some switching their telltale muscle groups. Moreover, while the automated classifier, reported here, outperformed untrained human detectors, their performance was correlated, suggesting reliance on shared features.

CONCLUSIONS

Our findings demonstrate the feasibility of using wearable electrode arrays in detecting human lies in a social setting and set the stage for future research on individual differences in deception expression.

Emotional Well-Being and Traditional Cypriot Easter Games: A Qualitative Analysis

The aim of the current study is to examine the effect of the Traditional Easter Games of Cyprus on the emotional well-being of the participants. Data were collected using a qualitative analysis. It consisted of interviews from 51 participants aged 32-93 years old, and observations were made from audiovisual material of the Traditional Cypriot Easter Games being played by a sample of 20 children aged 6-14 years old and 43 adults aged 18-65 years old. Demographic data were collected by using interviews and analyzed using IBM SPSS program. The observations of the audiovisual material focused on the emotions of the participants and were grouped into prevailing and secondary emotions according to frequency and duration. The results indicate that games produce emotions such as joy, excitement, and euphoria. Emotions such as embarrassment, frustration, and anger were also observed occasionally, specifically in situations of competitiveness and defeat. In addition, the differences and similarities between adults and children were recorded. The findings of the present study extend previous work by demonstrating the positive impact of the traditional games on children's and adult's emotional well-being.

Positive and Negative Emotion Classification Based on Multi-channel

The EEG features of different emotions were extracted based on multi-channel and forehead channels in this study. The EEG signals of 26 subjects were collected by the emotional video evoked method. The results show that the energy ratio and differential entropy of the frequency band can be used to classify positive and negative emotions effectively, and the best effect can be achieved by using an SVM classifier. When only the forehead and forehead signals are used, the highest classification accuracy can reach 66%. When the data of all channels are used, the highest accuracy of the model can reach 82%. After channel selection, the best model of this study can be obtained. The accuracy is more than 86%.

Atlas of voluntary facial muscle activation: Visualization of surface electromyographic activities of facial muscles during mimic exercises

Complex facial muscle movements are essential for many motoric and emotional functions. Facial muscles are unique in the musculoskeletal system as they are interwoven, so that the contraction of one muscle influences the contractility characteristic of other mimic muscles. The facial muscles act more as a whole than as single facial muscle movements. The standard for clinical and psychosocial experiments to detect these complex interactions is surface electromyography (sEMG). What is missing, is an atlas showing which facial muscles are activated during specific tasks. Based on high-resolution sEMG data of 10 facial muscles of both sides of the face simultaneously recorded during 29 different facial muscle tasks, an atlas visualizing voluntary facial muscle activation was developed. For each task, the mean normalized EMG amplitudes of the examined facial muscles were visualized by colors. The colors were spread between the lowest and highest EMG activity. Gray shades represent no to very low EMG activities, light and dark brown shades represent low to medium EMG activities and red shades represent high to very high EMG activities relatively with respect to each task. The present atlas should become a helpful tool to design sEMG experiments not only for clinical trials and psychological experiments, but also for speech therapy and orofacial rehabilitation studies.

Anxiety, not regulation tendency, predicts how individuals regulate in the laboratory: An exploratory comparison of self-report and psychophysiology

Anxiety influences how individuals experience and regulate emotions in a variety of ways. For example, individuals with lower anxiety tend to cognitively reframe (reappraise) negative emotion and those with higher anxiety tend to suppress negative emotion. Research has also investigated these individual differences with psychophysiology. These lines of research assume coherence between how individuals regulate outside the laboratory, typically measured with self-report, and how they regulate during an experiment. Indeed, performance during experiments is interpreted as an indication of future behavior outside the laboratory, yet this relationship is seldom directly explored. To address this gap, we computed psychophysiological profiles of uninstructed (natural) regulation in the laboratory and explored the coherence between these profiles and a) self-reported anxiety and b) self-reported regulation tendency. Participants viewed negative images and were instructed to reappraise, suppress or naturally engage. Electrodermal and facial electromyography signals were recorded to compute a multivariate psychophysiological profile of regulation. Participants with lower anxiety exhibited similar profiles when naturally regulating and following instructions to reappraise, suggesting they naturally reappraised more. Participants with higher anxiety exhibited similar profiles when naturally regulating and following instructions to suppress, suggesting they naturally suppressed more. However, there was no association between self-reported reappraisal or suppression tendency and psychophysiology. These exploratory results indicate that anxiety, but not regulation tendency, predicts how individuals regulate emotion in the laboratory. These findings suggest that how individuals report regulating in the real world does not map on to how they regulate in the laboratory. Taken together, this underscores the importance of developing emotion-regulation interventions and paradigms that more closely align to and predict real-world outcomes.

Facial features and head movements obtained with a webcam correlate with performance deterioration during prolonged wakefulness

We have performed a direct comparison between facial features obtained from a webcam and vigilance-task performance during prolonged wakefulness. Prolonged wakefulness deteriorates working performance due to changes in cognition, emotion, and by delayed response. Facial features can be potentially collected everywhere using webcams located in the workplace. If this type of device can obtain relevant information to predict performance deterioration, this technology can potentially reduce serious accidents and fatality. We extracted 34 facial indices, including head movements, facial expressions, and perceived facial emotions from 20 participants undergoing the psychomotor vigilance task (PVT) over 25 hours. We studied the correlation between facial indices and the performance indices derived from PVT, and evaluated the feasibility of facial indices as detectors of diminished reaction time during the PVT. Furthermore, we tested the feasibility of classifying performance as normal or impaired using several machine learning algorithms with correlated facial indices. Twenty-one indices were found significantly correlated with PVT indices. Pitch, from the head movement indices, and four perceived facial emotions-anger, surprise, sadness, and disgust-exhibited significant correlations with indices of performance. The eye-related facial expression indices showed especially strong correlation and higher feasibility of facial indices as classifiers. Significantly correlated indices were shown to explain more variance than the other indices for most of the classifiers. The facial indices obtained from a webcam strongly correlate with working performance during 25 hours of prolonged wakefulness.

The Study of Facial Muscle Movements for Non-Invasive Thermal Discomfort Detection via Bio-Sensing Technology. Part I: Development of the Experimental Design and Description of the Collected Data

In the time of climate change, as heat waves become a more regular occurrence, indoor thermal comfort is an important factor in day to day life. Due to such circumstances, many researchers have focused their studies on finding an effective solution that will not only enable thermal comfort, but also increase satisfaction within the indoor environment and, as a result, productivity. The fast development of the biometrical field encouraged the study focused on the investigation of how bio-markers, in combination with artificial intelligence algorithms, can be collected within an experimental setting to create a new approach for non-invasive thermal discomfort detection. The developed experimental design provides synergy between automatic facial coding, pulse, and galvanic skin response measurements via iMotions software in a controlled environment. The iMotions software has built-in machine vision algorithms, and with Shimmer sensors and a post-processing tool through Affectiva AFFDEX, is able to collect facial action data through detection of the facial muscle movements and various bio-markers. The Zero Emission Building (ZEB) Test Cell laboratory was used as the control environment and transformed to imitate an office space for the data collection campaign at NTNU in Trondheim. The given experimental design provides an opportunity to create an immense database with bio-markers that are linked to the subcortical level of the brain, indoor parameters, and direct feedback on the comfort level of occupants within an office-like environment. In total, 111 data collection sessions were registered with iMotions. The discomfort button was pressed 240 times and 1080 planned indoor comfort evaluation surveys were held during experiment. The discomfort button was pressed 49 times to indicate that participant felt discomfort due to low temperature and 52 due to high temperature. Collected data revealed a big deviation in the discomfort temperature values for experiment participants with respect to performed temperature ramps. While it is common to use the same predefined temperature range for facility management, it became clear that the complexity of the task is greater and should not be approached on a human computational level. Implementation of AI can potentially provide higher value accuracy within thermal discomfort detection and enable unique personal user experience at the workplace.

Developing a pain intensity prediction model using facial expression: A feasibility study with electromyography

The automatic detection of facial expressions of pain is needed to ensure accurate pain assessment of patients who are unable to self-report pain. To overcome the challenges of automatic systems for determining pain levels based on facial expressions in clinical patient monitoring, a surface electromyography method was tested for feasibility in healthy volunteers. In the current study, two types of experimental gradually increasing pain stimuli were induced in thirty-one healthy volunteers who attended the study. We used a surface electromyography method to measure the activity of five facial muscles to detect facial expressions during pain induction. Statistical tests were used to analyze the continuous electromyography data, and a supervised machine learning was applied for pain intensity prediction model. Muscle activation of corrugator supercilii was most strongly associated with self-reported pain, and the levator labii superioris and orbicularis oculi showed a statistically significant increase in muscle activation when the pain stimulus reached subjects' self -reported pain thresholds. The two strongest features associated with pain, the waveform length of the corrugator supercilii and levator labii superioris, were selected for a prediction model. The performance of the pain prediction model resulted in a c-index of 0.64. In the study results, the most detectable difference in muscle activity during the pain experience was connected to eyebrow lowering, nose wrinkling and upper lip raising. As the performance of the prediction model remains modest, yet with a statistically significant ordinal classification, we suggest testing with a larger sample size to further explore the variables that affect variation in expressiveness and subjective pain experience.

Read My Face: Automatic Facial Coding Versus Psychophysiological Indicators of Emotional Valence and Arousal

Facial expressions provide insight into a person's emotional experience. To automatically decode these expressions has been made possible by tremendous progress in the field of computer vision. Researchers are now able to decode emotional facial expressions with impressive accuracy in standardized images of prototypical basic emotions. We tested the sensitivity of a well-established automatic facial coding software program to detect spontaneous emotional reactions in individuals responding to emotional pictures. We compared automatically generated scores for valence and arousal of the Facereader (FR; Noldus Information Technology) with the current psychophysiological gold standard of measuring emotional valence (Facial Electromyography, EMG) and arousal (Skin Conductance, SC). We recorded physiological and behavioral measurements of 43 healthy participants while they looked at pleasant, unpleasant, or neutral scenes. When viewing pleasant pictures, FR Valence and EMG were both comparably sensitive. However, for unpleasant pictures, FR Valence showed an expected negative shift, but the signal differentiated not well between responses to neutral and unpleasant stimuli, that were distinguishable with EMG. Furthermore, FR Arousal values had a stronger correlation with self-reported valence than with arousal while SC was sensitive and specifically associated with self-reported arousal. This is the first study to systematically compare FR measurement of spontaneous emotional reactions to standardized emotional images with established psychophysiological measurement tools. This novel technology has yet to make strides to surpass the sensitivity of established psychophysiological measures. However, it provides a promising new measurement technique for non-contact assessment of emotional responses.

Assessing the Effectiveness of Automated Emotion Recognition in Adults and Children for Clinical Investigation

Recent success stories in automated object or face recognition, partly fuelled by deep learning artificial neural network (ANN) architectures, have led to the advancement of biometric research platforms and, to some extent, the resurrection of Artificial Intelligence (AI). In line with this general trend, inter-disciplinary approaches have been taken to automate the recognition of emotions in adults or children for the benefit of various applications, such as identification of children's emotions prior to a clinical investigation. Within this context, it turns out that automating emotion recognition is far from being straightforward, with several challenges arising for both science (e.g., methodology underpinned by psychology) and technology (e.g., the iMotions biometric research platform). In this paper, we present a methodology and experiment and some interesting findings, which raise the following research questions for the recognition of emotions and attention in humans: (a) the adequacy of well-established techniques such as the International Affective Picture System (IAPS), (b) the adequacy of state-of-the-art biometric research platforms, (c) the extent to which emotional responses may be different in children and adults. Our findings and first attempts to answer some of these research questions are based on a mixed sample of adults and children who took part in the experiment, resulting in a statistical analysis of numerous variables. These are related to both automatically and interactively captured responses of participants to a sample of IAPS pictures.

FaceSync: Open source framework for recording facial expressions with head-mounted cameras

Advances in computer vision and machine learning algorithms have enabled researchers to extract facial expression data from face video recordings with greater ease and speed than standard manual coding methods, which has led to a dramatic increase in the pace of facial expression research. However, there are many limitations in recording facial expressions in laboratory settings.  Conventional video recording setups using webcams, tripod-mounted cameras, or pan-tilt-zoom cameras require making compromises between cost, reliability, and flexibility. As an alternative, we propose the use of a mobile head-mounted camera that can be easily constructed from our open-source instructions and blueprints at a fraction of the cost of conventional setups. The head-mounted camera framework is supported by the open source Python toolbox FaceSync, which provides an automated method for synchronizing videos. We provide four proof-of-concept studies demonstrating the benefits of this recording system in reliably measuring and analyzing facial expressions in diverse experimental setups, including group interaction experiments.

Automatic Recognition of Posed Facial Expression of Emotion in Individuals with Autism Spectrum Disorder

Facial expression is impaired in autism spectrum disorder (ASD), but rarely systematically studied. We focus on the ability of individuals with ASD to produce facial expressions of emotions in response to a verbal prompt. We used the Janssen Autism Knowledge Engine (JAKE^®), including automated facial expression analysis software (FACET) to measure facial expressions in individuals with ASD (n = 144) and a typically developing (TD) comparison group (n = 41). Differences in ability to produce facial expressions were observed between ASD and TD groups, demonstrated by activation of facial action units (happy, scared, surprised, disgusted, but not angry or sad). Activation of facial action units correlated with parent-reported social communication skills. This approach has potential for diagnostic and response to intervention measures.Trial Registration NCT02299700.

Facial expression analysis with AFFDEX and FACET: A validation study

The goal of this study was to validate AFFDEX and FACET, two algorithms classifying emotions from facial expressions, in iMotions's software suite. In Study 1, pictures of standardized emotional facial expressions from three databases, the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP), the Amsterdam Dynamic Facial Expression Set (ADFES), and the Radboud Faces Database (RaFD), were classified with both modules. Accuracy (Matching Scores) was computed to assess and compare the classification quality. Results show a large variance in accuracy across emotions and databases, with a performance advantage for FACET over AFFDEX. In Study 2, 110 participants' facial expressions were measured while being exposed to emotionally evocative pictures from the International Affective Picture System (IAPS), the Geneva Affective Picture Database (GAPED) and the Radboud Faces Database (RaFD). Accuracy again differed for distinct emotions, and FACET performed better. Overall, iMotions can achieve acceptable accuracy for standardized pictures of prototypical (vs. natural) facial expressions, but performs worse for more natural facial expressions. We discuss potential sources for limited validity and suggest research directions in the broader context of emotion research.

Development of a methodology to combine fMRI and EMG to measure emotional responses in patients with anorexia nervosa

OBJECTIVE

Individuals with eating disorders are theorized to have basic impairments in affective appraisal and social-emotional processing that contribute to pathogenesis of the disease. We aimed to determine if facial electromyography could be used to discriminate between happy and disgust emotions during simultaneous acquisition of an fMRI BOLD sequence in efforts to establish a novel tool for investigating emotion-driven hypotheses about eating pathology. In line with standards for rigor and reproducibility, we provide detailed protocols and code to support each step of this project.

METHOD

Sixteen adolescents with low-weight eating disorders viewed emotional faces (Happy or Disgust) and were asked to mimic their facial expression during simultaneous BOLD and EMG (Corrugator supercilli, Lavator lavii, Zygomaticus major) acquisition. Trials were repeated with the scanner off and again with scanner on (i.e., fatigue).

RESULTS

The Levator and Zygomaticus activation patterns discriminated disgust and happy faces successfully. The pattern held between scanner on and off conditions, but muscle activation attenuated in the Fatigue condition, especially for the Zygomaticus.

DISCUSSION

Simultaneous fMRI-EMG is a new tool capable of discriminating specific emotions based on muscle activation patterns and can be leveraged to answer emotion-driven hypotheses about clinical populations characterized by difficulty labeling or processing emotions.

A Wearable High-Resolution Facial Electromyography for Long Term Recordings in Freely Behaving Humans

Human facial expressions are a complex capacity, carrying important psychological and neurological information. Facial expressions typically involve the co-activation of several muscles; they vary between individuals, between voluntary versus spontaneous expressions, and depend strongly on personal interpretation. Accordingly, while high-resolution recording of muscle activation in a non-laboratory setting offers exciting opportunities, it remains a major challenge. This paper describes a wearable and non-invasive method for objective mapping of facial muscle activation and demonstrates its application in a natural setting. We focus on muscle activation associated with "enjoyment", "social" and "masked" smiles; three categories with distinct social meanings. We use an innovative, dry, soft electrode array designed specifically for facial surface electromyography recording, a customized independent component analysis algorithm, and a short training procedure to achieve the desired mapping. First, identification of the orbicularis oculi and the levator labii superioris was demonstrated from voluntary expressions. Second, the zygomaticus major was identified from voluntary and spontaneous Duchenne and non-Duchenne smiles. Finally, using a wireless device in an unmodified work environment revealed expressions of diverse emotions in face-to-face interaction. Our high-resolution and crosstalk-free mapping, along with excellent user-convenience, opens new opportunities in gaming, virtual-reality, bio-feedback and objective psychological and neurological assessment.