Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research

Infrequent facial expressions of emotion do not bias attention

Despite the obvious importance of facial expressions of emotion, most studies have found that they do not bias attention. A critical limitation, however, is that these studies generally present face distractors on all trials of the experiment. For other kinds of emotional stimuli, such as emotional scenes, infrequently presented stimuli elicit greater attentional bias than frequently presented stimuli, perhaps due to suppression or habituation. The goal of the current study then was to test whether such modulation of attentional bias by distractor frequency generalizes to facial expressions of emotion. In Experiment 1, both angry and happy faces were unable to bias attention, despite being infrequently presented. Even when the location of these face cues were more unpredictable-presented in one of two possible locations-still no attentional bias was observed (Experiment 2). Moreover, there was no bottom-up influence for angry and happy faces shown under high or low perceptual load (Experiment 3). We conclude that task-irrelevant posed facial expressions of emotion cannot bias attention even when presented infrequently.

Quantifying the efficacy of an automated facial coding software using videos of parents

Introduction

This work explores the use of an automated facial coding software - FaceReader - as an alternative and/or complementary method to manual coding.

Methods

We used videos of parents (fathers, n = 36; mothers, n = 29) taken from the Avon Longitudinal Study of Parents and Children. The videos-obtained during real-life parent-infant interactions in the home-were coded both manually (using an existing coding scheme) and by FaceReader. We established a correspondence between the manual and automated coding categories - namely Positive, Neutral, Negative, and Surprise - before contingency tables were employed to examine the software's detection rate and quantify the agreement between manual and automated coding. By employing binary logistic regression, we examined the predictive potential of FaceReader outputs in determining manually classified facial expressions. An interaction term was used to investigate the impact of gender on our models, seeking to estimate its influence on the predictive accuracy.

Results

We found that the automated facial detection rate was low (25.2% for fathers, 24.6% for mothers) compared to manual coding, and discuss some potential explanations for this (e.g., poor lighting and facial occlusion). Our logistic regression analyses found that Surprise and Positive expressions had strong predictive capabilities, whilst Negative expressions performed poorly. Mothers' faces were more important for predicting Positive and Neutral expressions, whilst fathers' faces were more important in predicting Negative and Surprise expressions.

Discussion

We discuss the implications of our findings in the context of future automated facial coding studies, and we emphasise the need to consider gender-specific influences in automated facial coding research.

Coding infant engagement in the Face-to-Face Still-Face paradigm using deep neural networks

BACKGROUND

The Face-to-Face Still-Face (FFSF) task is a validated and commonly used observational measure of mother-infant socio-emotional interactions. With the ascendence of deep learning-based facial emotion recognition, it is possible that common complex tasks, such as the coding of FFSF videos, could be coded with a high degree of accuracy by deep neural networks (DNNs). The primary objective of this study was to test the accuracy of four DNN image classification models against the coding of infant engagement conducted by two trained independent manual raters.

METHODS

68 mother-infant dyads completed the FFSF task at three timepoints. Two trained independent raters undertook second-by-second manual coding of infant engagement into one of four classes: 1) positive affect, 2) neutral affect, 3) object/environment engagement, and 4) negative affect.

RESULTS

Training four different DNN models on 40,000 images, we achieved a maximum accuracy of 99.5% on image classification of infant frames taken from recordings of the FFSF task with a maximum inter-rater reliability (Cohen's κ-value) of 0.993.

LIMITATIONS

This study inherits all sampling and experimental limitations of the original study from which the data was taken, namely a relatively small and primarily White sample.

CONCLUSIONS

Based on the extremely high classification accuracy, these findings suggest that DNNs could be used to code infant engagement in FFSF recordings. DNN image classification models may also have the potential to improve the efficiency of coding all observational tasks with applications across multiple fields of human behavior research.

Dynamics of Affective Reactivity during Mother-Daughter Interactions: The Impact of Adolescent Non-Suicidal Self-Injury

Non-suicidal self-injury (NSSI) is an alarming public health concern that is particularly widespread among adolescents. The current study examined affective responses during mother-daughter interactions in adolescent girls with and without a history of NSSI. Participants were 60 girls aged 13-17 with (n = 27) and without (n = 33) a history of NSSI and their mothers. Adolescents and their mothers completed two interaction tasks: one positive and one negative. During these interactions, facial affect was assessed via electromyography (EMG). Results of Actor-Partner Interdependence Modeling (APIM) revealed several intra- and interpersonal disruptions in affect during both tasks among dyads in which the adolescent had an NSSI history. Findings suggest deficits in both self- and co-regulation of facial affect during mother-daughter interactions involving dyads in which the adolescents reports NSSI. Ultimately, if replicated and extended in longitudinal research, these disruptions may prove to be promising targets of intervention to reduce risk for future NSSI in adolescent girls.

Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism

Commonly used screening tools for autism spectrum disorder (ASD) generally rely on subjective caregiver questionnaires. While behavioral observation is more objective, it is also expensive, time-consuming, and requires significant expertise to perform. As such, there remains a critical need to develop feasible, scalable, and reliable tools that can characterize ASD risk behaviors. This study assessed the utility of a tablet-based behavioral assessment for eliciting and detecting one type of risk behavior, namely, patterns of facial expression, in 104 toddlers (ASD N = 22) and evaluated whether such patterns differentiated toddlers with and without ASD. The assessment consisted of the child sitting on his/her caregiver's lap and watching brief movies shown on a smart tablet while the embedded camera recorded the child's facial expressions. Computer vision analysis (CVA) automatically detected and tracked facial landmarks, which were used to estimate head position and facial expressions (Positive, Neutral, All Other). Using CVA, specific points throughout the movies were identified that reliably differentiate between children with and without ASD based on their patterns of facial movement and expressions (area under the curves for individual movies ranging from 0.62 to 0.73). During these instances, children with ASD more frequently displayed Neutral expressions compared to children without ASD, who had more All Other expressions. The frequency of All Other expressions was driven by non-ASD children more often displaying raised eyebrows and an open mouth, characteristic of engagement/interest. Preliminary results suggest computational coding of facial movements and expressions via a tablet-based assessment can detect differences in affective expression, one of the early, core features of ASD. LAY SUMMARY: This study tested the use of a tablet in the behavioral assessment of young children with autism. Children watched a series of developmentally appropriate movies and their facial expressions were recorded using the camera embedded in the tablet. Results suggest that computational assessments of facial expressions may be useful in early detection of symptoms of autism.

Emerging Opportunities Provided by Technology to Advance Research in Child Health Globally

Current approaches to longitudinal assessment of children's developmental and psychological well-being, as mandated in the United Nations Sustainable Development Goals, are expensive and time consuming. Substantive understanding of global progress toward these goals will require a suite of new robust, cost-effective research tools designed to assess key developmental processes in diverse settings. While first steps have been taken toward this end through efforts such as the National Institutes of Health's Toolbox, experience-near approaches including naturalistic observation have remained too costly and time consuming to scale to the population level. This perspective presents 4 emerging technologies with high potential for advancing the field of child health and development research, namely (1) affective computing, (2) ubiquitous computing, (3) eye tracking, and (4) machine learning. By drawing attention of scientists, policy makers, investors/funders, and the media to the applications and potential risks of these emerging opportunities, we hope to inspire a fresh wave of innovation and new solutions to the global challenges faced by children and their families.

Technology Use for Adolescent Health and Wellness

As avid users of technology, adolescents are a key demographic to engage when designing and developing technology applications for health. There are multiple opportunities for improving adolescent health, from promoting preventive behaviors to providing guidance for adolescents with chronic illness in supporting treatment adherence and transition to adult health care systems. This article will provide a brief overview of current technologies and then highlight new technologies being used specifically for adolescent health, such as artificial intelligence, virtual and augmented reality, and machine learning. Because there is paucity of evidence in this field, we will make recommendations for future research.