Validation of dynamic virtual faces for facial affect recognition

Differences in emotion recognition between nonimmersive versus immersive virtual reality: preliminary findings in schizophrenia and bipolar disorder

Deficits in social cognition may impair emotional processing and facial emotional recognition (FER) in patients with bipolar disorder (BD) and schizophrenia. FER is generally explored using photographs or images of static faces that do not fully capture the complexity of real-life facial stimuli. To overcome this limitation, we developed a set of dynamic virtual faces depicting six basic emotions (i.e. happiness, sadness, anger, fear, disgust, and surprise) and a neutral expression suitable for presentation in immersive and nonimmersive virtual realities. This study presents preliminary findings on the differences in FER accuracy from a frontal view between immersive and nonimmersive virtual realities among patients experiencing a relapse of schizophrenia ( n  = 10), a manic phase of BD ( n  = 10), and a group of healthy controls (HCs) ( n  = 10). As a secondary objective, we compare the FER accuracy across these three groups. Patients with schizophrenia and BD showed similar accuracy in recognizing emotions in immersive and nonimmersive virtual reality settings. However, patients with schizophrenia exhibited lower FER accuracy than HCs in both settings. Individuals with BD showed intermediate accuracy between those with schizophrenia and HCs, although these differences were not statistically significant. Notably, recognition of negative emotions was significantly impaired in both groups of patients.

Illumination Intelligent Adaptation and Analysis Framework: A comprehensive solution for enhancing nighttime driving fatigue monitoring

Nighttime driving presents a critical challenge to road safety due to insufficient lighting and increased risk of driver fatigue. Existing methods for monitoring driver fatigue, mainly focusing on behavioral analysis and biometric monitoring, face significant challenges under low-light conditions. Their effectiveness, especially in dynamic lighting environments, is limited by their dependency on specific environmental conditions and active driver participation, leading to reduced accuracy and practicality in real-world scenarios. This study introduces a novel 'Illumination Intelligent Adaptation and Analysis Framework (IIAAF)', aimed at addressing these limitations and enhancing the accuracy and practicality of driver fatigue monitoring under nighttime low-light conditions. The IIAAF framework employs a multidimensional technology integration, including comprehensive body posture analysis and facial fatigue feature detection, per-pixel dynamic illumination adjustment technology, and a light variation feature learning system based on Convolutional Neural Networks (CNN) and time-series analysis. Through this integrated approach, the framework is capable of accurately capturing subtle fatigue signals in nighttime driving environments and adapting in real-time to rapid changes in lighting conditions. Experimental results on two independent datasets indicate that the IIAAF framework significantly improves the accuracy of fatigue detection under nighttime low-light conditions. This breakthrough not only enhances the effectiveness of driving assistance systems but also provides reliable scientific support for reducing the risk of accidents caused by fatigued driving. These research findings have significant theoretical and practical implications for advancing intelligent driving assistance technology and improving nighttime road safety.

An exploratory study of the effect of age and gender on face scanning during affect recognition in immersive virtual reality

A person with impaired emotion recognition is not able to correctly identify facial expressions represented by other individuals. The aim of the present study is to assess eyes gaze and facial emotion recognition in a healthy population using dynamic avatars in immersive virtual reality (IVR). For the first time, the viewing of each area of interest of the face in IVR is studied by gender and age. This work in healthy people is conducted to assess the future usefulness of IVR in patients with deficits in the recognition of facial expressions. Seventy-four healthy volunteers participated in the study. The materials used were a laptop computer, a game controller, and a head-mounted display. Dynamic virtual faces randomly representing the six basic emotions plus neutral expression were used as stimuli. After the virtual human represented an emotion, a response panel was displayed with the seven possible options. Besides storing the hits and misses, the software program internally divided the faces into different areas of interest (AOIs) and recorded how long participants looked at each AOI. As regards the overall accuracy of the participants' responses, hits decreased from the youngest to the middle-aged and older adults. Also, all three groups spent the highest percentage of time looking at the eyes, but younger adults had the highest percentage. It is also noteworthy that attention to the face compared to the background decreased with age. Moreover, the hits between women and men were remarkably similar and, in fact, there were no statistically significant differences between them. In general, men paid more attention to the eyes than women, but women paid more attention to the forehead and mouth. In contrast to previous work, our study indicates that there are no differences between men and women in facial emotion recognition. Moreover, in line with previous work, the percentage of face viewing time for younger adults is higher than for older adults. However, contrary to earlier studies, older adults look more at the eyes than at the mouth.Consistent with other studies, the eyes are the AOI with the highest percentage of viewing time. For men the most viewed AOI is the eyes for all emotions in both hits and misses. Women look more at the eyes for all emotions, except for joy, fear, and anger on hits. On misses, they look more into the eyes for almost all emotions except surprise and fear.

Effect of Action Units, Viewpoint and Immersion on Emotion Recognition Using Dynamic Virtual Faces

Facial affect recognition is a critical skill in human interactions that is often impaired in psychiatric disorders. To address this challenge, tests have been developed to measure and train this skill. Recently, virtual human (VH) and virtual reality (VR) technologies have emerged as novel tools for this purpose. This study investigates the unique contributions of different factors in the communication and perception of emotions conveyed by VHs. Specifically, it examines the effects of the use of action units (AUs) in virtual faces, the positioning of the VH (frontal or mid-profile), and the level of immersion in the VR environment (desktop screen versus immersive VR). Thirty-six healthy subjects participated in each condition. Dynamic virtual faces (DVFs), VHs with facial animations, were used to represent the six basic emotions and the neutral expression. The results highlight the important role of the accurate implementation of AUs in virtual faces for emotion recognition. Furthermore, it is observed that frontal views outperform mid-profile views in both test conditions, while immersive VR shows a slight improvement in emotion recognition. This study provides novel insights into the influence of these factors on emotion perception and advances the understanding and application of these technologies for effective facial emotion recognition training.

Facial emotion recognition in patients with depression compared to healthy controls when using human avatars

The negative, mood-congruent cognitive bias described in depression, as well as excessive rumination, have been found to interfere with emotional processing. This study focuses on the assessment of facial recognition of emotions in patients with depression through a new set of dynamic virtual faces (DVFs). The sample consisted of 54 stable patients compared to 54 healthy controls. The experiment consisted in an emotion recognition task using non-immersive virtual reality (VR) with DVFs of six basic emotions and neutral expression. Patients with depression showed a worst performance in facial affect recognition compared to healthy controls. Age of onset was negatively correlated with emotion recognition and no correlation was observed for duration of illness or number of lifetime hospitalizations. There was no correlation for the depression group between emotion recognition and degree of psychopathology, excessive rumination, degree of functioning, or quality of life. Hence, it is important to improve and validate VR tools for emotion recognition to achieve greater methodological homogeneity of studies and to be able to establish more conclusive results.

How Interpersonal Distance Between Avatar and Human Influences Facial Affect Recognition in Immersive Virtual Reality

Purpose: The purpose of this study was to determine the optimal interpersonal distance (IPD) between humans and affective avatars in facial affect recognition in immersive virtual reality (IVR). The ideal IPD is the one in which the humans show the highest number of hits and the shortest reaction times in recognizing the emotions displayed by avatars. The results should help design future therapies to remedy facial affect recognition deficits.

Methods: A group of 39 healthy volunteers participated in an experiment in which participants were shown 65 dynamic faces in IVR and had to identify six basic emotions plus neutral expression presented by the avatars. We decided to limit the experiment to five different distances: D1 (35 cm), D2 (55 cm), D3 (75 cm), D4 (95 cm), and D5 (115 cm), all belonging to the intimate and personal interpersonal spaces. Of the total of 65 faces, 13 faces were presented for each of the included distances. The views were shown at different angles: 50% in frontal view, 25% from the right profile, and 25% from the left profile. The order of appearance of the faces presented to each participant was randomized.

Results: The overall success rate in facial emotion identification was 90.33%, being D3 the IPD with the best overall emotional recognition hits, although statistically significant differences could not be found between the IPDs. Consistent with results obtained in previous studies, identification rates for negative emotions were higher with increasing IPD, whereas the recognition task improved for positive emotions when IPD was closer. In addition, the study revealed irregular behavior in the facial detection of the emotion surprise.

Conclusions: IVR allows us to reliably assess facial emotion recognition using dynamic avatars as all the IPDs tested showed to be effective. However, no statistically significant differences in facial emotion recognition were found among the different IPDs.

Facial Affect Recognition by Patients with Schizophrenia Using Human Avatars

People with schizophrenia have difficulty recognizing the emotions in the facial expressions of others, which affects their social interaction and functioning in the community. Static stimuli such as photographs have been used traditionally to examine deficiencies in the recognition of emotions in patients with schizophrenia, which has been criticized by some authors for lacking the dynamism that real facial stimuli have. With the aim of overcoming these drawbacks, in recent years, the creation and validation of virtual humans has been developed. This work presents the results of a study that evaluated facial recognition of emotions through a new set of dynamic virtual humans previously designed by the research team, in patients diagnosed of schizophrenia. The study included 56 stable patients, compared with 56 healthy controls. Our results showed that patients with schizophrenia present a deficit in facial affect recognition, compared to healthy controls (average hit rate 71.6% for patients vs 90.0% for controls). Facial expressions with greater dynamism (compared to less dynamic ones), as well as those presented from frontal view (compared to profile view) were better recognized in both groups. Regarding clinical and sociodemographic variables, the number of hospitalizations throughout life did not correlate with recognition rates. There was also no correlation between functioning or quality of life and recognition. A trend showed a reduction in the emotional recognition rate as a result of increases in Positive and Negative Syndrome Scale (PANSS), being statistically significant for negative PANSS. Patients presented a learning effect during the progression of the task, slightly greater in comparison to the control group. This finding is relevant when designing training interventions for people with schizophrenia. Maintaining the attention of patients and getting them to improve in the proposed tasks is a challenge for today’s psychiatry.