Facial thermal variations: A new marker of emotional arousal

Facial Temperature Responses to Ostracism in Women: Exploring Nasal Thermal Signatures of Different Coping Behaviors

Ostracism (feeling ignored and excluded) triggers psychophysiological responses associated with distress. We investigated different coping responses after ostracism and explored whether these were preceded by unique facial thermal signatures, reflecting autonomic nervous system (ANS) activity. Using thermal infrared imaging, we recorded facial cutaneous temperature variations in female participants (N = 95) experiencing inclusion and ostracism using hypothetical Cyberball games. Coping after ostracism was assessed during a hypothetical Allocation Game, where participants could do nothing (withdrawal), reduce (antisocial), or increase (prosocial) the hypothetical earnings of their ostracizer. Contrary to expectations, most participants chose to withdraw (52%), with fewer opting for antisocial responses (30%) or prosocial responses (18%) after ostracism. Results from linear mixed-effects modeling revealed that substantial temperature variability occurred only in the nose region of the face. Both ostracism and inclusion showed a decrease in nasal temperature relative to baseline, but the average drop was greater during inclusion, suggesting stronger ANS activation during inclusion rather than ostracism. Crucially, exploratory findings showed that only participants who responded antisocially after ostracism exhibited steeper decreases in nasal temperature during ostracism compared to inclusion. This pattern suggests greater physiological reactivity among antisocial responders, particularly in contrast to those who chose to withdraw. Future research should integrate thermal imaging with other physiological measures and strengthen ostracism manipulations to understand the relationship between thermal responses and different coping behaviors.

ThermICA: Novel Approach for a Multivariate Analysis of Facial Thermal Responses

OBJECTIVE

Infrared Thermography (IRT) has been used to monitor skin temperature variation in a contactless manner, in both clinical medicine and psychophysiology. Here, we introduce a new methodology to obtain information about autonomic correlates related to perspiration, peripheral vasomotility, and respiration from infrared recordings.

METHODS

Our approach involves a model-based decomposition of facial thermograms using Independent Component Analysis (ICA) and an ad-hoc preprocessing procedure. We tested our approach on 30 healthy volunteers whose psychophysiological state was stimulated as part of an experimental protocol.

RESULTS

Within-subject ICA analysis identified three independent components demonstrating correlations with the reference physiological signals. Moreover, a linear combination of independent components effectively predicted each physiological signal, achieving median correlations of 0.9 for electrodermal activity, 0.8 for respiration, and 0.73 for photoplethysmography peaks envelope. In addition, we performed a cross-validated inter-subject analysis, which allows to predict physiological signals from facial thermograms of unseen subjects.

CONCLUSIONS/SIGNIFICANCE

Our findings validate the efficacy of features extracted from both original and thermal-derived signals for differentiating experimental conditions. This outcome emphasizes the sensitivity and promise of our approach, advocating for expanded investigations into thermal imaging within biomedical signal analysis. It underscores its potential for enhancing objective assessments of emotional states.

"Hot stuff": Behavioural and affective thermal responses to digital and non-digital disruptions during early mother-infant interaction

Responsive social exchanges are critical for infants' bio-behavioural regulation and healthy development. Parental smartphone use may disrupt early parent-infant interactions, leading to frequent interruptions. The impact of these interruptions on mother-infant behaviours and stress physiology is unclear. Infrared thermal imaging (ITI) offers a non-invasive approach to assess stress-related skin temperature changes reflecting autonomic activation. This study investigates mother-infant behavioural and thermal affective responses to parental digital and non-digital distractions. Thirty-eight mother-infant dyads (22 males) participated in a modified Still-Face Paradigm with five conditions: Free Play, Technoference Exposure (TF-E), Technoference Reunion (TF-R), Paperference Exposure (PF-E), Paperference Reunion (PF-R). During TF-E and PF-E mothers completed questionnaires on a smartphone or paper and were unresponsive to the infant. Mother-infant behaviours were coded microanalytically, while FLIR cameras detected changes in forehead and nasal tip temperatures. Maternal habitual smartphone use was assessed by self-report and passive sensing. Infants showed increased behavioural distress during TF-E and PF-E. ITI revealed lower infant forehead temperatures during TF-E compared to free play and reunions, while no significant changes at the nasal tip. Maternal forehead temperature dropped significantly during PF-E, compared to other episodes. Greater maternal habitual phone use was linked to infant responses and maternal behaviours. Parental unresponsiveness due to digital and non-digital distractions leads to infant behavioural distress, with digital disruptions also triggering a distinct thermal affective response. Findings highlight the impact of different parental distractions on early interactions and stress responding, with potential long-term implications.

Is it Possible to Recognize Emotions/Feelings?

It is true that emotions are the most important psychological functions in humans and are on the top of the hierarchical tree of the homeostasis. As a potent mental activity, emotions significantly influence various aspects of human functioning such as decision-making, perception, attention and memory. For scientific but also for practical reasons, the main question is: how to register and measure emotions? The aim of this article is to summarize the main known methods for measuring the emotions/feelings published in Medline basis. Different methodologies are presented in order to measure emotions such as: Face recognition, Bodily maps of emotions, Evaluation of autonomic nervous system activity, EEG, and especially fMRI and PET scan. Data are used from PubMed bases.

From feeling chilly to burning up: How thermal signals shape the physiological state of the body and impact physical, emotional, and social well-being

This review examines the role of thermal interoception-awareness of body temperature-in shaping physiological states and its impact on physical, emotional, and social well-being. We describe the neural pathways and mechanisms involved in thermal processing and environmental heat exchange and how thermal interoception influences both autonomic and behavioral responses, contributing to survival and homeostasis. Additionally, the review highlights the significance of thermal signals and thermoregulation in determining higher-order cognitive and emotional functions, such as regulating sleep patterns and activity levels, monitoring physical well-being, regulating emotions, and even social interactions. We describe the existing instruments for assessing body temperature and thermal awareness and call for further research to investigate the role of thermal interoception in emotional and social domains. We also discuss potential therapeutic applications, particularly in the context of psychosomatic and neurological disorders characterized by emotional dysregulation, disrupted sleep and mood patterns, social difficulties, and alterations in self-consciousness.

A Complete Pipeline to Extract Temperature from Thermal Images of Pigs

Using deep learning or artificial intelligence (AI) in research with animals is a new interdisciplinary area of research. In this study, we have explored the potential of thermal imaging and AI in pig research. Thermal cameras play a vital role in obtaining and collecting a large amount of data, and AI has the capabilities of processing and extracting valuable information from these data. The amount of data collected using thermal imaging is huge, and automation techniques are therefore crucial to find a meaningful interpretation of the changes in temperature. In this paper, we present a complete pipeline to extract temperature automatically from a selected Region of Interest (ROI). This system consists of three stages: the first one checks whether the ROI is completely visible to observe the thermal temperature, and then the second stage uses an encoder-decoder structure of a convolution neural network to segment the ROI, if the condition was met at stage one. In the last stage, the maximum temperature is extracted and saved in an external file. The segmentation model showed good performance, with a mean Pixel Class accuracy of 92.3%, and a mean Intersection over Union of 87.1%. The extracted temperature observed by the model entirely matched the manually observed temperature. The system showed reliable results to be used independently without human intervention to determine the temperature in the selected ROI in pigs.

Driver drowsiness is associated with altered facial thermal patterns: Machine learning insights from a thermal imaging approach

Driver drowsiness is a significant factor in road accidents. Thermal imaging has emerged as an effective tool for detecting drowsiness by enabling the analysis of facial thermal patterns. However, it is not clear which facial areas are most affected and correlate most strongly with drowsiness. This study examines the variations and importance of various facial areas and proposes an approach for detecting driver drowsiness. Twenty participants underwent tests in a driving simulator, and temperature changes in various facial regions were measured. The random forest method was employed to evaluate the importance of each facial region. The results revealed that temperature changes in the nasal area exhibited the highest value, while the eyes had the most correlated changes with drowsiness. Furthermore, drowsiness was classified with an accuracy of 88 % utilizing thermal variations in the facial region identified as the most important regions by the random forest feature importance model. These findings provide a comprehensive overview of facial thermal imaging for detecting driver drowsiness and introduce eye temperature as a novel and effective measure for investigating cognitive activities.

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Facial infrared thermography as an index of social anxiety

Previous research on physiological indices of social anxiety has offered unclear results. In this study, participants with low and high social anxiety performed five social interaction tasks while being recorded with a thermal camera. Each task was associated with a dimension assessed by the Social Anxiety Questionnaire for Adults (1 = Interactions with strangers. 2 = Speaking in public/Talking with people in authority, 3 = Criticism and embarrassment, 4 = Assertive expression of annoyance, disgust or displeasure, 5 = Interactions with the opposite sex). Mixed-effects models revealed that the temperature of the tip of the nose decreased significantly in participants with low (vs. high) social anxiety (p < 0.001), while no significant differences were found in other facial regions of interest: forehead (p = 0.999) and cheeks (p = 0.999). Furthermore, task 1 was the most effective at discriminating between the thermal change of the nose tip and social anxiety, with a trend for a higher nose temperature in participants with high social anxiety and a lower nose temperature for the low social anxiety group. We emphasize the importance of corroborating thermography with specific tasks as an ecological method, and tip of the nose thermal change as a psychophysiological index associated with social anxiety.

Reactivity of observers' facial skin blood flow depending on others' facial expressions and blushing

Facial skin blood flow (SkBF) has attracted attention as an autonomic indicator because it influences facial colour, which informs others of emotional states, and facial temperature related to social anxiety. Previous studies have examined the facial SkBF in people experiencing emotions; however, facial SkBF changes in the observers of emotions are poorly understood. Our study clarified facial SkBF changes related to observing others' emotions by comparing the changes with other physiological indices. Thirty healthy participants (24 females; mean age: 22.17) observed six types of facial expressions (neutral, angry, and embarrassed expressions with and without facial blushing) and rated the emotional intensity of the other person. We measured their facial SkBF, finger SkBF, and cardiac RR interval as they made their observations. Facial SkBF generally decreased in relation to observing emotional faces (angry and embarrassed faces) and significantly decreased for angry expressions with blushing. None of the participants noticed blushing of facial stimuli. For the RR interval and finger SkBF, there was no variation depending on the observed facial expressions, although there was a general increase related to observation. These results indicated that facial SkBF is sensitive and reactive to emotional faces-especially angry faces with blushing- compared with other autonomic indices. The facial SkBF changes were not related to either RR interval changes or the intensity rating, suggesting that facial SkBF changes may be caused by vasoconstriction and have potential functions for our emotions. The decrease in facial SkBF may have a role in calming observers by preventing them from adopting the same emotional state as a person with intense anger. These findings clarify daily facial SkBF fluctuations and their relationship with our emotional processing in interpersonal situations.

miR-214 could promote myocardial fibrosis and cardiac mesenchymal transition in VMC mice through regulation of the p53 or PTEN-PI3K-Akt signali pathway, promoting CF proliferation and inhibiting its ng pathway

INTRODUCTION

This study aimed to investigate the role of miR-214 in the bidirectional regulation of p53 and PTEN and its influence on myocardial fibrosis and cardiac mesenchymal transformation in mice with viral myocarditis (VMC).

METHODS

The study established a VMC model in BALB/c mice by injecting them with the CVB3 virus intraperitoneally. Techniques such as ELISA, H&E staining, Masson staining, immunohistochemical staining, RT-qPCR, western blot, and dual-luciferase reporter gene assay were used to detect the expression levels of relevant factors in tissues and cells. Isolation and culture of cardiac fibroblasts (CFs) were also conducted.

RESULTS

The study found that miR-214 bidirectional regulation of p53 and PTEN promotes myocardial fibrosis and cardiac mesenchymal transformation in mice with VMC. The expression levels of collagen-related peptides, inflammatory-related factors, miR-214, mesenchymal transformation-related factors, and fibrosis-related factors were significantly increased, while the expression levels of p53, PTEN, and epithelial/endothelial cell phenotype marker factors were significantly decreased. Downregulation of miR-214 or upregulation of p53 and PTEN expression inhibited inflammatory cell and fibroblast infiltration in VMC mouse myocardial tissue. It reduced the proliferation ability while increasing the apoptosis of cardiac fibroblasts.

CONCLUSION

miR-214 plays a significant role in the bidirectional inhibition of p53 and PTEN, which leads to myocardial fibrosis and cardiac mesenchymal transformation in mice with VMC. Downregulation of miR-214 or upregulation of p53 and PTEN expression may provide potential therapeutic targets for treating VMC-induced cardiac fibrosis and mesenchymal transformation.

Assessing Feasibility of Cognitive Impairment Testing Using Social Robotic Technology Augmented with Affective Computing and Emotional State Detection Systems

Social robots represent a valid opportunity to manage the diagnosis, treatment, care, and support of older people with dementia. The aim of this study is to validate the Mini-Mental State Examination (MMSE) test administered by the Pepper robot equipped with systems to detect psychophysical and emotional states in older patients. Our main result is that the Pepper robot is capable of administering the MMSE and that cognitive status is not a determinant in the effective use of a social robot. People with mild cognitive impairment appreciate the robot, as it interacts with them. Acceptability does not relate strictly to the user experience, but the willingness to interact with the robot is an important variable for engagement. We demonstrate the feasibility of a novel approach that, in the future, could lead to more natural human-machine interaction when delivering cognitive tests with the aid of a social robot and a Computational Psychophysiology Module (CPM).

Classification of User Emotional Experiences on B2C Websites Utilizing Infrared Thermal Imaging

The acquisition of physiological signals for analyzing emotional experiences has been intrusive, and potentially yields inaccurate results. This study employed infrared thermal images (IRTIs), a noninvasive technique, to classify user emotional experiences while interacting with business-to-consumer (B2C) websites. By manipulating the usability and aesthetics of B2C websites, the facial thermal images of 24 participants were captured as they engaged with the different websites. Machine learning techniques were leveraged to classify their emotional experiences, with participants' self-assessments serving as the ground truth. The findings revealed significant fluctuations in emotional valence, while the participants' arousal levels remained consistent, enabling the categorization of emotional experiences into positive and negative states. The support vector machine (SVM) model performed well in distinguishing between baseline and emotional experiences. Furthermore, this study identified key regions of interest (ROIs) and effective classification features in machine learning. These findings not only established a significant connection between user emotional experiences and IRTIs but also broadened the research perspective on the utility of IRTIs in the field of emotion analysis.

Facial thermal and blood perfusion patterns of human emotions: Proof-of-Concept

In this work, a preliminary study of proof-of-concept was conducted to evaluate the performance of the thermographic and blood perfusion data when emotions of positive and negative valence are applied, where the blood perfusion data are obtained from the thermographic data. The images were obtained for baseline, positive, and negative valence according to the protocol of the Geneva Affective Picture Database. Absolute and percentage differences of average values of the data between the valences and the baseline were calculated for different regions of interest (forehead, periorbital eyes, cheeks, nose and upper lips). For negative valence, a decrease in temperature and blood perfusion was observed in the regions of interest, and the effect was greater on the left side than on the right side. In positive valence, the temperature and blood perfusion increased in some cases, showing a complex pattern. The temperature and perfusion of the nose was reduced for both valences, which is indicative of the arousal dimension. The blood perfusion images were found to be greater contrast; the percentage differences in the blood perfusion images are greater than those obtained in thermographic images. Moreover, the blood perfusion images, and vasomotor answer are consistent, therefore, they can be a better biomarker than thermographic analysis in identifying emotions.

Mexican validation of the International Affective Digitized Sounds second edition (IADS-2) and additional sounds

Affective stimuli have been extensively used in emotion research for a better understanding of emotion regulation. Sound ratings, specifically non-verbal sounds, are biased by demographic indicators such as sex and nationality. Therefore, it is crucial to characterize sounds prior to their use in emotion research. This study aims to validate the IADS-2 database and additional sounds in a sample from the Mexican population. Three hundred twenty-nine participants born and raised in Mexico remotely listened to 174 sounds in monophonic format. They rated sounds according to the valence-arousal-dominance model using the Self-Assessment Manikin test. Results positively correlated to those of previous studies. Sex differences were observed only in dominance between female and male groups, contrary to the results from Portuguese, American and Japanese validations. Geographic region analysis demonstrated differences in arousal, indicating the need for additional research on occident and south regions. Furthermore, when conducting affective research, headphones and audio quality should be considered, primarily to reduce variability due to audio-related aspects, and to avoid changes in emotional experience. Finally, this study supports the feasibility of remote affective sound experiments over the internet as reported in previous research.

Abstract words processing induces parasympathetic activation: A thermal imaging study

Abstract words (e.g., freedom) compose a significant part of speech. Despite this, learning them is complicated. Abstract concepts collect more heterogeneous exemplars and are more detached from sensory modalities than concrete concepts. Recent views propose that, because of their complexity, other people are pivotal for abstract concepts’ acquisition and use, e.g., to explain their meaning. We tested this hypothesis using a combined behavioral and thermal imaging paradigm. Twenty-one Italian children (10\F, mean age: 6 years) determined whether acoustic stimuli (concrete and abstract words; non-words) were or not correct Italian words (lexical decision). Concrete terms yielded faster responses than abstract ones: for the first time, this effect appears with response times in very young children. More crucially, the higher increase in temperature of the nasal tip (i.e., typically associated with parasympathetic dominance of the neurovegetative response) suggests that, with abstract concepts, children might be more socially and cognitively engaged.

Face Thermal Map of the Mexican Population in the Basal State

There has been a wide use of thermal images of the human body in recent years, specifically images with thermal information of regions of interest (ROI) in the face; this information can be used for epidemiological, clinical, and/or psychological purposes. Due to this, it is important to have plenty of information on temperature in these ROIs in the basal state that allows their use as a reference in terms of their thermal analysis. In this work, a face thermal map of the Mexican population in the basal state (n = 196) is created, adding the comparison between different population groups, such as gender, age, and clinical status, obtaining results of great interest for future research. The t-test for independent samples was applied to the ROIs with normal distribution and Mann−Whitney u-test to the ones that did not present normal distribution. Statistically significant differences were found in some of the ROI comparisons like the corrugator, the supraorbitals, and the chin between the control and clinical groups, as well as in the differentiation by age (p < 0.05).

Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals

Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.

Reading Your Emotions in My Physiology? Reliable Emotion Interpretations in Absence of a Robust Physiological Resonance

Affective states are expressed in an individual’s physical appearance, ranging from facial expressions and body postures, to indicators of physiological arousal (e.g., a blush). Confirming the claimed communicative function of these markers, humans are capable of distinguishing between a variety of discrete emotion displays. In an attempt to explain the underlying mechanism, characteristic bodily changes within the observer, including physiological arousal and mimicry, have been suggested to facilitate the interpretation of an expression. The current study aims to create a holistic picture of emotion perception by (1) using three different sources of emotional information (prototypical facial expressions, bodily expressions, and subtle facial cues) and (2) measuring changes in multiple physiological signals (facial electromyography, skin conductance level, skin temperature, and pupil size). While participants clearly discriminated between perceived emotional expressions, there was no overall 1–1 correspondence with their physiological responses. Some specific but robust effects were observed. Angry facial expressions were consistently responded to with a peak in skin conductance level. Furthermore, sad body expressions were associated with a drop in skin temperature. In addition to being the best recognized expression, viewing happy faces elicited congruent facial muscle responses, which supports the potential role of embodied simulation in emotion recognition. Lastly, tears were not only rated as highly emotional intense but also evoked a peak in skin conductance level in the observer. The absence of distinct physiological responses to other expressions could be explained by the lacking functionality of affect sharing in a non-interactive experimental context. Consequentially, emotional alignment in body and mind might especially take place in real social situations, which should be considered in future research.

Automated Affective Computing Based on Bio-Signals Analysis and Deep Learning Approach

Extensive possibilities of applications have rendered emotion recognition ineluctable and challenging in the fields of computer science as well as in human-machine interaction and affective computing. Fields that, in turn, are increasingly requiring real-time applications or interactions in everyday life scenarios. However, while extremely desirable, an accurate and automated emotion classification approach remains a challenging issue. To this end, this study presents an automated emotion recognition model based on easily accessible physiological signals and deep learning (DL) approaches. As a DL algorithm, a Feedforward Neural Network was employed in this study. The network outcome was further compared with canonical machine learning algorithms such as random forest (RF). The developed DL model relied on the combined use of wearables and contactless technologies, such as thermal infrared imaging. Such a model is able to classify the emotional state into four classes, derived from the linear combination of valence and arousal (referring to the circumplex model of affect’s four-quadrant structure) with an overall accuracy of 70% outperforming the 66% accuracy reached by the RF model. Considering the ecological and agile nature of the technique used the proposed model could lead to innovative applications in the affective computing field.

Thermal imaging reveals audience-dependent effects during cooperation and competition in wild chimpanzees

Accessing animal minds has remained a challenge since the beginnings of modern science. Here, we used a little-tried method, functional infrared thermal imaging, with wild chimpanzees during common social interactions. After removing confounds, we found that chimpanzees involved in competitive events had lower nose skin temperatures whereas those involved in cooperative events had higher temperatures, the latter more so in high- than low-ranking males. Temperatures associated with grooming were akin to those of cooperative events, except when males interacted with a non-reciprocating alpha male. In addition, we found multiple audience effects. Notably, the alpha male's presence reduced positive effects associated with cooperation, whereas female presence buffered negative effects associated with competition. Copulation was perceived as competitive, especially during furtive mating when other males were absent. Overall, patterns suggest that chimpanzees categorise ordinary social events as cooperative or competitive and that these perceptions are moderated by specific audiences.

A Wearable Head Mounted Display Bio-Signals Pad System for Emotion Recognition

Multimodal bio-signals acquisition based on wearable devices and using virtual reality (VR) as stimulus source are promising techniques in emotion recognition research field. Numerous studies have shown that emotional states can be better evoked through Immersive Virtual Environments (IVE). The main goal of this paper is to provide researchers with a system for emotion recognition in VR environments. In this paper, we present a wearable forehead bio-signals acquisition pad which is attached to Head-Mounted Displays (HMD), termed HMD Bio Pad. This system can simultaneously record emotion-related two-channel electroencephalography (EEG), one-channel electrodermal activity (EDA), photoplethysmograph (PPG) and skin temperature (SKT) signals. In addition, we develop a human-computer interaction (HCI) interface which researchers can carry out emotion recognition research using VR HMD as stimulus presentation device. To evaluate the performance of the proposed system, we conducted different experiments to validate the multimodal bio-signals quality, respectively. To validate EEG signal, we have assessed the performance in terms of EEG eyes-blink task and eyes-open and eyes-closed task. The EEG eyes-blink task indicates that the proposed system can achieve comparable EEG signal quality in comparison to the dedicated bio-signals measuring device. The eyes-open and eyes-closed task proves that the proposed system can efficiently record alpha rhythm. Then we used signal-to-noise ratio (SNR) and Skin Conductance Reaction (SCR) signal to validate the performance for EDA acquisition system. A filtered EDA signal, with a high mean SNR of 28.52 dB, is plotted on HCI interface. Moreover, the SCR signal related to stimulus response can be correctly extracted from EDA signal. The SKT acquisition system has been validated effectively by the temperature change experiment when subjects are in unpleasant emotion. The pulse rate (PR) estimated from PPG signal achieved the low mean average absolute error (AAE), which is 1.12 beats per minute (BPM) over 8 recordings. In summary, the proposed HMD Bio Pad offers a portable, comfortable and easy-to-wear device for recording bio-signals. The proposed system could contribute to emotion recognition research in VR environments.

Brow and Masticatory Muscle Activity Senses Subjective Hedonic Experiences during Food Consumption

Sensing subjective hedonic or emotional experiences during eating using physiological activity is practically and theoretically important. A recent psychophysiological study has reported that facial electromyography (EMG) measured from the corrugator supercilii muscles was negatively associated with hedonic ratings, including liking, wanting, and valence, during the consumption of solid foods. However, the study protocol prevented participants from natural mastication (crushing of food between the teeth) during physiological data acquisition, which could hide associations between hedonic experiences and masticatory muscle activity during natural eating. We investigated this issue by assessing participants’ subjective ratings (liking, wanting, valence, and arousal) and recording physiological measures, including EMG of the corrugator supercilii, zygomatic major, masseter, and suprahyoid muscles while they consumed gel-type solid foods (water-based gellan gum jellies) of diverse flavors. Ratings of liking, wanting, and valence were negatively correlated with corrugator supercilii EMG and positively correlated with masseter and suprahyoid EMG. These findings imply that subjective hedonic experiences during food consumption can be sensed using EMG signals from the brow and masticatory muscles.

Driver drowsiness detection using facial thermal imaging in a driving simulator

Driver drowsiness causes fatal driving accidents. Thermal imaging is a suitable drowsiness detection method as it is non-invasive and robust against changes in the ambient light. In this paper, driver drowsiness is detected by measuring the forehead temperature at the region covering the supratrochlear artery and also the cheek temperature. About 30 subjects drove on a highway in a driving simulator in two sessions. A thermal camera was used to monitor the facial temperature pattern. The subjects' drowsiness levels were estimated by three human observers. The forehead and the cheek regions were located and tracked in each frame. The forehead and the cheek skin temperatures were obtained at three levels of drowsiness. The Support Vector Machine, the K-Nearest Neighbor, and the regression tree classifiers were used. From wakefulness to extreme drowsiness, the forehead skin temperature and the absolute cheek-forehead skin temperature gradient decreased by 0.46°C and 0.81°C, respectively. But the cheek skin temperature increased by 0.35°C in two sessions. The gradient difference is on average 50% higher than the forehead or the cheek temperature change alone. The results indicate that drowsiness can be detected with an accuracy of 82%, sensitivity of 85%, specificity of 90%, and precision of 84%. Driver drowsiness can be detected by monitoring changes in the forehead and the cheek temperature signal. Also, the temperature gradient can be used as a more robust and sensitive indicator of drowsiness.

Deep learning techniques for automated detection of autism spectrum disorder based on thermal imaging

Children with autism spectrum disorder have impairments in emotional processing which leads to the inability in recognizing facial expressions. Since emotion is a vital criterion for having fine socialisation, it is incredibly important for the autistic children to recognise emotions. In our study, we have chosen the facial skin temperature as a biomarker to measure emotions. To assess the facial skin temperature, the thermal imaging modality has been used in this study, since it has been recognised as a promising technique to evaluate emotional responses. The aim of this study was the following: (1) to compare the facial skin temperature of autistic and non-autistic children by using thermal imaging across various emotions; (2) to classify the thermal images obtained from the study using the customised convolutional neural network compared with the ResNet 50 network. Fifty autistic and fifty non-autistic participants were included for the study. Thermal imaging was used to obtain the temperature of specific facial regions such as the eyes, cheek, forehead and nose while we evoked emotions (Happiness, anger and sadness) in children using an audio-visual stimulus. Among the emotions considered, the emotion anger had the highest temperature difference between the autistic and non-autistic participants in the region's eyes (1.9%), cheek (2.38%) and nose (12.6%). The accuracy obtained by classifying the thermal images of the autistic and non-autistic children using Customised Neural Network and ResNet 50 Network was 96% and 90% respectively. This computer aided diagnostic tool can be a predictable and a steadfast method in the diagnosis of the autistic individuals.

The temperature of emotions

Emotions and temperature are closely related through embodied processes, and people seem to associate temperature concepts with emotions. While this relationship is often evidenced by everyday language (e.g., cold and warm feelings), what remains missing to date is a systematic study that holistically analyzes how and why people associate specific temperatures with emotions. The present research aimed to investigate the associations between temperature concepts and emotion adjectives on both explicit and implicit levels. In Experiment 1, we evaluated explicit associations between twelve pairs of emotion adjectives derived from the circumplex model of affect, and five different temperature concepts ranging from 0°C to 40°C, based on responses from 403 native speakers of four different languages (English, Spanish, Japanese, Chinese). The results of Experiment 1 revealed that, across languages, the temperatures were associated with different regions of the circumplex model. The 0°C and 10°C were associated with negative-valanced, low-arousal emotions, while 20°C was associated with positive-valanced, low-to-medium-arousal emotions. Moreover, 30°C was associated with positive-valanced, high-arousal emotions; and 40°C was associated with high-arousal and either positive- or negative-valanced emotions. In Experiment 2 (N = 102), we explored whether these temperature-emotion associations were also present at the implicit level, by conducting Implicit Association Tests (IATs) with temperature words (cold and hot) and opposing pairs of emotional adjectives for each dimension of valence (Unhappy/Dissatisfied vs. Happy/Satisfied) and arousal (Passive/Quiet vs. Active/Alert) on native English speakers. The results of Experiment 2 revealed that participants held implicit associations between the word hot and positive-valanced and high-arousal emotions. Additionally, the word cold was associated with negative-valanced and low-arousal emotions. These findings provide evidence for the existence of temperature-emotion associations at both explicit and implicit levels across languages.

Autonomic reactivity to emotion: A marker of sub-clinical anxiety and depression symptoms?

Anxiety and depression are both characterized by dysregulated autonomic reactivity to emotion. However, most experiments until now have focused on autonomic reactivity to stimuli presented in central vision (CV) even if affective saliency is also observed in peripheral vision (PV). We compared autonomic reactivity to CV and PV emotional stimulation in 58 participants with high anxious (HA) or low anxious (LA) and high depressive (HD) or low depressive (LD) symptomatology, based on STAI-B and BDI scores, respectively. Unpleasant (U), pleasant (P), and neutral (N) pictures from IAPS were presented at three eccentricities (0°: CV; -12 and 12°: PV). Skin conductance (SC), skin temperature, pupillary diameter, and heart rate (HR) were recorded. First, HA participants showed greater pupil dilation to emotional than to neutral stimuli in PV than in CV. Second, in contrast to HD, the valence effect indexed by SC and emotional arousal effect indexed by skin temperature were observed in LD. Third, both anxiety and depression lead to a valence effect indexed by pupillary light reflex and heart rate. These results suggest a hyperreactivity to emotion and hypervigilance to PV in anxiety. Depression is associated with an attenuation of positive effect and a global blunted autonomic reactivity to emotion. Moreover, anxiety mostly modulates the early processes of autonomic reactivity whereas depression mainly affects the later processes. The differential impact of emotional information over the visual field suggests the use of new stimulation strategies in order to attenuate anxious and depressive symptoms.

Physiological correlates of subjective emotional valence and arousal dynamics while viewing films

An exploration of the physiological correlates of subjective emotional states has theoretical and practical significance. Previous studies have reported that subjective valence and arousal correspond to facial electromyography (EMG) and electrodermal activity (EDA), respectively, across stimuli. However, the reported results were inconsistent, no study investigated subjective-physiological concordance across time, and measures of arousal remain controversial. To investigate these issues, while healthy adults (n = 20) viewed emotional films, we assessed overall and continuous ratings of valence and arousal and recorded EMG from the corrugator supercilii and zygomatic major, EDA from the palms and forehead, and nose-tip temperature. The corrugator and zygomatic EMG were negatively and positively associated with valence ratings, respectively, across stimuli and time. EDA (both sites) and nose-tip temperature were positively and negatively associated with arousal ratings, respectively, across stimuli and time. It is concluded that subjective emotional valence and arousal dynamics have specific physiological correlates.

Read My Face: Automatic Facial Coding Versus Psychophysiological Indicators of Emotional Valence and Arousal

Facial expressions provide insight into a person's emotional experience. To automatically decode these expressions has been made possible by tremendous progress in the field of computer vision. Researchers are now able to decode emotional facial expressions with impressive accuracy in standardized images of prototypical basic emotions. We tested the sensitivity of a well-established automatic facial coding software program to detect spontaneous emotional reactions in individuals responding to emotional pictures. We compared automatically generated scores for valence and arousal of the Facereader (FR; Noldus Information Technology) with the current psychophysiological gold standard of measuring emotional valence (Facial Electromyography, EMG) and arousal (Skin Conductance, SC). We recorded physiological and behavioral measurements of 43 healthy participants while they looked at pleasant, unpleasant, or neutral scenes. When viewing pleasant pictures, FR Valence and EMG were both comparably sensitive. However, for unpleasant pictures, FR Valence showed an expected negative shift, but the signal differentiated not well between responses to neutral and unpleasant stimuli, that were distinguishable with EMG. Furthermore, FR Arousal values had a stronger correlation with self-reported valence than with arousal while SC was sensitive and specifically associated with self-reported arousal. This is the first study to systematically compare FR measurement of spontaneous emotional reactions to standardized emotional images with established psychophysiological measurement tools. This novel technology has yet to make strides to surpass the sensitivity of established psychophysiological measures. However, it provides a promising new measurement technique for non-contact assessment of emotional responses.

Facilitating the Child–Robot Interaction by Endowing the Robot with the Capability of Understanding the Child Engagement: The Case of Mio Amico Robot

Social Robots (SRs) are substantially becoming part of modern society, given their frequent use in many areas of application including education, communication, assistance, and entertainment. The main challenge in human–robot interaction is in achieving human-like and affective interaction between the two groups. This study is aimed at endowing SRs with the capability of assessing the emotional state of the interlocutor, by analyzing his/her psychophysiological signals. The methodology is focused on remote evaluations of the subject’s peripheral neuro-vegetative activity by means of thermal infrared imaging. The approach was developed and tested for a particularly challenging use case: the interaction between children and a commercial educational robot, Mio Amico Robot, produced by LiscianiGiochi©. The emotional state classified from the thermal signal analysis was compared to the emotional state recognized by a facial action coding system. The proposed approach was reliable and accurate and favored a personalized and improved interaction of children with SRs.

Thermal Infrared Imaging to Evaluate Emotional Competences in Nursing Students: A First Approach through a Case Study

During university studies of nursing, it is important to develop emotional skills for their impact on academic performance and the quality of patient care. Thermography is a technology that could be applied during nursing training to evaluate emotional skills. The objective is to evaluate the effect of thermography as the tool for monitoring and improving emotional skills in student nurses through a case study. The student was subjected to different emotions. The stimuli applied were video and music. The process consisted of measuring the facial temperatures during each emotion and stimulus in three phases: acclimatization, stimulus, and response. Thermographic data acquisition was performed with an FLIR E6 camera. The analysis was complemented with the environmental data (temperature and humidity). With the video stimulus, the start and final forehead temperature from testing phases, showed a different behavior between the positive (joy: 34.5 °C-34.5 °C) and negative (anger: 36.1 °C-35.1 °C) emotions during the acclimatization phase, different from the increase experienced in the stimulus (joy: 34.7 °C-35.0 °C and anger: 35.0 °C-35.0 °C) and response phases (joy: 35.0 °C-35.0 °C and anger: 34.8 °C-35.0 °C). With the music stimulus, the emotions showed different patterns in each phase (joy: 34.2 °C-33.9 °C-33.4 °C and anger: 33.8 °C-33.4 °C-33.8 °C). Whenever the subject is exposed to a stimulus, there is a thermal bodily response. All of the facial areas follow a common thermal pattern in response to the stimulus, with the exception of the nose. Thermography is a technique suitable for the stimulation practices in emotional skills, given that it is non-invasive, it is quantifiable, and easy to access.

Thermal Infrared Imaging-Based Affective Computing and Its Application to Facilitate Human Robot Interaction: A Review

Over recent years, robots are increasingly being employed in several aspects of modern society. Among others, social robots have the potential to benefit education, healthcare, and tourism. To achieve this purpose, robots should be able to engage humans, recognize users’ emotions, and to some extent properly react and "behave" in a natural interaction. Most robotics applications primarily use visual information for emotion recognition, which is often based on facial expressions. However, the display of emotional states through facial expression is inherently a voluntary controlled process that is typical of human–human interaction. In fact, humans have not yet learned to use this channel when communicating with a robotic technology. Hence, there is an urgent need to exploit emotion information channels not directly controlled by humans, such as those that can be ascribed to physiological modulations. Thermal infrared imaging-based affective computing has the potential to be the solution to such an issue. It is a validated technology that allows the non-obtrusive monitoring of physiological parameters and from which it might be possible to infer affective states. This review is aimed to outline the advantages and the current research challenges of thermal imaging-based affective computing for human–robot interaction.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

Vascular response to social cognitive performance measured by infrared thermography: A translational study from mouse to man

To assess complex social recognition in mice, we previously developed the SocioBox paradigm. Unexpectedly, 4 weeks after performing in the SocioBox, mice displayed robust social avoidance during Y-maze sociability testing. This unique "sociophobia" acquisition could be documented in independent cohorts. We therefore employed infrared thermography as a non-invasive method of stress-monitoring during SocioBox testing (presentation of five other mice) versus empty box. A higher Centralization Index (body/tail temperature) in the SocioBox correlated negatively with social recognition memory and, after 4 weeks, with social preference in the Y-maze. Assuming that social stimuli might be associated with characteristic thermo-responses, we exposed healthy men (N = 103) with a comparably high intelligence level to a standardized test session including two cognitive tests with or without social component (face versus pattern recognition). In some analogy to the Centralization Index (within-subject measure) used in mice, the Reference Index (ratio nose/malar cheek temperature) was introduced to determine the autonomic facial response/flushing during social recognition testing. Whereas cognitive performance and salivary cortisol were comparable across human subjects and tests, the Face Recognition Test was associated with a characteristic Reference Index profile. Infrared thermography may have potential for discriminating disturbed social behaviors.

Nasal skin temperature reveals changes in arousal levels due to time on task: An experimental thermal infrared imaging study

Infrared thermography, thanks to technological developments and lowering prices, is now getting considerable attention as a potential arousal monitor in the safety industry. Nasal skin temperature might be a valid index to track physiological variations due to reduced arousal levels, and its use could prevent a drowsiness-related deterioration of performance. However, the few studies that have investigated nasal skin temperature in applied settings have had inconsistent results. Here, we assessed the validity of nasal skin temperature to monitor changes in arousal levels (from alertness to drowsiness). The participants performed a 2-h simulated driving task while we simultaneously recorded their nasal skin temperature, brain activity (we used frontal delta electroencephalographic [EEG] activity as the reference index of alertness), and driving performance (speeding time). For those variables, we calculated growth curve models. We also collected subjective ratings of alertness and fatigue before and after the driving session. We found that the nasal skin temperature showed a cubic trajectory (it increased for the first 75 min, and then it began to decrease, but such deceleration gradually diminished over time). As expected, frontal delta EEG activity showed an inverted U-shaped quadratic trend (EEG power increased for the first hour and half, and gradually decreased during the last thirty minutes). The speeding time exhibited a similar pattern of change. Subjective sleepiness and fatigue increased after the task. Overall, our results suggest that nasal skin temperature seems to be a valid measure of arousal variations while performing a complex and dynamic everyday task.

A Modular System for Detection, Tracking and Analysis of Human Faces in Thermal Infrared Recordings

We present a system that utilizes a range of image processing algorithms to allow fully automated thermal face analysis under both laboratory and real-world conditions. We implement methods for face detection, facial landmark detection, face frontalization and analysis, combining all of these into a fully automated workflow. The system is fully modular and allows implementing own additional algorithms for improved performance or specialized tasks. Our suggested pipeline contains a histogtam of oriented gradients support vector machine (HOG-SVM) based face detector and different landmark detecion methods implemented using feature-based active appearance models, deep alignment networks and a deep shape regression network. Face frontalization is achieved by utilizing piecewise affine transformations. For the final analysis, we present an emotion recognition system that utilizes HOG features and a random forest classifier and a respiratory rate analysis module that computes average temperatures from an automatically detected region of interest. Results show that our combined system achieves a performance which is comparable to current stand-alone state-of-the-art methods for thermal face and landmark datection and a classification accuracy of 65.75% for four basic emotions.

The use of infrared thermography to investigate emotions in common marmosets

Measuring body surface temperature changes with infrared thermography has recently been put forward as a non-invasive alternative measure of physiological correlates of emotional reactions. In particular, the nasal region seems to be highly sensitive to emotional reactions. Several studies suggest that nasal temperature is negatively correlated with the level of arousal in humans and other primates, but some studies provide inconsistent results. Our goal was to establish the use of infrared thermography to quantify emotional reactions in common marmosets (Callithrix jacchus), with a focus on the nasal region. To do so we exposed 17 common marmosets to a set of positive, negative and control stimuli (positive: preferred food, playback of food calls; negative: playback of aggressive vocalizations, teasing; control: no stimulus). We compared nasal temperature before and after the stimuli and expected that highly aroused emotional states would lead to a drop in nasal temperature. To validate the thermography measure, we coded piloerection of the tail as an independent measure of arousal and expected a negative correlation between the two measures. Finally, we coded physical activity to exclude its potential confounding impact on nasal temperature. Our results show that all predictions were met: the animals showed a strong decrease in nasal temperature after the presentation of negative arousing stimuli (teasing, playback of aggressive vocalizations). Furthermore, these changes in nasal temperature were correlated with piloerection of the tail and could not be explained by changes in physical activity. In the positive and the control conditions, we found systematic sex differences: in males, the preferred food, the playbacks of food calls, as well as the control stimulus led to an increase in nasal temperature, whereas in females the temperature remained stable (preferred food, control) or decreased (playback of food calls). Based on naturalistic observations that document higher food motivation and competition among female marmosets, as well as stronger reactions to separation from group members in male marmosets, these sex differences corroborate a negative correlation between arousal and nasal temperature. Overall, our results support that measuring nasal temperature by infrared thermography is a promising method to quantify emotional arousal in common marmosets in a fully non-invasive and highly objective way.

Exploratory Investigation of Infrared Thermography for Measuring Gorilla Emotional Responses to Interactions with Familiar Humans

Simple Summary

Interactions between zoo professionals and animals, such as positive reinforcement training, occur regularly and are thought to be enriching for animals. However, there is little empirical information on how animals perceive these interactions or on the interactions’ effects on animals’ emotional states. Our objective was to assess the effectiveness of infrared thermography for measuring the emotional responses of three western lowland gorillas at the Detroit Zoo to routine interactions (positive reinforcement training and cognitive tasks) with familiar humans. In addition to thermal images, we collected saliva samples for hormone analysis before and after human–animal interactions and a control condition, and we recorded behavioral data during all conditions. Nasal temperatures consistently decreased for two gorillas during interactions, while the third gorilla showed repeated increases. The behavior of all three gorillas suggested that they were engaged in the interactions, without exhibiting behaviors that could indicate negative welfare impacts. Oxytocin and cortisol both decreased following all conditions, including the control, and were thus equivocal for interpreting the meaning of the changes in nasal temperature. As mixed results in previous research show, infrared thermography may detect emotional arousal; however, additional indicators are necessary to determine the valence of the observed changes. The variability in responses we observed do not lend themselves to making firm conclusions about the validity of infrared thermography (IRT) for measuring emotion in this context or about how these gorillas responded to interactions. Challenges and suggestions for future studies using infrared thermography to examine interactions between humans and zoo animals are discussed.

Abstract

Interactions between zoo professionals and animals occur regularly and are believed to be enriching for animals. Little empirical information exists on how animals perceive these interactions, and particularly how the interactions affect the emotional states of animals. Infrared thermography (IRT) has shown some promise in the assessment of emotions in a variety of species, but further research is needed to determine if this method is useful in a zoo setting. We conducted a pilot study to determine if IRT is a valid measure of the emotional responses to routine interactions (positive reinforcement training and cognitive tasks, compared to a control condition) with familiar humans on three western lowland gorillas at the Detroit Zoo. We measured nasal temperatures associated with emotional change using IRT. To examine the validity of the IRT data, we collected saliva samples for hormone analysis before and after each condition, in addition to behavioral data during the interactions and control condition. Decreases in nasal temperatures for two gorillas and an increase in the third indicate that arousal changed consistently within individuals following the interactions but not the control condition. Pre-post cortisol levels and oxytocin concentrations decreased for all conditions, but the decreases seen did not differ among the conditions. The gorillas were highly engaged in the interactions, and two produced more grumble vocalizations during the human-animal interactions (HAIs) compared to the control condition. Additionally, the gorillas performed self-directed behaviors more often during the control condition, also suggesting HAIs were not a negative experience. In summary, we were able to measure changes in arousal using IRT, but we were unable to determine the emotional valence of these changes based on the additional indicators employed. Additionally, the inconsistency across these measures precluded firm conclusions about either the validity of IRT for measuring emotion in this context or how the interactions impacted these gorillas. These findings highlight the challenges of using IRT to measure emotional states in non-human animals, and we discuss further steps necessary to apply this method in future studies.

Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction

Child-Robot Interaction (CRI) has become increasingly addressed in research and applications. This work proposes a system for emotion recognition in children, recording facial images by both visual (RGB-red, green and blue) and Infrared Thermal Imaging (IRTI) cameras. For this purpose, the Viola-Jones algorithm is used on color images to detect facial regions of interest (ROIs), which are transferred to the thermal camera plane by multiplying a homography matrix obtained through the calibration process of the camera system. As a novelty, we propose to compute the error probability for each ROI located over thermal images, using a reference frame manually marked by a trained expert, in order to choose that ROI better placed according to the expert criteria. Then, this selected ROI is used to relocate the other ROIs, increasing the concordance with respect to the reference manual annotations. Afterwards, other methods for feature extraction, dimensionality reduction through Principal Component Analysis (PCA) and pattern classification by Linear Discriminant Analysis (LDA) are applied to infer emotions. The results show that our approach for ROI locations may track facial landmarks with significant low errors with respect to the traditional Viola-Jones algorithm. These ROIs have shown to be relevant for recognition of five emotions, specifically disgust, fear, happiness, sadness, and surprise, with our recognition system based on PCA and LDA achieving mean accuracy (ACC) and Kappa values of 85.75% and 81.84%, respectively. As a second stage, the proposed recognition system was trained with a dataset of thermal images, collected on 28 typically developing children, in order to infer one of five basic emotions (disgust, fear, happiness, sadness, and surprise) during a child-robot interaction. The results show that our system can be integrated to a social robot to infer child emotions during a child-robot interaction.

Emotion analysis in children through facial emissivity of infrared thermal imaging

Physiological signals may be used as objective markers to identify emotions, which play relevant roles in social and daily life. To measure these signals, the use of contact-free techniques, such as Infrared Thermal Imaging (IRTI), is indispensable to individuals who have sensory sensitivity. The goal of this study is to propose an experimental design to analyze five emotions (disgust, fear, happiness, sadness and surprise) from facial thermal images of typically developing (TD) children aged 7–11 years using emissivity variation, as recorded by IRTI. For the emotion analysis, a dataset considered emotional dimensions (valence and arousal), facial bilateral sides and emotion classification accuracy. The results evidence the efficiency of the experimental design with interesting findings, such as the correlation between the valence and the thermal decrement in nose; disgust and happiness as potent triggers of facial emissivity variations; and significant emissivity variations in nose, cheeks and periorbital regions associated with different emotions. Moreover, facial thermal asymmetry was revealed with a distinct thermal tendency in the cheeks, and classification accuracy reached a mean value greater than 85%. From the results, the emissivity variations were an efficient marker to analyze emotions in facial thermal images, and IRTI was confirmed to be an outstanding technique to study emotions. This study contributes a robust dataset to analyze the emotions of 7-11-year-old TD children, an age range for which there is a gap in the literature.

Network physiology of 'fight or flight' response in facial superficial blood vessels

OBJECTIVE

We introduced a novel framework to identify the dynamic pattern of blood flow changes in the cutaneous superficial blood vessels of the face for 'fight or flight' responses through facial thermal imaging.

APPROACH

For this purpose, a thermal dataset was collected from 41 subjects in a mock crime scenario. Five facial areas including periorbital, forehead, perinasal, cheek and chin were selected on the face. Due to the cause and effect movement of blood in the facial cutaneous vasculature, the effective connectivity approach and graph analysis were used to extract causality features. The effective connectivity was quantified using a modified version of the multivariate Granger causality (GC) method among each pair of facial region of interests.

MAIN RESULTS

Validation was performed using statistical analysis, and the results demonstrated that the proposed method was statistically significant in detecting the physiological pattern of deceptive anxiety on the face. Moreover, the obtained graph is visualized by different schemes to show these interactions more effectively. We used machine learning techniques to classify our data based on the GC values, which result in a greater than 87% accuracy rate in discriminating between deceptive and truthful subjects.

Physiological and behavioral reactivity to social exclusion: a functional infrared thermal imaging study in patients with psoriasis

Recent studies have shown that sympathetic nervous system (SNS) activity can be heavily impacted not only by basic threats to survival but also by threats to social bonds. In this study we explored the behavioral and physiological consequences of social exclusion/inclusion in patients with psoriasis, a disease frequently associated with the experience of being ostracized and with deficient emotion regulation skills. We employed a virtual ball-tossing game (Cyberball) to induce the experience of social exclusion/inclusion. We then used a Trust Game to measure the effects of this social modulation on trust. During Cyberball, infrared thermal imaging was used to record participants' facial temperature and thus obtain an online measure of SNS activation. Behavioral data showed that social exclusion shifted participants' trust toward unfamiliar players who had not previously excluded them. Physiological data indicated that in control participants, social exclusion triggered higher SNS activation than inclusion. No such effect was found in patients with psoriasis, whose SNS activity was the same during inclusion as it was during exclusion, suggesting that they benefit less from inclusive experiences than control participants. In addition, in patients but not in controls, higher SNS activation during social exclusion was linked to higher monetary investment toward unfamiliar players, a result in keeping with the social reconnection hypothesis, according to which emotions triggered by social rejection can be regulated by investing in new social interactions. We also found that an increase in periorbital temperature is accompanied by a decrease in happiness ratings after social exclusion was experienced during the Cyberball game. NEW & NOTEWORTHY Previous research on emotional processes in psoriasis has mainly employed self-report measures. In this study we used thermal imaging to obtain an online measure of the sympathetic nervous system (SNS) activity during social exclusion and tested how this experience influenced subsequent trust. We found that being included was a less positive experience for patients compared with controls and that SNS activity during exclusion had a stronger influence on subsequent trust in patients than in controls.

Cognitive load and emotional processing in psoriasis: a thermal imaging study

Psoriasis is a chronic dermatologic disease which is frequently associated with psychological distress. Although studies suggest a relationship between this condition and difficulties in emotion regulation, behavioral and physiological evidence about this link is scarce. We measured implicit emotion regulation abilities of psoriasis patients and a healthy control group by examining the impact of distracting emotional (positive, negative or neutral) images on a working memory task ("Emotional N-Back") which could present high (2-back) or low (1-back) cognitive workload. Moreover, we used Functional Infrared Thermal Imaging to record participants' facial temperature and obtain a measure of the activation of the autonomic system. Rising of temperature over the peri-orbital areas and the nose tip are believed to reflect the activation and the de-activation of the sympathetic system, respectively. Patients scored higher than controls on the "Lack of emotional clarity" sub-scale of the Difficulties in Emotion Regulation Scale. Compared to controls, who performed much better in the low vs. high cognitive load condition, patients showed a smaller accuracy difference between the two conditions. Moreover, patients showed less sympathetic (lower peri-orbital and higher nasal tip temperature) activity (especially in the negative and neutral blocks) during the high vs. low cognitive load condition, suggesting that the former condition might be less emotionally demanding for them. Patients benefit more than controls from the load-dependent interference effect when dealing with emotional information; thus, therapeutic techniques aiming at teaching how to use cognitive strategies to downregulate emotions might be particularly appropriated for them.