1
|
Kip A, Erle TM, van Beest I. Facial Temperature Responses to Ostracism in Women: Exploring Nasal Thermal Signatures of Different Coping Behaviors. Psychophysiology 2025; 62:e70081. [PMID: 40485184 DOI: 10.1111/psyp.70081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2024] [Revised: 05/10/2025] [Accepted: 05/20/2025] [Indexed: 06/11/2025]
Abstract
Ostracism (feeling ignored and excluded) triggers psychophysiological responses associated with distress. We investigated different coping responses after ostracism and explored whether these were preceded by unique facial thermal signatures, reflecting autonomic nervous system (ANS) activity. Using thermal infrared imaging, we recorded facial cutaneous temperature variations in female participants (N = 95) experiencing inclusion and ostracism using hypothetical Cyberball games. Coping after ostracism was assessed during a hypothetical Allocation Game, where participants could do nothing (withdrawal), reduce (antisocial), or increase (prosocial) the hypothetical earnings of their ostracizer. Contrary to expectations, most participants chose to withdraw (52%), with fewer opting for antisocial responses (30%) or prosocial responses (18%) after ostracism. Results from linear mixed-effects modeling revealed that substantial temperature variability occurred only in the nose region of the face. Both ostracism and inclusion showed a decrease in nasal temperature relative to baseline, but the average drop was greater during inclusion, suggesting stronger ANS activation during inclusion rather than ostracism. Crucially, exploratory findings showed that only participants who responded antisocially after ostracism exhibited steeper decreases in nasal temperature during ostracism compared to inclusion. This pattern suggests greater physiological reactivity among antisocial responders, particularly in contrast to those who chose to withdraw. Future research should integrate thermal imaging with other physiological measures and strengthen ostracism manipulations to understand the relationship between thermal responses and different coping behaviors.
Collapse
Affiliation(s)
- Anneloes Kip
- Department of Social Psychology, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, the Netherlands
- Department of Methodology and Statistics, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Thorsten M Erle
- Department of Social Psychology, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, the Netherlands
| | - Ilja van Beest
- Department of Social Psychology, Tilburg School of Social and Behavioral Sciences, Tilburg University, Tilburg, the Netherlands
| |
Collapse
|
2
|
Gioia F, Greco A, Callara AL, Vanello N, Scilingo EP, Citi L. ThermICA: Novel Approach for a Multivariate Analysis of Facial Thermal Responses. IEEE Trans Biomed Eng 2025; 72:1237-1247. [PMID: 39453803 DOI: 10.1109/tbme.2024.3486628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2024]
Abstract
OBJECTIVE Infrared Thermography (IRT) has been used to monitor skin temperature variation in a contactless manner, in both clinical medicine and psychophysiology. Here, we introduce a new methodology to obtain information about autonomic correlates related to perspiration, peripheral vasomotility, and respiration from infrared recordings. METHODS Our approach involves a model-based decomposition of facial thermograms using Independent Component Analysis (ICA) and an ad-hoc preprocessing procedure. We tested our approach on 30 healthy volunteers whose psychophysiological state was stimulated as part of an experimental protocol. RESULTS Within-subject ICA analysis identified three independent components demonstrating correlations with the reference physiological signals. Moreover, a linear combination of independent components effectively predicted each physiological signal, achieving median correlations of 0.9 for electrodermal activity, 0.8 for respiration, and 0.73 for photoplethysmography peaks envelope. In addition, we performed a cross-validated inter-subject analysis, which allows to predict physiological signals from facial thermograms of unseen subjects. CONCLUSIONS/SIGNIFICANCE Our findings validate the efficacy of features extracted from both original and thermal-derived signals for differentiating experimental conditions. This outcome emphasizes the sensitivity and promise of our approach, advocating for expanded investigations into thermal imaging within biomedical signal analysis. It underscores its potential for enhancing objective assessments of emotional states.
Collapse
|
3
|
Nazzari S, Morgese Zangrandi M, Bottini G, Salvato G, Provenzi L. "Hot stuff": Behavioural and affective thermal responses to digital and non-digital disruptions during early mother-infant interaction. Biol Psychol 2025; 196:109027. [PMID: 40204085 DOI: 10.1016/j.biopsycho.2025.109027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2024] [Revised: 03/27/2025] [Accepted: 04/04/2025] [Indexed: 04/11/2025]
Abstract
Responsive social exchanges are critical for infants' bio-behavioural regulation and healthy development. Parental smartphone use may disrupt early parent-infant interactions, leading to frequent interruptions. The impact of these interruptions on mother-infant behaviours and stress physiology is unclear. Infrared thermal imaging (ITI) offers a non-invasive approach to assess stress-related skin temperature changes reflecting autonomic activation. This study investigates mother-infant behavioural and thermal affective responses to parental digital and non-digital distractions. Thirty-eight mother-infant dyads (22 males) participated in a modified Still-Face Paradigm with five conditions: Free Play, Technoference Exposure (TF-E), Technoference Reunion (TF-R), Paperference Exposure (PF-E), Paperference Reunion (PF-R). During TF-E and PF-E mothers completed questionnaires on a smartphone or paper and were unresponsive to the infant. Mother-infant behaviours were coded microanalytically, while FLIR cameras detected changes in forehead and nasal tip temperatures. Maternal habitual smartphone use was assessed by self-report and passive sensing. Infants showed increased behavioural distress during TF-E and PF-E. ITI revealed lower infant forehead temperatures during TF-E compared to free play and reunions, while no significant changes at the nasal tip. Maternal forehead temperature dropped significantly during PF-E, compared to other episodes. Greater maternal habitual phone use was linked to infant responses and maternal behaviours. Parental unresponsiveness due to digital and non-digital distractions leads to infant behavioural distress, with digital disruptions also triggering a distinct thermal affective response. Findings highlight the impact of different parental distractions on early interactions and stress responding, with potential long-term implications.
Collapse
Affiliation(s)
- S Nazzari
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy.
| | | | - G Bottini
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy; Cognitive Neuropsychology Centre, ASST GOM Niguarda, Milano, Italy; Neuromi Milano Centre for Neuroscience, Milano, Italy
| | - G Salvato
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy; Cognitive Neuropsychology Centre, ASST GOM Niguarda, Milano, Italy; Neuromi Milano Centre for Neuroscience, Milano, Italy
| | - L Provenzi
- Department of Brain and Behavioral Sciences, University of Pavia, Pavia, Italy; Developmental Psychobiology Lab, IRCCS Mondino Foundation, Pavia, Italy
| |
Collapse
|
4
|
Pop-Jordanova N. Is it Possible to Recognize Emotions/Feelings? Pril (Makedon Akad Nauk Umet Odd Med Nauki) 2025; 46:15-26. [PMID: 40116144 DOI: 10.2478/prilozi-2025-0002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/23/2025]
Abstract
It is true that emotions are the most important psychological functions in humans and are on the top of the hierarchical tree of the homeostasis. As a potent mental activity, emotions significantly influence various aspects of human functioning such as decision-making, perception, attention and memory. For scientific but also for practical reasons, the main question is: how to register and measure emotions? The aim of this article is to summarize the main known methods for measuring the emotions/feelings published in Medline basis. Different methodologies are presented in order to measure emotions such as: Face recognition, Bodily maps of emotions, Evaluation of autonomic nervous system activity, EEG, and especially fMRI and PET scan. Data are used from PubMed bases.
Collapse
|
5
|
Vabba A, Scattolin M, Porciello G, Panasiti MS, Aglioti SM. From feeling chilly to burning up: How thermal signals shape the physiological state of the body and impact physical, emotional, and social well-being. Neurosci Biobehav Rev 2025; 169:105992. [PMID: 39793684 DOI: 10.1016/j.neubiorev.2024.105992] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2024] [Revised: 12/01/2024] [Accepted: 12/19/2024] [Indexed: 01/13/2025]
Abstract
This review examines the role of thermal interoception-awareness of body temperature-in shaping physiological states and its impact on physical, emotional, and social well-being. We describe the neural pathways and mechanisms involved in thermal processing and environmental heat exchange and how thermal interoception influences both autonomic and behavioral responses, contributing to survival and homeostasis. Additionally, the review highlights the significance of thermal signals and thermoregulation in determining higher-order cognitive and emotional functions, such as regulating sleep patterns and activity levels, monitoring physical well-being, regulating emotions, and even social interactions. We describe the existing instruments for assessing body temperature and thermal awareness and call for further research to investigate the role of thermal interoception in emotional and social domains. We also discuss potential therapeutic applications, particularly in the context of psychosomatic and neurological disorders characterized by emotional dysregulation, disrupted sleep and mood patterns, social difficulties, and alterations in self-consciousness.
Collapse
Affiliation(s)
- Alisha Vabba
- Sapienza University of Rome and CLN2S@Sapienza, Istituto Italiano di Tecnologia, Viale Regina Elena, 291, Rome 00161, Italy; IRCCS, Santa Lucia Foundation, Rome 00142, Italy.
| | - Marina Scattolin
- Sapienza University of Rome and CLN2S@Sapienza, Istituto Italiano di Tecnologia, Viale Regina Elena, 291, Rome 00161, Italy; IRCCS, Santa Lucia Foundation, Rome 00142, Italy; Department of Psychology, Sapienza University of Rome, Rome 00185, Italy
| | - Giuseppina Porciello
- IRCCS, Santa Lucia Foundation, Rome 00142, Italy; Department of Psychology, Sapienza University of Rome, Rome 00185, Italy
| | - Maria Serena Panasiti
- IRCCS, Santa Lucia Foundation, Rome 00142, Italy; Department of Psychology, Sapienza University of Rome, Rome 00185, Italy
| | - Salvatore Maria Aglioti
- Sapienza University of Rome and CLN2S@Sapienza, Istituto Italiano di Tecnologia, Viale Regina Elena, 291, Rome 00161, Italy; IRCCS, Santa Lucia Foundation, Rome 00142, Italy
| |
Collapse
|
6
|
Bekhit R, Reimert I. A Complete Pipeline to Extract Temperature from Thermal Images of Pigs. SENSORS (BASEL, SWITZERLAND) 2025; 25:643. [PMID: 39943283 PMCID: PMC11821139 DOI: 10.3390/s25030643] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/05/2024] [Revised: 01/17/2025] [Accepted: 01/20/2025] [Indexed: 02/16/2025]
Abstract
Using deep learning or artificial intelligence (AI) in research with animals is a new interdisciplinary area of research. In this study, we have explored the potential of thermal imaging and AI in pig research. Thermal cameras play a vital role in obtaining and collecting a large amount of data, and AI has the capabilities of processing and extracting valuable information from these data. The amount of data collected using thermal imaging is huge, and automation techniques are therefore crucial to find a meaningful interpretation of the changes in temperature. In this paper, we present a complete pipeline to extract temperature automatically from a selected Region of Interest (ROI). This system consists of three stages: the first one checks whether the ROI is completely visible to observe the thermal temperature, and then the second stage uses an encoder-decoder structure of a convolution neural network to segment the ROI, if the condition was met at stage one. In the last stage, the maximum temperature is extracted and saved in an external file. The segmentation model showed good performance, with a mean Pixel Class accuracy of 92.3%, and a mean Intersection over Union of 87.1%. The extracted temperature observed by the model entirely matched the manually observed temperature. The system showed reliable results to be used independently without human intervention to determine the temperature in the selected ROI in pigs.
Collapse
|
7
|
Aghamalizadeh A, Mazloumi A, Nikabadi A, Nahvi A, Khanehshenas F, Ebrahimian S. Driver drowsiness is associated with altered facial thermal patterns: Machine learning insights from a thermal imaging approach. Physiol Behav 2024; 283:114619. [PMID: 38917929 DOI: 10.1016/j.physbeh.2024.114619] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2024] [Revised: 06/21/2024] [Accepted: 06/22/2024] [Indexed: 06/27/2024]
Abstract
Driver drowsiness is a significant factor in road accidents. Thermal imaging has emerged as an effective tool for detecting drowsiness by enabling the analysis of facial thermal patterns. However, it is not clear which facial areas are most affected and correlate most strongly with drowsiness. This study examines the variations and importance of various facial areas and proposes an approach for detecting driver drowsiness. Twenty participants underwent tests in a driving simulator, and temperature changes in various facial regions were measured. The random forest method was employed to evaluate the importance of each facial region. The results revealed that temperature changes in the nasal area exhibited the highest value, while the eyes had the most correlated changes with drowsiness. Furthermore, drowsiness was classified with an accuracy of 88 % utilizing thermal variations in the facial region identified as the most important regions by the random forest feature importance model. These findings provide a comprehensive overview of facial thermal imaging for detecting driver drowsiness and introduce eye temperature as a novel and effective measure for investigating cognitive activities.
Collapse
Affiliation(s)
- Alireza Aghamalizadeh
- Department of Occupational Health, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran
| | - Adel Mazloumi
- Department of Occupational Health, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran; School of Data Science, Nagoya City University, Nagoya, Japan.
| | - Ahmad Nikabadi
- Computer Engineering Department, Amirkabir University of Technology, Tehran, Iran
| | - Ali Nahvi
- Virtual Reality Laboratory, K. N. Toosi University of Technology, Tehran 19697-6449, Iran
| | - Farin Khanehshenas
- Department of Ergonomics, School of Public Health, Iran University of Medical Sciences, Tehran, Iran
| | - Serajeddin Ebrahimian
- Virtual Reality Laboratory, K. N. Toosi University of Technology, Tehran 19697-6449, Iran; Department of Technical Physics, University of Eastern Finland, 70210 Kuopio, Finland
| |
Collapse
|
8
|
Talala S, Shvimmer S, Simhon R, Gilead M, Yitzhaky Y. Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning. SENSORS (BASEL, SWITZERLAND) 2024; 24:2620. [PMID: 38676235 PMCID: PMC11053953 DOI: 10.3390/s24082620] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/12/2024] [Revised: 04/16/2024] [Accepted: 04/17/2024] [Indexed: 04/28/2024]
Abstract
Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.
Collapse
Affiliation(s)
- Shlomi Talala
- Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel; (S.T.)
| | - Shaul Shvimmer
- Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel; (S.T.)
| | - Rotem Simhon
- School of Psychology, Tel Aviv University, Tel Aviv 39040, Israel
| | - Michael Gilead
- School of Psychology, Tel Aviv University, Tel Aviv 39040, Israel
| | - Yitzhak Yitzhaky
- Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel; (S.T.)
| |
Collapse
|
9
|
Fernández J, Albayay J, Gálvez-García G, Iborra O, Huertas C, Gómez-Milán E, Caballo VE. Facial infrared thermography as an index of social anxiety. ANXIETY, STRESS, AND COPING 2024; 37:114-126. [PMID: 37029987 DOI: 10.1080/10615806.2023.2199209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Accepted: 03/29/2023] [Indexed: 04/09/2023]
Abstract
Previous research on physiological indices of social anxiety has offered unclear results. In this study, participants with low and high social anxiety performed five social interaction tasks while being recorded with a thermal camera. Each task was associated with a dimension assessed by the Social Anxiety Questionnaire for Adults (1 = Interactions with strangers. 2 = Speaking in public/Talking with people in authority, 3 = Criticism and embarrassment, 4 = Assertive expression of annoyance, disgust or displeasure, 5 = Interactions with the opposite sex). Mixed-effects models revealed that the temperature of the tip of the nose decreased significantly in participants with low (vs. high) social anxiety (p < 0.001), while no significant differences were found in other facial regions of interest: forehead (p = 0.999) and cheeks (p = 0.999). Furthermore, task 1 was the most effective at discriminating between the thermal change of the nose tip and social anxiety, with a trend for a higher nose temperature in participants with high social anxiety and a lower nose temperature for the low social anxiety group. We emphasize the importance of corroborating thermography with specific tasks as an ecological method, and tip of the nose thermal change as a psychophysiological index associated with social anxiety.
Collapse
Affiliation(s)
- Jesús Fernández
- Centro de Investigación Mente, Cerebro y Comportamiento, Universidad de Granada, Granada, Spain
| | - Javier Albayay
- Centro Interdipartimentale Mente/Cervello, Università degli Studi di Trento, Rovereto, Italy
| | - Germán Gálvez-García
- Departamento de Psicología, Universidad de La Frontera, Temuco, Chile
- Departamento de Psicología Básica, Psicobiología y Metodología de las Ciencias del Comportamiento, Facultad de Psicología, Universidad de Salamanca, Salamanca, Spain
| | - Oscar Iborra
- Centro de Investigación Mente, Cerebro y Comportamiento, Universidad de Granada, Granada, Spain
| | - Carmen Huertas
- Centro de Investigación Mente, Cerebro y Comportamiento, Universidad de Granada, Granada, Spain
| | - Emilio Gómez-Milán
- Centro de Investigación Mente, Cerebro y Comportamiento, Universidad de Granada, Granada, Spain
| | - Vicente E Caballo
- Centro de Investigación Mente, Cerebro y Comportamiento, Universidad de Granada, Granada, Spain
| |
Collapse
|
10
|
Ishikawa N, Asahina M, Umeda S. Reactivity of observers' facial skin blood flow depending on others' facial expressions and blushing. Front Psychol 2023; 14:1259928. [PMID: 38130969 PMCID: PMC10733524 DOI: 10.3389/fpsyg.2023.1259928] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Accepted: 11/13/2023] [Indexed: 12/23/2023] Open
Abstract
Facial skin blood flow (SkBF) has attracted attention as an autonomic indicator because it influences facial colour, which informs others of emotional states, and facial temperature related to social anxiety. Previous studies have examined the facial SkBF in people experiencing emotions; however, facial SkBF changes in the observers of emotions are poorly understood. Our study clarified facial SkBF changes related to observing others' emotions by comparing the changes with other physiological indices. Thirty healthy participants (24 females; mean age: 22.17) observed six types of facial expressions (neutral, angry, and embarrassed expressions with and without facial blushing) and rated the emotional intensity of the other person. We measured their facial SkBF, finger SkBF, and cardiac RR interval as they made their observations. Facial SkBF generally decreased in relation to observing emotional faces (angry and embarrassed faces) and significantly decreased for angry expressions with blushing. None of the participants noticed blushing of facial stimuli. For the RR interval and finger SkBF, there was no variation depending on the observed facial expressions, although there was a general increase related to observation. These results indicated that facial SkBF is sensitive and reactive to emotional faces-especially angry faces with blushing- compared with other autonomic indices. The facial SkBF changes were not related to either RR interval changes or the intensity rating, suggesting that facial SkBF changes may be caused by vasoconstriction and have potential functions for our emotions. The decrease in facial SkBF may have a role in calming observers by preventing them from adopting the same emotional state as a person with intense anger. These findings clarify daily facial SkBF fluctuations and their relationship with our emotional processing in interpersonal situations.
Collapse
Affiliation(s)
- Naoki Ishikawa
- Graduate School of Human Relations, Keio University, Tokyo, Japan
- Research Fellow of Japan Society for the Promotion of Science, Tokyo, Japan
| | - Masato Asahina
- Research Fellow of Japan Society for the Promotion of Science, Tokyo, Japan
- Department of Neurology, Kanazawa Medical University, Ishikawa, Japan
| | - Satoshi Umeda
- Department of Neurology, Kanazawa Medical University, Ishikawa, Japan
| |
Collapse
|
11
|
Huang X, Zheng D, Liu C, Huang J, Chen X, Zhong J, Wang J, Lin X, Zhao C, Chen M, Su S, Chen Y, Xu C, Lin C, Huang Y, Zhang S. miR-214 could promote myocardial fibrosis and cardiac mesenchymal transition in VMC mice through regulation of the p53 or PTEN-PI3K-Akt signali pathway, promoting CF proliferation and inhibiting its ng pathway. Int Immunopharmacol 2023; 124:110765. [PMID: 37647681 DOI: 10.1016/j.intimp.2023.110765] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 07/25/2023] [Accepted: 08/02/2023] [Indexed: 09/01/2023]
Abstract
INTRODUCTION This study aimed to investigate the role of miR-214 in the bidirectional regulation of p53 and PTEN and its influence on myocardial fibrosis and cardiac mesenchymal transformation in mice with viral myocarditis (VMC). METHODS The study established a VMC model in BALB/c mice by injecting them with the CVB3 virus intraperitoneally. Techniques such as ELISA, H&E staining, Masson staining, immunohistochemical staining, RT-qPCR, western blot, and dual-luciferase reporter gene assay were used to detect the expression levels of relevant factors in tissues and cells. Isolation and culture of cardiac fibroblasts (CFs) were also conducted. RESULTS The study found that miR-214 bidirectional regulation of p53 and PTEN promotes myocardial fibrosis and cardiac mesenchymal transformation in mice with VMC. The expression levels of collagen-related peptides, inflammatory-related factors, miR-214, mesenchymal transformation-related factors, and fibrosis-related factors were significantly increased, while the expression levels of p53, PTEN, and epithelial/endothelial cell phenotype marker factors were significantly decreased. Downregulation of miR-214 or upregulation of p53 and PTEN expression inhibited inflammatory cell and fibroblast infiltration in VMC mouse myocardial tissue. It reduced the proliferation ability while increasing the apoptosis of cardiac fibroblasts. CONCLUSION miR-214 plays a significant role in the bidirectional inhibition of p53 and PTEN, which leads to myocardial fibrosis and cardiac mesenchymal transformation in mice with VMC. Downregulation of miR-214 or upregulation of p53 and PTEN expression may provide potential therapeutic targets for treating VMC-induced cardiac fibrosis and mesenchymal transformation.
Collapse
Affiliation(s)
- Xianggui Huang
- Department of Pediatrics, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China; Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China
| | - Danling Zheng
- Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China; Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Chong Liu
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Jianxiang Huang
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China; College of Pharmacy, Jinan University, Guangzhou 510220, PR China
| | - Xiaoshan Chen
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Jialin Zhong
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Jing Wang
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Xinyue Lin
- Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China; Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Chengkuan Zhao
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Meini Chen
- Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China; Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Siman Su
- Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China; Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Yun Chen
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Chengcheng Xu
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China
| | - Chaoxian Lin
- Department of Pharmacology, Shantou University Medical College, Shantou 515041, PR China; Department of Pharmacy, Shantou Chaonan Minsheng Hospital, Shantou 515000, PR China
| | - Yihui Huang
- Department of Pediatrics, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China.
| | - Shuyao Zhang
- Department of Pharmacy, Guangzhou Red Cross Hospital, Jinan University, Guangzhou 510220, PR China.
| |
Collapse
|
12
|
Russo S, Lorusso L, D’Onofrio G, Ciccone F, Tritto M, Nocco S, Cardone D, Perpetuini D, Lombardo M, Lombardo D, Sancarlo D, Greco A, Merla A, Giuliani F. Assessing Feasibility of Cognitive Impairment Testing Using Social Robotic Technology Augmented with Affective Computing and Emotional State Detection Systems. Biomimetics (Basel) 2023; 8:475. [PMID: 37887606 PMCID: PMC10604561 DOI: 10.3390/biomimetics8060475] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Revised: 09/26/2023] [Accepted: 09/27/2023] [Indexed: 10/28/2023] Open
Abstract
Social robots represent a valid opportunity to manage the diagnosis, treatment, care, and support of older people with dementia. The aim of this study is to validate the Mini-Mental State Examination (MMSE) test administered by the Pepper robot equipped with systems to detect psychophysical and emotional states in older patients. Our main result is that the Pepper robot is capable of administering the MMSE and that cognitive status is not a determinant in the effective use of a social robot. People with mild cognitive impairment appreciate the robot, as it interacts with them. Acceptability does not relate strictly to the user experience, but the willingness to interact with the robot is an important variable for engagement. We demonstrate the feasibility of a novel approach that, in the future, could lead to more natural human-machine interaction when delivering cognitive tests with the aid of a social robot and a Computational Psychophysiology Module (CPM).
Collapse
Affiliation(s)
- Sergio Russo
- Research & Innovation Unit, Foundation IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy;
| | - Letizia Lorusso
- Research & Innovation Unit, Foundation IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy;
- Interdisciplinary Department of Medicine, School of Medical Statistics and Biometry, University of Bari Aldo Moro, 70124 Bari, Italy
| | - Grazia D’Onofrio
- Clinical Psychology Service, Health Department, IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy; (G.D.); (F.C.)
| | - Filomena Ciccone
- Clinical Psychology Service, Health Department, IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy; (G.D.); (F.C.)
| | - Michele Tritto
- Next2U Srl, Via dei Peligni 137, 65127 Pescara, Italy; (M.T.); (S.N.)
| | - Sergio Nocco
- Next2U Srl, Via dei Peligni 137, 65127 Pescara, Italy; (M.T.); (S.N.)
| | - Daniela Cardone
- Department of Engineering and Geology, University G. D’Annunzio of Chieti-Pescara, 65127 Pescara, Italy; (D.C.); (D.P.); (A.M.)
| | - David Perpetuini
- Department of Engineering and Geology, University G. D’Annunzio of Chieti-Pescara, 65127 Pescara, Italy; (D.C.); (D.P.); (A.M.)
| | - Marco Lombardo
- Behaviour Labs S.r.l.s. Piazza Gen. di Brigata Luigi Sapienza 22, 95030 Sant’Agata Li Battiati, Italy (D.L.)
| | - Daniele Lombardo
- Behaviour Labs S.r.l.s. Piazza Gen. di Brigata Luigi Sapienza 22, 95030 Sant’Agata Li Battiati, Italy (D.L.)
| | - Daniele Sancarlo
- Geriatrics Unit, Foundation IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy; (D.S.); (A.G.)
| | - Antonio Greco
- Geriatrics Unit, Foundation IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy; (D.S.); (A.G.)
| | - Arcangelo Merla
- Department of Engineering and Geology, University G. D’Annunzio of Chieti-Pescara, 65127 Pescara, Italy; (D.C.); (D.P.); (A.M.)
| | - Francesco Giuliani
- Research & Innovation Unit, Foundation IRCCS Casa Sollievo della Sofferenza, 71013 San Giovanni Rotondo, Italy;
| |
Collapse
|
13
|
Li L, Tang W, Yang H, Xue C. Classification of User Emotional Experiences on B2C Websites Utilizing Infrared Thermal Imaging. SENSORS (BASEL, SWITZERLAND) 2023; 23:7991. [PMID: 37766045 PMCID: PMC10534612 DOI: 10.3390/s23187991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Revised: 09/18/2023] [Accepted: 09/18/2023] [Indexed: 09/29/2023]
Abstract
The acquisition of physiological signals for analyzing emotional experiences has been intrusive, and potentially yields inaccurate results. This study employed infrared thermal images (IRTIs), a noninvasive technique, to classify user emotional experiences while interacting with business-to-consumer (B2C) websites. By manipulating the usability and aesthetics of B2C websites, the facial thermal images of 24 participants were captured as they engaged with the different websites. Machine learning techniques were leveraged to classify their emotional experiences, with participants' self-assessments serving as the ground truth. The findings revealed significant fluctuations in emotional valence, while the participants' arousal levels remained consistent, enabling the categorization of emotional experiences into positive and negative states. The support vector machine (SVM) model performed well in distinguishing between baseline and emotional experiences. Furthermore, this study identified key regions of interest (ROIs) and effective classification features in machine learning. These findings not only established a significant connection between user emotional experiences and IRTIs but also broadened the research perspective on the utility of IRTIs in the field of emotion analysis.
Collapse
Affiliation(s)
- Lanxin Li
- School of Mechanical Engineering, Southeast University, 2 Southeast University Road, Nanjing 211189, China; (L.L.); (W.T.)
| | - Wenzhe Tang
- School of Mechanical Engineering, Southeast University, 2 Southeast University Road, Nanjing 211189, China; (L.L.); (W.T.)
| | - Han Yang
- School of Instrument Science and Engineering, Southeast University, 2 Southeast University Road, Nanjing 211189, China;
| | - Chengqi Xue
- School of Mechanical Engineering, Southeast University, 2 Southeast University Road, Nanjing 211189, China; (L.L.); (W.T.)
| |
Collapse
|
14
|
Aristizabal-Tique VH, Henao-Pérez M, López-Medina DC, Zambrano-Cruz R, Díaz-Londoño G. Facial thermal and blood perfusion patterns of human emotions: Proof-of-Concept. J Therm Biol 2023; 112:103464. [PMID: 36796909 DOI: 10.1016/j.jtherbio.2023.103464] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2021] [Revised: 12/07/2022] [Accepted: 12/22/2022] [Indexed: 01/11/2023]
Abstract
In this work, a preliminary study of proof-of-concept was conducted to evaluate the performance of the thermographic and blood perfusion data when emotions of positive and negative valence are applied, where the blood perfusion data are obtained from the thermographic data. The images were obtained for baseline, positive, and negative valence according to the protocol of the Geneva Affective Picture Database. Absolute and percentage differences of average values of the data between the valences and the baseline were calculated for different regions of interest (forehead, periorbital eyes, cheeks, nose and upper lips). For negative valence, a decrease in temperature and blood perfusion was observed in the regions of interest, and the effect was greater on the left side than on the right side. In positive valence, the temperature and blood perfusion increased in some cases, showing a complex pattern. The temperature and perfusion of the nose was reduced for both valences, which is indicative of the arousal dimension. The blood perfusion images were found to be greater contrast; the percentage differences in the blood perfusion images are greater than those obtained in thermographic images. Moreover, the blood perfusion images, and vasomotor answer are consistent, therefore, they can be a better biomarker than thermographic analysis in identifying emotions.
Collapse
Affiliation(s)
| | - Marcela Henao-Pérez
- School of Medicine, Universidad Cooperativa de Colombia, Medellín, 050012, Colombia.
| | | | - Renato Zambrano-Cruz
- School of Psychology, Universidad Cooperativa de Colombia, Medellín, 050012, Colombia.
| | - Gloria Díaz-Londoño
- School of Science, Universidad Nacional de Colombia-Sede, Medellín, 050034, Colombia.
| |
Collapse
|
15
|
Naal-Ruiz NE, Alonso-Valerdi LM, Ibarra-Zarate DI, Serrano-Cena A, Navas-Reascos G. Mexican validation of the International Affective Digitized Sounds second edition (IADS-2) and additional sounds. Sci Rep 2022; 12:21824. [PMID: 36528640 PMCID: PMC9758458 DOI: 10.1038/s41598-022-26320-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Accepted: 12/13/2022] [Indexed: 12/23/2022] Open
Abstract
Affective stimuli have been extensively used in emotion research for a better understanding of emotion regulation. Sound ratings, specifically non-verbal sounds, are biased by demographic indicators such as sex and nationality. Therefore, it is crucial to characterize sounds prior to their use in emotion research. This study aims to validate the IADS-2 database and additional sounds in a sample from the Mexican population. Three hundred twenty-nine participants born and raised in Mexico remotely listened to 174 sounds in monophonic format. They rated sounds according to the valence-arousal-dominance model using the Self-Assessment Manikin test. Results positively correlated to those of previous studies. Sex differences were observed only in dominance between female and male groups, contrary to the results from Portuguese, American and Japanese validations. Geographic region analysis demonstrated differences in arousal, indicating the need for additional research on occident and south regions. Furthermore, when conducting affective research, headphones and audio quality should be considered, primarily to reduce variability due to audio-related aspects, and to avoid changes in emotional experience. Finally, this study supports the feasibility of remote affective sound experiments over the internet as reported in previous research.
Collapse
Affiliation(s)
- Norberto E. Naal-Ruiz
- grid.419886.a0000 0001 2203 4701Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, 64849 Monterrey, Nuevo Leon Mexico
| | - Luz M. Alonso-Valerdi
- grid.419886.a0000 0001 2203 4701Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, 64849 Monterrey, Nuevo Leon Mexico
| | - David I. Ibarra-Zarate
- grid.419886.a0000 0001 2203 4701Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, 64849 Monterrey, Nuevo Leon Mexico
| | - Alba Serrano-Cena
- grid.419886.a0000 0001 2203 4701Escuela de Medicina y Ciencias de la Salud, Tecnologico de Monterrey, 64849 Monterrey, Nuevo Leon Mexico
| | - Gustavo Navas-Reascos
- grid.419886.a0000 0001 2203 4701Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey, 64849 Monterrey, Nuevo Leon Mexico
| |
Collapse
|
16
|
Paoletti M, Fini C, Filippini C, Massari GM, D’Abundo E, Merla A, Bellagamba F, Borghi AM. Abstract words processing induces parasympathetic activation: A thermal imaging study. Front Psychol 2022; 13:932118. [PMID: 36389444 PMCID: PMC9652150 DOI: 10.3389/fpsyg.2022.932118] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Accepted: 10/03/2022] [Indexed: 11/26/2022] Open
Abstract
Abstract words (e.g., freedom) compose a significant part of speech. Despite this, learning them is complicated. Abstract concepts collect more heterogeneous exemplars and are more detached from sensory modalities than concrete concepts. Recent views propose that, because of their complexity, other people are pivotal for abstract concepts’ acquisition and use, e.g., to explain their meaning. We tested this hypothesis using a combined behavioral and thermal imaging paradigm. Twenty-one Italian children (10\F, mean age: 6 years) determined whether acoustic stimuli (concrete and abstract words; non-words) were or not correct Italian words (lexical decision). Concrete terms yielded faster responses than abstract ones: for the first time, this effect appears with response times in very young children. More crucially, the higher increase in temperature of the nasal tip (i.e., typically associated with parasympathetic dominance of the neurovegetative response) suggests that, with abstract concepts, children might be more socially and cognitively engaged.
Collapse
Affiliation(s)
- Melania Paoletti
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
| | - Chiara Fini
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
| | - Chiara Filippini
- Department of Neurosciences and Imaging, University G. d’Annunzio of Chieti-Pescara, Chieti, Italy
| | - Giovanna M. Massari
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
| | - Emilia D’Abundo
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
| | - Arcangelo Merla
- Department of Engineering and Geology, University G. d’Annunzio of Chieti-Pescara, Pescara, Italy
| | - Francesca Bellagamba
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
| | - Anna M. Borghi
- Department of Dynamic and Clinical Psychology, and Health Studies, Sapienza University of Rome, Rome, Italy
- Institute of Cognitive Sciences and Technologies, National Research Council (CNR), Rome, Italy
- *Correspondence: Anna M. Borghi,
| |
Collapse
|
17
|
Jaramillo-Quintanar D, Cruz-Albarran IA, Dominguez-Trejo B, Rodriguez-Medina DA, Morales-Hernandez LA. Face Thermal Map of the Mexican Population in the Basal State. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:14208. [PMID: 36361088 PMCID: PMC9657387 DOI: 10.3390/ijerph192114208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/13/2022] [Revised: 10/18/2022] [Accepted: 10/24/2022] [Indexed: 06/16/2023]
Abstract
There has been a wide use of thermal images of the human body in recent years, specifically images with thermal information of regions of interest (ROI) in the face; this information can be used for epidemiological, clinical, and/or psychological purposes. Due to this, it is important to have plenty of information on temperature in these ROIs in the basal state that allows their use as a reference in terms of their thermal analysis. In this work, a face thermal map of the Mexican population in the basal state (n = 196) is created, adding the comparison between different population groups, such as gender, age, and clinical status, obtaining results of great interest for future research. The t-test for independent samples was applied to the ROIs with normal distribution and Mann−Whitney u-test to the ones that did not present normal distribution. Statistically significant differences were found in some of the ROI comparisons like the corrugator, the supraorbitals, and the chin between the control and clinical groups, as well as in the differentiation by age (p < 0.05).
Collapse
Affiliation(s)
- Daniel Jaramillo-Quintanar
- Mechatronics/Engineering Faculty, Campus San Juan del Rio, Autonomous University of Queretaro, San Juan del Rio 76807, Queretaro, Mexico
| | - Irving A. Cruz-Albarran
- Mechatronics/Engineering Faculty, Campus San Juan del Rio, Autonomous University of Queretaro, San Juan del Rio 76807, Queretaro, Mexico
| | - Benjamin Dominguez-Trejo
- Postgraduate Studies Division, Psychology Faculty, National Autonomous University of Mexico, Mexico City 04510, Mexico
| | - David A. Rodriguez-Medina
- Department of Sociology, Division of Social Sciences and Humanities, Universidad Autónoma Metropolitana, Mexico City 52919, Mexico
| | - Luis Alberto Morales-Hernandez
- Mechatronics/Engineering Faculty, Campus San Juan del Rio, Autonomous University of Queretaro, San Juan del Rio 76807, Queretaro, Mexico
| |
Collapse
|
18
|
Cardone D, Perpetuini D, Filippini C, Mancini L, Nocco S, Tritto M, Rinella S, Giacobbe A, Fallica G, Ricci F, Gallina S, Merla A. Classification of Drivers' Mental Workload Levels: Comparison of Machine Learning Methods Based on ECG and Infrared Thermal Signals. SENSORS (BASEL, SWITZERLAND) 2022; 22:7300. [PMID: 36236399 PMCID: PMC9572767 DOI: 10.3390/s22197300] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 09/19/2022] [Accepted: 09/22/2022] [Indexed: 06/16/2023]
Abstract
Mental workload (MW) represents the amount of brain resources required to perform concurrent tasks. The evaluation of MW is of paramount importance for Advanced Driver-Assistance Systems, given its correlation with traffic accidents risk. In the present research, two cognitive tests (Digit Span Test-DST and Ray Auditory Verbal Learning Test-RAVLT) were administered to participants while driving in a simulated environment. The tests were chosen to investigate the drivers' response to predefined levels of cognitive load to categorize the classes of MW. Infrared (IR) thermal imaging concurrently with heart rate variability (HRV) were used to obtain features related to the psychophysiology of the subjects, in order to feed machine learning (ML) classifiers. Six categories of models have been compared basing on unimodal IR/unimodal HRV/multimodal IR + HRV features. The best classifier performances were reached by the multimodal IR + HRV features-based classifiers (DST: accuracy = 73.1%, sensitivity = 0.71, specificity = 0.69; RAVLT: accuracy = 75.0%, average sensitivity = 0.75, average specificity = 0.87). The unimodal IR features based classifiers revealed high performances as well (DST: accuracy = 73.1%, sensitivity = 0.73, specificity = 0.73; RAVLT: accuracy = 71.1%, average sensitivity = 0.71, average specificity = 0.85). These results demonstrated the possibility to assess drivers' MW levels with high accuracy, also using a completely non-contact and non-invasive technique alone, representing a key advancement with respect to the state of the art in traffic accident prevention.
Collapse
Affiliation(s)
- Daniela Cardone
- Department of Engineering and Geology, University G. d’Annunzio of Chieti-Pescara, 65127 Pescara, Italy
| | - David Perpetuini
- Department of Neurosciences, Imaging and Clinical Sciences, University G. d’Annunzio of Chieti-Pescara, 66100 Chieti, Italy
| | - Chiara Filippini
- Department of Neurosciences, Imaging and Clinical Sciences, University G. d’Annunzio of Chieti-Pescara, 66100 Chieti, Italy
| | | | | | | | - Sergio Rinella
- Physiology Section, Department of Biomedical and Biotechnological Sciences, University of Catania, 95123 Catania, Italy
| | - Alberto Giacobbe
- Physiology Section, Department of Biomedical and Biotechnological Sciences, University of Catania, 95123 Catania, Italy
| | - Giorgio Fallica
- National Interuniversity Consortium of Science and Technology of Materials (INSTM), University of Messina, 98122 Messina, Italy
| | - Fabrizio Ricci
- Department of Neurosciences, Imaging and Clinical Sciences, University G. d’Annunzio of Chieti-Pescara, 66100 Chieti, Italy
| | - Sabina Gallina
- Department of Neurosciences, Imaging and Clinical Sciences, University G. d’Annunzio of Chieti-Pescara, 66100 Chieti, Italy
| | - Arcangelo Merla
- Department of Engineering and Geology, University G. d’Annunzio of Chieti-Pescara, 65127 Pescara, Italy
- Next2U s.r.l., 65127 Pescara, Italy
| |
Collapse
|
19
|
Abdul-Al M, Kyeremeh GK, Abd-Alhameed RA, Qahwaji R, Abdul-Atty MM, Parchin NO, Rodriguez J, Amar AS. Types of Infrareds Focusing on Face Recognition: Promises, Advances and Challenges. 2022 INTERNATIONAL TELECOMMUNICATIONS CONFERENCE (ITC-EGYPT) 2022. [DOI: 10.1109/itc-egypt55520.2022.9855672] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/02/2023]
Affiliation(s)
- Mohamed Abdul-Al
- University of Bradford,Department of Biomedical and Elecronics Engineering,Bradford,England
| | - George Kumi Kyeremeh
- University of Bradford,Department of Biomedical and Elecronics Engineering,Bradford,England
| | - Raed A. Abd-Alhameed
- University of Bradford,Department of Biomedical and Elecronics Engineering,Bradford,England
| | - Rami Qahwaji
- University of Bradford,Department of Biomedical and Elecronics Engineering,Bradford,England
| | | | - Naser Ojaroudi Parchin
- Edinburgh Napier University,School of Engineering and the Built Environment,Edinburgh,UK
| | | | | |
Collapse
|
20
|
Folz J, Fiacchino D, Nikolić M, van Steenbergen H, Kret ME. Reading Your Emotions in My Physiology? Reliable Emotion Interpretations in Absence of a Robust Physiological Resonance. AFFECTIVE SCIENCE 2022; 3:480-497. [PMID: 35282156 PMCID: PMC8901434 DOI: 10.1007/s42761-021-00083-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Accepted: 09/12/2021] [Indexed: 11/26/2022]
Abstract
Affective states are expressed in an individual’s physical appearance, ranging from facial expressions and body postures, to indicators of physiological arousal (e.g., a blush). Confirming the claimed communicative function of these markers, humans are capable of distinguishing between a variety of discrete emotion displays. In an attempt to explain the underlying mechanism, characteristic bodily changes within the observer, including physiological arousal and mimicry, have been suggested to facilitate the interpretation of an expression. The current study aims to create a holistic picture of emotion perception by (1) using three different sources of emotional information (prototypical facial expressions, bodily expressions, and subtle facial cues) and (2) measuring changes in multiple physiological signals (facial electromyography, skin conductance level, skin temperature, and pupil size). While participants clearly discriminated between perceived emotional expressions, there was no overall 1–1 correspondence with their physiological responses. Some specific but robust effects were observed. Angry facial expressions were consistently responded to with a peak in skin conductance level. Furthermore, sad body expressions were associated with a drop in skin temperature. In addition to being the best recognized expression, viewing happy faces elicited congruent facial muscle responses, which supports the potential role of embodied simulation in emotion recognition. Lastly, tears were not only rated as highly emotional intense but also evoked a peak in skin conductance level in the observer. The absence of distinct physiological responses to other expressions could be explained by the lacking functionality of affect sharing in a non-interactive experimental context. Consequentially, emotional alignment in body and mind might especially take place in real social situations, which should be considered in future research.
Collapse
Affiliation(s)
- Julia Folz
- Department of Cognitive Psychology, Institute of Psychology, Leiden University, Leiden, 2333 AK The Netherlands
- Leiden Institute for Brain and Cognition (LIBC), Leiden University, Leiden, 2300 RC The Netherlands
| | - Donatella Fiacchino
- Department of Cognitive Psychology, Institute of Psychology, Leiden University, Leiden, 2333 AK The Netherlands
| | - Milica Nikolić
- Department of Cognitive Psychology, Institute of Psychology, Leiden University, Leiden, 2333 AK The Netherlands
- Leiden Institute for Brain and Cognition (LIBC), Leiden University, Leiden, 2300 RC The Netherlands
- Research Institute of Child Development and Education, University of Amsterdam, Amsterdam, 1018 WS The Netherlands
| | - Henk van Steenbergen
- Department of Cognitive Psychology, Institute of Psychology, Leiden University, Leiden, 2333 AK The Netherlands
- Leiden Institute for Brain and Cognition (LIBC), Leiden University, Leiden, 2300 RC The Netherlands
| | - Mariska E. Kret
- Department of Cognitive Psychology, Institute of Psychology, Leiden University, Leiden, 2333 AK The Netherlands
- Leiden Institute for Brain and Cognition (LIBC), Leiden University, Leiden, 2300 RC The Netherlands
| |
Collapse
|
21
|
Filippini C, Di Crosta A, Palumbo R, Perpetuini D, Cardone D, Ceccato I, Di Domenico A, Merla A. Automated Affective Computing Based on Bio-Signals Analysis and Deep Learning Approach. SENSORS 2022; 22:s22051789. [PMID: 35270936 PMCID: PMC8914721 DOI: 10.3390/s22051789] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Revised: 02/21/2022] [Accepted: 02/22/2022] [Indexed: 12/18/2022]
Abstract
Extensive possibilities of applications have rendered emotion recognition ineluctable and challenging in the fields of computer science as well as in human-machine interaction and affective computing. Fields that, in turn, are increasingly requiring real-time applications or interactions in everyday life scenarios. However, while extremely desirable, an accurate and automated emotion classification approach remains a challenging issue. To this end, this study presents an automated emotion recognition model based on easily accessible physiological signals and deep learning (DL) approaches. As a DL algorithm, a Feedforward Neural Network was employed in this study. The network outcome was further compared with canonical machine learning algorithms such as random forest (RF). The developed DL model relied on the combined use of wearables and contactless technologies, such as thermal infrared imaging. Such a model is able to classify the emotional state into four classes, derived from the linear combination of valence and arousal (referring to the circumplex model of affect’s four-quadrant structure) with an overall accuracy of 70% outperforming the 66% accuracy reached by the RF model. Considering the ecological and agile nature of the technique used the proposed model could lead to innovative applications in the affective computing field.
Collapse
Affiliation(s)
- Chiara Filippini
- Department of Neurosciences, Imaging and Clinical Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (C.F.); (D.P.); (D.C.); (I.C.)
| | - Adolfo Di Crosta
- Department of Psychological, Health and Territorial Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (A.D.C.); (R.P.); (A.D.D.)
| | - Rocco Palumbo
- Department of Psychological, Health and Territorial Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (A.D.C.); (R.P.); (A.D.D.)
| | - David Perpetuini
- Department of Neurosciences, Imaging and Clinical Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (C.F.); (D.P.); (D.C.); (I.C.)
| | - Daniela Cardone
- Department of Neurosciences, Imaging and Clinical Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (C.F.); (D.P.); (D.C.); (I.C.)
| | - Irene Ceccato
- Department of Neurosciences, Imaging and Clinical Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (C.F.); (D.P.); (D.C.); (I.C.)
| | - Alberto Di Domenico
- Department of Psychological, Health and Territorial Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (A.D.C.); (R.P.); (A.D.D.)
| | - Arcangelo Merla
- Department of Neurosciences, Imaging and Clinical Sciences, University G. D’Annunzio of Chieti-Pescara, 9, 66100 Chieti, Italy; (C.F.); (D.P.); (D.C.); (I.C.)
- Correspondence: ; Tel.: +39-0871-3556-954
| |
Collapse
|
22
|
Thermal imaging reveals audience-dependent effects during cooperation and competition in wild chimpanzees. Sci Rep 2022; 12:2972. [PMID: 35194084 PMCID: PMC8863809 DOI: 10.1038/s41598-022-07003-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2021] [Accepted: 02/09/2022] [Indexed: 12/28/2022] Open
Abstract
Accessing animal minds has remained a challenge since the beginnings of modern science. Here, we used a little-tried method, functional infrared thermal imaging, with wild chimpanzees during common social interactions. After removing confounds, we found that chimpanzees involved in competitive events had lower nose skin temperatures whereas those involved in cooperative events had higher temperatures, the latter more so in high- than low-ranking males. Temperatures associated with grooming were akin to those of cooperative events, except when males interacted with a non-reciprocating alpha male. In addition, we found multiple audience effects. Notably, the alpha male's presence reduced positive effects associated with cooperation, whereas female presence buffered negative effects associated with competition. Copulation was perceived as competitive, especially during furtive mating when other males were absent. Overall, patterns suggest that chimpanzees categorise ordinary social events as cooperative or competitive and that these perceptions are moderated by specific audiences.
Collapse
|
23
|
Wan C, Chen D, Huang Z, Luo X. A Wearable Head Mounted Display Bio-Signals Pad System for Emotion Recognition. SENSORS (BASEL, SWITZERLAND) 2021; 22:142. [PMID: 35009684 PMCID: PMC8749721 DOI: 10.3390/s22010142] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/22/2021] [Revised: 12/14/2021] [Accepted: 12/22/2021] [Indexed: 06/14/2023]
Abstract
Multimodal bio-signals acquisition based on wearable devices and using virtual reality (VR) as stimulus source are promising techniques in emotion recognition research field. Numerous studies have shown that emotional states can be better evoked through Immersive Virtual Environments (IVE). The main goal of this paper is to provide researchers with a system for emotion recognition in VR environments. In this paper, we present a wearable forehead bio-signals acquisition pad which is attached to Head-Mounted Displays (HMD), termed HMD Bio Pad. This system can simultaneously record emotion-related two-channel electroencephalography (EEG), one-channel electrodermal activity (EDA), photoplethysmograph (PPG) and skin temperature (SKT) signals. In addition, we develop a human-computer interaction (HCI) interface which researchers can carry out emotion recognition research using VR HMD as stimulus presentation device. To evaluate the performance of the proposed system, we conducted different experiments to validate the multimodal bio-signals quality, respectively. To validate EEG signal, we have assessed the performance in terms of EEG eyes-blink task and eyes-open and eyes-closed task. The EEG eyes-blink task indicates that the proposed system can achieve comparable EEG signal quality in comparison to the dedicated bio-signals measuring device. The eyes-open and eyes-closed task proves that the proposed system can efficiently record alpha rhythm. Then we used signal-to-noise ratio (SNR) and Skin Conductance Reaction (SCR) signal to validate the performance for EDA acquisition system. A filtered EDA signal, with a high mean SNR of 28.52 dB, is plotted on HCI interface. Moreover, the SCR signal related to stimulus response can be correctly extracted from EDA signal. The SKT acquisition system has been validated effectively by the temperature change experiment when subjects are in unpleasant emotion. The pulse rate (PR) estimated from PPG signal achieved the low mean average absolute error (AAE), which is 1.12 beats per minute (BPM) over 8 recordings. In summary, the proposed HMD Bio Pad offers a portable, comfortable and easy-to-wear device for recording bio-signals. The proposed system could contribute to emotion recognition research in VR environments.
Collapse
Affiliation(s)
- Chunting Wan
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; (C.W.); (Z.H.); (X.L.)
- School of Electronic Engineering and Automation, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Dongyi Chen
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; (C.W.); (Z.H.); (X.L.)
| | - Zhiqi Huang
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; (C.W.); (Z.H.); (X.L.)
| | - Xi Luo
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China; (C.W.); (Z.H.); (X.L.)
| |
Collapse
|
24
|
Sato W, Ikegami A, Ishihara S, Nakauma M, Funami T, Yoshikawa S, Fushiki T. Brow and Masticatory Muscle Activity Senses Subjective Hedonic Experiences during Food Consumption. Nutrients 2021; 13:nu13124216. [PMID: 34959773 PMCID: PMC8708739 DOI: 10.3390/nu13124216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2021] [Revised: 11/21/2021] [Accepted: 11/22/2021] [Indexed: 11/16/2022] Open
Abstract
Sensing subjective hedonic or emotional experiences during eating using physiological activity is practically and theoretically important. A recent psychophysiological study has reported that facial electromyography (EMG) measured from the corrugator supercilii muscles was negatively associated with hedonic ratings, including liking, wanting, and valence, during the consumption of solid foods. However, the study protocol prevented participants from natural mastication (crushing of food between the teeth) during physiological data acquisition, which could hide associations between hedonic experiences and masticatory muscle activity during natural eating. We investigated this issue by assessing participants’ subjective ratings (liking, wanting, valence, and arousal) and recording physiological measures, including EMG of the corrugator supercilii, zygomatic major, masseter, and suprahyoid muscles while they consumed gel-type solid foods (water-based gellan gum jellies) of diverse flavors. Ratings of liking, wanting, and valence were negatively correlated with corrugator supercilii EMG and positively correlated with masseter and suprahyoid EMG. These findings imply that subjective hedonic experiences during food consumption can be sensed using EMG signals from the brow and masticatory muscles.
Collapse
Affiliation(s)
- Wataru Sato
- Psychological Process Team, Guardian Robot Project, RIKEN, Kyoto 619-0288, Japan
- Field Science Education and Research Center, Kyoto University, Kyoto 606-78502, Japan;
- Correspondence: ; Tel.: +81-774-95-1360
| | - Akira Ikegami
- San-Ei Gen F. F. I., Inc., Osaka 561-8588, Japan; (A.I.); (S.I.); (M.N.); (T.F.)
| | - Sayaka Ishihara
- San-Ei Gen F. F. I., Inc., Osaka 561-8588, Japan; (A.I.); (S.I.); (M.N.); (T.F.)
| | - Makoto Nakauma
- San-Ei Gen F. F. I., Inc., Osaka 561-8588, Japan; (A.I.); (S.I.); (M.N.); (T.F.)
| | - Takahiro Funami
- San-Ei Gen F. F. I., Inc., Osaka 561-8588, Japan; (A.I.); (S.I.); (M.N.); (T.F.)
| | - Sakiko Yoshikawa
- Field Science Education and Research Center, Kyoto University, Kyoto 606-78502, Japan;
- Faculty of the Arts, Kyoto University of the Arts, Kyoto 606-8271, Japan
| | - Tohru Fushiki
- Faculty of Agriculture, Ryukoku University, Otsu 520-2194, Japan;
| |
Collapse
|
25
|
Tashakori M, Nahvi A, Ebrahimian Hadi Kiashari S. Driver drowsiness detection using facial thermal imaging in a driving simulator. Proc Inst Mech Eng H 2021; 236:43-55. [PMID: 34477030 DOI: 10.1177/09544119211044232] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Driver drowsiness causes fatal driving accidents. Thermal imaging is a suitable drowsiness detection method as it is non-invasive and robust against changes in the ambient light. In this paper, driver drowsiness is detected by measuring the forehead temperature at the region covering the supratrochlear artery and also the cheek temperature. About 30 subjects drove on a highway in a driving simulator in two sessions. A thermal camera was used to monitor the facial temperature pattern. The subjects' drowsiness levels were estimated by three human observers. The forehead and the cheek regions were located and tracked in each frame. The forehead and the cheek skin temperatures were obtained at three levels of drowsiness. The Support Vector Machine, the K-Nearest Neighbor, and the regression tree classifiers were used. From wakefulness to extreme drowsiness, the forehead skin temperature and the absolute cheek-forehead skin temperature gradient decreased by 0.46°C and 0.81°C, respectively. But the cheek skin temperature increased by 0.35°C in two sessions. The gradient difference is on average 50% higher than the forehead or the cheek temperature change alone. The results indicate that drowsiness can be detected with an accuracy of 82%, sensitivity of 85%, specificity of 90%, and precision of 84%. Driver drowsiness can be detected by monitoring changes in the forehead and the cheek temperature signal. Also, the temperature gradient can be used as a more robust and sensitive indicator of drowsiness.
Collapse
Affiliation(s)
- Masoumeh Tashakori
- Virtual Reality Laboratory, K.N. Toosi University of Technology, Tehran, Iran
| | - Ali Nahvi
- Virtual Reality Laboratory, K.N. Toosi University of Technology, Tehran, Iran
| | | |
Collapse
|
26
|
Ganesh K, Umapathy S, Thanaraj Krishnan P. Deep learning techniques for automated detection of autism spectrum disorder based on thermal imaging. Proc Inst Mech Eng H 2021; 235:1113-1127. [PMID: 34105405 DOI: 10.1177/09544119211024778] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Children with autism spectrum disorder have impairments in emotional processing which leads to the inability in recognizing facial expressions. Since emotion is a vital criterion for having fine socialisation, it is incredibly important for the autistic children to recognise emotions. In our study, we have chosen the facial skin temperature as a biomarker to measure emotions. To assess the facial skin temperature, the thermal imaging modality has been used in this study, since it has been recognised as a promising technique to evaluate emotional responses. The aim of this study was the following: (1) to compare the facial skin temperature of autistic and non-autistic children by using thermal imaging across various emotions; (2) to classify the thermal images obtained from the study using the customised convolutional neural network compared with the ResNet 50 network. Fifty autistic and fifty non-autistic participants were included for the study. Thermal imaging was used to obtain the temperature of specific facial regions such as the eyes, cheek, forehead and nose while we evoked emotions (Happiness, anger and sadness) in children using an audio-visual stimulus. Among the emotions considered, the emotion anger had the highest temperature difference between the autistic and non-autistic participants in the region's eyes (1.9%), cheek (2.38%) and nose (12.6%). The accuracy obtained by classifying the thermal images of the autistic and non-autistic children using Customised Neural Network and ResNet 50 Network was 96% and 90% respectively. This computer aided diagnostic tool can be a predictable and a steadfast method in the diagnosis of the autistic individuals.
Collapse
Affiliation(s)
- Kavya Ganesh
- Department of Biomedical Engineering, College of Engineering and Technology, SRM Institute of Science and Technology, Chennai, Tamil Nadu, India
| | - Snekhalatha Umapathy
- Department of Biomedical Engineering, College of Engineering and Technology, SRM Institute of Science and Technology, Chennai, Tamil Nadu, India
| | - Palani Thanaraj Krishnan
- Department of Electronics and Instrumentation Engineering, St. Joseph's College of Engineering, Anna University, Chennai, Tamil Nadu, India
| |
Collapse
|
27
|
Barbosa Escobar F, Velasco C, Motoki K, Byrne DV, Wang QJ. The temperature of emotions. PLoS One 2021; 16:e0252408. [PMID: 34081750 PMCID: PMC8174739 DOI: 10.1371/journal.pone.0252408] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Accepted: 05/14/2021] [Indexed: 11/18/2022] Open
Abstract
Emotions and temperature are closely related through embodied processes, and people seem to associate temperature concepts with emotions. While this relationship is often evidenced by everyday language (e.g., cold and warm feelings), what remains missing to date is a systematic study that holistically analyzes how and why people associate specific temperatures with emotions. The present research aimed to investigate the associations between temperature concepts and emotion adjectives on both explicit and implicit levels. In Experiment 1, we evaluated explicit associations between twelve pairs of emotion adjectives derived from the circumplex model of affect, and five different temperature concepts ranging from 0°C to 40°C, based on responses from 403 native speakers of four different languages (English, Spanish, Japanese, Chinese). The results of Experiment 1 revealed that, across languages, the temperatures were associated with different regions of the circumplex model. The 0°C and 10°C were associated with negative-valanced, low-arousal emotions, while 20°C was associated with positive-valanced, low-to-medium-arousal emotions. Moreover, 30°C was associated with positive-valanced, high-arousal emotions; and 40°C was associated with high-arousal and either positive- or negative-valanced emotions. In Experiment 2 (N = 102), we explored whether these temperature-emotion associations were also present at the implicit level, by conducting Implicit Association Tests (IATs) with temperature words (cold and hot) and opposing pairs of emotional adjectives for each dimension of valence (Unhappy/Dissatisfied vs. Happy/Satisfied) and arousal (Passive/Quiet vs. Active/Alert) on native English speakers. The results of Experiment 2 revealed that participants held implicit associations between the word hot and positive-valanced and high-arousal emotions. Additionally, the word cold was associated with negative-valanced and low-arousal emotions. These findings provide evidence for the existence of temperature-emotion associations at both explicit and implicit levels across languages.
Collapse
Affiliation(s)
- Francisco Barbosa Escobar
- Department of Food Science, Faculty of Science and Technology, Aarhus University, Aarhus, Denmark
- * E-mail:
| | - Carlos Velasco
- Department of Marketing, Centre for Multisensory Marketing, BI Norwegian Business School, Oslo, Norway
| | - Kosuke Motoki
- Department of Food Science and Business, Miyagi University, Sendai, Japan
| | - Derek Victor Byrne
- Department of Food Science, Faculty of Science and Technology, Aarhus University, Aarhus, Denmark
| | - Qian Janice Wang
- Department of Food Science, Faculty of Science and Technology, Aarhus University, Aarhus, Denmark
| |
Collapse
|
28
|
Che X, Luo X, Chen Y, Li B, Li X, Li X, Qiao L. Social touch modulates pain-evoked increases in facial temperature. CURRENT PSYCHOLOGY 2021. [DOI: 10.1007/s12144-020-01212-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
29
|
De Zorzi L, Ranfaing S, Honoré J, Sequeira H. Autonomic reactivity to emotion: A marker of sub-clinical anxiety and depression symptoms? Psychophysiology 2021; 58:e13774. [PMID: 33538013 DOI: 10.1111/psyp.13774] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2020] [Revised: 11/04/2020] [Accepted: 12/23/2020] [Indexed: 12/12/2022]
Abstract
Anxiety and depression are both characterized by dysregulated autonomic reactivity to emotion. However, most experiments until now have focused on autonomic reactivity to stimuli presented in central vision (CV) even if affective saliency is also observed in peripheral vision (PV). We compared autonomic reactivity to CV and PV emotional stimulation in 58 participants with high anxious (HA) or low anxious (LA) and high depressive (HD) or low depressive (LD) symptomatology, based on STAI-B and BDI scores, respectively. Unpleasant (U), pleasant (P), and neutral (N) pictures from IAPS were presented at three eccentricities (0°: CV; -12 and 12°: PV). Skin conductance (SC), skin temperature, pupillary diameter, and heart rate (HR) were recorded. First, HA participants showed greater pupil dilation to emotional than to neutral stimuli in PV than in CV. Second, in contrast to HD, the valence effect indexed by SC and emotional arousal effect indexed by skin temperature were observed in LD. Third, both anxiety and depression lead to a valence effect indexed by pupillary light reflex and heart rate. These results suggest a hyperreactivity to emotion and hypervigilance to PV in anxiety. Depression is associated with an attenuation of positive effect and a global blunted autonomic reactivity to emotion. Moreover, anxiety mostly modulates the early processes of autonomic reactivity whereas depression mainly affects the later processes. The differential impact of emotional information over the visual field suggests the use of new stimulation strategies in order to attenuate anxious and depressive symptoms.
Collapse
Affiliation(s)
- Lucas De Zorzi
- UMR 9193 - SCALab - Sciences Cognitives et Sciences Affectives, CNRS, University of Lille, Lille, France
| | - Stéphane Ranfaing
- UMR 9193 - SCALab - Sciences Cognitives et Sciences Affectives, CNRS, University of Lille, Lille, France
| | - Jacques Honoré
- UMR 9193 - SCALab - Sciences Cognitives et Sciences Affectives, CNRS, University of Lille, Lille, France
| | - Henrique Sequeira
- UMR 9193 - SCALab - Sciences Cognitives et Sciences Affectives, CNRS, University of Lille, Lille, France
| |
Collapse
|
30
|
Sato W, Kochiyama T, Yoshikawa S. Physiological correlates of subjective emotional valence and arousal dynamics while viewing films. Biol Psychol 2020; 157:107974. [PMID: 33086090 DOI: 10.1016/j.biopsycho.2020.107974] [Citation(s) in RCA: 33] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2020] [Revised: 09/27/2020] [Accepted: 10/13/2020] [Indexed: 12/15/2022]
Abstract
An exploration of the physiological correlates of subjective emotional states has theoretical and practical significance. Previous studies have reported that subjective valence and arousal correspond to facial electromyography (EMG) and electrodermal activity (EDA), respectively, across stimuli. However, the reported results were inconsistent, no study investigated subjective-physiological concordance across time, and measures of arousal remain controversial. To investigate these issues, while healthy adults (n = 20) viewed emotional films, we assessed overall and continuous ratings of valence and arousal and recorded EMG from the corrugator supercilii and zygomatic major, EDA from the palms and forehead, and nose-tip temperature. The corrugator and zygomatic EMG were negatively and positively associated with valence ratings, respectively, across stimuli and time. EDA (both sites) and nose-tip temperature were positively and negatively associated with arousal ratings, respectively, across stimuli and time. It is concluded that subjective emotional valence and arousal dynamics have specific physiological correlates.
Collapse
Affiliation(s)
- Wataru Sato
- Psychological Process Team, BZP, RIKEN, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0288, Japan.
| | - Takanori Kochiyama
- Brain Activity Imaging Center, ATR-Promotions, 2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0288, Japan
| | - Sakiko Yoshikawa
- Kyoto University of the Arts, 2-116 Uryuyama Kitashirakawa, Sakyo, Kyoto, 606-8271, Japan
| |
Collapse
|
31
|
Höfling TTA, Gerdes ABM, Föhl U, Alpers GW. Read My Face: Automatic Facial Coding Versus Psychophysiological Indicators of Emotional Valence and Arousal. Front Psychol 2020; 11:1388. [PMID: 32636788 PMCID: PMC7316962 DOI: 10.3389/fpsyg.2020.01388] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2019] [Accepted: 05/25/2020] [Indexed: 12/12/2022] Open
Abstract
Facial expressions provide insight into a person's emotional experience. To automatically decode these expressions has been made possible by tremendous progress in the field of computer vision. Researchers are now able to decode emotional facial expressions with impressive accuracy in standardized images of prototypical basic emotions. We tested the sensitivity of a well-established automatic facial coding software program to detect spontaneous emotional reactions in individuals responding to emotional pictures. We compared automatically generated scores for valence and arousal of the Facereader (FR; Noldus Information Technology) with the current psychophysiological gold standard of measuring emotional valence (Facial Electromyography, EMG) and arousal (Skin Conductance, SC). We recorded physiological and behavioral measurements of 43 healthy participants while they looked at pleasant, unpleasant, or neutral scenes. When viewing pleasant pictures, FR Valence and EMG were both comparably sensitive. However, for unpleasant pictures, FR Valence showed an expected negative shift, but the signal differentiated not well between responses to neutral and unpleasant stimuli, that were distinguishable with EMG. Furthermore, FR Arousal values had a stronger correlation with self-reported valence than with arousal while SC was sensitive and specifically associated with self-reported arousal. This is the first study to systematically compare FR measurement of spontaneous emotional reactions to standardized emotional images with established psychophysiological measurement tools. This novel technology has yet to make strides to surpass the sensitivity of established psychophysiological measures. However, it provides a promising new measurement technique for non-contact assessment of emotional responses.
Collapse
Affiliation(s)
- T. Tim A. Höfling
- Department of Psychology, School of Social Sciences, University of Mannheim, Mannheim, Germany
| | - Antje B. M. Gerdes
- Department of Psychology, School of Social Sciences, University of Mannheim, Mannheim, Germany
| | - Ulrich Föhl
- Business Unit, Pforzheim University of Applied Sciences, Pforzheim, Germany
| | - Georg W. Alpers
- Department of Psychology, School of Social Sciences, University of Mannheim, Mannheim, Germany
| |
Collapse
|
32
|
Facilitating the Child–Robot Interaction by Endowing the Robot with the Capability of Understanding the Child Engagement: The Case of Mio Amico Robot. Int J Soc Robot 2020. [DOI: 10.1007/s12369-020-00661-w] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
AbstractSocial Robots (SRs) are substantially becoming part of modern society, given their frequent use in many areas of application including education, communication, assistance, and entertainment. The main challenge in human–robot interaction is in achieving human-like and affective interaction between the two groups. This study is aimed at endowing SRs with the capability of assessing the emotional state of the interlocutor, by analyzing his/her psychophysiological signals. The methodology is focused on remote evaluations of the subject’s peripheral neuro-vegetative activity by means of thermal infrared imaging. The approach was developed and tested for a particularly challenging use case: the interaction between children and a commercial educational robot, Mio Amico Robot, produced by LiscianiGiochi©. The emotional state classified from the thermal signal analysis was compared to the emotional state recognized by a facial action coding system. The proposed approach was reliable and accurate and favored a personalized and improved interaction of children with SRs.
Collapse
|
33
|
Cannas Aghedu F, Cardone D, Merla A, Troïan J, Bisiacchi PS, Lux-Sterritt L, Graziani P. The colours of love: facial thermal reactions of people thinking about their lovers. PSYCHOLOGY & SEXUALITY 2020. [DOI: 10.1080/19419899.2020.1756392] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Affiliation(s)
| | - Daniela Cardone
- Institute of Advanced Biomedical Technologies, G. d’Annunzio University, Chieti-Pescara, Italy
- Department of Neuroscience, Imaging and Clinical Sciences, G. d’Annunzio University, Chieti-Pescara, Italy
| | - Arcangelo Merla
- Institute of Advanced Biomedical Technologies, G. d’Annunzio University, Chieti-Pescara, Italy
- Department of Neuroscience, Imaging and Clinical Sciences, G. d’Annunzio University, Chieti-Pescara, Italy
| | - Jaïs Troïan
- LPS, Aix Marseille University, Aix En Provence, France
| | - Patrizia S. Bisiacchi
- Department of General Psychology, University of Padova, Padova, Italy
- Padova Neuroscience Center, University of Padova, Italy
| | | | - Pierluigi Graziani
- LPS, Aix Marseille University, Aix En Provence, France
- Department of Psychology, Languages, Letters, History, University of Nimes, Nimes, France
| |
Collapse
|
34
|
Marqués-Sánchez P, Liébana-Presa C, Benítez-Andrades JA, Gundín-Gallego R, Álvarez-Barrio L, Rodríguez-Gonzálvez P. Thermal Infrared Imaging to Evaluate Emotional Competences in Nursing Students: A First Approach through a Case Study. SENSORS 2020; 20:s20092502. [PMID: 32354094 PMCID: PMC7248891 DOI: 10.3390/s20092502] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/08/2020] [Revised: 04/25/2020] [Accepted: 04/27/2020] [Indexed: 12/18/2022]
Abstract
During university studies of nursing, it is important to develop emotional skills for their impact on academic performance and the quality of patient care. Thermography is a technology that could be applied during nursing training to evaluate emotional skills. The objective is to evaluate the effect of thermography as the tool for monitoring and improving emotional skills in student nurses through a case study. The student was subjected to different emotions. The stimuli applied were video and music. The process consisted of measuring the facial temperatures during each emotion and stimulus in three phases: acclimatization, stimulus, and response. Thermographic data acquisition was performed with an FLIR E6 camera. The analysis was complemented with the environmental data (temperature and humidity). With the video stimulus, the start and final forehead temperature from testing phases, showed a different behavior between the positive (joy: 34.5 °C-34.5 °C) and negative (anger: 36.1 °C-35.1 °C) emotions during the acclimatization phase, different from the increase experienced in the stimulus (joy: 34.7 °C-35.0 °C and anger: 35.0 °C-35.0 °C) and response phases (joy: 35.0 °C-35.0 °C and anger: 34.8 °C-35.0 °C). With the music stimulus, the emotions showed different patterns in each phase (joy: 34.2 °C-33.9 °C-33.4 °C and anger: 33.8 °C-33.4 °C-33.8 °C). Whenever the subject is exposed to a stimulus, there is a thermal bodily response. All of the facial areas follow a common thermal pattern in response to the stimulus, with the exception of the nose. Thermography is a technique suitable for the stimulation practices in emotional skills, given that it is non-invasive, it is quantifiable, and easy to access.
Collapse
Affiliation(s)
- Pilar Marqués-Sánchez
- SALBIS Research Group, Faculty of Health Sciences, Campus of Ponferrada, University of León, 24401 Ponferrada, Spain;
| | - Cristina Liébana-Presa
- SALBIS Research Group, Faculty of Health Sciences, Campus of Ponferrada, University of León, 24401 Ponferrada, Spain;
- Correspondence:
| | - José Alberto Benítez-Andrades
- SALBIS Research Group, Department of Electric, Systems and Automatics Engineering, University of León, 24071 León, Spain;
| | | | - Lorena Álvarez-Barrio
- Department of Nursing and Physiotherapy, Faculty of Health Sciences, Campus of Ponferrada, University of León, 24401 Ponferrada, Spain;
| | - Pablo Rodríguez-Gonzálvez
- Department of Mining, Surveying and Structure, Campus of Ponferrada, University of León, 24401 Ponferrada, Spain;
| |
Collapse
|
35
|
Thermal Infrared Imaging-Based Affective Computing and Its Application to Facilitate Human Robot Interaction: A Review. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10082924] [Citation(s) in RCA: 44] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
Over recent years, robots are increasingly being employed in several aspects of modern society. Among others, social robots have the potential to benefit education, healthcare, and tourism. To achieve this purpose, robots should be able to engage humans, recognize users’ emotions, and to some extent properly react and "behave" in a natural interaction. Most robotics applications primarily use visual information for emotion recognition, which is often based on facial expressions. However, the display of emotional states through facial expression is inherently a voluntary controlled process that is typical of human–human interaction. In fact, humans have not yet learned to use this channel when communicating with a robotic technology. Hence, there is an urgent need to exploit emotion information channels not directly controlled by humans, such as those that can be ascribed to physiological modulations. Thermal infrared imaging-based affective computing has the potential to be the solution to such an issue. It is a validated technology that allows the non-obtrusive monitoring of physiological parameters and from which it might be possible to infer affective states. This review is aimed to outline the advantages and the current research challenges of thermal imaging-based affective computing for human–robot interaction.
Collapse
|
36
|
Dzedzickis A, Kaklauskas A, Bucinskas V. Human Emotion Recognition: Review of Sensors and Methods. SENSORS (BASEL, SWITZERLAND) 2020; 20:E592. [PMID: 31973140 PMCID: PMC7037130 DOI: 10.3390/s20030592] [Citation(s) in RCA: 115] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/11/2019] [Revised: 01/10/2020] [Accepted: 01/12/2020] [Indexed: 11/16/2022]
Abstract
Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.
Collapse
Affiliation(s)
- Andrius Dzedzickis
- Faculty of Mechanics, Vilnius Gediminas Technical University, J. Basanaviciaus g. 28, LT-03224 Vilnius, Lithuania;
| | - Artūras Kaklauskas
- Faculty of Civil engineering, Vilnius Gediminas Technical University, Sauletekio ave. 11, LT-10223 Vilnius, Lithuania;
| | - Vytautas Bucinskas
- Faculty of Mechanics, Vilnius Gediminas Technical University, J. Basanaviciaus g. 28, LT-03224 Vilnius, Lithuania;
| |
Collapse
|
37
|
Seidel J, Bockhop F, Mitkovski M, Martin S, Ronnenberg A, Krueger‐Burg D, Schneider K, Röhse H, Wüstefeld L, Cosi F, Bröking K, Schacht A, Ehrenreich H. Vascular response to social cognitive performance measured by infrared thermography: A translational study from mouse to man. FASEB Bioadv 2020; 2:18-32. [PMID: 32123854 PMCID: PMC6996302 DOI: 10.1096/fba.2019-00085] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2019] [Revised: 10/25/2019] [Accepted: 11/12/2019] [Indexed: 01/27/2023] Open
Abstract
To assess complex social recognition in mice, we previously developed the SocioBox paradigm. Unexpectedly, 4 weeks after performing in the SocioBox, mice displayed robust social avoidance during Y-maze sociability testing. This unique "sociophobia" acquisition could be documented in independent cohorts. We therefore employed infrared thermography as a non-invasive method of stress-monitoring during SocioBox testing (presentation of five other mice) versus empty box. A higher Centralization Index (body/tail temperature) in the SocioBox correlated negatively with social recognition memory and, after 4 weeks, with social preference in the Y-maze. Assuming that social stimuli might be associated with characteristic thermo-responses, we exposed healthy men (N = 103) with a comparably high intelligence level to a standardized test session including two cognitive tests with or without social component (face versus pattern recognition). In some analogy to the Centralization Index (within-subject measure) used in mice, the Reference Index (ratio nose/malar cheek temperature) was introduced to determine the autonomic facial response/flushing during social recognition testing. Whereas cognitive performance and salivary cortisol were comparable across human subjects and tests, the Face Recognition Test was associated with a characteristic Reference Index profile. Infrared thermography may have potential for discriminating disturbed social behaviors.
Collapse
Affiliation(s)
- Jan Seidel
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Fabian Bockhop
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Miso Mitkovski
- Light Microscopy FacilityMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Sabine Martin
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Anja Ronnenberg
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Dilja Krueger‐Burg
- Department of Molecular NeurobiologyMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Katharina Schneider
- Light Microscopy FacilityMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Heiko Röhse
- Light Microscopy FacilityMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Liane Wüstefeld
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Filippo Cosi
- Biomedical Physics GroupMax Planck Institute for Dynamics and Self‐OrganizationGöttingenGermany
| | - Kai Bröking
- Light Microscopy FacilityMax Planck Institute of Experimental MedicineGöttingenGermany
| | - Annekathrin Schacht
- Department of Affective Neuroscience and PsychophysiologyGeorg‐Elias‐Müller‐Institute of PsychologyGeorg‐August UniversityGöttingenGermany
| | - Hannelore Ehrenreich
- Clinical NeuroscienceMax Planck Institute of Experimental MedicineGöttingenGermany
| |
Collapse
|
38
|
Diaz-Piedra C, Gomez-Milan E, Di Stasi LL. Nasal skin temperature reveals changes in arousal levels due to time on task: An experimental thermal infrared imaging study. APPLIED ERGONOMICS 2019; 81:102870. [PMID: 31422278 DOI: 10.1016/j.apergo.2019.06.001] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/14/2018] [Revised: 05/27/2019] [Accepted: 06/03/2019] [Indexed: 06/10/2023]
Abstract
Infrared thermography, thanks to technological developments and lowering prices, is now getting considerable attention as a potential arousal monitor in the safety industry. Nasal skin temperature might be a valid index to track physiological variations due to reduced arousal levels, and its use could prevent a drowsiness-related deterioration of performance. However, the few studies that have investigated nasal skin temperature in applied settings have had inconsistent results. Here, we assessed the validity of nasal skin temperature to monitor changes in arousal levels (from alertness to drowsiness). The participants performed a 2-h simulated driving task while we simultaneously recorded their nasal skin temperature, brain activity (we used frontal delta electroencephalographic [EEG] activity as the reference index of alertness), and driving performance (speeding time). For those variables, we calculated growth curve models. We also collected subjective ratings of alertness and fatigue before and after the driving session. We found that the nasal skin temperature showed a cubic trajectory (it increased for the first 75 min, and then it began to decrease, but such deceleration gradually diminished over time). As expected, frontal delta EEG activity showed an inverted U-shaped quadratic trend (EEG power increased for the first hour and half, and gradually decreased during the last thirty minutes). The speeding time exhibited a similar pattern of change. Subjective sleepiness and fatigue increased after the task. Overall, our results suggest that nasal skin temperature seems to be a valid measure of arousal variations while performing a complex and dynamic everyday task.
Collapse
Affiliation(s)
- Carolina Diaz-Piedra
- Mind, Brain, and Behavior Research Center-CIMCYC, University of Granada, Campus de Cartuja s/n, 18071 Granada, Spain; College of Health Solutions, Arizona State University, 500 N. 3rd St., 85004 Phoenix, AZ, USA.
| | - Emilo Gomez-Milan
- Mind, Brain, and Behavior Research Center-CIMCYC, University of Granada, Campus de Cartuja s/n, 18071 Granada, Spain
| | - Leandro L Di Stasi
- Mind, Brain, and Behavior Research Center-CIMCYC, University of Granada, Campus de Cartuja s/n, 18071 Granada, Spain; Joint Center University of Granada - Spanish Army Training and Doctrine Command, C/ Gran Via de Colon, 48, 18071 Granada, Spain.
| |
Collapse
|
39
|
A Modular System for Detection, Tracking and Analysis of Human Faces in Thermal Infrared Recordings. SENSORS 2019; 19:s19194135. [PMID: 31554260 PMCID: PMC6806182 DOI: 10.3390/s19194135] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/30/2019] [Revised: 09/05/2019] [Accepted: 09/16/2019] [Indexed: 11/17/2022]
Abstract
We present a system that utilizes a range of image processing algorithms to allow fully automated thermal face analysis under both laboratory and real-world conditions. We implement methods for face detection, facial landmark detection, face frontalization and analysis, combining all of these into a fully automated workflow. The system is fully modular and allows implementing own additional algorithms for improved performance or specialized tasks. Our suggested pipeline contains a histogtam of oriented gradients support vector machine (HOG-SVM) based face detector and different landmark detecion methods implemented using feature-based active appearance models, deep alignment networks and a deep shape regression network. Face frontalization is achieved by utilizing piecewise affine transformations. For the final analysis, we present an emotion recognition system that utilizes HOG features and a random forest classifier and a respiratory rate analysis module that computes average temperatures from an automatically detected region of interest. Results show that our combined system achieves a performance which is comparable to current stand-alone state-of-the-art methods for thermal face and landmark datection and a classification accuracy of 65.75% for four basic emotions.
Collapse
|
40
|
Ermatinger FA, Brügger RK, Burkart JM. The use of infrared thermography to investigate emotions in common marmosets. Physiol Behav 2019; 211:112672. [PMID: 31487492 DOI: 10.1016/j.physbeh.2019.112672] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2019] [Revised: 08/29/2019] [Accepted: 09/01/2019] [Indexed: 12/21/2022]
Abstract
Measuring body surface temperature changes with infrared thermography has recently been put forward as a non-invasive alternative measure of physiological correlates of emotional reactions. In particular, the nasal region seems to be highly sensitive to emotional reactions. Several studies suggest that nasal temperature is negatively correlated with the level of arousal in humans and other primates, but some studies provide inconsistent results. Our goal was to establish the use of infrared thermography to quantify emotional reactions in common marmosets (Callithrix jacchus), with a focus on the nasal region. To do so we exposed 17 common marmosets to a set of positive, negative and control stimuli (positive: preferred food, playback of food calls; negative: playback of aggressive vocalizations, teasing; control: no stimulus). We compared nasal temperature before and after the stimuli and expected that highly aroused emotional states would lead to a drop in nasal temperature. To validate the thermography measure, we coded piloerection of the tail as an independent measure of arousal and expected a negative correlation between the two measures. Finally, we coded physical activity to exclude its potential confounding impact on nasal temperature. Our results show that all predictions were met: the animals showed a strong decrease in nasal temperature after the presentation of negative arousing stimuli (teasing, playback of aggressive vocalizations). Furthermore, these changes in nasal temperature were correlated with piloerection of the tail and could not be explained by changes in physical activity. In the positive and the control conditions, we found systematic sex differences: in males, the preferred food, the playbacks of food calls, as well as the control stimulus led to an increase in nasal temperature, whereas in females the temperature remained stable (preferred food, control) or decreased (playback of food calls). Based on naturalistic observations that document higher food motivation and competition among female marmosets, as well as stronger reactions to separation from group members in male marmosets, these sex differences corroborate a negative correlation between arousal and nasal temperature. Overall, our results support that measuring nasal temperature by infrared thermography is a promising method to quantify emotional arousal in common marmosets in a fully non-invasive and highly objective way.
Collapse
Affiliation(s)
- F A Ermatinger
- Department of Anthropology, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland
| | - R K Brügger
- Department of Anthropology, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland.
| | - J M Burkart
- Department of Anthropology, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland
| |
Collapse
|
41
|
Exploratory Investigation of Infrared Thermography for Measuring Gorilla Emotional Responses to Interactions with Familiar Humans. Animals (Basel) 2019; 9:ani9090604. [PMID: 31450716 PMCID: PMC6770422 DOI: 10.3390/ani9090604] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2019] [Revised: 08/08/2019] [Accepted: 08/21/2019] [Indexed: 12/29/2022] Open
Abstract
Simple Summary Interactions between zoo professionals and animals, such as positive reinforcement training, occur regularly and are thought to be enriching for animals. However, there is little empirical information on how animals perceive these interactions or on the interactions’ effects on animals’ emotional states. Our objective was to assess the effectiveness of infrared thermography for measuring the emotional responses of three western lowland gorillas at the Detroit Zoo to routine interactions (positive reinforcement training and cognitive tasks) with familiar humans. In addition to thermal images, we collected saliva samples for hormone analysis before and after human–animal interactions and a control condition, and we recorded behavioral data during all conditions. Nasal temperatures consistently decreased for two gorillas during interactions, while the third gorilla showed repeated increases. The behavior of all three gorillas suggested that they were engaged in the interactions, without exhibiting behaviors that could indicate negative welfare impacts. Oxytocin and cortisol both decreased following all conditions, including the control, and were thus equivocal for interpreting the meaning of the changes in nasal temperature. As mixed results in previous research show, infrared thermography may detect emotional arousal; however, additional indicators are necessary to determine the valence of the observed changes. The variability in responses we observed do not lend themselves to making firm conclusions about the validity of infrared thermography (IRT) for measuring emotion in this context or about how these gorillas responded to interactions. Challenges and suggestions for future studies using infrared thermography to examine interactions between humans and zoo animals are discussed. Abstract Interactions between zoo professionals and animals occur regularly and are believed to be enriching for animals. Little empirical information exists on how animals perceive these interactions, and particularly how the interactions affect the emotional states of animals. Infrared thermography (IRT) has shown some promise in the assessment of emotions in a variety of species, but further research is needed to determine if this method is useful in a zoo setting. We conducted a pilot study to determine if IRT is a valid measure of the emotional responses to routine interactions (positive reinforcement training and cognitive tasks, compared to a control condition) with familiar humans on three western lowland gorillas at the Detroit Zoo. We measured nasal temperatures associated with emotional change using IRT. To examine the validity of the IRT data, we collected saliva samples for hormone analysis before and after each condition, in addition to behavioral data during the interactions and control condition. Decreases in nasal temperatures for two gorillas and an increase in the third indicate that arousal changed consistently within individuals following the interactions but not the control condition. Pre-post cortisol levels and oxytocin concentrations decreased for all conditions, but the decreases seen did not differ among the conditions. The gorillas were highly engaged in the interactions, and two produced more grumble vocalizations during the human-animal interactions (HAIs) compared to the control condition. Additionally, the gorillas performed self-directed behaviors more often during the control condition, also suggesting HAIs were not a negative experience. In summary, we were able to measure changes in arousal using IRT, but we were unable to determine the emotional valence of these changes based on the additional indicators employed. Additionally, the inconsistency across these measures precluded firm conclusions about either the validity of IRT for measuring emotion in this context or how the interactions impacted these gorillas. These findings highlight the challenges of using IRT to measure emotional states in non-human animals, and we discuss further steps necessary to apply this method in future studies.
Collapse
|
42
|
Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction. SENSORS 2019; 19:s19132844. [PMID: 31248004 PMCID: PMC6650968 DOI: 10.3390/s19132844] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/20/2019] [Revised: 06/21/2019] [Accepted: 06/22/2019] [Indexed: 11/17/2022]
Abstract
Child-Robot Interaction (CRI) has become increasingly addressed in research and applications. This work proposes a system for emotion recognition in children, recording facial images by both visual (RGB-red, green and blue) and Infrared Thermal Imaging (IRTI) cameras. For this purpose, the Viola-Jones algorithm is used on color images to detect facial regions of interest (ROIs), which are transferred to the thermal camera plane by multiplying a homography matrix obtained through the calibration process of the camera system. As a novelty, we propose to compute the error probability for each ROI located over thermal images, using a reference frame manually marked by a trained expert, in order to choose that ROI better placed according to the expert criteria. Then, this selected ROI is used to relocate the other ROIs, increasing the concordance with respect to the reference manual annotations. Afterwards, other methods for feature extraction, dimensionality reduction through Principal Component Analysis (PCA) and pattern classification by Linear Discriminant Analysis (LDA) are applied to infer emotions. The results show that our approach for ROI locations may track facial landmarks with significant low errors with respect to the traditional Viola-Jones algorithm. These ROIs have shown to be relevant for recognition of five emotions, specifically disgust, fear, happiness, sadness, and surprise, with our recognition system based on PCA and LDA achieving mean accuracy (ACC) and Kappa values of 85.75% and 81.84%, respectively. As a second stage, the proposed recognition system was trained with a dataset of thermal images, collected on 28 typically developing children, in order to infer one of five basic emotions (disgust, fear, happiness, sadness, and surprise) during a child-robot interaction. The results show that our system can be integrated to a social robot to infer child emotions during a child-robot interaction.
Collapse
|
43
|
Goulart C, Valadão C, Delisle-Rodriguez D, Caldeira E, Bastos T. Emotion analysis in children through facial emissivity of infrared thermal imaging. PLoS One 2019; 14:e0212928. [PMID: 30893343 PMCID: PMC6426206 DOI: 10.1371/journal.pone.0212928] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2018] [Accepted: 02/12/2019] [Indexed: 11/25/2022] Open
Abstract
Physiological signals may be used as objective markers to identify emotions, which play relevant roles in social and daily life. To measure these signals, the use of contact-free techniques, such as Infrared Thermal Imaging (IRTI), is indispensable to individuals who have sensory sensitivity. The goal of this study is to propose an experimental design to analyze five emotions (disgust, fear, happiness, sadness and surprise) from facial thermal images of typically developing (TD) children aged 7–11 years using emissivity variation, as recorded by IRTI. For the emotion analysis, a dataset considered emotional dimensions (valence and arousal), facial bilateral sides and emotion classification accuracy. The results evidence the efficiency of the experimental design with interesting findings, such as the correlation between the valence and the thermal decrement in nose; disgust and happiness as potent triggers of facial emissivity variations; and significant emissivity variations in nose, cheeks and periorbital regions associated with different emotions. Moreover, facial thermal asymmetry was revealed with a distinct thermal tendency in the cheeks, and classification accuracy reached a mean value greater than 85%. From the results, the emissivity variations were an efficient marker to analyze emotions in facial thermal images, and IRTI was confirmed to be an outstanding technique to study emotions. This study contributes a robust dataset to analyze the emotions of 7-11-year-old TD children, an age range for which there is a gap in the literature.
Collapse
Affiliation(s)
- Christiane Goulart
- Northeast Network of Biotechnology (RENORBIO), Postgraduate Program in Biotechnology, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
- * E-mail:
| | - Carlos Valadão
- Postgraduate Program in Electrical Engineering, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
| | - Denis Delisle-Rodriguez
- Postgraduate Program in Electrical Engineering, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
- Center of Medical Biophysics, University of Oriente, Santiago de Cuba, Cuba
| | - Eliete Caldeira
- Postgraduate Program in Electrical Engineering, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
| | - Teodiano Bastos
- Northeast Network of Biotechnology (RENORBIO), Postgraduate Program in Biotechnology, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
- Postgraduate Program in Electrical Engineering, Assistive Technology Group, Federal University of Espirito Santo, Vitoria, Espirito Santo, Brazil
| |
Collapse
|
44
|
Derakhshan A, Mikaeili M, Nasrabadi AM, Gedeon T. Network physiology of 'fight or flight' response in facial superficial blood vessels. Physiol Meas 2019; 40:014002. [PMID: 30523843 DOI: 10.1088/1361-6579/aaf089] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE We introduced a novel framework to identify the dynamic pattern of blood flow changes in the cutaneous superficial blood vessels of the face for 'fight or flight' responses through facial thermal imaging. APPROACH For this purpose, a thermal dataset was collected from 41 subjects in a mock crime scenario. Five facial areas including periorbital, forehead, perinasal, cheek and chin were selected on the face. Due to the cause and effect movement of blood in the facial cutaneous vasculature, the effective connectivity approach and graph analysis were used to extract causality features. The effective connectivity was quantified using a modified version of the multivariate Granger causality (GC) method among each pair of facial region of interests. MAIN RESULTS Validation was performed using statistical analysis, and the results demonstrated that the proposed method was statistically significant in detecting the physiological pattern of deceptive anxiety on the face. Moreover, the obtained graph is visualized by different schemes to show these interactions more effectively. We used machine learning techniques to classify our data based on the GC values, which result in a greater than 87% accuracy rate in discriminating between deceptive and truthful subjects.
Collapse
Affiliation(s)
- Amin Derakhshan
- Biomedical Engineering Department, Shahed University, Tehran, Iran
| | | | | | | |
Collapse
|
45
|
Ponsi G, Monachesi B, Panasiti V, Aglioti SM, Panasiti MS. Physiological and behavioral reactivity to social exclusion: a functional infrared thermal imaging study in patients with psoriasis. J Neurophysiol 2018; 121:38-49. [PMID: 30379630 DOI: 10.1152/jn.00555.2018] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022] Open
Abstract
Recent studies have shown that sympathetic nervous system (SNS) activity can be heavily impacted not only by basic threats to survival but also by threats to social bonds. In this study we explored the behavioral and physiological consequences of social exclusion/inclusion in patients with psoriasis, a disease frequently associated with the experience of being ostracized and with deficient emotion regulation skills. We employed a virtual ball-tossing game (Cyberball) to induce the experience of social exclusion/inclusion. We then used a Trust Game to measure the effects of this social modulation on trust. During Cyberball, infrared thermal imaging was used to record participants' facial temperature and thus obtain an online measure of SNS activation. Behavioral data showed that social exclusion shifted participants' trust toward unfamiliar players who had not previously excluded them. Physiological data indicated that in control participants, social exclusion triggered higher SNS activation than inclusion. No such effect was found in patients with psoriasis, whose SNS activity was the same during inclusion as it was during exclusion, suggesting that they benefit less from inclusive experiences than control participants. In addition, in patients but not in controls, higher SNS activation during social exclusion was linked to higher monetary investment toward unfamiliar players, a result in keeping with the social reconnection hypothesis, according to which emotions triggered by social rejection can be regulated by investing in new social interactions. We also found that an increase in periorbital temperature is accompanied by a decrease in happiness ratings after social exclusion was experienced during the Cyberball game. NEW & NOTEWORTHY Previous research on emotional processes in psoriasis has mainly employed self-report measures. In this study we used thermal imaging to obtain an online measure of the sympathetic nervous system (SNS) activity during social exclusion and tested how this experience influenced subsequent trust. We found that being included was a less positive experience for patients compared with controls and that SNS activity during exclusion had a stronger influence on subsequent trust in patients than in controls.
Collapse
Affiliation(s)
- Giorgia Ponsi
- Department of Psychology, Sapienza University of Rome , Rome , Italy.,IRCCS Santa Lucia Foundation , Rome , Italy
| | - Bianca Monachesi
- Department of Psychology, Sapienza University of Rome , Rome , Italy
| | - Vincenzo Panasiti
- Plastic and Reconstructive Surgery Unit, Campus Bio-Medico University of Rome , Rome , Italy
| | - Salvatore Maria Aglioti
- Department of Psychology, Sapienza University of Rome , Rome , Italy.,IRCCS Santa Lucia Foundation , Rome , Italy
| | - Maria Serena Panasiti
- Department of Psychology, Sapienza University of Rome , Rome , Italy.,IRCCS Santa Lucia Foundation , Rome , Italy
| |
Collapse
|
46
|
Panasiti MS, Ponsi G, Monachesi B, Lorenzini L, Panasiti V, Aglioti SM. Cognitive load and emotional processing in psoriasis: a thermal imaging study. Exp Brain Res 2018; 237:211-222. [PMID: 30374785 DOI: 10.1007/s00221-018-5416-y] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2018] [Accepted: 10/24/2018] [Indexed: 10/28/2022]
Abstract
Psoriasis is a chronic dermatologic disease which is frequently associated with psychological distress. Although studies suggest a relationship between this condition and difficulties in emotion regulation, behavioral and physiological evidence about this link is scarce. We measured implicit emotion regulation abilities of psoriasis patients and a healthy control group by examining the impact of distracting emotional (positive, negative or neutral) images on a working memory task ("Emotional N-Back") which could present high (2-back) or low (1-back) cognitive workload. Moreover, we used Functional Infrared Thermal Imaging to record participants' facial temperature and obtain a measure of the activation of the autonomic system. Rising of temperature over the peri-orbital areas and the nose tip are believed to reflect the activation and the de-activation of the sympathetic system, respectively. Patients scored higher than controls on the "Lack of emotional clarity" sub-scale of the Difficulties in Emotion Regulation Scale. Compared to controls, who performed much better in the low vs. high cognitive load condition, patients showed a smaller accuracy difference between the two conditions. Moreover, patients showed less sympathetic (lower peri-orbital and higher nasal tip temperature) activity (especially in the negative and neutral blocks) during the high vs. low cognitive load condition, suggesting that the former condition might be less emotionally demanding for them. Patients benefit more than controls from the load-dependent interference effect when dealing with emotional information; thus, therapeutic techniques aiming at teaching how to use cognitive strategies to downregulate emotions might be particularly appropriated for them.
Collapse
Affiliation(s)
- Maria Serena Panasiti
- Department of Psychology, Sapienza University of Rome, Via dei Marsi, 78, 00185, Rome, Italy. .,IRCCS Santa Lucia Foundation, Rome, Italy.
| | - Giorgia Ponsi
- Department of Psychology, Sapienza University of Rome, Via dei Marsi, 78, 00185, Rome, Italy.,IRCCS Santa Lucia Foundation, Rome, Italy
| | - Bianca Monachesi
- Department of Psychology, Sapienza University of Rome, Via dei Marsi, 78, 00185, Rome, Italy.,IRCCS Santa Lucia Foundation, Rome, Italy
| | - Luigi Lorenzini
- Department of Psychology, Sapienza University of Rome, Via dei Marsi, 78, 00185, Rome, Italy.,IRCCS Santa Lucia Foundation, Rome, Italy
| | - Vincenzo Panasiti
- Plastic and Reconstructive Surgery Unit, Campus Bio-Medico University of Rome, Rome, Italy
| | - Salvatore Maria Aglioti
- Department of Psychology, Sapienza University of Rome, Via dei Marsi, 78, 00185, Rome, Italy.,IRCCS Santa Lucia Foundation, Rome, Italy
| |
Collapse
|