1
|
Rudokaite J, Ertugrul IO, Ong S, Janssen MP, Huis In 't Veld E. Predicting Vasovagal Reactions to Needles from Facial Action Units. J Clin Med 2023; 12. [PMID: 36836177 DOI: 10.3390/jcm12041644] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 02/14/2023] [Accepted: 02/16/2023] [Indexed: 02/22/2023] Open
Abstract
BACKGROUND Merely the sight of needles can cause extreme emotional and physical (vasovagal) reactions (VVRs). However, needle fear and VVRs are not easy to measure nor prevent as they are automatic and difficult to self-report. This study aims to investigate whether a blood donors' unconscious facial microexpressions in the waiting room, prior to actual blood donation, can be used to predict who will experience a VVR later, during the donation. METHODS The presence and intensity of 17 facial action units were extracted from video recordings of 227 blood donors and were used to classify low and high VVR levels using machine-learning algorithms. We included three groups of blood donors as follows: (1) a control group, who had never experienced a VVR in the past (n = 81); (2) a 'sensitive' group, who experienced a VVR at their last donation (n = 51); and (3) new donors, who are at increased risk of experiencing a VVR (n = 95). RESULTS The model performed very well, with an F1 (=the weighted average of precision and recall) score of 0.82. The most predictive feature was the intensity of facial action units in the eye regions. CONCLUSIONS To our knowledge, this study is the first to demonstrate that it is possible to predict who will experience a vasovagal response during blood donation through facial microexpression analyses prior to donation.
Collapse
|
2
|
Auflem M, Kohtala S, Jung M, Steinert M. Facing the FACS-Using AI to Evaluate and Control Facial Action Units in Humanoid Robot Face Development. Front Robot AI 2022; 9:887645. [PMID: 35774595 PMCID: PMC9237251 DOI: 10.3389/frobt.2022.887645] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Accepted: 05/11/2022] [Indexed: 11/13/2022] Open
Abstract
This paper presents a new approach for evaluating and controlling expressive humanoid robotic faces using open-source computer vision and machine learning methods. Existing research in Human-Robot Interaction lacks flexible and simple tools that are scalable for evaluating and controlling various robotic faces; thus, our goal is to demonstrate the use of readily available AI-based solutions to support the process. We use a newly developed humanoid robot prototype intended for medical training applications as a case example. The approach automatically captures the robot’s facial action units through a webcam during random motion, which are components traditionally used to describe facial muscle movements in humans. Instead of manipulating the actuators individually or training the robot to express specific emotions, we propose using action units as a means for controlling the robotic face, which enables a multitude of ways to generate dynamic motion, expressions, and behavior. The range of action units achieved by the robot is thus analyzed to discover its expressive capabilities and limitations and to develop a control model by correlating action units to actuation parameters. Because the approach is not dependent on specific facial attributes or actuation capabilities, it can be used for different designs and continuously inform the development process. In healthcare training applications, our goal is to establish a prerequisite of expressive capabilities of humanoid robots bounded by industrial and medical design constraints. Furthermore, to mediate human interpretation and thus enable decision-making based on observed cognitive, emotional, and expressive cues, our approach aims to find the minimum viable expressive capabilities of the robot without having to optimize for realism. The results from our case example demonstrate the flexibility and efficiency of the presented AI-based solutions to support the development of humanoid facial robots.
Collapse
Affiliation(s)
- Marius Auflem
- TrollLABS, Department of Mechanical and Industrial Engineering, Faculty of Engineering, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Sampsa Kohtala
- TrollLABS, Department of Mechanical and Industrial Engineering, Faculty of Engineering, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Malte Jung
- Robots in Groups Lab, Department of Information Science, Cornell University, Ithaca, NY, United States
| | - Martin Steinert
- TrollLABS, Department of Mechanical and Industrial Engineering, Faculty of Engineering, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| |
Collapse
|
3
|
Yap CH, Cunningham R, Davison AK, Yap MH. Synthesising Facial Macro- and Micro-Expressions Using Reference Guided Style Transfer. J Imaging 2021; 7:142. [PMID: 34460778 PMCID: PMC8404916 DOI: 10.3390/jimaging7080142] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Revised: 08/01/2021] [Accepted: 08/06/2021] [Indexed: 11/17/2022] Open
Abstract
Long video datasets of facial macro- and micro-expressions remains in strong demand with the current dominance of data-hungry deep learning methods. There are limited methods of generating long videos which contain micro-expressions. Moreover, there is a lack of performance metrics to quantify the generated data. To address the research gaps, we introduce a new approach to generate synthetic long videos and recommend assessment methods to inspect dataset quality. For synthetic long video generation, we use the state-of-the-art generative adversarial network style transfer method-StarGANv2. Using StarGANv2 pre-trained on the CelebA dataset, we transfer the style of a reference image from SAMM long videos (a facial micro- and macro-expression long video dataset) onto a source image of the FFHQ dataset to generate a synthetic dataset (SAMM-SYNTH). We evaluate SAMM-SYNTH by conducting an analysis based on the facial action units detected by OpenFace. For quantitative measurement, our findings show high correlation on two Action Units (AUs), i.e., AU12 and AU6, of the original and synthetic data with a Pearson's correlation of 0.74 and 0.72, respectively. This is further supported by evaluation method proposed by OpenFace on those AUs, which also have high scores of 0.85 and 0.59. Additionally, optical flow is used to visually compare the original facial movements and the transferred facial movements. With this article, we publish our dataset to enable future research and to increase the data pool of micro-expressions research, especially in the spotting task.
Collapse
Affiliation(s)
- Chuin Hong Yap
- Department of Computing and Mathematics, Manchester Metropolitan University, Manchester M15 6BH, UK; (R.C.); (M.H.Y.)
| | - Ryan Cunningham
- Department of Computing and Mathematics, Manchester Metropolitan University, Manchester M15 6BH, UK; (R.C.); (M.H.Y.)
| | - Adrian K. Davison
- Faculty of Biology, Medicine and Health, The University of Manchester, Manchester M13 9PL, UK;
| | - Moi Hoon Yap
- Department of Computing and Mathematics, Manchester Metropolitan University, Manchester M15 6BH, UK; (R.C.); (M.H.Y.)
| |
Collapse
|
4
|
Andersen PH, Broomé S, Rashid M, Lundblad J, Ask K, Li Z, Hernlund E, Rhodin M, Kjellström H. Towards Machine Recognition of Facial Expressions of Pain in Horses. Animals (Basel) 2021; 11:1643. [PMID: 34206077 PMCID: PMC8229776 DOI: 10.3390/ani11061643] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2021] [Revised: 05/26/2021] [Accepted: 05/28/2021] [Indexed: 01/30/2023] Open
Abstract
Automated recognition of human facial expressions of pain and emotions is to a certain degree a solved problem, using approaches based on computer vision and machine learning. However, the application of such methods to horses has proven difficult. Major barriers are the lack of sufficiently large, annotated databases for horses and difficulties in obtaining correct classifications of pain because horses are non-verbal. This review describes our work to overcome these barriers, using two different approaches. One involves the use of a manual, but relatively objective, classification system for facial activity (Facial Action Coding System), where data are analyzed for pain expressions after coding using machine learning principles. We have devised tools that can aid manual labeling by identifying the faces and facial keypoints of horses. This approach provides promising results in the automated recognition of facial action units from images. The second approach, recurrent neural network end-to-end learning, requires less extraction of features and representations from the video but instead depends on large volumes of video data with ground truth. Our preliminary results suggest clearly that dynamics are important for pain recognition and show that combinations of recurrent neural networks can classify experimental pain in a small number of horses better than human raters.
Collapse
Affiliation(s)
- Pia Haubro Andersen
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Sofia Broomé
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
| | - Maheen Rashid
- Department of Computer Science, University of California at Davis, California, CA 95616, USA;
| | - Johan Lundblad
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Katrina Ask
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Zhenghong Li
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
- Department of Computer Science, Stony Brook University, New York, NY 11794, USA
| | - Elin Hernlund
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Marie Rhodin
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Hedvig Kjellström
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
| |
Collapse
|
5
|
Gunes H, Celiktutan O, Sariyanidi E. Live human-robot interactive public demonstrations with automatic emotion and personality prediction. Philos Trans R Soc Lond B Biol Sci 2019; 374:20180026. [PMID: 30853000 PMCID: PMC6452249 DOI: 10.1098/rstb.2018.0026] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/14/2019] [Indexed: 02/05/2023] Open
Abstract
Communication with humans is a multi-faceted phenomenon where the emotions, personality and non-verbal behaviours, as well as the verbal behaviours, play a significant role, and human-robot interaction (HRI) technologies should respect this complexity to achieve efficient and seamless communication. In this paper, we describe the design and execution of five public demonstrations made with two HRI systems that aimed at automatically sensing and analysing human participants' non-verbal behaviour and predicting their facial action units, facial expressions and personality in real time while they interacted with a small humanoid robot. We describe an overview of the challenges faced together with the lessons learned from those demonstrations in order to better inform the science and engineering fields to design and build better robots with more purposeful interaction capabilities. This article is part of the theme issue 'From social brains to social robots: applying neurocognitive insights to human-robot interaction'.
Collapse
Affiliation(s)
- Hatice Gunes
- Department of Computer Science and Technology, University of Cambridge, Cambridge CB3 0FD, UK
| | - Oya Celiktutan
- Centre for Robotics Research, Department of Informatics, King’s College London, London WC2R 2LS, UK
| | | |
Collapse
|
6
|
Atee M, Hoti K, Parsons R, Hughes JD. A novel pain assessment tool incorporating automated facial analysis: interrater reliability in advanced dementia. Clin Interv Aging 2018; 13:1245-1258. [PMID: 30038491 PMCID: PMC6052926 DOI: 10.2147/cia.s168024] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022] Open
Abstract
OBJECTIVES Regardless of its severity, dementia does not negate the experience of pain. Rather, dementia hinders self-reporting mechanisms in affected individuals because they lose the ability to do so. The primary aim of this study was to examine the interrater reliability of the electronic Pain Assessment Tool (ePAT) among raters when assessing pain in residents with moderate-to-severe dementia. Secondly, it sought to examine the relationship between total instrument scores and facial scores, as determined by automated facial expression analysis. STUDY DESIGN A 2-week observational study. SETTING An accredited, high-care, and dementia-specific residential aged care facility in Perth, Western Australia. PARTICIPANTS Subjects were 10 residents (age range: 63.1-84.4 years old) predominantly with severe dementia (Dementia Severity Rating Scale score: 46.3±8.4) rated for pain by 11 aged care staff. Raters (female: 82%; mean age: 44.1±12.6 years) consisted of one clinical nurse, four registered nurses, five enrolled nurses, and one care worker. MEASUREMENTS ePAT measured pain using automated detection of facial action codes and recordings of pain behaviors. RESULTS A total of 76 assessments (rest =38 [n=19 pairs], movement =38 [n=19 pairs]) were conducted. At rest, raters' agreement was excellent on overall total scores (coefficient of concordance =0.92 [95% CI: 0.85-0.96]) and broad category scores (κ=1.0). Agreement was moderate (κ=0.59) on categorical scores upon movement, while it was exact in 68.4% of the cases. Agreement in actual pain category scores gave κw=0.72 (95% CI: 0.58-0.86) at rest and κw=0.69 (95% CI: 0.50-0.87) with movement. All raters scored residents with higher total scores post-mobilization compared to rest. More facial action unit codes were also detected during pain (mean: 2.5 vs 1.9; p<0.0012) and following mobilization (mean: 2.5 vs 1.7; p<0.0001) compared to no pain and rest, respectively. CONCLUSIONS ePAT, which combines automated facial expression analysis and clinical behavioral indicators in a single observational pain assessment tool, demonstrates good reliability properties, which supports its appropriateness for use in residents with advanced dementia.
Collapse
Affiliation(s)
- Mustafa Atee
- School of Pharmacy and Biomedical Sciences, Faculty of Health Sciences, Curtin University, Bentley, WA, Australia,
| | - Kreshnik Hoti
- School of Pharmacy and Biomedical Sciences, Faculty of Health Sciences, Curtin University, Bentley, WA, Australia,
- Division of Pharmacy, Faculty of Medicine, University of Pristina, Prishtina, Kosovo
| | - Richard Parsons
- School of Pharmacy and Biomedical Sciences, Faculty of Health Sciences, Curtin University, Bentley, WA, Australia,
| | - Jeffery D Hughes
- School of Pharmacy and Biomedical Sciences, Faculty of Health Sciences, Curtin University, Bentley, WA, Australia,
| |
Collapse
|
7
|
Xu X, Craig KD, Diaz D, Goodwin MS, Akcakaya M, Susam BT, Huang JS, de Sa VR. Automated Pain Detection in Facial Videos of Children using Human-Assisted Transfer Learning. CEUR Workshop Proc 2018; 2142:10-21. [PMID: 30713485 PMCID: PMC6352979] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Accurately determining pain levels in children is difficult, even for trained professionals and parents. Facial activity provides sensitive and specific information about pain, and computer vision algorithms have been developed to automatically detect Facial Action Units (AUs) defined by the Facial Action Coding System (FACS). Our prior work utilized information from computer vision, i.e., automatically detected facial AUs, to develop classifiers to distinguish between pain and no-pain conditions. However, application of pain/no-pain classifiers based on automated AU codings across different environmental domains results in diminished performance. In contrast, classifiers based on manually coded AUs demonstrate reduced environmentally-based variability in performance. In this paper, we train a machine learning model to recognize pain using AUs coded by a computer vision system embedded in a software package called iMotions. We also study the relationship between iMotions (automatically) and human (manually) coded AUs. We find that AUs coded automatically are different from those coded by a human trained in the FACS system, and that the human coder is less sensitive to environmental changes. To improve classification performance in the current work, we applied transfer learning by training another machine learning model to map automated AU codings to a subspace of manual AU codings to enable more robust pain recognition performance when only automatically coded AUs are available for the test data. With this transfer learning method, we improved the Area Under the ROC Curve (AUC) on independent data from new participants in our target domain from 0.67 to 0.72.
Collapse
Affiliation(s)
- Xiaojing Xu
- Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA,
| | - Kenneth D Craig
- Department of Psychology, University of British Columbia Vancouver, BC, Canada,
| | - Damaris Diaz
- Rady Childrens Hospital and Department of Pediatrics, University of California San Diego, CA, USA, ,
| | - Matthew S Goodwin
- Department of Health Sciences, Northeastern University, Boston, MA, USA,
| | - Murat Akcakaya
- Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA, ,
| | - Büşra Tuğçe Susam
- Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, USA, ,
| | - Jeannie S Huang
- Rady Childrens Hospital and Department of Pediatrics, University of California San Diego, CA, USA, ,
| | - Virginia R de Sa
- Department of Cognitive Science, University of California San Diego, La Jolla, CA, USA,
| |
Collapse
|
8
|
Abstract
The authors investigated children's ability to recognize emotions from the information available in the lower, middle, or upper face. School-age children were shown partial or complete facial expressions and asked to say whether they corresponded to a given emotion (anger, fear, surprise, or disgust). The results indicate that 5-year-olds were able to recognize fear, anger, and surprise from partial facial expressions. Fear was better recognized from the information located in the upper face than those located in the lower face. A similar pattern of results was found for anger, but only in girls. Recognition improved between 5 and 10 years old for surprise and anger, but not for fear and disgust.
Collapse
|