1
|
Pagano TP, dos Santos LL, Santos VR, Sá PHM, Bonfim YDS, Paranhos JVD, Ortega LL, Nascimento LFS, Santos A, Rönnau MM, Winkler I, Nascimento EGS. Remote Heart Rate Prediction in Virtual Reality Head-Mounted Displays Using Machine Learning Techniques. Sensors (Basel) 2022; 22:9486. [PMID: 36502188 PMCID: PMC9738680 DOI: 10.3390/s22239486] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Revised: 11/22/2022] [Accepted: 11/30/2022] [Indexed: 06/17/2023]
Abstract
Head-mounted displays are virtual reality devices that may be equipped with sensors and cameras to measure a patient's heart rate through facial regions. Heart rate is an essential body signal that can be used to remotely monitor users in a variety of situations. There is currently no study that predicts heart rate using only highlighted facial regions; thus, an adaptation is required for beats per minute predictions. Likewise, there are no datasets containing only the eye and lower face regions, necessitating the development of a simulation mechanism. This work aims to remotely estimate heart rate from facial regions that can be captured by the cameras of a head-mounted display using state-of-the-art EVM-CNN and Meta-rPPG techniques. We developed a region of interest extractor to simulate a dataset from a head-mounted display device using stabilizer and video magnification techniques. Then, we combined support vector machine and FaceMash to determine the regions of interest and adapted photoplethysmography and beats per minute signal predictions to work with the other techniques. We observed an improvement of 188.88% for the EVM and 55.93% for the Meta-rPPG. In addition, both models were able to predict heart rate using only facial regions as input. Moreover, the adapted technique Meta-rPPG outperformed the original work, whereas the EVM adaptation produced comparable results for the photoplethysmography signal.
Collapse
Affiliation(s)
- Tiago Palma Pagano
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | - Lucas Lisboa dos Santos
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | - Victor Rocha Santos
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | - Paulo H. Miranda Sá
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | - Yasmin da Silva Bonfim
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | | | - Lucas Lemos Ortega
- Computational Modeling Department, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | | | - Alexandre Santos
- HP Inc. Brazil R&D, Porto Alegre 90619-900, Rio Grande do Sul, Brazil
| | | | - Ingrid Winkler
- Department of Management and Industrial Technology, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
| | - Erick G. Sperandio Nascimento
- Department of Management and Industrial Technology, SENAI CIMATEC University Center, Salvador 41650-010, Bahia, Brazil
- Faculty of Engineering and Physical Sciences, School of Computer Science and Electronic Engineering, Surrey Institute for People-Centred AI, University of Surrey, Guildford GU2 7XH, UK
| |
Collapse
|