MOMBAT: Heart rate monitoring from face video using pulse modeling and Bayesian tracking

Remote physiological signal recovery with efficient spatio-temporal modeling

Contactless physiological signal measurement has great applications in various fields, such as affective computing and health monitoring. Physiological measurements based on remote photoplethysmography (rPPG) are realized by capturing the weak periodic color changes. The changes are caused by the variation in the light absorption of skin surface during systole and diastole stages of a functioning heart. This measurement mode has advantages of contactless measurement, simple operation, low cost, etc. In recent years, several deep learning-based rPPG measurement methods have been proposed. However, the features learned by deep learning models are vulnerable to motion and illumination artefacts, and are unable to fully exploit the intrinsic temporal characteristics of the rPPG. This paper presents an efficient spatiotemporal modeling-based rPPG recovery method for physiological signal measurements. First, two modules are utilized in the rPPG task: 1) 3D central difference convolution for temporal context modeling with enhanced representation and generalization capacity, and 2) Huber loss for robust intensity-level rPPG recovery. Second, a dual branch structure for both motion and appearance modeling and a soft attention mask are adapted to take full advantage of the central difference convolution. Third, a multi-task setting for joint cardiac and respiratory signals measurements is introduced to benefit from the internal relevance between two physiological signals. Last, extensive experiments performed on three public databases show that the proposed method outperforms prior state-of-the-art methods with the Pearson's correlation coefficient higher than 0.96 on all three datasets. The generalization ability of the proposed method is also evaluated by cross-database and video compression experiments. The effectiveness and necessity of each module are confirmed by ablation studies.

Deep learning and remote photoplethysmography powered advancements in contactless physiological measurement

In recent decades, there has been ongoing development in the application of computer vision (CV) in the medical field. As conventional contact-based physiological measurement techniques often restrict a patient's mobility in the clinical environment, the ability to achieve continuous, comfortable and convenient monitoring is thus a topic of interest to researchers. One type of CV application is remote imaging photoplethysmography (rPPG), which can predict vital signs using a video or image. While contactless physiological measurement techniques have an excellent application prospect, the lack of uniformity or standardization of contactless vital monitoring methods limits their application in remote healthcare/telehealth settings. Several methods have been developed to improve this limitation and solve the heterogeneity of video signals caused by movement, lighting, and equipment. The fundamental algorithms include traditional algorithms with optimization and developing deep learning (DL) algorithms. This article aims to provide an in-depth review of current Artificial Intelligence (AI) methods using CV and DL in contactless physiological measurement and a comprehensive summary of the latest development of contactless measurement techniques for skin perfusion, respiratory rate, blood oxygen saturation, heart rate, heart rate variability, and blood pressure.

Robust Pulse Rate Measurements from Facial Videos in Diverse Environments

Pulse wave and pulse rate are important indicators of cardiovascular health. Technologies that can check the pulse by contacting the skin with optical sensors built into smart devices have been developed. However, this may cause inconvenience, such as foreign body sensation. Accordingly, studies have been conducted on non-contact pulse rate measurements using facial videos focused on the indoors. Moreover, since the majority of studies are conducted indoors, the error in the pulse rate measurement in outdoor environments, such as an outdoor bench, car and drone, is high. In this paper, to deal with this issue, we focus on developing a robust pulse measurement method based on facial videos taken in diverse environments. The proposed method stably detects faces by removing high-frequency components of face coordinate signals derived from fine body tremors and illumination conditions. It optimizes for extracting skin color changes by reducing illumination-caused noise using the Cg color difference component. The robust pulse wave is extracted from the Cg signal using FFT-iFFT with zero-padding. It can eliminate signal-filtering distortion effectively. We demonstrate that the proposed method relieves pulse rate measurement problems, producing 3.36, 5.81, and 6.09 bpm RMSE for an outdoor bench, driving car, and flying drone, respectively.

Unobtrusive Heart Rate Monitoring using Near-Infrared Imaging During Driving

In-vehicle health monitoring allows for continuous vital sign measurement in everyday life. Eventually, this could lead to early detection of cardiovascular diseases. In this work, we propose non-contact heart rate (HR) monitoring utilizing near-infrared (NIR) camera technology. Ten healthy volunteers are monitored in a realistic driving simulator during resting (5 min) and driving (10 min). We synchronously acquire videos using an out-of-the-shelf, low-cost NIR camera and 3-lead electrocardiography (ECG) serves as ground truth. The MediaPipe face detector delivers the region of interest (ROI) and we determine the HR from the peak with maximum amplitude within the power spectrum of skin color changes. We compare video-based with ECG-based HR, resulting in a mean absolute error (MAE) of 7.8 bpm and 13.0 bpm in resting and driving condition, respectively. As we apply only a simple signal processing pipeline without sophisticated filtering, we conclude that NIR camera-based HR measurements enables unobtrusive and non-contact monitoring to a certain extent, but artifacts from subject movement pose a challenge. If these issues can be addressed, continuous vital sign measurement in everyday life could become reality.

Availability and performance of face based non-contact methods for heart rate and oxygen saturation estimations: A systematic review

BACKGROUND

Consumer-level cameras have provided an advantage of designing cost-effective, non-contact physiological parameters estimation approaches which is not possible with gold standard estimation techniques. This encourages the development of non-contact estimation methods using camera technology. Therefore, this work aims to present a systematic review summarizing the currently existing face-based non-contact methods along with their performance.

METHODS

This review includes all heart rate (HR) and oxygen saturation (SpO2) studies published in journals and a few reputed conferences, which have compared the proposed estimation methods with one or more standard reference devices. The articles were collected from the following research databases: Institute of Electrical and Electronics Engineers (IEEE), PubMed, Web of Science (WoS), Science Direct, and Association of Computer Machinery (ACM) digital library. All database searches were completed on May 20, 2021. Each study was assessed using a finite set of identified factors for reporting bias.

RESULTS

Out of 332 identified studies, 32 studies were selected for the final review. Additionally, 18 studies were included by thoroughly checking these studies. 3 out of 50 (6%) studies were performed in clinical conditions, while the remaining studies were carried out on a healthy population. 42 out of 50 (84%) studies have estimated HR, while 5/50 (10%) studies have measured SpO₂ only. The remaining three studies have estimated both parameters. The majority of the studies have used 1-3 min videos for estimation. Among the estimation methods, Deep Learning and Independent component analysis (ICA) were used by 11/42 (26.19%) and 9/42 (21.42%) studies, respectively. According to the Bland-Altman analysis, only 8/45 (17.77%) HR studies achieved the clinically accepted error limits whereas, for SpO₂, 4/5 (80%) studies have matched the industry standards (±3%).

DISCUSSION

Deep Learning and ICA have been predominantly used for HR estimations. Among deep learning estimation methods, convolutional neural networks have been employed till date due to their good generalization ability. Most non-contact HR estimation methods need significant improvements to implement these methods in a clinical environment. Furthermore, these methods need to be tested on the subjects suffering from any related disease. SpO₂ estimation studies are challenging and need to be tested by conducting hypoxemic events. The authors would encourage reporting the detailed information about the study population, the use of longer videos, and appropriate performance metrics and testing under abnormal HR and SpO₂ ranges for future estimation studies.

Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review

In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.

AND-rPPG: A novel denoising-rPPG network for improving remote heart rate estimation

Heart rate (HR) estimation is an essential physiological parameter in the field of biomedical imaging. Remote Photoplethysmography (r-PPG) is a pathbreaking development in this field wherein the PPG signal is extracted from non-contact face videos. In the COVID-19 pandemic, rPPG plays a vital role for doctors and patients to perform telehealthcare. Existing rPPG methods provide incorrect HR estimation when face video contains facial deformations induced by facial expression. These methods process the entire face and utilize the same knowledge to mitigate different noises. It limits the performance of these methods because different facial expressions induce different noise characteristics depending on the facial region. Another limitation is that these methods neglect the facial expression for denoising even though it is the prominent noise source in temporal signals. These issues are mitigated in this paper by proposing a novel HR estimation method AND-rPPG, that is, A Novel Denoising-rPPG. We initiate the utilization of Action Units (AUs) for denoising temporal signals. Our denoising network models the temporal signals better than sequential architectures and mitigate the AUs-based (or face expression-based) noises effectively. The experiments performed on publicly available datasets reveal that our proposed method outperforms state-of-the-art HR estimation methods, and our denoising model can be easily integrated with existing methods to improve their HR estimation.