A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation

Motion-Tolerant Non-Contact Heart-Rate Measurements from Radar Sensor Fusion

Microwave radar technology is very attractive for ubiquitous short-range health monitoring due to its non-contact, see-through, privacy-preserving and safe features compared to the competing remote technologies such as optics. The possibility of radar-based approaches for breathing and cardiac sensing was demonstrated a few decades ago. However, investigation regarding the robustness of radar-based vital-sign monitoring (VSM) is not available in the current radar literature. In this paper, we aim to close this gap by presenting an extensive experimental study of vital-sign radar approach. We consider diversity in test subjects, fitness levels, poses/postures, and, more importantly, random body movement (RBM) in the study. We discuss some new insights that lead to robust radar heart-rate (HR) measurements. A novel active motion cancellation signal-processing technique is introduced, exploiting dual ultra-wideband (UWB) radar system for motion-tolerant HR measurements. Additionally, we propose a spectral pruning routine to enhance HR estimation performance. We validate the proposed method theoretically and experimentally. Totally, we record and analyze about 3500 seconds of radar measurements from multiple human subjects.

A measurement of illumination variation-resistant noncontact heart rate based on the combination of singular spectrum analysis and sub-band method

BACKGROUND AND OBJECTIVE

The imaging photoplethysmography method is a non-contact and non-invasive measurement method that usually uses surrounding illumination as an illuminant, which can be easily influenced by the surrounding illumination variations. Thus, it has a practical value to develop an efficient method of heart rate measurement that can remove the interference of illumination variations robustly.

METHOD

We propose a novel framework of heart rate measurement that is robust to illumination variations by combining singular spectrum analysis and sub-band method. At first, we extract the blood volume pulse signal by applying the modified sub-band method to the raw facial RGB trace signals. Then the spectra for the interference of illumination variations are extracted from the raw signal obtained from facial regions of interest by using singular spectrum analysis. Finally, we estimate the more robust heart rate through comparison analysis between the spectra of the extracted blood volume pulse signal and the illumination variations.

RESULTS

We compared our method with several state-of-the-art methods through the analysis using the self-collected data and the UBFC-RPPG database. Bland-Altman plots and Pearson correlation coefficients pointed out that the proposed method could measure the heart rate more accurately than the state-of-the-art methods. For the self-collected data and the UBFC-RPPG database, Bland-Altman plots showed that proposed method caused better agreement with 95% limits from -4 bpm to 10 bpm and from -2 bpm to 4 bpm respectively than the other state-of-the-art methods. It also revealed that the highly linear relation was held between the estimated heart rate and ground truth with the correlation coefficients of 0.89 and 0.99, respectively.

CONCLUSION

By extracting illumination variation directly from the facial region of interest rather than from the background region of interest, the proposed method demonstrates that it can overcome the drawbacks of the previous methods due to the illumination variation difference between the background and facial regions of interest. It can be found that the proposed method has a relatively good robustness regardless of whether illumination variation exists or not.

Non-Contact HR Monitoring via Smartphone and Webcam During Different Respiratory Maneuvers and Body Movements

As a reliable indicator for individual's healthiness conditions, heart rate (HR) has been widely considered and used. Imaging photoplethysmography (iPPG) is recently highlighted as a promising HR measurement method, due to its non-contact characteristics, by extracting the HR from facial video recordings. In this study, we propose a camera-based HR monitoring technique that estimates HR information from iPPG signals extracted from a video sequence. Videos were recorded using a smartphone or a laptop camera. We adopted the plane-orthogonal-to-skin (POS) method to compute iPPG. The proposed method is evaluated by applying it to extract HR of 9 subjects at rest and during two motion conditions (lateral and frontal) while they were performing several respiratory maneuvers-spontaneous, metronome, and forced. Automatic face detection algorithms were implemented in the proposed method. Our experimental results show that mean values of HR have 0.56% error and 99.4% accuracy when compared to HR calculated from the gold-standard electrocardiography (ECG) reference in diverse conditions of motions and respiratory maneuvers.

Noncontact Sensing of Contagion

The World Health Organization (WHO) has declared COVID-19 a pandemic. We review and reduce the clinical literature on diagnosis of COVID-19 through symptoms that might be remotely detected as of early May 2020. Vital signs associated with respiratory distress and fever, coughing, and visible infections have been reported. Fever screening by temperature monitoring is currently popular. However, improved noncontact detection is sought. Vital signs including heart rate and respiratory rate are affected by the condition. Cough, fatigue, and visible infections are also reported as common symptoms. There are non-contact methods for measuring vital signs remotely that have been shown to have acceptable accuracy, reliability, and practicality in some settings. Each has its pros and cons and may perform well in some challenges but be inadequate in others. Our review shows that visible spectrum and thermal spectrum cameras offer the best options for truly noncontact sensing of those studied to date, thermal cameras due to their potential to measure all likely symptoms on a single camera, especially temperature, and video cameras due to their availability, cost, adaptability, and compatibility. Substantial supply chain disruptions during the pandemic and the widespread nature of the problem means that cost-effectiveness and availability are important considerations.

Research on the combination of color channels in heart rate measurement based on photoplethysmography imaging

SIGNIFICANCE

The measurement of human vital signs based on photoplethysmography imaging (PPGI) can be severely affected by the interference of various factors in the measurement process; therefore, a lot of complex signal processing techniques are used to remove the influence of the interference.

AIM

We comprehensively analyze several methods for color channel combination in the color spaces currently used in PPGI and determine the combination method that can improve the quality of the pulse signal, which results in a modified plane-orthogonal-to-skin based method (POS).

APPROACH

Based on the analysis of the previous studies, 13 methods for color channel combination in the different color spaces, which can be seen as having potential abilities in measuring vital signs, were compared by employing the average value of signal-to-noise ratio (SNR) and the box-plot in the public databases UBFC-RPPG and PURE. In addition, the pulse signal was extracted through the dual-color space transformation (sRGB → intensity normalized RGB → YCbCr) and fine-tuning on the CbCr plane.

RESULTS

Among the 13 methods for color channel combination, the signal extracted by the Cb+Cr combination in the YCbCr color space includes the most pulse information. Furthermore, the average SNR of the modified POS for all the used databases is improved by 69.3% compared to POS.

CONCLUSIONS

The methods using prior knowledge are not only simple to calculate but can significantly increase the SNR, which will provide a great help in the practical use of vital sign measurements based on PPGI.

Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation

Remote photo-plethysmography (rPPG) uses a camera to estimate a person’s heart rate (HR). Similar to how heart rate can provide useful information about a person’s vital signs, insights about the underlying physio/psychological conditions can be obtained from heart rate variability (HRV). HRV is a measure of the fine fluctuations in the intervals between heart beats. However, this measure requires temporally locating heart beats with a high degree of precision. We introduce a refined and efficient real-time rPPG pipeline with novel filtering and motion suppression that not only estimates heart rates, but also extracts the pulse waveform to time heart beats and measure heart rate variability. This unsupervised method requires no rPPG specific training and is able to operate in real-time. We also introduce a new multi-modal video dataset, VicarPPG 2, specifically designed to evaluate rPPG algorithms on HR and HRV estimation. We validate and study our method under various conditions on a comprehensive range of public and self-recorded datasets, showing state-of-the-art results and providing useful insights into some unique aspects. Lastly, we make available CleanerPPG, a collection of human-verified ground truth peak/heart-beat annotations for existing rPPG datasets. These verified annotations should make future evaluations and benchmarking of rPPG algorithms more accurate, standardized and fair.

Robot Assisted Instantaneous Heart Rate Estimator using Camera based Remote Photoplethysmograpy via Plane-Orthogonal-to-Skin and Finite State Machine

In this paper, we have presented Turtulebot-assisted instantaneous heart rate (HR) estimator using camera based remote photoplethysmography. We used a Turtlebot with a camera to record human face. For the face detection, we used Haar Cascade algorithm. To increase the accuracy of the HR estimation, we combined a plane-orthogonal-to-skin (POS) model with finite state machine (FSM) framework. By combining POS and FSM framework, we achieved 1.08 bpm of MAE, which is the lowest error comparing to the state-of-art methods.

A computer vision approach for classifying isometric grip force exertion levels

Exposure to high and/or repetitive force exertions can lead to musculoskeletal injuries. However, measuring worker force exertion levels is challenging, and existing techniques can be intrusive, interfere with human-machine interface, and/or limited by subjectivity. In this work, computer vision techniques are developed to detect isometric grip exertions using facial videos and wearable photoplethysmogram. Eighteen participants (19-24 years) performed isometric grip exertions at varying levels of maximum voluntary contraction. Novel features that predict forces were identified and extracted from video and photoplethysmogram data. Two experiments with two (High/Low) and three (0%MVC/50%MVC/100%MVC) labels were performed to classify exertions. The Deep Neural Network classifier performed the best with 96% and 87% accuracy for two- and three-level classifications, respectively. This approach was robust to leave subjects out during cross-validation (86% accuracy when 3-subjects were left out) and robust to noise (i.e. 89% accuracy for correctly classifying talking activities as low force exertions). Practitioner summary: Forceful exertions are contributing factors to musculoskeletal injuries, yet it remains difficult to measure in work environments. This paper presents an approach to estimate force exertion levels, which is less distracting to workers, easier to implement by practitioners, and could potentially be used in a wide variety of workplaces. Abbreviations: MSD: musculoskeletal disorders; ACGIH: American Conference of Governmental Industrial Hygienists; HAL: hand activity level; MVC: maximum voluntary contraction; PPG: photoplethysmogram; DNN: deep neural networks; LOSO: leave-one-subject-out; ROC: receiver operating characteristic; AUC: area under curve.

Continuous-Spectrum Infrared Illuminator for Camera-PPG in Darkness

Many camera-based remote photoplethysmography (PPG) applications require sensing in near infrared (NIR). The performance of PPG systems benefits from multi-wavelength processing. The illumination source in such system is explored in this paper. We demonstrate that multiple narrow-band LEDs have inferior color homogeneity compared to broadband light sources. Therefore, we consider the broadband option based on phosphor material excited by LEDs. A first prototype was realized and its details are discussed. It was tested within a remote-PPG monitoring scenario in darkness and the full system demonstrates robust pulse-rate measurement. Given its accuracy in pulse rate extraction, the proposed illumination principle is considered a valuable asset for large-scale NIR-PPG applications as it enables multi-wavelength processing, lightweight set-ups with relatively low-power infrared light sources.

Impact of makeup on remote-PPG monitoring

Camera-based remote photoplethysmography (remote-PPG) enables contactless measurement of blood volume pulse from the human skin. Skin visibility is essential to remote-PPG as the camera needs to capture the light reflected from the skin that penetrates deep into skin tissues and carries blood pulsation information. The use of facial makeup may jeopardize this measurement by reducing the amount of light penetrating into and reflecting from the skin. In this paper, we conduct an empirical study to thoroughly investigate the impact of makeup on remote-PPG monitoring, in both the visible (RGB) and invisible (Near Infrared, NIR) lighting conditions. The experiment shows that makeup has negative influence on remote-PPG, which reduces the relative PPG strength (AC/DC) at different wavelengths and changes the normalized PPG signature across multiple wavelengths. It makes (i) the pulse-rate extraction more difficult in both the RGB and NIR, although NIR is less affected than RGB, and (ii) the blood oxygen saturation extraction in NIR impossible. To the best of our knowledge, this is the first work that systematically investigate the impact of makeup on camera-based remote-PPG monitoring.

Analysis of CNN-based remote-PPG to understand limitations and sensitivities

Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. We also find that the PPG-related prior knowledge may still be helpful for the CNN-based PPG extraction, and recommend further investigation of hybrid CNN-based methods that include prior knowledge in their design.

Modified RGB Cameras for Infrared Remote-PPG

Multi-wavelength cameras play an essential role in remote photoplethysmography (PPG). Whereas these are readily available for visible light, this is not the case for near infrared (NIR). We propose to modify existing RGB cameras to make them suited for NIR-PPG. In particular, we exploit the spectral leakage of the RGB channels in infrared in combination with a narrow dual-band optical filter. Such camera modification is simple, cost-effective, easy to implement, and it is shown to attain a pulse-rate extraction performance comparable to that of multiple narrow-band NIR cameras.

Feasibility of assessing ultra-short-term pulse rate variability from video recordings

Objectives

Remote photoplethysmography (rPPG) is a promising non-contact measurement technique for assessing numerous physiological parameters: pulse rate, pulse rate variability (PRV), respiratory rate, pulse wave velocity, blood saturation, blood pressure, etc. To justify its use in ultra-short-term (UST) PRV analysis, which is of great benefit for several healthcare applications, the agreement between rPPG- and PPG-derived UST-PRV metrics was studied.

Approach

Three time-domain metrics—standard deviation of normal-to-normal (NN) intervals (SDNN), root mean square of successive NN interval differences (RMSSD), and the percentage of adjacent NN intervals that differ from each other by more than 50 ms (pNN50)—were extracted from 56 video recordings in a publicly available data set. The selected metrics were calculated on the basis of three groups of 10 s recordings and their average, two groups of 30 s recordings and their average, and a group of 60 s recordings taken from the full-length recordings and then compared with metrics derived from the corresponding reference (PPG) pulse waveform signals by using correlation and effect size parameters, and Bland–Altman plots.

Main results

The results show there is stronger agreement as the recording length increases for SDNN and RMSSD, yet there is no significant change for pNN50. The agreement parameters reach r = 0.841 (p < 0.001), r = 0.529 (p < 0.001), and r = 0.657 (p < 0.001), estimated median bias −1.52, −2.28 ms and −1.95% and a small effect size for SDNN, RMSSD, and pNN50 derived from the 60 s recordings, respectively.

Significance

Remote photoplethysmography-derived UST-PRV metrics manage to capture UST-PRV metrics derived from reference (PPG) recordings well. This feature is highly desirable in numerous applications for the assessment of one’s health and well-being. In future research, the validity of rPPG-derived UST-PRV metrics compared to the gold standard electrocardiography recordings is to be assessed.

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.

Remote Monitoring of Vital Signs in Diverse Non-Clinical and Clinical Scenarios Using Computer Vision Systems: A Review

Techniques for noncontact measurement of vital signs using camera imaging technologies have been attracting increasing attention. For noncontact physiological assessments, computer vision-based methods appear to be an advantageous approach that could be robust, hygienic, reliable, safe, cost effective and suitable for long distance and long-term monitoring. In addition, video techniques allow measurements from multiple individuals opportunistically and simultaneously in groups. This paper aims to explore the progress of the technology from controlled clinical scenarios with fixed monitoring installations and controlled lighting, towards uncontrolled environments, crowds and moving sensor platforms. We focus on the diversity of applications and scenarios being studied in this topic. From this review it emerges that automatic multiple regions of interest (ROIs) selection, removal of noise artefacts caused by both illumination variations and motion artefacts, simultaneous multiple person monitoring, long distance detection, multi-camera fusion and accepted publicly available datasets are topics that still require research to enable the technology to mature into many real-world applications.

Robust heart-rate estimation from facial videos using Project_ICA

OBJECTIVE

Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described.

APPROACH

After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed 'Project_ICA', based on a skin reflection model, is employed to extract the pulse signal from the original signal.

MAIN RESULTS

To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants.

SIGNIFICANCE

The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

A Broader Look: Camera-Based Vital Sign Estimation across the Spectrum

OBJECTIVES

Camera-based vital sign estimation allows the contactless assessment of important physiological parameters. Seminal contributions were made in the 1930s, 1980s, and 2000s, and the speed of development seems ever increasing. In this suivey, we aim to overview the most recent works in this area, describe their common features as well as shortcomings, and highlight interesting "outliers".

METHODS

We performed a comprehensive literature research and quantitative analysis of papers published between 2016 and 2018. Quantitative information about the number of subjects, studies with healthy volunteers vs. pathological conditions, public datasets, laboratory vs. real-world works, types of camera, usage of machine learning, and spectral properties of data was extracted. Moreover, a qualitative analysis of illumination used and recent advantages in terms of algorithmic developments was also performed.

RESULTS

Since 2016, 116 papers were published on camera-based vital sign estimation and 59% of papers presented results on 20 or fewer subjects. While the average number of participants increased from 15.7 in 2016 to 22.9 in 2018, the vast majority of papers (n=100) were on healthy subjects. Four public datasets were used in 10 publications. We found 27 papers whose application scenario could be considered a real-world use case, such as monitoring during exercise or driving. These include 16 papers that dealt with non-healthy subjects. The majority of papers (n=61) presented results based on visual, red-green-blue (RGB) information, followed by RGB combined with other parts of the electromagnetic spectrum (n=18), and thermography only (n=12), while other works (n=25) used other mono- or polychromatic non-RGB data. Surprisingly, a minority of publications (n=39) made use of consumer-grade equipment. Lighting conditions were primarily uncontrolled or ambient. While some works focused on specialized aspects such as the removal of vital sign information from video streams to protect privacy or the influence of video compression, most algorithmic developments were related to three areas: region of interest selection, tracking, or extraction of a one-dimensional signal. Seven papers used deep learning techniques, 17 papers used other machine learning approaches, and 92 made no explicit use of machine learning.

CONCLUSION

Although some general trends and frequent shortcomings are obvious, the spectrum of publications related to camera-based vital sign estimation is broad. While many creative solutions and unique approaches exist, the lack of standardization hinders comparability of these techniques and of their performance. We believe that sharing algorithms and/ or datasets will alleviate this and would allow the application of newer techniques such as deep learning.

Game-Calibrated and User-Tailored Remote Detection of Stress and Boredom in Games

Emotion detection based on computer vision and remote extraction of user signals commonly rely on stimuli where users have a passive role with limited possibilities for interaction or emotional involvement, e.g., images and videos. Predictive models are also trained on a group level, which potentially excludes or dilutes key individualities of users. We present a non-obtrusive, multifactorial, user-tailored emotion detection method based on remotely estimated psychophysiological signals. A neural network learns the emotional profile of a user during the interaction with calibration games, a novel game-based emotion elicitation material designed to induce emotions while accounting for particularities of individuals. We evaluate our method in two experiments ( n = 20 and n = 62 ) with mean classification accuracy of 61.6%, which is statistically significantly better than chance-level classification. Our approach and its evaluation present unique circumstances: our model is trained on one dataset (calibration games) and tested on another (evaluation game), while preserving the natural behavior of subjects and using remote acquisition of signals. Results of this study suggest our method is feasible and an initiative to move away from questionnaires and physical sensors into a non-obtrusive, remote-based solution for detecting emotions in a context involving more naturalistic user behavior and games.

Cardiovascular assessment by imaging photoplethysmography - a review

Over the last few years, the contactless acquisition of cardiovascular parameters using cameras has gained immense attention. The technique provides an optical means to acquire cardiovascular information in a very convenient way. This review provides an overview on the technique's background and current realizations. Besides giving detailed information on the most widespread application of the technique, namely the contactless acquisition of heart rate, we outline further concepts and we critically discuss the current state.

Current progress of photoplethysmography and SPO2 for health monitoring

A photoplethysmograph (PPG) is a simple medical device for monitoring blood flow and transportation of substances in the blood. It consists of a light source and a photodetector for measuring transmitted and reflected light signals. Clinically, PPGs are used to monitor the pulse rate, oxygen saturation, blood pressure, and blood vessel stiffness. Wearable unobtrusive PPG monitors are commercially available. Here, we review the principle issues and clinical applications of PPG for monitoring oxygen saturation.

A wavelet-based decomposition method for a robust extraction of pulse rate from video recordings

Background

Remote photoplethysmography (rPPG) is a promising optical method for non-contact assessment of pulse rate (PR) from video recordings. In order to implement the method in real-time applications, it is necessary for the rPPG algorithms to be capable of eliminating as many distortions from the pulse signal as possible.

Methods

In order to increase the degrees-of-freedom of the distortion elimination, the dimensionality of the RGB video signals is increased by the wavelet transform decomposition using the generalized Morse wavelet. The proposed Continuous-Wavelet-Transform-based Sub-Band rPPG method (SB-CWT) is evaluated on the 101 publicly available RGB facial video recordings and corresponding reference blood volume pulse (BVP) signals taken from the MMSE-HR database. The performance of the SB-CWT is compared with the performance of the state-of-the-art Sub-band rPPG (SB).

Results

Median signal-to-noise ratio (SNR) for the proposed SB-CWT ranges from 6.63 to 10.39 dB and for the SB from 4.23 to 6.24 dB. The agreement between the estimated PRs from rPPG pulse signals and the reference signals in terms of the coefficients of determination ranges from 0.81 to 0.91 for SB-CWT and from 0.41 to 0.47 for SB. All the correlation coefficients are statistically significant (p < 0.001). The Bland-Altman plots show that mean difference range from 5.37 to 1.82 BPM for SB-CWT and from 22.18 to 18.80 BPM for SB.

Discussion

The results show that the proposed SB-CWT outperforms SB in terms of SNR and the agreement between the estimated PRs from RGB video signals and PRs from the reference BVP signals.

Single Element Remote-PPG

Camera-based remote photoplethysmography technology (remote-PPG) has shown great potential for contactless pulse-rate monitoring. However, remote-PPG systems typically analyze face images, which may restrict applications in view of privacy-preserving regulations such as the recently announced General Data Protection Regulation in the European Union. In this paper, we investigate the case of using single-element sensing as an input for remote-PPG extraction, which prohibits facial analysis and thus evades privacy issues. It also improves the efficiency of data storage and transmission. In contrast to known remote-PPG solutions using skin-selection techniques, the input signals in a single-element setup will contain a non-negligible degree of signal components associated with non-skin areas. Current remote-PPG extraction methods based on physiological and optical properties of skin reflections are therefore no longer valid. A new remote-PPG method, named Soft Signature based extraction (SoftSig), is proposed to deal with this situation by softening the dependence of pulse extraction on prior knowledge. A large scale experiment validates the concept of single-element remote-PPG monitoring and shows the improvement of SoftSig over general purpose solutions.

Using imaging photoplethysmography for heart rate estimation in non-human primates

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.

Effect of motion artifact on digital camera based heart rate measurement

Remote health monitoring is an emerging field in biomedical technology. Digital camera based heart rate measurement method is a recent development which would make remote health monitoring reliable and sustainable in future. This paper presents an investigation on the effect of motion artifact on digital camera-based heart rate measurement. The paper will discuss details on the principles and effects of motion artifacts on photoplethysmography signals. An experiment is conducted using publicly available MAHNOB-HCI database. We have investigated the effects of static scenarios, scenarios involving rigid motion and scenarios involving non-rigid motion. The experiment was tested on state of the art digital camera based heart rate measuring methods. The results showed the effectiveness of the methods and provide a direction to overcome/minimize the effect of motion artifacts for future research.

Full video pulse extraction

This paper introduces a new method to automate heart-rate detection using remote photoplethysmography (rPPG). The method replaces the commonly used region of interest (RoI) detection and tracking, and does not require initialization. Instead, it combines a number of candidate pulse-signals computed in the parallel, each biased towards differently colored objects in the scene. The method is based on the observation that the temporally averaged colors of video objects (skin and background) are usually quite stable over time in typical application-driven scenarios, such as the monitoring of a subject sleeping in bed, or an infant in an incubator. The resulting system, called full video pulse extraction (FVP), allows the direct use of raw video streams for pulse extraction. Our benchmark set of diverse videos shows that FVP enables long-term sleep monitoring in visible light and in infrared, and works for adults and neonates. Although we only demonstrate the concept for heart-rate monitoring, we foresee the adaptation to a range of vital signs, thus benefiting the larger video health monitoring field.

Performance Assessment of a Dedicated Reflectance Pulse Oximeter in a Neonatal Intensive Care Unit

The measurement of peripheral oxygen saturation (SpO2) in neonatal intensive care units (NICUs) poses a significant challenge. Motion artifacts due to the patient's limb motion induce many false alarms, which in turn cause an additional workload for the medical staff and anxiety for the parents. We developed a reflectance pulse oximeter dedicated to be placed at the patient's forehead, which is less prone to such artifacts. We trained our algorithms for SpO2 estimation on 8 adult healthy volunteers participating in a controlled desaturation study. We then validated our SpO2 monitoring system on 25 newborn patients monitored in an NICU. We further evaluated the versatility and resilience to low signal-tonoise ratios (SNR) of our solution by testing it on signals acquired in a low-perfusion region (upper right part of the chest) of our adult volunteers. We obtained an SpO2 estimation accuracy ($A _{\mathbf {rms}}$) of 1.9 % and 3.1 % at the forehead and the chest in our adult volunteers, respectively. These performances were obtained after automatic rejection of 0.1 % and 30.0 %, respectively, of low-SNR signals by our dedicated quality index. In the dataset recorded on newborn patients in the NICU, we obtained an accuracy of 3.9 % after automatic rejection of 11.7 % of low-SNR signals by our quality index. These analyses were carried out following the procedures suggested by the ISO 80601-2-61:2011 standard, which specifies a target $A _{\mathbf {rms}} \le $ 4 % for SpO2 monitoring applications. These promising results suggest that reflectance pulse oximeters can achieve clinically acceptable accuracy, while being placed at locations less sensitive to limb motion artifacts - such as the forehead - thereby reducing the amount of SpO2-related false alarms in NICUs.

Lock-in technique for extraction of pulse rates and associated confidence levels from video

We investigate the practical applicability of video photoplethysmography (VPPG) to extract heart rates of subjects using noncontact color video recordings of human faces collected under typical indoor laboratory conditions using commercial video cameras. Videos were processed following three previously described simple VPPG algorithms to produce a time-varying plethysmographic signal. These time signals were then analyzed using, to the best of our knowledge, a novel, lock-in algorithm that was developed to extract the pulsatile frequency component. A protocol to associate confidence estimates for the extracted heart rates for each video stream is presented. Results indicate that the difference between heart rates extracted using the lock-in technique and gold-standard measurements, for videos with high-confidence metrics, was less than 4 beats per minute. Constraints on video acquisition and processing, including natural subject motion and the total duration of video recorded required for evaluating these confidence metrics, are discussed.

Bounded Kalman filter method for motion-robust, non-contact heart rate estimation

The authors of this work present a real-time measurement of heart rate across different lighting conditions and motion categories. This is an advancement over existing remote Photo Plethysmography (rPPG) methods that require a static, controlled environment for heart rate detection, making them impractical for real-world scenarios wherein a patient may be in motion, or remotely connected to a healthcare provider through telehealth technologies. The algorithm aims to minimize motion artifacts such as blurring and noise due to head movements (uniform, random) by employing i) a blur identification and denoising algorithm for each frame and ii) a bounded Kalman filter technique for motion estimation and feature tracking. A case study is presented that demonstrates the feasibility of the algorithm in non-contact estimation of the pulse rate of subjects performing everyday head and body movements. The method in this paper outperforms state of the art rPPG methods in heart rate detection, as revealed by the benchmarked results.

Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement

Remote camera-based measurement of physiology has great potential for healthcare and affective computing. Recent advances in computer vision and signal processing have enabled photoplethysmography (PPG) measurement using commercially available cameras. However, there remain challenges in recovering accurate noncontact PPG measurements in the presence of rigid head motion. When a subject is moving, their face may be turned away from one camera, be obscured by an object, or move out of the frame resulting in missing observations. As the calculation of pulse rate variability (PRV) requires analysis over a time window of several minutes, the effect of missing observations on such features is deleterious. We present an approach for fusing partial color-channel signals from an array of cameras that enable physiology measurements to be made from moving subjects, even if they leave the frame of one or more cameras, which would not otherwise be possible with only a single camera. We systematically test our method on subjects ( N=25) using a set of six, 5-min tasks (each repeated twice) involving different levels of head motion. This results in validation across 25 h of measurement. We evaluate pulse rate and PRV parameter estimation including statistical, geometric, and frequency-based measures. The median absolute error in pulse rate measurements was 0.57 beats-per-minute (BPM). In all but two tasks with the greatest motion, the median error was within 0.4 BPM of that from a contact PPG device. PRV estimates were significantly improved using our proposed approach compared to an alternative not designed to handle missing values and multiple camera signals; the error was reduced by over 50%. Without our proposed method, errors in pulse rate would be very high, and estimation of PRV parameters would not be feasible due to significant data loss.

Novel health monitoring method using an RGB camera

In this paper we present a novel health monitoring method by estimating the heart rate and respiratory rate using an RGB camera. The heart rate and the respiratory rate are estimated from the photoplethysmography (PPG) and the respiratory motion. The method mainly operates by using the green spectrum of the RGB camera to generate a multivariate PPG signal to perform multivariate de-noising on the video signal to extract the resultant PPG signal. A periodicity based voting scheme (PVS) was used to measure the heart rate and respiratory rate from the estimated PPG signal. We evaluated our proposed method with a state of the art heart rate measuring method for two scenarios using the MAHNOB-HCI database and a self collected naturalistic environment database. The methods were furthermore evaluated for various scenarios at naturalistic environments such as a motion variance session and a skin tone variance session. Our proposed method operated robustly during the experiments and outperformed the state of the art heart rate measuring methods by compensating the effects of the naturalistic environment.

Simultaneous Tracking of Cardiorespiratory Signals for Multiple Persons Using a Machine Vision System With Noise Artifact Removal

Most existing non-contact monitoring systems are limited to detecting physiological signs from a single subject at a time. Still, another challenge facing these systems is that they are prone to noise artifacts resulting from motion of subjects, facial expressions, talking, skin tone, and illumination variations. This paper proposes an efficient non-contact system based on a digital camera to track the cardiorespiratory signal from a number of subjects (up to six persons) at the same time with a new method for noise artifact removal. The proposed system relied on the physiological and physical effects as a result of the activity of the cardiovascular and respiratory systems, such as skin color changes and head motion. Since these effects are imperceptible to the human eye and highly affected by the noise variations, we used advanced signal and video processing techniques, including developing video magnification technique, complete ensemble empirical mode decomposition with adaptive noise, and canonical correlation analysis to extract the heart rate and respiratory rate from multiple subjects under the noise artifact assumptions. The experimental results of the proposed system had a significant correlation (Pearson's correlation coefficient = 0.9994, Spearman correlation coefficient = 0.9987, and root mean square error = 0.32) when compared with the conventional contact methods (pulse oximeter and piezorespiratory belt), which makes the proposed system a promising candidate for novel applications.

Hybrid Optical Unobtrusive Blood Pressure Measurements

Blood pressure (BP) is critical in diagnosing certain cardiovascular diseases such as hypertension. Some previous studies have proved that BP can be estimated by pulse transit time (PTT) calculated by a pair of photoplethysmography (PPG) signals at two body sites. Currently, contact PPG (cPPG) and imaging PPG (iPPG) are two feasible ways to obtain PPG signals. In this study, we proposed a hybrid system (called the ICPPG system) employing both methods that can be implemented on a wearable device, facilitating the measurement of BP in an inconspicuous way. The feasibility of the ICPPG system was validated on a dataset with 29 subjects. It has been proved that the ICPPG system is able to estimate PTT values. Moreover, the PTT measured by the new system shows a correlation on average with BP variations for most subjects, which could facilitate a new generation of BP measurement using wearable and mobile devices.

Noncontact spirometry with a webcam

We present an imaging-based method for noncontact spirometry. The method tracks the subtle respiratory-induced shoulder movement of a subject, builds a calibration curve, and determines the flow-volume spirometry curve and vital respiratory parameters, including forced expiratory volume in the first second, forced vital capacity, and peak expiratory flow rate. We validate the accuracy of the method by comparing the data with those simultaneously recorded with a gold standard reference method and examine the reliability of the noncontact spirometry with a pilot study including 16 subjects. This work demonstrates that the noncontact method can provide accurate and reliable spirometry tests with a webcam. Compared to the traditional spirometers, the present noncontact spirometry does not require using a spirometer, breathing into a mouthpiece, or wearing a nose clip, thus making spirometry test more easily accessible for the growing population of asthma and chronic obstructive pulmonary diseases.

Amplitude-selective filtering for remote-PPG

Biometric signatures of remote photoplethysmography (rPPG), including the pulse-induced characteristic color absorptions and pulse frequency range, have been used to design robust algorithms for extracting the pulse-signal from a video. In this paper, we look into a new biometric signature, i.e., the relative pulsatile amplitude, and use it to design a very effective yet computationally low-cost filtering method for rPPG, namely "amplitude-selective filtering" (ASF). Based on the observation that the human relative pulsatile amplitude varies in a specific lower range as a function of RGB channels, our basic idea is using the spectral amplitude of, e.g., the R-channel, to select the RGB frequency components inside the assumed pulsatile amplitude-range for pulse extraction. Similar to band-pass filtering (BPF), the proposed ASF can be applied to a broad range of rPPG algorithms to pre-process the RGB-signals before extracting the pulse. The benchmark in challenging fitness use-cases shows that applying ASF (ASF+BPF) as a pre-processing step brings significant and consistent improvements to all multi-channel pulse extraction methods. It improves different (multi-wavelength) rPPG algorithms to the extent where quality differences between the individual approaches almost disappear. The novelty of the proposed method is its simplicity and effectiveness in providing a solution for the extremely challenging application of rPPG to a fitness setting. The proposed method is easy to understand, simple to implement, and low-cost in running. It is the first time that the physiological property of pulsatile amplitude is used as a biometric signature for generic signal filtering.

Living-Skin Classification via Remote-PPG

Detecting living-skin tissue in a video on the basis of induced color changes due to blood pulsation is emerging for automatic region of interest localization in remote photoplethysmography (rPPG). However, the state-of-the-art method performing unsupervised living-skin detection in a video is rather time consuming, which is mainly due to the high complexity of its unsupervised online learning for pulse/noise separation. In this paper, we address this issue by proposing a fast living-skin classification method. Our basic idea is to transform the time-variant rPPG-signals into signal shape descriptors called "multiresolution iterative spectrum," where pulse and noise have different patterns enabling accurate binary classification. The proposed technique is a proof-of-concept that has only been validated in lab conditions but not in real clinical conditions. The benchmark, including synthetic and realistic (nonclinical) experiments, shows that it achieves a high detection accuracy better than the state-of-the-art method, and a high detection speed at hundreds of frames per second in MATLAB, enabling real-time living-skin detection.

Motion robust remote photoplethysmography in CIELab color space

Remote photoplethysmography (rPPG) is attractive for tracking a subject’s physiological parameters without wearing a device. However, rPPG is known to be prone to body movement-induced artifacts, making it unreliable in realistic situations. Here we report a method to minimize the movement-induced artifacts. The method selects an optimal region of interest (ROI) automatically, prunes frames in which the ROI is not clearly captured (e.g., subject moves out of the view), and analyzes rPPG using an algorithm in CIELab color space, rather than the widely used RGB color space. We show that body movement primarily affects image intensity, rather than chromaticity, and separating chromaticity from intensity in CIELab color space thus helps achieve effective reduction of the movement-induced artifacts. We validate the method by performing a pilot study including 17 people with diverse skin tones.

Algorithmic Principles of Remote PPG

This paper introduces a mathematical model that incorporates the pertinent optical and physiological properties of skin reflections with the objective to increase our understanding of the algorithmic principles behind remote photoplethysmography (rPPG). The model is used to explain the different choices that were made in existing rPPG methods for pulse extraction. The understanding that comes from the model can be used to design robust or application-specific rPPG solutions. We illustrate this by designing an alternative rPPG method, where a projection plane orthogonal to the skin tone is used for pulse extraction. A large benchmark on the various discussed rPPG methods shows that their relative merits can indeed be understood from the proposed model.