Advancements in noncontact, multiparameter physiological measurements using a webcam

A wavelet-based decomposition method for a robust extraction of pulse rate from video recordings

Background

Remote photoplethysmography (rPPG) is a promising optical method for non-contact assessment of pulse rate (PR) from video recordings. In order to implement the method in real-time applications, it is necessary for the rPPG algorithms to be capable of eliminating as many distortions from the pulse signal as possible.

Methods

In order to increase the degrees-of-freedom of the distortion elimination, the dimensionality of the RGB video signals is increased by the wavelet transform decomposition using the generalized Morse wavelet. The proposed Continuous-Wavelet-Transform-based Sub-Band rPPG method (SB-CWT) is evaluated on the 101 publicly available RGB facial video recordings and corresponding reference blood volume pulse (BVP) signals taken from the MMSE-HR database. The performance of the SB-CWT is compared with the performance of the state-of-the-art Sub-band rPPG (SB).

Results

Median signal-to-noise ratio (SNR) for the proposed SB-CWT ranges from 6.63 to 10.39 dB and for the SB from 4.23 to 6.24 dB. The agreement between the estimated PRs from rPPG pulse signals and the reference signals in terms of the coefficients of determination ranges from 0.81 to 0.91 for SB-CWT and from 0.41 to 0.47 for SB. All the correlation coefficients are statistically significant (p < 0.001). The Bland-Altman plots show that mean difference range from 5.37 to 1.82 BPM for SB-CWT and from 22.18 to 18.80 BPM for SB.

Discussion

The results show that the proposed SB-CWT outperforms SB in terms of SNR and the agreement between the estimated PRs from RGB video signals and PRs from the reference BVP signals.

Single Element Remote-PPG

Camera-based remote photoplethysmography technology (remote-PPG) has shown great potential for contactless pulse-rate monitoring. However, remote-PPG systems typically analyze face images, which may restrict applications in view of privacy-preserving regulations such as the recently announced General Data Protection Regulation in the European Union. In this paper, we investigate the case of using single-element sensing as an input for remote-PPG extraction, which prohibits facial analysis and thus evades privacy issues. It also improves the efficiency of data storage and transmission. In contrast to known remote-PPG solutions using skin-selection techniques, the input signals in a single-element setup will contain a non-negligible degree of signal components associated with non-skin areas. Current remote-PPG extraction methods based on physiological and optical properties of skin reflections are therefore no longer valid. A new remote-PPG method, named Soft Signature based extraction (SoftSig), is proposed to deal with this situation by softening the dependence of pulse extraction on prior knowledge. A large scale experiment validates the concept of single-element remote-PPG monitoring and shows the improvement of SoftSig over general purpose solutions.

Head-motion Robust Video-based Heart Rate Estimation Using Facial Feature Point Fluctuations

Interest in measuring heart rates (HRs) without physical contact has increased in the area of stress checking and health care. In this paper, we propose head-motion robust video-based heart rate estimation using facial feature point fluctuations. The proposed method adaptively estimates and removes such rigid-noise components as noise stemming from horizontal head motion and extracts relatively small heart signals. Rigid-noise components can be accurately estimated and removed by using changes in facial feature points which are not dominant over heart signals and are more dominant over noise signals than are such luminance signals as RGB. In evaluation experiments on a benchmark dataset, our method achieved the highest accuracy among state-of-the-art methods.

Spectral Estimation Methods for Evaluating iPPG Pulse Rate Variability

Non-contact measurement of physiological parameters, like pulse rate variability (PRV), has numerous applications in medicine and affective computing. PRV is an informative measure of autonomic nervous system activity. Spectral estimation from unevenly sampled, non-stationary data is integral to pulse rate variability frequency-domain analysis. We present the first comparison of results of PRV computation using the Lomb-Scargle method and Bayesian Spectral Estimation. The Lomb-Scargle method performs well, even in the presence of missing beats. However, the Bayesian Spectral Estimation method has advantages when tracking changes in amplitude and frequency. We illustrate these characteristics with results from synthetic data and real non-contact imaging photoplethysmography measurements.

A Maximum Likelihood Formulation To Exploit Heart Rate Variability for Robust Heart Rate Estimation From Facial Video

The problem of estimating the heart rate (HR) from a racial video is considered. A typical approach for this problem is to use independent component analysis (ICA) on the red, blue, green intensity prof iles averaged over the facial region. This provides estimates of the underlying source signals, whose spectral peaks are used to predict HR in every analysis window. In this work, we propose a maximum likelihood formulation to optimally select a source signal in each window such that the predicted HR trajectory not only corresponds to the most likely spectral peaks but also ensures a realistic HR variability (HRV) across analysis windows. The likelihood function is efficiently optimized using dynamic programming in a manner similar to Viterbi decoding. The proposed scheme for HR estimation is denoted by vICA. The performance of vICA is compared with a typical ICA approach as well as a recently proposed sparse spectral peak tracking (SSPT) technique that ensures that the predicted HR does not vary drastically across analysis windows. Experiments are performed in a five fold setup using racial videos of 15 subjects recorded using two types of smartphones (Samsung Galaxy and iPhone) at three different distances (6inches, lfoot, 2feet) between the phone camera and the subject. Mean absolute error (MAE) between the original and predicted HR reveals that the proposed vICA scheme performs better than the best of the baseline schemes, namely SSPT by -8.69%, 52.77% and 8.00% when Samsung Galaxy phone was used at a distance of 6inches, lfoot, and 2feet respectively. These improvements are 12.13%, 13.59% and 18.34% when iPhone was used. This, in turn, suggests that the HR predicted from a racial video becomes more accurate when the smoothness of HRV is utilized in predicting the HR trajectory as done in the proposed vICA.

InPhysible: Camouflage Against Video-Based Physiological Measurement

Imaging photoplethysmography (iPPG) is a powerful set of methods for measuring physiological signals from video. Recent advances have shown that a low-cost webcam can be used to measure heart rate, blood flow, respiration, blood oxygen levels and stress. While these methods have many beneficial applications, the unobtrusive and ubiquitous nature of the sensors risk exposing people to unwanted measurement. We present InPhysible the first camouflage system against video- based physiological measurement. The infra-red system can be embedded into any pair of glasses, or other headwear, and disrupts the measurement of the iPPG signal while being imperceptible by the human eye. Our system is flexible and can simulate realistic pulse signals to hinder heart rate measurement. In this paper we present the design of our prototype and a user study validating its efficacy. Finally, we discuss the limitations and implications for data privacy and security.

Sensitivity of Video-Based Pulse Arrival Time to Dynamic Blood Pressure Changes

Estimating blood pressure (BP) from pulse arrival time (PAT) by image-based (skin video) photoplethysmography (iPPG) is of increasing interest due to the non-contact method advantage (over cuff-based methods) and potential for BP measurement to be built into portable devices such as mobile phones. The relationship between pulse transit time extracted from iPPG has been investigated during stable BP. The sensitivity of beat-to-beat iPPG-PAT to dynamic changes in BP has not been explored. This study investigated the correlation between iPPG-PAT and diastolic BP (DBP) during 1-minute seated rest and 3-minute isometric handgrip exercise. 15 healthy participants (9 female, 34±13 years) were recruited. Video was recorded from subjects' faces at 30 frames per second using a standard web-camera with simultaneous measurement of the electrocardiogram and noninvasive finger BP. The iPPG waveform was from the averaged green channel intensity of regions of the forehead or cheek. PAT was calculated from the R-wave ofthe electrocardiogram to the foot of the iPPG or finger BP waveform respectively for direct comparison. Handgrip exercise caused a steady increase in DBP (75±9 to 87±13 mmHg, p<0.001). Beat-to-beat iPPG-PAT and DBP was negatively correlated (mena ±SE -1.33±1.70 ms/mmHg, P=0.0024) as was finger-PAT (mean ±SE -0.5S ±0.39 ms/mmHg, P<0.001). The proportion of individual significant negative regression slopes between DBP and finger-PAT and between DBP and iPPG-PAT was not significantly different. Despite high variability of the correlation between iPPG-PAT and DBP among subjects, iPPG-PAT can track dynamic changes in BP.

Remote Heart Rate Measurement from RGB-NIR Video Based on Spatial and Spectral Face Patch Selection

In this paper, we propose a novel heart rate (HR) estimation method using simultaneously recorded RGB and near-infrared (NIR) face videos. The key idea of our method is to automatically select suitable face patches for HR estimation in both spatial and spectral domains. The spatial and spectral face patch selection enables us to robustly estimate HR under various situations, including scenes under which existing RGB camera-based methods fail to accurately estimate HR. For a challenging scene in low light and with light fluctuations, our method can successfully estimate HR for all 20 subjects $( \pm 3$ beats per minute), while the RGB camera-based methods succeed only for 25% of the subjects.

Heart Rate and Heart Rate Variability From Single-Channel Video and ICA Integration of Multiple Signals

Unobtrusive monitoring of vital signs is relevant for both medical (patient monitoring) and non-medical applications (e.g., stress and fatigue monitoring). In this paper, we focus on the use of imaging photoplethysmography (iPPG). High frame rate videos were acquired by using a monochrome camera and an optical band-pass filter ([Formula: see text] nm). To enhance iPPG signal, we investigated the use of independent component analysis (ICA) pre-processing applied to iPPG signal from different regions of the face. Methodology was tested on [Formula: see text] healthy volunteers. Heart rate (HR) and standard time and frequency domain descriptors of heart rate variability (HRV), simultaneously extracted from videos and ECG data, were compared. A mean absolute error (MAE) about 3.812 ms was observed for normal-to-normal intervals with or without ICA pre-processing. Smaller MAE values of frequency domain descriptors were observed when ICA pre-processing was used. The impact of both video frame rate and video signal interval were also analyzed. All the results support the conclusion that proposed ICA pre-processing can effectively improve the HR and HRV assessment from iPPG.

Estimating carotid pulse and breathing rate from near-infrared video of the neck

OBJECTIVE

Non-contact physiological measurement is a growing research area that allows capturing vital signs such as heart rate (HR) and breathing rate (BR) comfortably and unobtrusively with remote devices. However, most of the approaches work only in bright environments in which subtle photoplethysmographic and ballistocardiographic signals can be easily analyzed and/or require expensive and custom hardware to perform the measurements.

APPROACH

This work introduces a low-cost method to measure subtle motions associated with the carotid pulse and breathing movement from the neck using near-infrared (NIR) video imaging. A skin reflection model of the neck was established to provide a theoretical foundation for the method. In particular, the method relies on template matching for neck detection, principal component analysis for feature extraction, and hidden Markov models for data smoothing.

MAIN RESULTS

We compared the estimated HR and BR measures with ones provided by an FDA-cleared device in a 12-participant laboratory study: the estimates achieved a mean absolute error of 0.36 beats per minute and 0.24 breaths per minute under both bright and dark lighting.

SIGNIFICANCE

This work advances the possibilities of non-contact physiological measurement in real-life conditions in which environmental illumination is limited and in which the face of the person is not readily available or needs to be protected. Due to the increasing availability of NIR imaging devices, the described methods are readily scalable.

Infrared Camera-Based Non-contact Measurement of Brain Activity From Pupillary Rhythms

Pupillary responses are associated with affective processing, cognitive function, perception, memory, attention, and other brain activities involving neural pathways. The present study aimed to develop a noncontact system to measure brain activity based on pupillary rhythms using an infra-red web camera. Electroencephalogram (EEG) signals and pupil imaging of 70 undergraduate volunteers (35 female, 35 male) were measured in response to sound stimuli designed to evoke arousal, relaxation, happiness, sadness, or neutral responses. This study successfully developed a real-time system that could detect an EEG spectral index (relative power: low beta in FP1; mid beta in FP1; SMR in FP1; beta in F3; high beta in F8; gamma P4; mu in C4) from pupillary rhythms using the synchronization phenomenon in harmonic frequency (1/100 f) between the pupil and brain oscillations. This method was effective in measuring and evaluating brain activity using a simple, low-cost, noncontact system, and may be an alternative to previous methods used to evaluate brain activity.

Inter-Method Reliability of Pulse Volume Related Measures Derived Using Finger-Photoplethysmography

Abstract. Pulse volume (PV) and its related measures, such as modified normalized pulse volume (mNPV), direct-current component (DC), and pulse rate (PR), derived from the finger-photoplethysmogram (FPPG), are useful psychophysiological measures. Although considerable uncertainties exist in finger-photoplethysmography, little is known about the extent of the adverse effects on the measures. In this study, we therefore examined the inter-method reliability of each index across sensor positions and light intensities, which are major disturbance factors of FPPG. From the tips of the index fingers of 12 participants in a resting state, three simultaneous FPPGs having overlapping optical paths were recorded, with their light intensity being changed in three steps. The analysis revealed that the minimum values of three coefficients of Cronbach’s α for ln PV, ln mNPV, ln DC, and PR across positions were .948, .850, .922, and 1.000, respectively, and that those across intensities were .774, .985, .485, and .998, respectively. These findings suggest that ln mNPV and PR can be used for psychophysiological studies irrespective of minor differences in sensor attachment positions and light source intensity, whereas and ln DC can also be used for such studies but under the condition of light intensity being fixed.

Using imaging photoplethysmography for heart rate estimation in non-human primates

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.

Effect of motion artifact on digital camera based heart rate measurement

Remote health monitoring is an emerging field in biomedical technology. Digital camera based heart rate measurement method is a recent development which would make remote health monitoring reliable and sustainable in future. This paper presents an investigation on the effect of motion artifact on digital camera-based heart rate measurement. The paper will discuss details on the principles and effects of motion artifacts on photoplethysmography signals. An experiment is conducted using publicly available MAHNOB-HCI database. We have investigated the effects of static scenarios, scenarios involving rigid motion and scenarios involving non-rigid motion. The experiment was tested on state of the art digital camera based heart rate measuring methods. The results showed the effectiveness of the methods and provide a direction to overcome/minimize the effect of motion artifacts for future research.

Full video pulse extraction

This paper introduces a new method to automate heart-rate detection using remote photoplethysmography (rPPG). The method replaces the commonly used region of interest (RoI) detection and tracking, and does not require initialization. Instead, it combines a number of candidate pulse-signals computed in the parallel, each biased towards differently colored objects in the scene. The method is based on the observation that the temporally averaged colors of video objects (skin and background) are usually quite stable over time in typical application-driven scenarios, such as the monitoring of a subject sleeping in bed, or an infant in an incubator. The resulting system, called full video pulse extraction (FVP), allows the direct use of raw video streams for pulse extraction. Our benchmark set of diverse videos shows that FVP enables long-term sleep monitoring in visible light and in infrared, and works for adults and neonates. Although we only demonstrate the concept for heart-rate monitoring, we foresee the adaptation to a range of vital signs, thus benefiting the larger video health monitoring field.

Transdermal optical imaging revealed different spatiotemporal patterns of facial cardiovascular activities

Human cardiovascular activities are important indicators of a variety of physiological and psychological activities in human neuroscience research. The present proof-of-concept study aimed to reveal the spatiotemporal patterns of cardiovascular activities from the dynamic changes in hemoglobin concentrations in the face. We first recorded the dynamics of facial transdermal blood flow using a digital video camera and the Electrocardiography (ECG) signals using an ECG system simultaneously. Then we decomposed the video imaging data extracted from different sub-regions of a face into independent components using group independent component analysis (group ICA). Finally, the ICA components that included cardiovascular activities were identified by correlating their magnitude spectrum to those obtained from the ECG. We found that cardiovascular activities were associated with five independent components reflecting different spatiotemporal dynamics of facial blood flow changes. The strongest strengths of these ICA components were observed in the bilateral forehead, the left chin, and the left cheek, respectively. Our findings suggest that the cardiovascular activities presented different dynamic properties within different facial sub-regions, respectively. More broadly, the present findings point to the potential of the transdermal optical imaging technology as a new neuroscience methodology to study human physiology and psychology, noninvasively and remotely in a contactless manner.

Self-adaptive signal separation for non-contact heart rate estimation from facial video in realistic environments

OBJECTIVE

Recent research indicates that facial epidermis color varies with the rhythm of heat beats. It can be captured by consumer-level cameras and, astonishingly, be adopted to estimate heart rate (HR). The HR estimated remains not as precise as required in a practical environment where illumination interference, facial expressions, or motion artifacts are involved, although numerous methods have been proposed in the last few years. A novel algorithm is proposed to make non-contact HR estimation technique more robust.

APPROACH

First, the face of the subject is detected and tracked to follow the head movement. The facial region then falls into several blocks, and the chrominance feature of each block is extracted to establish a raw HR sub-signal. Self-adaptive signal separation is performed to separate the noiseless HR sub-signals from raw sub-signals. On that basis, the noiseless sub-signals full of HR information are selected using a weight-based scheme to establish the holistic HR signal, from which the average HR is computed adopting wavelet transform and data filtering.

MAIN RESULTS

Forty subjects took part in our experiments, whose facial videos were recorded by a normal webcam with the frame rate of 30 fps under ambient lighting conditions. The average HR estimated by our method correlates strongly with ground truth measurements, as indicated in experimental results measured in a static scenario with the Pearson correlation r  =  0.980 and a dynamic scenario with the Pearson correlation r  =  0.897. In addition, our method, compared to the newest method, decreases the error rate by 38.63% and increases the Pearson correlation by 15.59%.

SIGNIFICANCE

This work proposes a robust method for non-contact HR measurement in a realistic environment. Results of comparative experiments indicate that our method out-performs state-of-the-art non-contact HR estimation methods in realistic environments.

A study of color illumination effect on the SNR of rPPG signals

Remote photoplethysmography (rPPG) can be used to measure cardiac activity by detecting the subtle color variation of the human skin tissue using an RGB camera. Recent studies have presented the feasibility and proposed multiple methods to improve the motion robustness for the subject movements. However, enhancing the signal-to-noise ratio (SNR) of the rPPG signal is still an important issue for the contactless measurement. In this paper, we conducted an experiment to study the lighting effect on the SNR of rPPG signals. The results point out that different colors of light sources provide different SNR in each RGB channel. By providing the dedicated light sources (λ= 490-620) nm, the SNR of rPPG signals captured from the green color channel can be enhanced. Among the tested light sources, light green provides the most significant improvement from -11.09 to -6.6 dB compared with the fluorescent light.

The non-contact biometric identified bio signal measurement sensor and algorithms

In these days, wearable devices have been developed for effectively measuring biological data. However, these devices have tissue allege and noise problem. To solve these problems, biometric measurement based on a non-contact method, such as face image sequencing is developed. This makes it possible to measure biometric data without any operation and side effects. However, it is impossible for a remote center to identify the person whose data are measured by the novel methods. In this paper, we propose the novel non-contact heart rate and blood pressure imaging system, Deep Health Eye. This system has authentication process at the same time as measuring bio signals, through non-contact method. In the future, this system can be convenient home bio signal monitoring system by combined with smart mirror.

Diagnostic Accuracy of a Novel Mobile Phone Application for the Detection and Monitoring of Atrial Fibrillation

Atrial fibrillation (AF) is the most common sustained cardiac arrhythmia in adults, associated with significant morbidity, increased mortality, and rising health-care costs. Simple and available tools for the accurate detection of arrhythmia recurrence in patients after electrical cardioversion (CV) or ablation procedures for AF can help to guide therapeutic decisions. We conducted a prospective, single-center study to evaluate the accuracy of Cardiio Rhythm Mobile Application (CRMA) for AF detection. Patients >18 years of age who were scheduled for elective CV for AF were enrolled in the study. CRMA finger pulse recordings, utilizing an iPhone camera, were obtained before (pre-CV) and after (post-CV) the CV. The findings were validated against surface electrocardiograms. Ninety-eight patients (75.5% men), mean age of 67.7 ± 10.5 years, were enrolled. No electrocardiogram for validation was available in 1 case. Pre-CV CRMA readings were analyzed in 97 of the 98 patients. Post-CV CRMA readings were analyzed for 92 of 93 patients who underwent CV. One patient left before the recording was obtained. The Cardiio Rhythm Mobile Application correctly identified 94 of 101 AF recordings (93.1%) as AF and 80 of 88 non-AF recordings (90.1%) as non-AF. The sensitivity was 93.1% (95% confidence interval [CI] = 86.9% to 97.2%) and the specificity was 90.9% (95% CI = 82.9% to 96.0%). The positive predictive value was 92.2% (95% CI = 85.8% to 95.8%) and the negative predictive value was 92.0% (95% CI = 94.8% to 95.9%). In conclusion, the CRMA demonstrates promising potential in accurate detection and discrimination of AF from normal sinus rhythm in patients with a history of AF.

Accurate face alignment and adaptive patch selection for heart rate estimation from videos under realistic scenarios

Non-contact heart rate (HR) measurement from facial videos has attracted high interests due to its convenience and cost effectiveness. However, accurate and robust HR estimation under various realistic scenarios remain a very challenging problem. In this paper, we develop a novel system which can achieve a robust and accurate HR estimation under those challenging scenarios. First, to minimize tracking-artifacts arising from large head motions and facial expressions, we propose a joint face detection and alignment method which can produce alignment-friendly facial bounding boxes with reliable initial facial shapes, facilitating accurate and robust face alignment even in the presence of large pose variations and expressions. Second, different from most existing methods [1–5] which derive pulse signals from predetermined grid cells (i.e. local patches), our patches are varying-sized triangles generated adaptively to exclude negative effects from non-rigid facial motions. Third, we propose an adaptive patch selection method to choose patches which contain skin regions and are more likely to contain useful information, followed by an independent component analysis, for an accurate HR estimate. Extensive experiments on both public datasets and our own dataset demonstrated that, comparing with the state-of-the-art methods [1–3], our method reduces the root mean square error (RMSE) by a large margin, ranging from 12% to 63%, and can provide a robust and accurate estimation under various challenging scenarios.

A Comparative Survey of Methods for Remote Heart Rate Detection From Frontal Face Videos

Remotely measuring physiological activity can provide substantial benefits for both the medical and the affective computing applications. Recent research has proposed different methodologies for the unobtrusive detection of heart rate (HR) using human face recordings. These methods are based on subtle color changes or motions of the face due to cardiovascular activities, which are invisible to human eyes but can be captured by digital cameras. Several approaches have been proposed such as signal processing and machine learning. However, these methods are compared with different datasets, and there is consequently no consensus on method performance. In this article, we describe and evaluate several methods defined in literature, from 2008 until present day, for the remote detection of HR using human face recordings. The general HR processing pipeline is divided into three stages: face video processing, face blood volume pulse (BVP) signal extraction, and HR computation. Approaches presented in the paper are classified and grouped according to each stage. At each stage, algorithms are analyzed and compared based on their performance using the public database MAHNOB-HCI. Results found in this article are limited on MAHNOB-HCI dataset. Results show that extracted face skin area contains more BVP information. Blind source separation and peak detection methods are more robust with head motions for estimating HR.

Detection of human vital signs in hazardous environments by means of video magnification

In cases of natural disasters, epidemics or even in dangerous situations like an act of terrorism, battle fields, a shooting or a mountain accident, finding survivors is a challenge. In these kind of situations it is sometimes critical to know if a person has vital signs or not, without the need to be in contact with the victim, thus avoiding jeopardizing the lives of the rescue workers. In this work, we propose the use of video magnification techniques to detect small movements in human bodies due to breathing that are invisible to the naked eye. Two different video magnification techniques, intensity-based and phase-based, were tested. The utility of these techniques to detect people who are alive but injured in risk situations was verified by simulating a scene with three people involved in an accident. Several factors such as camera stability, distance to the object, light conditions, magnification factor or computing time were analyzed. The results obtained were quite positive for both techniques, intensity-based method proving more adequate if the interest is in almost instant results whereas the phase-based method is more appropriate if processing time is not so relevant but the degree of magnification without excessive image noise.

Contact-Free Screening of Atrial Fibrillation by a Smartphone Using Facial Pulsatile Photoplethysmographic Signals

BACKGROUND

We aimed to evaluate a novel method of atrial fibrillation (AF) screening using an iPhone camera to detect and analyze photoplethysmographic signals from the face without physical contact by extracting subtle beat-to-beat variations of skin color that reflect the cardiac pulsatile signal.

METHODS AND RESULTS

Patients admitted to the cardiology ward of the hospital for clinical reasons were recruited. Simultaneous facial and fingertip photoplethysmographic measurements were obtained from 217 hospital inpatients (mean age, 70.3±13.9 years; 71.4% men) facing the front camera and with an index finger covering the back camera of 2 independent iPhones before a 12-lead ECG was recorded. Backdrop and background light intensity was monitored during signal acquisition. Three successive 20-second (total, 60 seconds) recordings were acquired per patient and analyzed for heart rate regularity by Cardiio Rhythm (Cardiio Inc, Cambridge, MA) smartphone application. Pulse irregularity in ≥1 photoplethysmographic readings or 3 uninterpretable photoplethysmographic readings were considered a positive AF screening result. AF was present on 12-lead ECG in 34.6% (n=75/217) patients. The Cardiio Rhythm facial photoplethysmographic application demonstrated high sensitivity (95%; 95% confidence interval, 87%-98%) and specificity (96%; 95% confidence interval, 91%-98%) in discriminating AF from sinus rhythm compared with 12-lead ECG. The positive and negative predictive values were 92% (95% confidence interval, 84%-96%) and 97% (95% confidence interval, 93%-99%), respectively.

CONCLUSIONS

Detection of a facial photoplethysmographic signal to determine pulse irregularity attributable to AF is feasible. The Cardiio Rhythm smartphone application showed high sensitivity and specificity, with low negative likelihood ratio for AF from facial photoplethysmographic signals. The convenience of a contact-free approach is attractive for community screening and has the potential to be useful for distant AF screening.

Multimodal Observation and Classification of People Engaged in Problem Solving: Application to Chess Players

In this paper we present the first results of a pilot experiment in the interpretation of multimodal observations of human experts engaged in solving challenging chess problems. Our goal is to investigate the extent to which observations of eye-gaze, posture, emotion and other physiological signals can be used to model the cognitive state of subjects, and to explore the integration of multiple sensor modalities to improve the reliability of detection of human displays of awareness and emotion. Domains of application for such cognitive model based systems are, for instance, healthy autonomous ageing or automated training systems. Abilities to observe cognitive abilities and emotional reactions can allow artificial systems to provide appropriate assistance in such contexts. We observed chess players engaged in problems of increasing difficulty while recording their behavior. Such recordings can be used to estimate a participant’s awareness of the current situation and to predict ability to respond effectively to challenging situations. Feature selection has been performed to construct a multimodal classifier relying on the most relevant features from each modality. Initial results indicate that eye-gaze, body posture and emotion are good features to capture such awareness. This experiment also validates the use of our equipment as a general and reproducible tool for the study of participants engaged in screen-based interaction and/or problem solving.

Heart rate estimation from facial photoplethysmography during dynamic illuminance changes

Camera-based remote photoplethysmography (rPPG) enables low-cost, non-contact cardiovascular activity monitoring. However, applying rPPG to practical use has some limitations caused from the artifacts by illuminance changes. During watching a video in a dark room, for example, watching a TV at night without illuminance, there is a high correlation between the brightness changes of a video and the illuminance variation on the skin of the viewer's face. In this study, we propose an artifact reduction method in rPPG, which is caused by the variation of the illuminance. The method subtracts the artifacts from the raw facial rPPG signal by applying multi-order curve fitting between the illuminance information from the facial rPPG signal and the brightness information from a video. On average, the results showed that signal-to-noise ratio (SNR) increased from -11.74 to -4.19 dB and from -15.27 to 7.99 dB for low-dynamic-brightness and high-dynamic-brightness video, respectively. In addition, the root-mean-square-error (RMSE) of estimated heart rate decreased from 11.00 to 1.82 bpm and from 9.88 to 4.65 bpm for the videos, respectively.

Contactless vision-based pulse rate detection of Infants Under Neurological Examinations

In this paper, we propose a method for detecting variations in the Pulse Rate (PR) of infants undergoing the Hammersmith Infant Neurological Examinations (HINE) using video data. As in every other medical examination the measurement of the PR is critical to underpin the physiological state of living beings. During HINE, measuring the infant's PR is important as its variations against physical conditions, age and other factors must be studied and correlated against developmental scores. However, this becomes highly complicated with active infants where their movements often lead to inconsistent PR estimation. We propose the use of a non-linear dimensionality reduction technique, called Laplacian Eigenmap (LE), to uncover the pulse information encapsulated within the high dimensional visual manifold characterized by normalized RGB feature vectors. Furthermore, low-level image filtering is applied to accurately detect PR within a chosen region-of-interest (ROI) from different parts of the infant's body. For validation and analysis, a set of 14 video sequences of infants undergoing five important tests of HINE have been chosen. Experimental results suggest that a bi-parametrized combination of color features from the RG and GB channels provide more valuable information in comparison to the RB and RGB channels. Results have demonstrated that this contactless method of PR detection has promising prospects for its future use in other clinical examinations of infants.

Remote photoplethysmography with constrained ICA using periodicity and chrominance constraints

BACKGROUND

Remote photoplethysmography (rPPG) has been in the forefront recently for measuring cardiac pulse rates from live or recorded videos. It finds advantages in scenarios requiring remote monitoring, such as medicine and fitness, where contact based monitoring is limiting and cumbersome. The blood volume pulse, defined as the pulsative flow of arterial blood, gives rise to periodic changes in the skin color which are then quantified to estimate a temporal signal. This temporal signal can be analysed using various methods to extract the representative cardiac signal.

METHODS

We present a novel method for measuring rPPG signals using constrained independent component analysis (cICA). We incorporate a priori information into the cICA algorithm to aid in the extraction of the most prominent rPPG signal. This a priori information is implemented using two constraints: first, based on periodicity using autocorrelation, and second, a chrominance-based constraint exploiting the physical characteristics of the skin.

RESULTS AND CONCLUSION

Our method showed improved performances over traditional blind source separation methods like ICA and chrominance based methods with mean absolute errors of 0.62 beats per minute (BPM) and 3.14 BPM for the two datasets in our inhouse video database UBFC-RPPG, and 4.69 BPM for the public MMSE-HR dataset. Its performance was also better in comparison to other state of the art methods in terms of accuracy and robustness. Our UBFC-RPPG database is also made publicly available and is specifically aimed towards testing rPPG measurements.

Bounded Kalman filter method for motion-robust, non-contact heart rate estimation

The authors of this work present a real-time measurement of heart rate across different lighting conditions and motion categories. This is an advancement over existing remote Photo Plethysmography (rPPG) methods that require a static, controlled environment for heart rate detection, making them impractical for real-world scenarios wherein a patient may be in motion, or remotely connected to a healthcare provider through telehealth technologies. The algorithm aims to minimize motion artifacts such as blurring and noise due to head movements (uniform, random) by employing i) a blur identification and denoising algorithm for each frame and ii) a bounded Kalman filter technique for motion estimation and feature tracking. A case study is presented that demonstrates the feasibility of the algorithm in non-contact estimation of the pulse rate of subjects performing everyday head and body movements. The method in this paper outperforms state of the art rPPG methods in heart rate detection, as revealed by the benchmarked results.

Video pulse rate variability analysis in stationary and motion conditions

BACKGROUND

In the last few years, some studies have measured heart rate (HR) or heart rate variability (HRV) parameters using a video camera. This technique focuses on the measurement of the small changes in skin colour caused by blood perfusion. To date, most of these works have obtained HRV parameters in stationary conditions, and there are practically no studies that obtain these parameters in motion scenarios and by conducting an in-depth statistical analysis.

METHODS

In this study, a video pulse rate variability (PRV) analysis is conducted by measuring the pulse-to-pulse (PP) intervals in stationary and motion conditions. Firstly, given the importance of the sampling rate in a PRV analysis and the low frame rate of commercial cameras, we carried out an analysis of two models to evaluate their performance in the measurements. We propose a selective tracking method using the Viola-Jones and KLT algorithms, with the aim of carrying out a robust video PRV analysis in stationary and motion conditions. Data and results of the proposed method are contrasted with those reported in the state of the art.

RESULTS

The webcam achieved better results in the performance analysis of video cameras. In stationary conditions, high correlation values were obtained in PRV parameters with results above 0.9. The PP time series achieved an RMSE (mean ± standard deviation) of 19.45 ± 5.52 ms (1.70 ± 0.75 bpm). In the motion analysis, most of the PRV parameters also achieved good correlation results, but with lower values as regards stationary conditions. The PP time series presented an RMSE of 21.56 ± 6.41 ms (1.79 ± 0.63 bpm).

CONCLUSIONS

The statistical analysis showed good agreement between the reference system and the proposed method. In stationary conditions, the results of PRV parameters were improved by our method in comparison with data reported in related works. An overall comparative analysis of PRV parameters in motion conditions was more limited due to the lack of studies or studies containing insufficient data analysis. Based on the results, the proposed method could provide a low-cost, contactless and reliable alternative for measuring HR or PRV parameters in non-clinical environments.

The PhysioCam: A Novel Non-Contact Sensor to Measure Heart Rate Variability in Clinical and Field Applications

Heart rate variability (HRV) is a reliable indicator of health status and a sensitive index of autonomic stress reactivity. Stress negatively affects physical and psychological wellness by decreasing cardiovascular health and reducing quality of life. Wearable sensors have made it possible to track HRV during daily activity, and recent advances in mobile technology have reduced the cost and difficulty of applying this powerful technique. Although advances have made sensors smaller and lighter, some burden on the subject remains. Chest-worn electrocardiogram (ECG) sensors provide the optimal source signal for HRV analysis, but they require obtrusive electrode or conductive material adherence. A less invasive surrogate of HRV can be derived from the arterial pulse obtained using the photoplethysmogram (PPG), but sensor placement requirements limit the application of PPG in field research. Factors including gender, age, height, and weight also affect PPG-HRV level, but PPG-HRV is sufficient to track individual HRV reactions to physical and mental challenges. To overcome the limitations of contact sensors, we developed the PhysioCam (PhyC), a non-contact system capable of measuring arterial pulse with sufficient precision to derive HRV during different challenges. This passive sensor uses an off the shelf digital color video camera to extract arterial pulse from the light reflected from an individual’s face. In this article, we validate this novel non-contact measure against criterion signals (ECG and PPG) in a controlled laboratory setting. Data from 12 subjects are presented under the following physiological conditions: rest, single deep breath and hold, and rapid breathing. The following HRV parameters were validated: interbeat interval (IBI), respiratory sinus arrhythmia (RSA), and low frequency HRV (LF). When testing the PhyC against ECG or PPG: the Bland–Altman plots for the IBIs show no systematic bias; correlation coefficients (all p values < 0.05) comparing ECG to PhyC for IBI and LF approach 1, while RSA correlations average 0.82 across conditions. We discuss future refinements of the HRV metrics derived from the PhyC that will enable this technology to unobtrusively track indicators of health and wellness.

Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement

Remote camera-based measurement of physiology has great potential for healthcare and affective computing. Recent advances in computer vision and signal processing have enabled photoplethysmography (PPG) measurement using commercially available cameras. However, there remain challenges in recovering accurate noncontact PPG measurements in the presence of rigid head motion. When a subject is moving, their face may be turned away from one camera, be obscured by an object, or move out of the frame resulting in missing observations. As the calculation of pulse rate variability (PRV) requires analysis over a time window of several minutes, the effect of missing observations on such features is deleterious. We present an approach for fusing partial color-channel signals from an array of cameras that enable physiology measurements to be made from moving subjects, even if they leave the frame of one or more cameras, which would not otherwise be possible with only a single camera. We systematically test our method on subjects ( N=25) using a set of six, 5-min tasks (each repeated twice) involving different levels of head motion. This results in validation across 25 h of measurement. We evaluate pulse rate and PRV parameter estimation including statistical, geometric, and frequency-based measures. The median absolute error in pulse rate measurements was 0.57 beats-per-minute (BPM). In all but two tasks with the greatest motion, the median error was within 0.4 BPM of that from a contact PPG device. PRV estimates were significantly improved using our proposed approach compared to an alternative not designed to handle missing values and multiple camera signals; the error was reduced by over 50%. Without our proposed method, errors in pulse rate would be very high, and estimation of PRV parameters would not be feasible due to significant data loss.

Novel health monitoring method using an RGB camera

In this paper we present a novel health monitoring method by estimating the heart rate and respiratory rate using an RGB camera. The heart rate and the respiratory rate are estimated from the photoplethysmography (PPG) and the respiratory motion. The method mainly operates by using the green spectrum of the RGB camera to generate a multivariate PPG signal to perform multivariate de-noising on the video signal to extract the resultant PPG signal. A periodicity based voting scheme (PVS) was used to measure the heart rate and respiratory rate from the estimated PPG signal. We evaluated our proposed method with a state of the art heart rate measuring method for two scenarios using the MAHNOB-HCI database and a self collected naturalistic environment database. The methods were furthermore evaluated for various scenarios at naturalistic environments such as a motion variance session and a skin tone variance session. Our proposed method operated robustly during the experiments and outperformed the state of the art heart rate measuring methods by compensating the effects of the naturalistic environment.

Comparison of motion-based analysis to thermal-based analysis of thermal video in the extraction of respiration patterns

Non-contact methods of extracting vital signals has become a popular area of research. This is likely due to the world's aging population and the increased need for long term and remote monitoring. This paper examines and compares the potential for one modality to capture a vital sign, specifically respiration, in the presence of signal abnormalities. This paper compares temperature based-methods to motion-based methods of extracting respiration rate from thermal video of a subject performing computationally difficult respiration tests. The thermal video was subjected to segmentation-based image processing and region tracking to encompass temperature changes over time. All methods were successful in identifying regular breathing and the absence of breathing, but differed in performance identifying hyperventilation and obstructive sleep apnea simulated breathing. The temperature-based method better depicted airflow volume, while the motion-based method better depicted absence of breath and chest movement; neither signal on its own was able to accurately depict OSA breathing. These results suggest that the fusion of information from different physical phenomenon (i.e. motion and temperature) is important here in detecting abnormal breathing patterns, but also in the detection of all vital signals, adding algorithmic robustness in the presence of signal abnormalities.

Serial fusion of Eulerian and Lagrangian approaches for accurate heart-rate estimation using face videos

Camera-equipped devices are ubiquitous and proliferating in the day-to-day life. Accurate heart rate (HR) estimation from the face videos acquired from the low cost cameras in a non-contact manner, can be used in many real-world scenarios and hence, require rigorous exploration. This paper has presented an accurate and near real-time HR estimation system using these face videos. It is based on the phenomenon that the color and motion variations in the face video are closely related to the heart beat. The variations also contain the noise due to facial expressions, respiration, eye blinking and environmental factors which are handled by the proposed system. Neither Eulerian nor Lagrangian temporal signals can provide accurate HR in all the cases. The cases where Eulerian temporal signals perform spuriously are determined using a novel poorness measure and then both the Eulerian and Lagrangian temporal signals are employed for better HR estimation. Such a fusion is referred as serial fusion. Experimental results reveal that the error introduced in the proposed algorithm is 1.8±3.6 which is significantly lower than the existing well known systems.

Assessment of instantaneous cardiovascular dynamics from video plethysmography

Although there is growing interest in estimating cardiovascular information using contactless video plethysmography (VP), an in-depth validation of time-varying, nonlinear dynamics of the related pulse rate variability is still missing. In this study we estimate the heartbeat through VP and standard ECG, and employ inhomogeneous point-process nonlinear models to assess instantaneous heart rate variability measures defined in the time, frequency, and bispectral domains. Experimental data were gathered from 60 young healthy subjects (age: 24±3 years) undergoing postural changes (rest-to-stand maneuver). Video recordings are processed using our recently proposed method based on zero-phase component analysis. Results show that, at a group level, there is an overall agreement between linear and nonlinear indices computed from ECG and VP during resting state conditions. However, significant differences are found, especially in the bispectral domain, when considering data gathered while standing. Although significant differences exist between cardiovascular estimates from VP and ECG, results can be considered very promising as instantaneous sympatho-vagal changes were correctly identified. More research is indeed needed to improve on the precise estimation of nonlinear sympatho-vagal interactions.

A Survey on Computer Vision for Assistive Medical Diagnosis From Faces

Automatic medical diagnosis is an emerging center of interest in computer vision as it provides unobtrusive objective information on a patient's condition. The face, as a mirror of health status, can reveal symptomatic indications of specific diseases. Thus, the detection of facial abnormalities or atypical features is at upmost importance when it comes to medical diagnostics. This survey aims to give an overview of the recent developments in medical diagnostics from facial images based on computer vision methods. Various approaches have been considered to assess facial symptoms and to eventually provide further help to the practitioners. However, the developed tools are still seldom used in clinical practice, since their reliability is still a concern due to the lack of clinical validation of the methodologies and their inadequate applicability. Nonetheless, efforts are being made to provide robust solutions suitable for healthcare environments, by dealing with practical issues such as real-time assessment or patients positioning. This survey provides an updated collection of the most relevant and innovative solutions in facial images analysis. The findings show that with the help of computer vision methods, over 30 medical conditions can be preliminarily diagnosed from the automatic detection of some of their symptoms. Furthermore, future perspectives, such as the need for interdisciplinary collaboration and collecting publicly available databases, are highlighted.

Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging

The ability to monitor the respiratory rate, one of the vital signs, is extremely important for the medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake everyday activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Alternatively, contactless digital image sensor based remote-photoplethysmography (PPG) can be used. However, remote PPG requires an ambient source of light, and does not work properly in dark places or under varying lighting conditions. Recent advances in thermographic systems have shrunk their size, weight and cost, to the point where it is possible to create smart-phone based respiration rate monitoring devices that are not affected by lighting conditions. However, mobile thermal imaging is challenged in scenes with high thermal dynamic ranges (e.g. due to the different environmental temperature distributions indoors and outdoors). This challenge is further amplified by general problems such as motion artifacts and low spatial resolution, leading to unreliable breathing signals. In this paper, we propose a novel and robust approach for respiration tracking which compensates for the negative effects of variations in the ambient temperature and motion artifacts and can accurately extract breathing rates in highly dynamic thermal scenes. The approach is based on tracking the nostril of the user and using local temperature variations to infer inhalation and exhalation cycles. It has three main contributions. The first is a novel Optimal Quantization technique which adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking. The second is the Thermal Gradient Flow method that computes thermal gradient magnitude maps to enhance the accuracy of the nostril region tracking. Finally, we introduce the Thermal Voxel method to increase the reliability of the captured respiration signals compared to the traditional averaging method. We demonstrate the extreme robustness of our system to track the nostril-region and measure the respiratory rate by evaluating it during controlled respiration exercises in high thermal dynamic scenes (e.g. strong correlation (r = 0.9987) with the ground truth from the respiration-belt sensor). We also demonstrate how our algorithm outperformed standard algorithms in settings with different amounts of environmental thermal changes and human motion. We open the tracked ROI sequences of the datasets collected for these studies (i.e. under both controlled and unconstrained real-world settings) to the community to foster work in this area.

Noncontact measurement of emotional and physiological changes in heart rate from a webcam

Heart rate, measured in beats per minute, can be used as an index of an individual's physiological state. Each time the heart beats, blood is expelled and travels through the body. This blood flow can be detected in the face using a standard webcam that is able to pick up subtle changes in color that cannot be seen by the naked eye. Due to the light absorption spectrum of blood, we are able to detect differences in the amount of light absorbed by the blood traveling just below the skin (i.e., photoplethysmography). By modulating emotional and physiological stress-that is, viewing arousing images and sitting versus standing, respectively-to elicit changes in heart rate, we explored the feasibility of using a webcam as a psychophysiological measurement of autonomic activity. We found a high level of agreement between established physiological measures, electrocardiogram, and blood pulse oximetry, and heart rate estimates obtained from the webcam. We thus suggest webcams can be used as a noninvasive and readily available method for measuring psychophysiological changes, easily integrated into existing stimulus presentation software and hardware setups.