Advancements in noncontact, multiparameter physiological measurements using a webcam

Is Continuous Heart Rate Monitoring of Livestock a Dream or Is It Realistic? A Review

For all homoeothermic living organisms, heart rate (HR) is a core variable to control the metabolic energy production in the body, which is crucial to realize essential bodily functions. Consequently, HR monitoring is becoming increasingly important in research of farm animals, not only for production efficiency, but also for animal welfare. Real-time HR monitoring for humans has become feasible though there are still shortcomings for continuously accurate measuring. This paper is an effort to estimate whether it is realistic to get a continuous HR sensor for livestock that can be used for long term monitoring. The review provides the reported techniques to monitor HR of living organisms by emphasizing their principles, advantages, and drawbacks. Various properties and capabilities of these techniques are compared to check the potential to transfer the mostly adequate sensor technology of humans to livestock in term of application. Based upon this review, we conclude that the photoplethysmographic (PPG) technique seems feasible for implementation in livestock. Therefore, we present the contributions to overcome challenges to evolve to better solutions. Our study indicates that it is realistic today to develop a PPG sensor able to be integrated into an ear tag for mid-sized and larger farm animals for continuously and accurately monitoring their HRs.

Contactless Vital Signs Measurement System Using RGB-Thermal Image Sensors and Its Clinical Screening Test on Patients with Seasonal Influenza

Background: In the last two decades, infrared thermography (IRT) has been applied in quarantine stations for the screening of patients with suspected infectious disease. However, the fever-based screening procedure employing IRT suffers from low sensitivity, because monitoring body temperature alone is insufficient for detecting infected patients. To overcome the drawbacks of fever-based screening, this study aims to develop and evaluate a multiple vital sign (i.e., body temperature, heart rate and respiration rate) measurement system using RGB-thermal image sensors. Methods: The RGB camera measures blood volume pulse (BVP) through variations in the light absorption from human facial areas. IRT is used to estimate the respiration rate by measuring the change in temperature near the nostrils or mouth accompanying respiration. To enable a stable and reliable system, the following image and signal processing methods were proposed and implemented: (1) an RGB-thermal image fusion approach to achieve highly reliable facial region-of-interest tracking, (2) a heart rate estimation method including a tapered window for reducing noise caused by the face tracker, reconstruction of a BVP signal with three RGB channels to optimize a linear function, thereby improving the signal-to-noise ratio and multiple signal classification (MUSIC) algorithm for estimating the pseudo-spectrum from limited time-domain BVP signals within 15 s and (3) a respiration rate estimation method implementing nasal or oral breathing signal selection based on signal quality index for stable measurement and MUSIC algorithm for rapid measurement. We tested the system on 22 healthy subjects and 28 patients with seasonal influenza, using the support vector machine (SVM) classification method. Results: The body temperature, heart rate and respiration rate measured in a non-contact manner were highly similarity to those measured via contact-type reference devices (i.e., thermometer, ECG and respiration belt), with Pearson correlation coefficients of 0.71, 0.87 and 0.87, respectively. Moreover, the optimized SVM model with three vital signs yielded sensitivity and specificity values of 85.7% and 90.1%, respectively. Conclusion: For contactless vital sign measurement, the system achieved a performance similar to that of the reference devices. The multiple vital sign-based screening achieved higher sensitivity than fever-based screening. Thus, this system represents a promising alternative for further quarantine procedures to prevent the spread of infectious diseases.

Detail-preserving pulse wave extraction from facial videos using consumer-level camera

With the popularity of smart phones, non-contact video-based vital sign monitoring using a camera has gained increased attention over recent years. Especially, imaging photoplethysmography (IPPG), a technique for extracting pulse waves from videos, conduces to monitor physiological information on a daily basis, including heart rate, respiration rate, blood oxygen saturation, and so on. The main challenge for accurate pulse wave extraction from facial videos is that the facial color intensity change due to cardiovascular activities is subtle and is often badly disturbed by noise, such as illumination variation, facial expression changes, and head movements. Even a tiny interference could bring a big obstacle for pulse wave extraction and reduce the accuracy of the calculated vital signs. In recent years, many novel approaches have been proposed to eliminate noise such as filter banks, adaptive filters, Distance-PPG, and machine learning, but these methods mainly focus on heart rate detection and neglect the retention of useful details of pulse wave. For example, the pulse wave extracted by the filter bank method has no dicrotic wave and approaching sine wave, but dicrotic waves are essential for calculating vital signs like blood viscosity and blood pressure. Therefore, a new framework is proposed to achieve accurate pulse wave extraction that contains mainly two steps: 1) preprocessing procedure to remove baseline offset and high frequency random noise; and 2) a self-adaptive singular spectrum analysis algorithm to obtain cyclical components and remove aperiodic irregular noise. Experimental results show that the proposed method can extract detail-preserved pulse waves from facial videos under realistic situations and outperforms state-of-the-art methods in terms of detail-preserving and real time heart rate estimation. Furthermore, the pulse wave extracted by our approach enabled the non-contact estimation of atrial fibrillation, heart rate variability, blood pressure, as well as other physiological indices that require standard pulse wave.

Impact of makeup on remote-PPG monitoring

Camera-based remote photoplethysmography (remote-PPG) enables contactless measurement of blood volume pulse from the human skin. Skin visibility is essential to remote-PPG as the camera needs to capture the light reflected from the skin that penetrates deep into skin tissues and carries blood pulsation information. The use of facial makeup may jeopardize this measurement by reducing the amount of light penetrating into and reflecting from the skin. In this paper, we conduct an empirical study to thoroughly investigate the impact of makeup on remote-PPG monitoring, in both the visible (RGB) and invisible (Near Infrared, NIR) lighting conditions. The experiment shows that makeup has negative influence on remote-PPG, which reduces the relative PPG strength (AC/DC) at different wavelengths and changes the normalized PPG signature across multiple wavelengths. It makes (i) the pulse-rate extraction more difficult in both the RGB and NIR, although NIR is less affected than RGB, and (ii) the blood oxygen saturation extraction in NIR impossible. To the best of our knowledge, this is the first work that systematically investigate the impact of makeup on camera-based remote-PPG monitoring.

Analysis of CNN-based remote-PPG to understand limitations and sensitivities

Deep learning based on convolutional neural network (CNN) has shown promising results in various vision-based applications, recently also in camera-based vital signs monitoring. The CNN-based photoplethysmography (PPG) extraction has, so far, been focused on performance rather than understanding. In this paper, we try to answer four questions with experiments aiming at improving our understanding of this methodology as it gains popularity. We conclude that the network exploits the blood absorption variation to extract the physiological signals, and that the choice and parameters (phase, spectral content, etc.) of the reference-signal may be more critical than anticipated. The availability of multiple convolutional kernels is necessary for CNN to arrive at a flexible channel combination through the spatial operation, but may not provide the same motion-robustness as a multi-site measurement using knowledge-based PPG extraction. We also find that the PPG-related prior knowledge may still be helpful for the CNN-based PPG extraction, and recommend further investigation of hybrid CNN-based methods that include prior knowledge in their design.

Multi-Modal, Remote Breathing Monitor

Monitoring breathing is important for a plethora of applications including, but not limited to, baby monitoring, sleep monitoring, and elderly care. This paper presents a way to fuse both vision-based and RF-based modalities for the task of estimating the breathing rate of a human. The modalities used are the F200 Intel^® RealSense^TM RGB and depth (RGBD) sensor, and an ultra-wideband (UWB) radar. RGB image-based features and their corresponding image coordinates are detected on the human body and are tracked using the famous optical flow algorithm of Lucas and Kanade. The depth at these coordinates is also tracked. The synced-radar received signal is processed to extract the breathing pattern. All of these signals are then passed to a harmonic signal detector which is based on a generalized likelihood ratio test. Finally, a spectral estimation algorithm based on the reformed Pisarenko algorithm tracks the breathing fundamental frequencies in real-time, which are then fused into a one optimal breathing rate in a maximum likelihood fashion. We tested this multimodal set-up on 14 human subjects and we report a maximum error of 0.5 BPM compared to the true breathing rate.

Non-Contact Vital Signs Monitoring of Dog and Cat Using a UWB Radar

Keywords: cat; dog; vital signs monitoring; radar; ultra-wideband (UWB); variational mode decomposition (VMD).

Illumination Robust Heart-rate Extraction from Single-wavelength Infrared Camera Using Spatial-channel Expansion

Heart rate (HR) is one of the most important vital signs for indicating the health condition of a person. Contactless camera-based HR measurement is particularly beneficial for sleep monitoring, as it is comfortable and convenient. However, compared with ambient light, the skin pulsation in near infrared range is much weaker and more susceptible to distortions (e.g. body motion, light changes). In this paper, we propose a method to expand the single-wavelength channel of a near infrared camera to multiple channels for illumination noise reduction, where the channel expansion is performed in the spatial domain using skin and non-skin pixels. The essence is using illumination changes of non-skin pixels to eliminate such a distortion on skin pixels and thus improve pulse extraction. On average, measurement coverage increased from 50% to 83% for the methods of subtraction and Segment Principal Component Analysis (Seg-PCA), and Signal-to-Noise Ratio (SNR) is increased from -8.40 dB to -4.62 dB for the method of Segment Independent Component Analysis (Seg-ICA).

Using Non-Contact Imaging Photoplethysmography to Recover Diurnal Patterns in Heart Rate

Daily patterns in cardiovascular signals can reveal important information about physiological processes, health and well-being. Traditionally, contact sensors have been used to collect longitudinal data of this kind. However, recent advances in non-contact imaging techniques have led to algorithms that can be used to measure vital signs unobtrusively. Imaging methods are highly scalable due to the availability of webcams and computing devices making them attractive for longitudinal, in-situ measurement. Using a software tool we captured over 1,000 hours of non-contact heart rate measurements, via imaging photoplethysmography. Using these data we were able to recover diurnal patterns in heart rate during the working day. Non-contact sensing techniques hold much promise but also raise ethical issues that need to be addressed seriously within the biomedical engineering community.

Inter-Beat Interval Estimation from Facial Video Based on Reliability of BVP Signals

Inter-beat interval (IBI) and heart rate variability (HRV) are important cardiac parameters that provide physiological and emotional states of a person. In this paper, we present a framework for accurate IBI and HRV estimation from a facial video based on the reliability of extracted blood volume pulse (BVP) signals. Our framework first extracts candidate BVP signals from randomly sampled multiple face patches. The BVP signals are then assessed based on a reliability metric to select the most reliable BVP signal, from which IBI and HRV are calculated. In experiments, we evaluate three reliability metrics and demonstrate that our framework can estimate IBI and HRV more accurately than a conventional single face region-based framework.

Feasibility of assessing ultra-short-term pulse rate variability from video recordings

Objectives

Remote photoplethysmography (rPPG) is a promising non-contact measurement technique for assessing numerous physiological parameters: pulse rate, pulse rate variability (PRV), respiratory rate, pulse wave velocity, blood saturation, blood pressure, etc. To justify its use in ultra-short-term (UST) PRV analysis, which is of great benefit for several healthcare applications, the agreement between rPPG- and PPG-derived UST-PRV metrics was studied.

Approach

Three time-domain metrics—standard deviation of normal-to-normal (NN) intervals (SDNN), root mean square of successive NN interval differences (RMSSD), and the percentage of adjacent NN intervals that differ from each other by more than 50 ms (pNN50)—were extracted from 56 video recordings in a publicly available data set. The selected metrics were calculated on the basis of three groups of 10 s recordings and their average, two groups of 30 s recordings and their average, and a group of 60 s recordings taken from the full-length recordings and then compared with metrics derived from the corresponding reference (PPG) pulse waveform signals by using correlation and effect size parameters, and Bland–Altman plots.

Main results

The results show there is stronger agreement as the recording length increases for SDNN and RMSSD, yet there is no significant change for pNN50. The agreement parameters reach r = 0.841 (p < 0.001), r = 0.529 (p < 0.001), and r = 0.657 (p < 0.001), estimated median bias −1.52, −2.28 ms and −1.95% and a small effect size for SDNN, RMSSD, and pNN50 derived from the 60 s recordings, respectively.

Significance

Remote photoplethysmography-derived UST-PRV metrics manage to capture UST-PRV metrics derived from reference (PPG) recordings well. This feature is highly desirable in numerous applications for the assessment of one’s health and well-being. In future research, the validity of rPPG-derived UST-PRV metrics compared to the gold standard electrocardiography recordings is to be assessed.

Towards health monitoring using remote heart rate measurement using digital camera: A feasibility study

The paper presents a feasibility study for heart rate measurement using a digital camera to perform health monitoring. The feasibility study investigates the reliability of the state of the art heart rate measuring methods in realistic situations. Therefore, an experiment was designed and carried out on 45 subjects to investigate the effects caused by illumination, motion, skin tone, and distance variance. The experiment was conducted for two main scenarios; human-computer interaction scenario and health monitoring scenario. The human-computer scenario investigated the effects caused by illumination variance, motion variance, and skin tone variance. The health monitoring scenario investigates the feasibility of health monitoring at public spaces (i.e. airports, subways, malls). Five state of the art heart rate measuring methods were re-implemented and tested with the feasibility study database. The results were compared with ground truth to estimate the heart rate measurement error. The heart rate measurement error was analyzed using mean error, standard deviation; root means square error and Pearson correlation coefficient. The findings of this experiment inferred promising results for health monitoring of subjects standing at a distance of 500 cm.

A Non-Contact Photoplethysmography Technique for the Estimation of Heart Rate via Smartphone

This paper describes the development of an application for mobile devices under the iOS platform which has the objective of monitoring patients with alterations or affections from cardiac pathologies. The software tool developed for mobile devices provides a patient and a specialist doctor the ability to handle and treat disease remotely while monitoring through the technique of non-contact photoplethysmography (PPG). The mobile application works by processing red, green, and blue (RGB) color video images on a specific region of the face, thus obtaining the intensity of the pixels in the green channel. The results are then processed using mathematical algorithms and Fourier transform, moving from the time domain to the frequency domain to ensure proper interpretation and to obtain the pulses per minute (PPM). The results are favorable because a comparison of the results was made with respect to the application of a medical-grade pulse-oximeter, where an error rate of 3% was obtained, indicating the acceptable performance of our application. The present technological development provides an application tool with significant potential in the area of health.

Non-contact physiological monitoring of preterm infants in the Neonatal Intensive Care Unit

The implementation of video-based non-contact technologies to monitor the vital signs of preterm infants in the hospital presents several challenges, such as the detection of the presence or the absence of a patient in the video frame, robustness to changes in lighting conditions, automated identification of suitable time periods and regions of interest from which vital signs can be estimated. We carried out a clinical study to evaluate the accuracy and the proportion of time that heart rate and respiratory rate can be estimated from preterm infants using only a video camera in a clinical environment, without interfering with regular patient care. A total of 426.6 h of video and reference vital signs were recorded for 90 sessions from 30 preterm infants in the Neonatal Intensive Care Unit (NICU) of the John Radcliffe Hospital in Oxford. Each preterm infant was recorded under regular ambient light during daytime for up to four consecutive days. We developed multi-task deep learning algorithms to automatically segment skin areas and to estimate vital signs only when the infant was present in the field of view of the video camera and no clinical interventions were undertaken. We propose signal quality assessment algorithms for both heart rate and respiratory rate to discriminate between clinically acceptable and noisy signals. The mean absolute error between the reference and camera-derived heart rates was 2.3 beats/min for over 76% of the time for which the reference and camera data were valid. The mean absolute error between the reference and camera-derived respiratory rate was 3.5 breaths/min for over 82% of the time. Accurate estimates of heart rate and respiratory rate could be derived for at least 90% of the time, if gaps of up to 30 seconds with no estimates were allowed.

Cardio-respiratory signal extraction from video camera data for continuous non-contact vital sign monitoring using deep learning

Non-contact vital sign monitoring enables the estimation of vital signs, such as heart rate, respiratory rate and oxygen saturation (SpO2), by measuring subtle color changes on the skin surface using a video camera. For patients in a hospital ward, the main challenges in the development of continuous and robust non-contact monitoring techniques are the identification of time periods and the segmentation of skin regions of interest (ROIs) from which vital signs can be estimated. We propose a deep learning framework to tackle these challenges.

APPROACH

This paper presents two convolutional neural network (CNN) models. The first network was designed for detecting the presence of a patient and segmenting the patient's skin area. The second network combined the output from the first network with optical flow for identifying time periods of clinical intervention so that these periods can be excluded from the estimation of vital signs. Both networks were trained using video recordings from a clinical study involving 15 pre-term infants conducted in the high dependency area of the neonatal intensive care unit (NICU) of the John Radcliffe Hospital in Oxford, UK.

MAIN RESULTS

Our proposed methods achieved an accuracy of 98.8% for patient detection, a mean intersection-over-union (IOU) score of 88.6% for skin segmentation and an accuracy of 94.5% for clinical intervention detection using two-fold cross validation. Our deep learning models produced accurate results and were robust to different skin tones, changes in light conditions, pose variations and different clinical interventions by medical staff and family visitors.

SIGNIFICANCE

Our approach allows cardio-respiratory signals to be continuously derived from the patient's skin during which the patient is present and no clinical intervention is undertaken.

Non-contact heart and respiratory rate monitoring of preterm infants based on a computer vision system: a method comparison study

BACKGROUND

Non-contact heart rate (HR) and respiratory rate (RR) monitoring is necessary for preterm infants due to the potential for the adhesive electrodes of conventional electrocardiogram (ECG) to cause damage to the epidermis. This study was performed to evaluate the agreement between HR and RR measurements of preterm infants using a non-contact computer vision system with comparison to measurements obtained by the ECG.

METHODS

A single-centre, cross-sectional observational study was conducted in a Neonatal Unit. Ten infants and their ECG monitors were videoed using two Nikon cameras for 10 min. HR and RR measurements obtained from the non-contact system were extracted using advanced signal processing techniques and later compared to the ECG readings using Bland-Altman analysis.

RESULTS

The non-contact system was able to detect an apnoea when the ECG determined movement as respirations. Although the mean bias between both methods was relatively low, the limits of agreement for HR were -8.3 to 17.4 beats per minute (b.p.m.) and for RR, -22 to 23.6 respirations per minute (r.p.m.).

CONCLUSIONS

This study provides necessary data for improving algorithms to address confounding variables common to the neonatal population. Further studies investigating the robustness of the proposed system for premature infants are therefore required.

Neuroscience in service research: an overview and discussion of its possibilities

Purpose

The purpose of this paper is to discuss recent developments in neuroscientific methods and demonstrate its potential for the service field. This work is a call to action for more service researchers to adopt promising and increasingly accessible neuro-tools that allow the service field to benefit from neuroscience theories and insights.

Design/methodology/approach

The paper synthesizes key literature from a variety of domains (e.g. neuroscience, consumer neuroscience and organizational neuroscience) to provide an in-depth background to start applying neuro-tools. Specifically, this paper outlines the most important neuro-tools today and discusses their theoretical and empirical value.

Findings

To date, the use of neuro-tools in the service field is limited. This is surprising given the great potential they hold to advance service research. To stimulate the use of neuro-tools in the service area, the authors provide a roadmap to enable neuroscientific service studies and conclude with a discussion on promising areas (e.g. service experience and servicescape) ripe for neuroscientific input.

Originality/value

The paper offers service researchers a starting point to understand the potential benefits of adopting the neuroscientific method and shows their complementarity with traditional service research methods like surveys, experiments and qualitative research. In addition, this paper may also help reviewers and editors to better assess the quality of neuro-studies in service.

Unconstrained Vital Sign Monitoring System Using an Aortic Pulse Wave Sensor

This paper proposes a novel unconstrained monitoring system that measures heart and respiratory rates and evaluates autonomic nervous activity based on heart rate variability. The proposed system measures the aortic pulse waves (APWs) of a patient via an APW sensor that comprises a single microphone integrated into a mattress. Vital signs (i.e., heart rate, respiratory rate) and autonomic nervous activity were analyzed using the measured APWs. In an experiment with supine and seated participants, vital signs calculated by the proposed system were compared with vital signs measured with commercial devices, and we obtained the correlations of r > 0.8 for the heart rates, r > 0.7 for the respiratory rates, and r > 0.8 for the heart rate variability indices. These results indicate that the proposed system can produce accurate vital sign measurements. In addition, we performed the experiment of image stimulus presentation and explored the relationships between the self-reported psychological states evoked by the stimulus and the measured vital signs. The results indicated that vital signs reflect psychological states. In conclusion, the proposed system demonstrated its ability to monitor health conditions by actions as simple as sitting or lying on the APW sensor.

Noise Reduction Technique for Single-Color Video Plethysmography Using Singular Spectrum Analysis

Recently, a contactless method for measuring a biological signal using a video camera has garnered attention. Especially, video plethysmography, a technique for obtaining a pulse wave from a video, is useful for managing the health of people on a daily basis. However, any body movement of a person subjected to the measurement leads to the generation of irregular noise in video plethysmography and reduces the accuracy of the recorded biological information, e.g., heart rate, during the measurement. Blind source separation is a popular technique for eliminating noise from the results of video plethysmography comprising different multiple-color channels. However, it is difficult to apply this technique to a single-color video such as a near-infrared video. Herein, a new method that combines singular spectrum analysis with the circular autocorrelation function is introduced to eliminate irregular noise in single-color video plethysmography. Applying the proposed method on videos collected from 39 individuals improved the estimation accuracy of instantaneous heart rate by approximately 44% over a conventional method using a linear filter. Furthermore, the proposed method also enabled more precise estimations of the heart rate than that achieved using multi-color video plethysmography.

Stable Contactless Sensing of Vital Signs Using RGB-Thermal Image Fusion System with Facial Tracking for Infection Screening

Infrared thermography (IRT) has been used to screen febrile passengers in international airports for over a decade. However, fever-based infection screening using IRT suffered from low sensitivity because measurements can be affected by ambient temperature, humidity, etc. In our previous study, we proposed an RGB-thermal image fusion system to measure vital signs i.e., the RGB camera detects tiny changes in color from facial skin, associated with blood flow, to estimate heart rate, and IRT senses temperature changes around the nasal area, caused by respiration, to measure respiratory rate). The inclusion of heart and respiratory rates lead to increased screening accuracy. In the present study, to promote the widespread use of our system in real-world settings, a face detection and tracking method was developed and implemented into the system, thereby enabling the accurate and stable measurement of vital signs. We assessed heart and respiratory rate estimation via an RGB-thermal image fusion system using Bland-Altman plots and statistical analysis.

[Mobile seizure monitoring in epilepsy patients]

Wearables are receiving much attention from both epilepsy patients and treating physicians, for monitoring of seizure frequency and warning of seizures. They are also of interest for the detection of seizure-associated risks of patients, for differential diagnosis of rare seizure types and prediction of seizure-prone periods. Accelerometry, electromyography (EMG), heart rate and further autonomic parameters are recorded to capture clinical seizure manifestations. Currently, a clinical use to document nocturnal motor seizures is feasible. In this review the available devices, data on the performance in the documentation of seizures, current options for clinical use and developments in data analysis are presented and critically discussed.

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HRVIPL-HR is available at: ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the state-of-the-art methods on both the public-domain and our VIPL-HR databases.

Remote Monitoring of Vital Signs in Diverse Non-Clinical and Clinical Scenarios Using Computer Vision Systems: A Review

Techniques for noncontact measurement of vital signs using camera imaging technologies have been attracting increasing attention. For noncontact physiological assessments, computer vision-based methods appear to be an advantageous approach that could be robust, hygienic, reliable, safe, cost effective and suitable for long distance and long-term monitoring. In addition, video techniques allow measurements from multiple individuals opportunistically and simultaneously in groups. This paper aims to explore the progress of the technology from controlled clinical scenarios with fixed monitoring installations and controlled lighting, towards uncontrolled environments, crowds and moving sensor platforms. We focus on the diversity of applications and scenarios being studied in this topic. From this review it emerges that automatic multiple regions of interest (ROIs) selection, removal of noise artefacts caused by both illumination variations and motion artefacts, simultaneous multiple person monitoring, long distance detection, multi-camera fusion and accepted publicly available datasets are topics that still require research to enable the technology to mature into many real-world applications.

3D Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video

Remote pulse rate measurement from facial video has gained particular attention over the last few years. Research exhibits significant advancements and demonstrates that common video cameras correspond to reliable devices that can be employed to measure a large set of biomedical parameters without any contact with the subject. A new framework for measuring and mapping pulse rate from video is presented in this pilot study. The method, which relies on convolutional 3D networks, is fully automatic and does not require any special image preprocessing. In addition, the network ensures concurrent mapping by producing a prediction for each local group of pixels. A particular training procedure that employs only synthetic data is proposed. Preliminary results demonstrate that this convolutional 3D network can effectively extract pulse rate from video without the need for any processing of frames. The trained model was compared with other state-of-the-art methods on public data. Results exhibit significant agreement between estimated and ground-truth measurements: the root mean square error computed from pulse rate values assessed with the convolutional 3D network is equal to 8.64 bpm, which is superior to 10 bpm for the other state-of-the-art methods. The robustness of the method to natural motion and increases in performance correspond to the two main avenues that will be considered in future works.

Video-Based Contactless Heart-Rate Detection and Counting via Joint Blind Source Separation with Adaptive Noise Canceller

Driver assistance systems are a major focus of the automotive industry. Although technological functions that help drivers are improving, the monitoring of driver state functions receives less attention. In this respect, the human heart rate (HR) is one of the most important bio-signals, and it can be detected remotely using consumer-grade cameras. Based on this, a video-based driver state monitoring system using HR signals is proposed in this paper. In a practical automotive environment, monitoring the HR is very challenging due to changes in illumination, vibrations, and human motion. In order to overcome these problems, source separation strategies were employed using joint blind source separation, and feature combination was adopted to maximize HR variation. Noise-assisted data analysis was then adopted using ensemble empirical mode decomposition to extract the pure HR. Finally, power spectral density analysis was conducted in the frequency domain, and a post-processing smoothing filter was applied. The performance of the proposed approach was tested based on commonly employed metrics using the MAHNOB-HCI public dataset and compared with recently proposed competing methods. The experimental results proved that our method is robust for a variety of driving conditions based on testing using a driving dataset and static indoor environments.

Continuous Distant Measurement of the User's Heart Rate in Human-Computer Interaction Applications

In real world scenarios, the task of estimating heart rate (HR) using video plethysmography (VPG) methods is difficult because many factors could contaminate the pulse signal (i.e., a subjects’ movement, illumination changes). This article presents the evaluation of a VPG system designed for continuous monitoring of the user’s heart rate during typical human-computer interaction scenarios. The impact of human activities while working at the computer (i.e., reading and writing text, playing a game) on the accuracy of HR VPG measurements was examined. Three commonly used signal extraction methods were evaluated: green (G), green-red difference (GRD), blind source separation (ICA). A new method based on an excess green (ExG) image representation was proposed. Three algorithms for estimating pulse rate were used: power spectral density (PSD), autoregressive modeling (AR) and time domain analysis (TIME). In summary, depending on the scenario being studied, different combinations of signal extraction methods and the pulse estimation algorithm ensure optimal heart rate detection results. The best results were obtained for the ICA method: average RMSE = 6.1 bpm (beats per minute). The proposed ExG signal representation outperforms other methods except ICA (RMSE = 11.2 bpm compared to 14.4 bpm for G and 13.0 bmp for GRD). ExG also is the best method in terms of proposed success rate metric (sRate).

Discriminative Signatures for Remote-PPG

Near-infrared (NIR) remote photoplethysmography (PPG) promises attractive applications in darkness, as it involves unobtrusive, invisible light. However, since the PPG strength (AC/DC) is much lower in the NIR spectrum than in the RGB spectrum, robust vital signs monitoring is more challenging. In this paper, we propose a new PPG-extraction method, DIScriminative signature based extraction (DIS), to significantly improve the pulse-rate measurement in NIR. Our core idea is to use both the color signals containing blood absorption variations and additional disturbance signals as input for PPG extraction. By defining a discriminative signature, we use one-step least-squares regression (joint optimization) to retrieve the pulsatile component from color signals and suppress disturbance signals simultaneously. A large-scale lab experiment, recorded in NIR with heavy body motions, shows the significant improvement of DIS over the state-of-the-art method, whereas its principle is simple and generally applicable.

Robust heart-rate estimation from facial videos using Project_ICA

OBJECTIVE

Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described.

APPROACH

After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed 'Project_ICA', based on a skin reflection model, is employed to extract the pulse signal from the original signal.

MAIN RESULTS

To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants.

SIGNIFICANCE

The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

An Empirical Study Comparing Unobtrusive Physiological Sensors for Stress Detection in Computer Work

Several unobtrusive sensors have been tested in studies to capture physiological reactions to stress in workplace settings. Lab studies tend to focus on assessing sensors during a specific computer task, while in situ studies tend to offer a generalized view of sensors' efficacy for workplace stress monitoring, without discriminating different tasks. Given the variation in workplace computer activities, this study investigates the efficacy of unobtrusive sensors for stress measurement across a variety of tasks. We present a comparison of five physiological measurements obtained in a lab experiment, where participants completed six different computer tasks, while we measured their stress levels using a chest-band (ECG, respiration), a wristband (PPG and EDA), and an emerging thermal imaging method (perinasal perspiration). We found that thermal imaging can detect increased stress for most participants across all tasks, while wrist and chest sensors were less generalizable across tasks and participants. We summarize the costs and benefits of each sensor stream, and show how some computer use scenarios present usability and reliability challenges for stress monitoring with certain physiological sensors. We provide recommendations for researchers and system builders for measuring stress with physiological sensors during workplace computer use.

Noncontact Heart and Respiratory Rate Monitoring of Preterm Infants Based on a Computer Vision System: Protocol for a Method Comparison Study

BACKGROUND

Biomedical research in the application of noncontact methods to measure heart rate (HR) and respiratory rate (RR) in the neonatal population has produced mixed results. This paper describes and discusses a protocol for conducting a method comparison study, which aims to determine the accuracy of a proposed noncontact computer vision system to detect HR and RR relative to the HR and RR obtained by 3-lead electrocardiogram (ECG) in preterm infants in the neonatal unit.

OBJECTIVE

The aim of this preliminary study is to determine the accuracy of a proposed noncontact computer vision system to detect HR and RR relative to the HR and RR obtained by 3-lead ECG in preterm infants in the neonatal unit.

METHODS

A single-center cross-sectional study was planned to be conducted in the neonatal unit at Flinders Medical Centre, South Australia, in May 2018. A total of 10 neonates and their ECG monitors will be filmed concurrently for 10 min using digital cameras. Advanced image processing techniques are to be applied later to determine their physiological data at 3 intervals. These data will then be compared with the ECG readings at the same points in time.

RESULTS

Study enrolment began in May 2018. Results of this study were published in July 2019.

CONCLUSIONS

The study will analyze the data obtained by the noncontact system in comparison to data obtained by ECG, identify factors that may influence data extraction and accuracy when filming infants, and provide recommendations for how this noncontact system may be implemented into clinical applications.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR1-10.2196/13400.

iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox

Imaging-based, non-contact measurement of physiology (including imaging photoplethysmography and imaging ballistocardiography) is a growing field of research. There are several strengths of imaging methods that make them attractive. They remove the need for uncomfortable contact sensors and can enable spatial and concomitant measurement from a single sensor. Furthermore, cameras are ubiquitous and often low-cost solutions for sensing. Open source toolboxes help accelerate the progress of research by providing a means to compare new approaches against standard implementations of the state-of-the-art. We present an open source imaging-based physiological measurement toolbox with implementations of many of the most frequently employed computational methods. We hope that this toolbox will contribute to the advancement of noncontact physiological sensing methods. Code: https://github.com/danmcduff/iphys-toolbox.

Game-Calibrated and User-Tailored Remote Detection of Stress and Boredom in Games

Emotion detection based on computer vision and remote extraction of user signals commonly rely on stimuli where users have a passive role with limited possibilities for interaction or emotional involvement, e.g., images and videos. Predictive models are also trained on a group level, which potentially excludes or dilutes key individualities of users. We present a non-obtrusive, multifactorial, user-tailored emotion detection method based on remotely estimated psychophysiological signals. A neural network learns the emotional profile of a user during the interaction with calibration games, a novel game-based emotion elicitation material designed to induce emotions while accounting for particularities of individuals. We evaluate our method in two experiments ( n = 20 and n = 62 ) with mean classification accuracy of 61.6%, which is statistically significantly better than chance-level classification. Our approach and its evaluation present unique circumstances: our model is trained on one dataset (calibration games) and tested on another (evaluation game), while preserving the natural behavior of subjects and using remote acquisition of signals. Results of this study suggest our method is feasible and an initiative to move away from questionnaires and physical sensors into a non-obtrusive, remote-based solution for detecting emotions in a context involving more naturalistic user behavior and games.

Non-Contact Monitoring of Breathing Pattern and Respiratory Rate via RGB Signal Measurement

Among all the vital signs, respiratory rate remains the least measured in several scenarios, mainly due to the intrusiveness of the sensors usually adopted. For this reason, all contactless monitoring systems are gaining increasing attention in this field. In this paper, we present a measuring system for contactless measurement of the respiratory pattern and the extraction of breath-by-breath respiratory rate. The system consists of a laptop’s built-in RGB camera and an algorithm for post-processing of acquired video data. From the recording of the chest movements of a subject, the analysis of the pixel intensity changes yields a waveform indicating respiratory pattern. The proposed system has been tested on 12 volunteers, both males and females seated in front of the webcam, wearing both slim-fit and loose-fit t-shirts. The pressure-drop signal recorded at the level of nostrils with a head-mounted wearable device was used as reference respiratory pattern. The two methods have been compared in terms of mean of absolute error, standard error, and percentage error. Additionally, a Bland–Altman plot was used to investigate the bias between methods. Results show the ability of the system to record accurate values of respiratory rate, with both slim-fit and loose-fit clothing. The measuring system shows better performance on females. Bland–Altman analysis showed a bias of −0.01 breaths·min−1, with respiratory rate values between 10 and 43 breaths·min−1. Promising performance has been found in the preliminary tests simulating tachypnea.

Influence of acquisition frame-rate and video compression techniques on pulse-rate variability estimation from vPPG signal

In this paper, common time- and frequency-domain variability indexes obtained by pulse rate variability (PRV) series extracted from video-photoplethysmographic signal (vPPG) were compared with heart rate variability (HRV) parameters calculated from synchronized ECG signals. The dual focus of this study was to analyze the effect of different video acquisition frame-rates starting from 60 frames-per-second (fps) down to 7.5 fps and different video compression techniques using both lossless and lossy codecs on PRV parameters estimation. Video recordings were acquired through an off-the-shelf GigE Sony XCG-C30C camera on 60 young, healthy subjects (age 23±4 years) in the supine position. A fully automated, signal extraction method based on the Kanade-Lucas-Tomasi (KLT) algorithm for regions of interest (ROI) detection and tracking, in combination with a zero-phase principal component analysis (ZCA) signal separation technique was employed to convert the video frames sequence to a pulsatile signal. The frame-rate degradation was simulated on video recordings by directly sub-sampling the ROI tracking and signal extraction modules, to correctly mimic videos recorded at a lower speed. The compression of the videos was configured to avoid any frame rejection caused by codec quality leveling, FFV1 codec was used for lossless compression and H.264 with variable quality parameter as lossy codec. The results showed that a reduced frame-rate leads to inaccurate tracking of ROIs, increased time-jitter in the signals dynamics and local peak displacements, which degrades the performances in all the PRV parameters. The root mean square of successive differences (RMSSD) and the proportion of successive differences greater than 50 ms (PNN50) indexes in time-domain and the low frequency (LF) and high frequency (HF) power in frequency domain were the parameters which highly degraded with frame-rate reduction. Such a degradation can be partially mitigated by up-sampling the measured signal at a higher frequency (namely 60 Hz). Concerning the video compression, the results showed that compression techniques are suitable for the storage of vPPG recordings, although lossless or intra-frame compression are to be preferred over inter-frame compression methods. FFV1 performances are very close to the uncompressed (UNC) version with less than 45% disk size. H.264 showed a degradation of the PRV estimation directly correlated with the increase of the compression ratio.

Cardiovascular assessment by imaging photoplethysmography - a review

Over the last few years, the contactless acquisition of cardiovascular parameters using cameras has gained immense attention. The technique provides an optical means to acquire cardiovascular information in a very convenient way. This review provides an overview on the technique's background and current realizations. Besides giving detailed information on the most widespread application of the technique, namely the contactless acquisition of heart rate, we outline further concepts and we critically discuss the current state.

A Clinically Evaluated Interferometric Continuous-Wave Radar System for the Contactless Measurement of Human Vital Parameters

Vital parameters are key indicators for the assessment of health. Conventional methods rely on direct contact with the patients’ skin and can hence cause discomfort and reduce autonomy. This article presents a bistatic 24 GHz radar system based on an interferometric six-port architecture and features a precision of 1 µm in distance measurements. Placed at a distance of 40 cm in front of the human chest, it detects vibrations containing respiratory movements, pulse waves and heart sounds. For the extraction of the respiration rate, time-domain approaches like autocorrelation, peaksearch and zero crossing rate are compared to the Fourier transform, while template matching and a hidden semi-Markov model are utilized for the detection of the heart rate from sphygmograms and heart sounds. A medical study with 30 healthy volunteers was conducted to collect 5.5 h of data, where impedance cardiogram and electrocardiogram were used as gold standard for synchronously recording respiration and heart rate, respectively. A low root mean square error for the breathing rate (0.828 BrPM) and a high overall F1 score for heartbeat detection (93.14%) could be achieved using the proposed radar system and signal processing.

Non-Contact Physiological Parameters Extraction Using Facial Video Considering Illumination, Motion, Movement and Vibration

OBJECTIVE

In this paper, four physiological parameters, i.e., heart rate (HR), inter-beat-interval (IBI), heart rate variability (HRV), and oxygen saturation (SpO2), are extracted from facial video recordings.

METHODS

Facial videos were recorded for 10 min each in 30 test subjects while driving a simulator. Four regions of interest (ROIs) are automatically selected in each facial image frame based on 66 facial landmarks. Red-green-blue color signals are extracted from the ROIs and four physiological parameters are extracted from the color signals. For the evaluation, physiological parameters are also recorded simultaneously using a traditional sensor "cStress," which is attached to hands and fingers of test subjects.

RESULTS

The Bland Altman plots show 95% agreement between the camera system and "cStress" with the highest correlation coefficient R = 0.96 for both HR and SpO2. The quality index is estimated for IBI considering 100 ms R-peak error; the accumulated percentage achieved is 97.5%. HRV features in both time and frequency domains are compared and the highest correlation coefficient achieved is 0.93. One-way analysis of variance test shows that there are no statistically significant differences between the measurements by camera and reference sensors.

CONCLUSION

These results present high degrees of accuracy of HR, IBI, HRV, and SpO2 extraction from facial image sequences.

SIGNIFICANCE

The proposed non-contact approach could broaden the dimensionality of physiological parameters extraction using cameras. This proposed method could be applied for driver monitoring application under realistic conditions, i.e., illumination, motion, movement, and vibration.

Effects of Video Encoding on Camera-Based Heart Rate Estimation

OBJECTIVE

Public databases are important for evaluating and comparing different methods and algorithms for camera-based heart rate estimation. Because uncompressed video requires huge file sizes, a need for compression algorithms exists to store and share video data. Due to the optimization of modern video codecs for human perception, video compression can influence heart rate estimation negatively by reducing or eliminating small color changes of the skin (PPG) that are needed for camera based heart rate estimation. In this paper, we contribute a comprehensive analysis to answer the question of how to compress video without compromising PPG information.

METHODS

To analyze the influence of video compression, we compare the effect of several encoding parameters: two modern encoders (H264, H265), compression rate, resolution changes using different scaling algorithms, color subsampling, and file size on two publicly available datasets.

RESULTS

We show that increasing the compression rate decreases the accuracy of heart rate estimation, but that resolution can be reduced (up to a cutoff point) and color subsampling can be applied for reducing file size without a big impact on heart rate estimation.

CONCLUSIONS

From the results, we derive and propose guidelines for the recording and encoding of video data for camera-based heart rate estimation.

SIGNIFICANCE

The paper can sensitize the research community toward the problems of video encoding, and the proposed recommended practices can help with conducting future experiments and creating valuable datasets that can be shared publicly. Such datasets would improve comparability and reproducibility in the research field.

Measurement of emotional contagion using synchronization of heart rhythm pattern between two persons: Application to sales managers and sales force synchronization

The purpose of this study was to measure emotional contagion, determine its direction, and compare the intensity between positive and negative contagion using the synchronization of heart rhythm pattern (HRP). A total of 64 undergraduate students (32 women and 32 men) participated in the experiment, and were randomly categorized as either leaders or followers. Followers were required to imitate the facial expression (happy and sad) of the leader (emotional contagion) or of a facial image (emotional non-contagion). We found that emotional contagion significantly increased the correlation coefficient between leaders and followers' HRP for both positive and negative emotions, but emotional non-contagion did not. There was no significant difference in leaders' HRP before and after contagion, while followers' HRP changed significantly. During emotional contagion, the correlation coefficient for negative emotion was significantly higher than for positive emotion. The proposed method could measure low or high emotional contagion and determine its direction quantitatively. In our application study, a sales manager (leader) transmitted a positive emotion to a sales employee (follower), and the groups are organized as HEC or LEC (high or low emotional contagion) groups by evaluating the intensity of emotional contagion based on HRP synchrony between them. HEC group's performance was enhanced compared to the LEC group.

Estimation of respiratory rate from motion contaminated photoplethysmography signals incorporating accelerometry

Estimation of respiratory rate (RR) from photoplethysmography (PPG) signals has important applications in the healthcare sector, from assisting doctors onwards to monitoring patients in their own homes. The problem is still very challenging, particularly during the motion for large segments of data, where results from different methods often do not agree. The authors aim to propose a new technique which performs motion reduction from PPG signals with the help of simultaneous acceleration signals where the PPG and accelerometer sensors need to be embedded in the same sensor unit. This method also reconstructs motion corrupted PPG signals in the Hilbert domain. An auto-regressive (AR) based technique has been used to estimate the RR from reconstructed PPGs. The proposed method has provided promising results for the estimation of RRs and their variations from PPG signals corrupted with motion artefact. The proposed platform is able to contribute to continuous in-hospital and home-based monitoring of patients using PPG signals under various conditions such as rest and motion states.

Current progress of photoplethysmography and SPO2 for health monitoring

A photoplethysmograph (PPG) is a simple medical device for monitoring blood flow and transportation of substances in the blood. It consists of a light source and a photodetector for measuring transmitted and reflected light signals. Clinically, PPGs are used to monitor the pulse rate, oxygen saturation, blood pressure, and blood vessel stiffness. Wearable unobtrusive PPG monitors are commercially available. Here, we review the principle issues and clinical applications of PPG for monitoring oxygen saturation.

Non-Contact, Simple Neonatal Monitoring by Photoplethysmography

This paper presents non-contact vital sign monitoring in neonates, based on image processing, where a standard color camera captures the plethysmographic signal and the heart and breathing rates are processed and estimated online. It is important that the measurements are taken in a non-invasive manner, which is imperceptible to the patient. Currently, many methods have been proposed for non-contact measurement. However, to the best of the authors’ knowledge, it has not been possible to identify methods with low computational costs and a high tolerance to artifacts. With the aim of improving contactless measurement results, the proposed method based on the computer vision technique is enhanced to overcome the mentioned drawbacks. The camera is attached to an incubator in the Neonatal Intensive Care Unit and a single area in the neonate’s diaphragm is monitored. Several factors are considered in the stages of image acquisition, as well as in the plethysmographic signal formation, pre-filtering and filtering. The pre-filter step uses numerical analysis techniques to reduce the signal offset. The proposed method decouples the breath rate from the frequency of sinus arrhythmia. This separation makes it possible to analyze independently any cardiac and respiratory dysrhythmias. Nine newborns were monitored with our proposed method. A Bland-Altman analysis of the data shows a close correlation of the heart rates measured with the two approaches (correlation coefficient of 0.94 for heart rate (HR) and 0.86 for breath rate (BR)) with an uncertainty of 4.2 bpm for HR and 4.9 for BR (k = 1). The comparison of our method and another non-contact method considered as a standard independent component analysis (ICA) showed lower central processing unit (CPU) usage for our method (75% less CPU usage).

Frame Registration for Motion Compensation in Imaging Photoplethysmography

Imaging photoplethysmography (iPPG) is an emerging technology used to assess microcirculation and cardiovascular signs by collecting backscattered light from illuminated tissue using optical imaging sensors. An engineering approach is used to evaluate whether a silicone cast of a human palm might be effectively utilized to predict the results of image registration schemes for motion compensation prior to their application on live human tissue. This allows us to establish a performance baseline for each of the algorithms and to isolate performance and noise fluctuations due to the induced motion from the temporally changing physiological signs. A multi-stage evaluation model is developed to qualitatively assess the influence of the region of interest (ROI), system resolution and distance, reference frame selection, and signal normalization on extracted iPPG waveforms from live tissue. We conclude that the application of image registration is able to deliver up to 75% signal-to-noise (SNR) improvement (4.75 to 8.34) over an uncompensated iPPG signal by employing an intensity-based algorithm with a moving reference frame.