Single Element Remote-PPG

Privacy-Protected Contactless Sleep Parameters Measurement Using a Defocused Camera

Sleep monitoring plays a vital role in various scenarios such as hospitals and living-assisted homes, contributing to the prevention of sleep accidents as well as the assessment of sleep health. Contactless camera-based sleep monitoring is promising due to its user-friendly nature and rich visual semantics. However, the privacy concern of video cameras limits their applications in sleep monitoring. In this paper, we explored the opportunity of using a defocused camera that does not allow identification of the monitored subject when measuring sleep-related parameters, as face detection and recognition are impossible on optically blurred images. We proposed a novel privacy-protected sleep parameters measurement framework, including a physiological measurement branch and a semantic analysis branch based on ResNet-18. Four important sleep parameters are measured: heart rate (HR), respiration rate (RR), sleep posture, and movement. The results of HR, RR, and movement have strong correlations with the reference (HR: R = 0.9076; RR: R = 0.9734; Movement: R = 0.9946). The overall mean absolute errors (MAE) for HR and RR are 5.2 bpm and 1.5 bpm respectively. The measurement of HR and RR achieve reliable estimation coverage of 72.1% and 93.6%, respectively. The sleep posture detection achieves an overall accuracy of 94.5%. Experimental results show that the defocused camera is promising for sleep monitoring as it fundamentally eliminates the privacy issue while still allowing the measurement of multiple parameters that are essential for sleep health informatics.

Multispectral Depolarization Based Living-Skin Detection: A New Measurement Principle

Camera-based photoplethysmographic imaging enabled the segmentation of living-skin tissues in a video, but it has inherent limitations to be used in real-life applications such as video health monitoring and face anti-spoofing. Inspired by the use of polarization for improving vital signs monitoring (i.e. specular reflection removal), we observed that skin tissues have an attractive property of wavelength-dependent depolarization due to its multi-layer structure containing different absorbing chromophores, i.e. polarized light photons with longer wavelengths (R) have deeper skin penetrability and thus experience thorougher depolarization than those with shorter wavelengths (G and B). Thus we proposed a novel dual-polarization setup and an elegant algorithm (named "MSD") that exploits the nature of multispectral depolarization of skin tissues to detect living-skin pixels, which only requires two images sampled at the parallel and cross polarizations to estimate the characteristic chromaticity changes (R/G) caused by tissue depolarization. Our proposal was verified in both the laboratory and hospital settings (ICU and NICU) focused on anti-spoofing and patient skin segmentation. The clinical experiments in ICU also indicate the potential of MSD for skin perfusion analysis, which may lead to a new diagnostic imaging approach in the future.

Insights on Using Time-of-Flight Camera for Recovering Cardiac Pulse From Chest Motion in Depth Videos

In this article, we introduce a novel use of depth camera to extract cardiac pulse signal from human chest area, in which the depth information is obtained from a near infrared sensor using time-of-flight technology. We successfully isolate weak chest motion due to heartbeat by processing a sequence of depth images without raising privacy concern. We discuss motion sensitivity in depth video with examples from actuator simulation and human chest motion. Compared to other imaging modalities, the depth image intensity can be directly used for micromotion reconstruction. To deal with the challenges of recovering heartbeat from the chest area, we develop a set of coherent processing techniques to suppress the unwanted motion interference from breathing motion and involuntary body motion and eventually obtain clean cardiac pulse signal. We, thus, derive inter-beat-interval, showing high consistency to the contact photoplethysmography. Additionally, we develop a graphical interpretation of the most and the less pulsatile principal components in eigen space. For validation, we test our method on ten healthy human subjects with different resting heart rates. More importantly, we conduct a set of experiments to study the robustness and weakness of our methods, including extended range, multi-subject, thickness of clothes and generation to other measurement site.

Your Health Is Leaked: PPG Waveform Reconstruction Using Stealthily Recorded Physiological Signals

In this study, we propose a method to reconstruct photoplethysmogram (PPG) waveforms from other stealthily recorded physiological signals. The proposed method focuses on the frequency characteristics between two physiological signals and reconstructs the target PPG waveform using a regression model. We investigate the feasibility of the proposed method to reconstruct target PPG signals from respiratory (RSP) and PPG signals recorded at non-genuine measurement sites using the two datasets of physiological signals. The results indicate that the proposed method achieves similarities between the target PPG and reconstructed PPG signals with correlation coefficients more than 0.860.

Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review

In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.

Rational selection of RGB channels for disease classification based on IPPG technology

The green channel is usually selected as the optimal channel for vital signs monitoring in image photoplethysmography (IPPG) technology. However, some controversies arising from the different penetrability of skin tissue in visible light remain unresolved, i.e., making the optical and physiological information carried by the IPPG signals of the RGB channels inconsistent. This study clarifies that the optimal channels for different diseases are different when IPPG technology is used for disease classification. We further verified this conclusion in the classification model of heart disease and diabetes mellitus based on the random forest classification algorithm. The experimental results indicate that the green channel has a considerably excellent performance in classifying heart disease patients and the healthy with an average Accuracy value of 88.43% and an average F1score value of 93.72%. The optimal channel for classifying diabetes mellitus patients and the healthy is the red channel with an average Accuracy value of 82.12% and the average F1score value of 89.31%. Due to the limited penetration depth of the blue channel into the skin tissue, the blue channel is not as effective as the green and red channels as a disease classification channel. This investigation is of great significance to the development of IPPG technology and its application in disease classification.

Fundamentals of Camera-PPG based Magnetic Resonance Imaging

In Magnetic Resonance Imaging (MRI), cardiac triggering that synchronizes data acquisition with cardiac contractions is an essential technique for acquiring high-quality images. Triggering is typically based on the Electrocardiogram (ECG) signal (e.g. R-peak). Since ECG acquisition involves extra workflow steps like electrode placement and ECG signals are usually disturbed by magnetic fields in high Magnetic Resonance (MR) systems, we explored camera-based photoplethysmography (PPG) as an alternative. We used the in-bore camera of a clinical MR system to investigate the feasibility and challenges of camera-based cardiac triggering. Data from ECG, finger oximeter and camera were synchronously collected. We found that that camera-PPG provides a higher availability of signal (and trigger) measurement, and the PPG signals measured from the forehead show considerably less delay w.r.t. the coupled ECG R-peak than the finger PPG signals in terms of trigger detection. The insights obtained in this study provide a basis for an envisioned system design phase.

Non-contact Measurement of Pulse Rate Variability Using a Webcam and Application to Mental Illness Screening System

The COVID-19 pandemic is a global health crisis. Mental health is critical in such uncertain situations, particularly when people are required to significantly restrict their movements and change their lifestyles. Under these conditions, many countries have turned to telemedicine to strengthen and expand mental health services. Our research group previously developed a mental illness screening system based on heart rate variability (HRV) analysis, enabling an objective and easy mental health self-check. This screening system cannot be used for telemedicine because it uses electrocardiography (ECG) and contact photoplethysmography (PPG), that are not widely available outside of a clinical setting. The purpose of this study is to enable the extension of the aforementioned system to telemedicine by the application of non-contact PPG using an RGB webcam, also called imaging- photoplethysmography (iPPG). The iPPG measurement errors occur due to changes in the relative position between the camera and the target, and due to changes in light. Conventionally, in image processing, the pixel value of the entire face region is used. We propose skin pixel extraction to eliminate blinks, eye movements, and changes in light and shadow. In signal processing, the green channel signal is conventionally used as a pulse wave owing to the absorption characteristics of blood flow. Taking advantage of the fact that the red and blue channels contain noise, we propose a signal reconstruction method for removing noise and strengthening the signal in the pulse rate variability (PRV) frequency band by weighting the three signals of the RGB camera. We conducted an experiment with 13 healthy subjects, and showed that the PRV index and pulse rate (PR) errors estimated by the proposed method were smaller than those of the conventional method. The correlation coefficients between estimated values by the proposed method and reference values of LF, HF, and PR were 0.86, 0.69, and 0.96, respectively.

Motion-Tolerant Non-Contact Heart-Rate Measurements from Radar Sensor Fusion

Microwave radar technology is very attractive for ubiquitous short-range health monitoring due to its non-contact, see-through, privacy-preserving and safe features compared to the competing remote technologies such as optics. The possibility of radar-based approaches for breathing and cardiac sensing was demonstrated a few decades ago. However, investigation regarding the robustness of radar-based vital-sign monitoring (VSM) is not available in the current radar literature. In this paper, we aim to close this gap by presenting an extensive experimental study of vital-sign radar approach. We consider diversity in test subjects, fitness levels, poses/postures, and, more importantly, random body movement (RBM) in the study. We discuss some new insights that lead to robust radar heart-rate (HR) measurements. A novel active motion cancellation signal-processing technique is introduced, exploiting dual ultra-wideband (UWB) radar system for motion-tolerant HR measurements. Additionally, we propose a spectral pruning routine to enhance HR estimation performance. We validate the proposed method theoretically and experimentally. Totally, we record and analyze about 3500 seconds of radar measurements from multiple human subjects.

Continuous-Spectrum Infrared Illuminator for Camera-PPG in Darkness

Many camera-based remote photoplethysmography (PPG) applications require sensing in near infrared (NIR). The performance of PPG systems benefits from multi-wavelength processing. The illumination source in such system is explored in this paper. We demonstrate that multiple narrow-band LEDs have inferior color homogeneity compared to broadband light sources. Therefore, we consider the broadband option based on phosphor material excited by LEDs. A first prototype was realized and its details are discussed. It was tested within a remote-PPG monitoring scenario in darkness and the full system demonstrates robust pulse-rate measurement. Given its accuracy in pulse rate extraction, the proposed illumination principle is considered a valuable asset for large-scale NIR-PPG applications as it enables multi-wavelength processing, lightweight set-ups with relatively low-power infrared light sources.

Contactless Vital Signs Measurement System Using RGB-Thermal Image Sensors and Its Clinical Screening Test on Patients with Seasonal Influenza

Background: In the last two decades, infrared thermography (IRT) has been applied in quarantine stations for the screening of patients with suspected infectious disease. However, the fever-based screening procedure employing IRT suffers from low sensitivity, because monitoring body temperature alone is insufficient for detecting infected patients. To overcome the drawbacks of fever-based screening, this study aims to develop and evaluate a multiple vital sign (i.e., body temperature, heart rate and respiration rate) measurement system using RGB-thermal image sensors. Methods: The RGB camera measures blood volume pulse (BVP) through variations in the light absorption from human facial areas. IRT is used to estimate the respiration rate by measuring the change in temperature near the nostrils or mouth accompanying respiration. To enable a stable and reliable system, the following image and signal processing methods were proposed and implemented: (1) an RGB-thermal image fusion approach to achieve highly reliable facial region-of-interest tracking, (2) a heart rate estimation method including a tapered window for reducing noise caused by the face tracker, reconstruction of a BVP signal with three RGB channels to optimize a linear function, thereby improving the signal-to-noise ratio and multiple signal classification (MUSIC) algorithm for estimating the pseudo-spectrum from limited time-domain BVP signals within 15 s and (3) a respiration rate estimation method implementing nasal or oral breathing signal selection based on signal quality index for stable measurement and MUSIC algorithm for rapid measurement. We tested the system on 22 healthy subjects and 28 patients with seasonal influenza, using the support vector machine (SVM) classification method. Results: The body temperature, heart rate and respiration rate measured in a non-contact manner were highly similarity to those measured via contact-type reference devices (i.e., thermometer, ECG and respiration belt), with Pearson correlation coefficients of 0.71, 0.87 and 0.87, respectively. Moreover, the optimized SVM model with three vital signs yielded sensitivity and specificity values of 85.7% and 90.1%, respectively. Conclusion: For contactless vital sign measurement, the system achieved a performance similar to that of the reference devices. The multiple vital sign-based screening achieved higher sensitivity than fever-based screening. Thus, this system represents a promising alternative for further quarantine procedures to prevent the spread of infectious diseases.

Impact of makeup on remote-PPG monitoring

Camera-based remote photoplethysmography (remote-PPG) enables contactless measurement of blood volume pulse from the human skin. Skin visibility is essential to remote-PPG as the camera needs to capture the light reflected from the skin that penetrates deep into skin tissues and carries blood pulsation information. The use of facial makeup may jeopardize this measurement by reducing the amount of light penetrating into and reflecting from the skin. In this paper, we conduct an empirical study to thoroughly investigate the impact of makeup on remote-PPG monitoring, in both the visible (RGB) and invisible (Near Infrared, NIR) lighting conditions. The experiment shows that makeup has negative influence on remote-PPG, which reduces the relative PPG strength (AC/DC) at different wavelengths and changes the normalized PPG signature across multiple wavelengths. It makes (i) the pulse-rate extraction more difficult in both the RGB and NIR, although NIR is less affected than RGB, and (ii) the blood oxygen saturation extraction in NIR impossible. To the best of our knowledge, this is the first work that systematically investigate the impact of makeup on camera-based remote-PPG monitoring.

Analysis of Facial Information for Healthcare Applications: A Survey on Computer Vision-Based Approaches

This paper gives an overview of the cutting-edge approaches that perform facial cue analysis in the healthcare area. The document is not limited to global face analysis but it also concentrates on methods related to local cues (e.g., the eyes). A research taxonomy is introduced by dividing the face in its main features: eyes, mouth, muscles, skin, and shape. For each facial feature, the computer vision-based tasks aiming at analyzing it and the related healthcare goals that could be pursued are detailed.

Modified RGB Cameras for Infrared Remote-PPG

Multi-wavelength cameras play an essential role in remote photoplethysmography (PPG). Whereas these are readily available for visible light, this is not the case for near infrared (NIR). We propose to modify existing RGB cameras to make them suited for NIR-PPG. In particular, we exploit the spectral leakage of the RGB channels in infrared in combination with a narrow dual-band optical filter. Such camera modification is simple, cost-effective, easy to implement, and it is shown to attain a pulse-rate extraction performance comparable to that of multiple narrow-band NIR cameras.

Discriminative Signatures for Remote-PPG

Near-infrared (NIR) remote photoplethysmography (PPG) promises attractive applications in darkness, as it involves unobtrusive, invisible light. However, since the PPG strength (AC/DC) is much lower in the NIR spectrum than in the RGB spectrum, robust vital signs monitoring is more challenging. In this paper, we propose a new PPG-extraction method, DIScriminative signature based extraction (DIS), to significantly improve the pulse-rate measurement in NIR. Our core idea is to use both the color signals containing blood absorption variations and additional disturbance signals as input for PPG extraction. By defining a discriminative signature, we use one-step least-squares regression (joint optimization) to retrieve the pulsatile component from color signals and suppress disturbance signals simultaneously. A large-scale lab experiment, recorded in NIR with heavy body motions, shows the significant improvement of DIS over the state-of-the-art method, whereas its principle is simple and generally applicable.