A survey of remote optical photoplethysmographic imaging methods

Remote Vital Sensing in Clinical Veterinary Medicine: A Comprehensive Review of Recent Advances, Accomplishments, Challenges, and Future Perspectives

Remote vital sensing in veterinary medicine is a relatively new area of practice, which involves the acquisition of data without invasion of the body cavities of live animals. This paper aims to review several technologies in remote vital sensing: infrared thermography, remote photoplethysmography (rPPG), radar, wearable sensors, and computer vision and machine learning. In each of these technologies, we outline its concepts, uses, strengths, and limitations in multiple animal species, and its potential to reshape health surveillance, welfare evaluation, and clinical medicine in animals. The review also provides information about the problems associated with applying these technologies, including species differences, external conditions, and the question of the reliability and classification of these technologies. Additional topics discussed in this review include future developments such as the use of artificial intelligence, combining different sensing methods, and creating monitoring solutions tailored to specific animal species. This contribution gives a clear understanding of the status and future possibilities of remote vital sensing in veterinary applications and stresses the importance of that technology for the development of the veterinary field in terms of animal health and science.

Understanding the physiological transmission mechanisms of photoplethysmography signals: a comprehensive review

Objective. The widespread adoption of Photoplethysmography (PPG) as a non-invasive method for detecting blood volume variations and deriving vital physiological parameters reflecting health status has surged, primarily due to its accessibility, cost-effectiveness, and non-intrusive nature. This has led to extensive research around this technique in both daily life and clinical applications. Interestingly, despite the existence of contradictory explanations of the underlying mechanism of PPG signals across various applications, a systematic investigation into this crucial matter has not been conducted thus far. This gap in understanding hinders the full exploitation of PPG technology and undermines its accuracy and reliability in numerous applications.Approach. Building upon a comprehensive review of the fundamental principles and technological advancements in PPG, this paper initially attributes the origin of PPG signals to a combination of physical and physiological transmission processes. Furthermore, three distinct models outlining the concerned physiological transmission processes are synthesized, with each model undergoing critical examination based on theoretical underpinnings, empirical evidence, and constraints.Significance. The ultimate objective is to form a fundamental framework for a better understanding of physiological transmission processes in PPG signal generation and to facilitate the development of more reliable technologies for detecting physiological signals.

PhysioSens1D-NET: A 1D Convolution Network for Extracting Heart Rate from Facial Videos

Non-contact heart rate (HR) monitoring from video streams is the most established approach to unobtrusive vitals monitoring. A multitude of classical signal processing algorithms and cutting-edge deep learning models have been developed for non-contact HR extraction. Classical signal processing algorithms excel in real-time application, even on low-end CPUs, while deep learning models offer higher accuracy at the cost of computational complexity. In this study, we introduce PhysioSens1DNET- a novel 1D convolutional neural network, that deliver both computational efficiency and accurate HR measures. In contrast to classical rPPG algorithms like ICA, POS, CHROM, PBV, LGI, and GREEN, the PhysioSens1D-NET demonstrates significant improvements, achieving reductions in Mean Absolute Error (MAE) by 91.4%, 72.5%, 70.7%, 93.1%, 76.7%, and 95.1%, respectively. When compared to state-of-the-art deep learning models, including DeepPhys, EfficientNet, PhysNet, and TS-CAN, our 1D-NET exhibits comparable performance. A performance analysis on low specification CPU's, indicated that PhysioSens1DNET outperforms deep learning models, showcasing a considerable speed advantage-being 180 times faster than the bestperforming DL model. Furthermore, our 1D-NET aligns closely with classical algorithms with a computational time of only 2.3 ms.

Enhancing Non-Contact Heart Rate Monitoring: An Intelligent Multi-ROI Approach with Face Masking and CNN-Based Feature Adaptation

Heart rate (HR) estimation from facial video streams has emerged in the recent years as a promising method of unobtrusive vitals monitoring. Conventional non-contact HR monitoring algorithms like POS, CHROM, ICA are often applied to a single region of interest (ROI), typically the forehead. However, this approach has a lot of disadvantages, such as not utilizing other facial regions, poor tolerance to movement of the subject or face. To address this, we propose a MultiROI approach with face Masking and CNN-based facial feature adaptation. We introduce an novel face masking technique method using facial landmarks alone, effectively eliminating non-skin pixels like background, hair, eyes, lips, and eyebrows. Additionally, a CNN model was designed to classify individuals based on facial features, dynamically adjusting ROI positions and ROI numbers accordingly. The proposed comprehensive approach significantly reduced the Mean Absolute Error (MAE) in HR measurement by 58.2%, 47.1%, and 33.2% for POS, CHROM, and ICA algorithms respectively, when compared to the traditional single ROI approach. The multi-ROI approach can thus improve measurement reliability and robustness to motion.

Insights on Using Time-of-Flight Camera for Recovering Cardiac Pulse From Chest Motion in Depth Videos

In this article, we introduce a novel use of depth camera to extract cardiac pulse signal from human chest area, in which the depth information is obtained from a near infrared sensor using time-of-flight technology. We successfully isolate weak chest motion due to heartbeat by processing a sequence of depth images without raising privacy concern. We discuss motion sensitivity in depth video with examples from actuator simulation and human chest motion. Compared to other imaging modalities, the depth image intensity can be directly used for micromotion reconstruction. To deal with the challenges of recovering heartbeat from the chest area, we develop a set of coherent processing techniques to suppress the unwanted motion interference from breathing motion and involuntary body motion and eventually obtain clean cardiac pulse signal. We, thus, derive inter-beat-interval, showing high consistency to the contact photoplethysmography. Additionally, we develop a graphical interpretation of the most and the less pulsatile principal components in eigen space. For validation, we test our method on ten healthy human subjects with different resting heart rates. More importantly, we conduct a set of experiments to study the robustness and weakness of our methods, including extended range, multi-subject, thickness of clothes and generation to other measurement site.

Non-contact high precision pulse-rate monitoring system for moving subjects in different motion states

Remote photoplethysmography (rPPG) enables contact-free monitoring of the pulse rate by using a color camera. The fundamental limitation is that motion artifacts and changes in ambient light conditions greatly affect the accuracy of pulse-rate monitoring. We propose use of a high-speed camera and a motion suppression algorithm with high computational efficiency. This system incorporates a number of major improvements including reproduction of pulse wave details, high-precision pulse-rate monitoring of moving subjects, and excellent scene scalability. A series of quantization methods were used to evaluate the effect of different frame rates and different algorithms in pulse-rate monitoring of moving subjects. The experimental results show that use of 180-fps video and a Plane-Orthogonal-to-Skin (POS) algorithm can produce high-precision pulse-rate monitoring results with mean absolute error can be less than 5 bpm and the relative accuracy reaching 94.5%. Thus, it has significant potential to improve personal health care and intelligent health monitoring.

Benchmarking of Contactless Heart Rate Measurement Systems in ARM-Based Embedded Platforms

Heart rate monitoring is especially important for aging individuals because it is associated with longevity and cardiovascular risk. Typically, this vital parameter can be measured using wearable sensors, which are widely available commercially. However, wearable sensors have some disadvantages in terms of acceptability, especially when used by elderly people. Thus, contactless solutions have increasingly attracted the scientific community in recent years. Camera-based photoplethysmography (also known as remote photoplethysmography) is an emerging method of contactless heart rate monitoring that uses a camera and a processing unit on the hardware side, and appropriate image processing methodologies on the software side. This paper describes the design and implementation of a novel pipeline for heart rate estimation using a commercial and low-cost camera as the input device. The pipeline's performance was tested and compared on a desktop PC, a laptop, and three different ARM-based embedded platforms (Raspberry Pi 4, Odroid N2+, and Jetson Nano). The results showed that the designed and implemented pipeline achieved an average accuracy of about 96.7% for heart rate estimation, with very low variance (between 1.5% and 2.5%) across processing platforms, user distances from the camera, and frame resolutions. Furthermore, benchmark analysis showed that the Odroid N2+ platform was the most convenient in terms of CPU load, RAM usage, and average execution time of the algorithmic pipeline.

RGB camera-based simultaneous measurements of percutaneous arterial oxygen saturation, tissue oxygen saturation, pulse rate, and respiratory rate

We propose a method to perform simultaneous measurements of percutaneous arterial oxygen saturation (SpO₂), tissue oxygen saturation (StO₂), pulse rate (PR), and respiratory rate (RR) in real-time, using a digital red–green–blue (RGB) camera. Concentrations of oxygenated hemoglobin (C_HbO), deoxygenated hemoglobin (C_HbR), total hemoglobin (C_HbT), and StO₂ were estimated from videos of the human face using a method based on a tissue-like light transport model of the skin. The photoplethysmogram (PPG) signals are extracted from the temporal fluctuations in C_HbO, C_HbR, and C_HbT using a finite impulse response (FIR) filter (low and high cut-off frequencies of 0.7 and 3 Hz, respectively). The PR is calculated from the PPG signal for C_HbT. The ratio of pulse wave amplitude for C_HbO and that for C_HbR are associated with the reference value of SpO₂ measured by a commercially available pulse oximeter, which provides an empirical formula to estimate SpO₂ from videos. The respiration-dependent oscillation in C_HbT was extracted from another FIR filter (low and high cut-off frequencies of 0.05 and 0.5 Hz, respectively) and used to calculate the RR. In vivo experiments with human volunteers while varying the fraction of inspired oxygen were performed to evaluate the comparability of the proposed method with commercially available devices. The Bland–Altman analysis showed that the mean bias for PR, RR, SpO₂, and StO₂ were -1.4 (bpm), -1.2(rpm), 0.5 (%), and -3.0 (%), respectively. The precisions for PR, RR, Sp O₂, and StO₂ were ±3.1 (bpm), ±3.5 (rpm), ±4.3 (%), and ±4.8 (%), respectively. The resulting precision and RMSE for StO₂ were pretty close to the clinical accuracy requirement. The accuracy of the RR is considered a little less accurate than clinical requirements. This is the first demonstration of a low-cost RGB camera-based method for contactless simultaneous measurements of the heart rate, percutaneous arterial oxygen saturation, and tissue oxygen saturation in real-time.

Performance analysis of remote photoplethysmography deep filtering using long short-term memory neural network

BACKGROUND

Remote photoplethysmography (rPPG) is a technique developed to estimate heart rate using standard video cameras and ambient light. Due to the multiple sources of noise that deteriorate the quality of the signal, conventional filters such as the bandpass and wavelet-based filters are commonly used. However, after using conventional filters, some alterations remain, but interestingly an experienced eye can easily identify them.

RESULTS

We studied a long short-term memory (LSTM) network in the rPPG filtering task to identify these alterations using many-to-one and many-to-many approaches. We used three public databases in intra-dataset and cross-dataset scenarios, along with different protocols to analyze the performance of the method. We demonstrate how the network can be easily trained with a set of 90 signals totaling around 45 min. On the other hand, we show the stability of the LSTM performance with six state-of-the-art rPPG methods.

CONCLUSIONS

This study demonstrates the superiority of the LSTM-based filter experimentally compared with conventional filters in an intra-dataset scenario. For example, we obtain on the VIPL database an MAE of 3.9 bpm, whereas conventional filtering improves performance on the same dataset from 10.3 bpm to 7.7 bpm. The cross-dataset approach presents a dependence in the network related to the average signal-to-noise ratio on the rPPG signals, where the closest signal-to-noise ratio values in the training and testing set the better. Moreover, it was demonstrated that a relatively small amount of data are sufficient to successfully train the network and outperform the results obtained by classical filters. More precisely, we have shown that about 45 min of rPPG signal could be sufficient to train an effective LSTM deep-filter.

Intelligent Remote Photoplethysmography-Based Methods for Heart Rate Estimation from Face Videos: A Survey

Over the last few years, a rich amount of research has been conducted on remote vital sign monitoring of the human body. Remote photoplethysmography (rPPG) is a camera-based, unobtrusive technology that allows continuous monitoring of changes in vital signs and thereby helps to diagnose and treat diseases earlier in an effective manner. Recent advances in computer vision and its extensive applications have led to rPPG being in high demand. This paper specifically presents a survey on different remote photoplethysmography methods and investigates all facets of heart rate analysis. We explore the investigation of the challenges of the video-based rPPG method and extend it to the recent advancements in the literature. We discuss the gap within the literature and suggestions for future directions.

On the spatial phase distribution of cutaneous low-frequency perfusion oscillations

Distributed cutaneous tissue blood volume oscillations contain information on autonomic nervous system (ANS) regulation of cardiorespiratory activity as well as dominating thermoregulation. ANS associated with low-frequency oscillations can be quantified in terms of frequencies, amplitudes, and phase shifts. The relative order between these faculties may be disturbed by conditions colloquially termed ‘stress’. Photoplethysmography imaging, an optical non-invasive diagnostic technique provides information on cutaneous tissue perfusion in the temporal and spatial domains. Using the cold pressure test (CPT) in thirteen healthy volunteers as a well-studied experimental intervention, we present a method for evaluating phase shifts in low- and intermediate frequency bands in forehead cutaneous perfusion mapping. Phase shift changes were analysed in low- and intermediate frequency ranges from 0.05 Hz to 0.18 Hz. We observed that time waveforms increasingly desynchronised in various areas of the scanned area throughout measurements. An increase of IM band phase desynchronization observed throughout measurements was comparable in experimental and control group, suggesting a time effect possibly due to overshooting the optimal relaxation duration. CPT triggered an increase in the number of points phase-shifted to the reference that was specific to the low frequency range for phase-shift thresholds defined as π/4, 3π/8, and π/2 rad, respectively. Phase shifts in forehead blood oscillations may infer changes of vascular tone due to activity of various neural systems. We present an innovative method for the phase shift analysis of cutaneous tissue perfusion that appears promising to assess ANS change processes related to physical or psychological stress. More comprehensive studies are needed to further investigate the reliability and physiological significance of findings.

Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing

Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camera-based physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing. We provide empirical evidence that heart and breathing rate measurement accuracy increases with the number of synthetic avatars in the training set. Furthermore, training with avatars with darker skin types leads to better overall performance than training with avatars with lighter skin types. Finally, we discuss the opportunities that synthetics present in the domain of camera-based physiological sensing and limitations that need to be overcome.

Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda

Heart rate (HR) is one of the essential vital signs used to indicate the physiological health of the human body. While traditional HR monitors usually require contact with skin, remote photoplethysmography (rPPG) enables contactless HR monitoring by capturing subtle light changes of skin through a video camera. Given the vast potential of this technology in the future of digital healthcare, remote monitoring of physiological signals has gained significant traction in the research community. In recent years, the success of deep learning (DL) methods for image and video analysis has inspired researchers to apply such techniques to various parts of the remote physiological signal extraction pipeline. In this paper, we discuss several recent advances of DL-based methods specifically for remote HR measurement, categorizing them based on model architecture and application. We further detail relevant real-world applications of remote physiological monitoring and summarize various common resources used to accelerate related research progress. Lastly, we analyze the implications of research findings and discuss research gaps to guide future explorations.

A Review of Deep Learning-Based Contactless Heart Rate Measurement Methods

The interest in contactless or remote heart rate measurement has been steadily growing in healthcare and sports applications. Contactless methods involve the utilization of a video camera and image processing algorithms. Recently, deep learning methods have been used to improve the performance of conventional contactless methods for heart rate measurement. After providing a review of the related literature, a comparison of the deep learning methods whose codes are publicly available is conducted in this paper. The public domain UBFC dataset is used to compare the performance of these deep learning methods for heart rate measurement. The results obtained show that the deep learning method PhysNet generates the best heart rate measurement outcome among these methods, with a mean absolute error value of 2.57 beats per minute and a mean square error value of 7.56 beats per minute.

Noncontact Monitoring of Heart Rate and Heart Rate Variability in Geriatric Patients Using Photoplethysmography Imaging

OBJECTIVE

Geriatric patients, especially those with dementia or in a delirious state, do not accept conventional contact-based monitoring. Therefore, we propose to measure heart rate (HR) and heart rate variability (HRV) of geriatric patients in a noncontact and unobtrusive way using photoplethysmography imaging (PPGI).

METHODS

PPGI video sequences were recorded from 10 geriatric patients and 10 healthy elderly people using a monochrome camera operating in the near-infrared spectrum and a colour camera operating in the visible spectrum. PPGI waveforms were extracted from both cameras using superpixel-based regions of interests (ROI). A classifier based on bagged trees was trained to automatically select artefact-free ROIs for HR estimation. HRV was calculated in the time-domain and frequency-domain.

RESULTS

an RMSE of 1.03 bpm and a correlation of 0.8 with the reference was achieved using the NIR camera for HR estimation. Using the RGB camera, RMSE and correlation improved to 0.48 bpm and 0.95, respectively. Correlation for HRV in the frequency-domain (LF/HF-ratio) was 0.50 using the NIR camera and 0.70 using the RGB camera.

CONCLUSION

We were able to demonstrate that PPGI is very suitable to measure HR and HRV in geriatric patients. We strongly believe that PPGI will become clinically relevant in monitoring of geriatric patients.

SIGNIFICANCE

we are the first group to measure both HR and HRV in awake geriatric patients using PPGI. Moreover, we systematically evaluate the effects of the spectrum (near-infrared vs. visible), ROI, and additional motion artefact reduction algorithms on the accuracy of estimated HR and HRV.

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

Noncontact Respiratory Monitoring Using Depth Sensing Cameras: A Review of Current Literature

There is considerable interest in the noncontact monitoring of patients as it allows for reduced restriction of patients, the avoidance of single-use consumables and less patient-clinician contact and hence the reduction of the spread of disease. A technology that has come to the fore for noncontact respiratory monitoring is that based on depth sensing camera systems. This has great potential for the monitoring of a range of respiratory information including the provision of a respiratory waveform, the calculation of respiratory rate and tidal volume (and hence minute volume). Respiratory patterns and apneas can also be observed in the signal. Here we review the ability of this method to provide accurate and clinically useful respiratory information.

Spatio-temporal and -spectral feature maps in photoplethysmography imaging and infrared thermograph

BACKGROUND

Only a small fraction of the information available is generally used in the majority of camera-based sensing approaches for vital sign monitoring. Dedicated skin pixels, for example, fall into this category while other regions are often disregarded early in the processing chain.

METHODS

We look at a simple processing chain for imaging where a video stream is converted to several other streams to investigate whether other image regions should also be considered. These streams are generated by mapping spatio-temporal and -spectral features of video segments and, thus, compressing the information contained in several seconds of video and encoding these in a new image. Two typical scenarios are provided as examples to study the applicability of these maps: face videos in a laboratory setting and measurements of a baby in the neonatal intensive care unit. Each measurement consists of the synchronous recording of photoplethysmography imaging (PPGI) and infrared thermography (IRT). We report the results of a visual inspection of those maps, evaluate the root mean square (RMS) contrast of foreground and background regions, and use histogram intersections as a tool for similarity measurements.

RESULTS

The maps allow us to distinguish visually between pulsatile foreground objects and an image background, which is found to be a noisy pattern. Distortions in the maps could be localized and the origin could be discovered. The IRT highlights subject contours for the heart frequency band, while silhouettes show strong signals in PPGI. Reflections and shadows were found to be sources of signals and distortions. We can testify advantages for the use of near-infrared light for PPGI. Furthermore, a difference in RMS contrast for pulsatile and non-pulsatile regions could be demonstrated. Histogram intersections allowed us to differentiate between the background and foreground.

CONCLUSIONS

We introduced new maps for the two sensing modalities and presented an overview for three different wavelength ranges. The maps can be used as a tool for visualizing aspects of the dynamic information hidden in video streams without automation. We propose focusing on an indirect method to detect pulsatile regions by using the noisy background pattern characteristic, for example, based on the histogram approach introduced.

Biometric Signals Estimation Using Single Photon Camera and Deep Learning

The problem of performing remote biomedical measurements using just a video stream of a subject face is called remote photoplethysmography (rPPG). The aim of this work is to propose a novel method able to perform rPPG using single-photon avalanche diode (SPAD) cameras. These are extremely accurate cameras able to detect even a single photon and are already used in many other applications. Moreover, a novel method that mixes deep learning and traditional signal analysis is proposed in order to extract and study the pulse signal. Experimental results show that this system achieves accurate results in the estimation of biomedical information such as heart rate, respiration rate, and tachogram. Lastly, thanks to the adoption of the deep learning segmentation method and dependability checks, this method could be adopted in non-ideal working conditions—for example, in the presence of partial facial occlusions.

Non-contact imaging of peripheral hemodynamics during cognitive and psychological stressors

Peripheral hemodynamics, measured via the blood volume pulse and vasomotion, provide a valuable way of monitoring physiological state. Camera imaging-based systems can be used to measure these peripheral signals without contact with the body, at distances of multiple meters. While researchers have paid attention to non-contact imaging photoplethysmography, the study of peripheral hemodynamics and the effect of autonomic nervous system activity on these signals has received less attention. Using a method, based on a tissue-like model of the skin, we extract melanin [Formula: see text] and hemoglobin [Formula: see text] concentrations from videos of the hand and face and show that significant decreases in peripheral pulse signal power (by 36% ± 29%) and vasomotion signal power (by 50% ± 26%) occur during periods of cognitive and psychological stress. Via three experiments we show that similar results are achieved across different stimuli and regions of skin (face and hand). While changes in peripheral pulse and vasomotion power were significant the changes in pulse rate variability were less consistent across subjects and tasks.

Is Continuous Heart Rate Monitoring of Livestock a Dream or Is It Realistic? A Review

For all homoeothermic living organisms, heart rate (HR) is a core variable to control the metabolic energy production in the body, which is crucial to realize essential bodily functions. Consequently, HR monitoring is becoming increasingly important in research of farm animals, not only for production efficiency, but also for animal welfare. Real-time HR monitoring for humans has become feasible though there are still shortcomings for continuously accurate measuring. This paper is an effort to estimate whether it is realistic to get a continuous HR sensor for livestock that can be used for long term monitoring. The review provides the reported techniques to monitor HR of living organisms by emphasizing their principles, advantages, and drawbacks. Various properties and capabilities of these techniques are compared to check the potential to transfer the mostly adequate sensor technology of humans to livestock in term of application. Based upon this review, we conclude that the photoplethysmographic (PPG) technique seems feasible for implementation in livestock. Therefore, we present the contributions to overcome challenges to evolve to better solutions. Our study indicates that it is realistic today to develop a PPG sensor able to be integrated into an ear tag for mid-sized and larger farm animals for continuously and accurately monitoring their HRs.

An open-source remote heart rate imaging method with practical apparatus and algorithms

Recent developments in computer science and digital image processing have enabled the extraction of an individual’s heart pulsations from pixel changes in recorded video images of human skin surfaces. This method is termed remote photoplethysmography (rPPG) and can be achieved with consumer-level cameras (e.g., a webcam or mobile camera). The goal of the present publication is two-fold. First, we aim to organize future rPPG software developments in a tractable and nontechnical manner, such that the public gains access to a basic open-source rPPG code, comes to understand its utility, and can follow its most recent progressions. The second goal is to investigate rPPG’s accuracy in detecting heart rates from the skin surfaces of several body parts after physical exercise and under ambient lighting conditions with a consumer-level camera. We report that rPPG is highly accurate when the camera is aimed at facial skin tissue, but that the heart rate recordings from wrist regions are less reliable, and recordings from the calves are unreliable. Facial rPPG remained accurate despite the high heart rates after exercise. The proposed research procedures and the experimental findings provide guidelines for future studies on rPPG.

Illumination Robust Heart-rate Extraction from Single-wavelength Infrared Camera Using Spatial-channel Expansion

Heart rate (HR) is one of the most important vital signs for indicating the health condition of a person. Contactless camera-based HR measurement is particularly beneficial for sleep monitoring, as it is comfortable and convenient. However, compared with ambient light, the skin pulsation in near infrared range is much weaker and more susceptible to distortions (e.g. body motion, light changes). In this paper, we propose a method to expand the single-wavelength channel of a near infrared camera to multiple channels for illumination noise reduction, where the channel expansion is performed in the spatial domain using skin and non-skin pixels. The essence is using illumination changes of non-skin pixels to eliminate such a distortion on skin pixels and thus improve pulse extraction. On average, measurement coverage increased from 50% to 83% for the methods of subtraction and Segment Principal Component Analysis (Seg-PCA), and Signal-to-Noise Ratio (SNR) is increased from -8.40 dB to -4.62 dB for the method of Segment Independent Component Analysis (Seg-ICA).

Using Non-Contact Imaging Photoplethysmography to Recover Diurnal Patterns in Heart Rate

Daily patterns in cardiovascular signals can reveal important information about physiological processes, health and well-being. Traditionally, contact sensors have been used to collect longitudinal data of this kind. However, recent advances in non-contact imaging techniques have led to algorithms that can be used to measure vital signs unobtrusively. Imaging methods are highly scalable due to the availability of webcams and computing devices making them attractive for longitudinal, in-situ measurement. Using a software tool we captured over 1,000 hours of non-contact heart rate measurements, via imaging photoplethysmography. Using these data we were able to recover diurnal patterns in heart rate during the working day. Non-contact sensing techniques hold much promise but also raise ethical issues that need to be addressed seriously within the biomedical engineering community.

Continuous respiratory rate monitoring during an acute hypoxic challenge using a depth sensing camera

Respiratory rate is a well-known to be a clinically important parameter with numerous clinical uses including the assessment of disease state and the prediction of deterioration. It is frequently monitored using simple spot checks where reporting is intermittent and often prone to error. We report here on an algorithm to determine respiratory rate continuously and robustly using a non-contact method based on depth sensing camera technology. The respiratory rate of 14 healthy volunteers was studied during an acute hypoxic challenge where blood oxygen saturation was reduced in steps to a target 70% oxygen saturation and which elicited a wide range of respiratory rates. Depth sensing data streams were acquired and processed to generate a respiratory rate (RR_depth). This was compared to a reference respiratory rate determined from a capnograph (RR_cap). The bias and root mean squared difference (RMSD) accuracy between RR_depth and the reference RR_cap was found to be 0.04 bpm and 0.66 bpm respectively. The least squares fit regression equation was determined to be: RR_depth = 0.99 × RR_cap + 0.13 and the resulting Pearson correlation coefficient, R, was 0.99 (p < 0.001). These results were achieved with a 100% reporting uptime. In conclusion, excellent agreement was found between RR_depth and RR_cap. Further work should include a larger cohort combined with a protocol to further test algorithmic performance in the face of motion and interference typical of that experienced in the clinical setting.

A Broader Look: Camera-Based Vital Sign Estimation across the Spectrum

OBJECTIVES

Camera-based vital sign estimation allows the contactless assessment of important physiological parameters. Seminal contributions were made in the 1930s, 1980s, and 2000s, and the speed of development seems ever increasing. In this suivey, we aim to overview the most recent works in this area, describe their common features as well as shortcomings, and highlight interesting "outliers".

METHODS

We performed a comprehensive literature research and quantitative analysis of papers published between 2016 and 2018. Quantitative information about the number of subjects, studies with healthy volunteers vs. pathological conditions, public datasets, laboratory vs. real-world works, types of camera, usage of machine learning, and spectral properties of data was extracted. Moreover, a qualitative analysis of illumination used and recent advantages in terms of algorithmic developments was also performed.

RESULTS

Since 2016, 116 papers were published on camera-based vital sign estimation and 59% of papers presented results on 20 or fewer subjects. While the average number of participants increased from 15.7 in 2016 to 22.9 in 2018, the vast majority of papers (n=100) were on healthy subjects. Four public datasets were used in 10 publications. We found 27 papers whose application scenario could be considered a real-world use case, such as monitoring during exercise or driving. These include 16 papers that dealt with non-healthy subjects. The majority of papers (n=61) presented results based on visual, red-green-blue (RGB) information, followed by RGB combined with other parts of the electromagnetic spectrum (n=18), and thermography only (n=12), while other works (n=25) used other mono- or polychromatic non-RGB data. Surprisingly, a minority of publications (n=39) made use of consumer-grade equipment. Lighting conditions were primarily uncontrolled or ambient. While some works focused on specialized aspects such as the removal of vital sign information from video streams to protect privacy or the influence of video compression, most algorithmic developments were related to three areas: region of interest selection, tracking, or extraction of a one-dimensional signal. Seven papers used deep learning techniques, 17 papers used other machine learning approaches, and 92 made no explicit use of machine learning.

CONCLUSION

Although some general trends and frequent shortcomings are obvious, the spectrum of publications related to camera-based vital sign estimation is broad. While many creative solutions and unique approaches exist, the lack of standardization hinders comparability of these techniques and of their performance. We believe that sharing algorithms and/ or datasets will alleviate this and would allow the application of newer techniques such as deep learning.

iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox

Imaging-based, non-contact measurement of physiology (including imaging photoplethysmography and imaging ballistocardiography) is a growing field of research. There are several strengths of imaging methods that make them attractive. They remove the need for uncomfortable contact sensors and can enable spatial and concomitant measurement from a single sensor. Furthermore, cameras are ubiquitous and often low-cost solutions for sensing. Open source toolboxes help accelerate the progress of research by providing a means to compare new approaches against standard implementations of the state-of-the-art. We present an open source imaging-based physiological measurement toolbox with implementations of many of the most frequently employed computational methods. We hope that this toolbox will contribute to the advancement of noncontact physiological sensing methods. Code: https://github.com/danmcduff/iphys-toolbox.

Game-Calibrated and User-Tailored Remote Detection of Stress and Boredom in Games

Emotion detection based on computer vision and remote extraction of user signals commonly rely on stimuli where users have a passive role with limited possibilities for interaction or emotional involvement, e.g., images and videos. Predictive models are also trained on a group level, which potentially excludes or dilutes key individualities of users. We present a non-obtrusive, multifactorial, user-tailored emotion detection method based on remotely estimated psychophysiological signals. A neural network learns the emotional profile of a user during the interaction with calibration games, a novel game-based emotion elicitation material designed to induce emotions while accounting for particularities of individuals. We evaluate our method in two experiments ( n = 20 and n = 62 ) with mean classification accuracy of 61.6%, which is statistically significantly better than chance-level classification. Our approach and its evaluation present unique circumstances: our model is trained on one dataset (calibration games) and tested on another (evaluation game), while preserving the natural behavior of subjects and using remote acquisition of signals. Results of this study suggest our method is feasible and an initiative to move away from questionnaires and physical sensors into a non-obtrusive, remote-based solution for detecting emotions in a context involving more naturalistic user behavior and games.

Optimum Wavelengths in the Near Infrared for Imaging Photoplethysmography

OBJECTIVE

The purpose of this contribution is to determine the ideal near infrared wavelength bands for monochromatic and dual-band remote heartbeat detection using imaging photoplethysmography (iPPG) of the forehead.

METHODS

Experimental data of 38 healthy volunteers has been recorded and analyzed. For the data acquisition, a fast hyperspectral imager has been used. A new combination approach has been implemented that computes the quotient of the bands and, therefore, reduces motion artifacts.

RESULTS

With this dual-band method excellent results (1.67 beats per minute mean deviation from electrocardiogram measurements for 73 recordings) have been obtained using a simple algorithm to analyze images at 799 and 861 nm.

CONCLUSION

It can be concluded that excellent imaging photoplethysmography measurements can be performed at low cost using conventional silicon-based image sensors with invisible light in the near infrared region.

SIGNIFICANCE

This approach is a contribution to the development of non-contact heart rate measurement systems that can be used for medical diagnosis or other applications.

A wavelet-based decomposition method for a robust extraction of pulse rate from video recordings

Background

Remote photoplethysmography (rPPG) is a promising optical method for non-contact assessment of pulse rate (PR) from video recordings. In order to implement the method in real-time applications, it is necessary for the rPPG algorithms to be capable of eliminating as many distortions from the pulse signal as possible.

Methods

In order to increase the degrees-of-freedom of the distortion elimination, the dimensionality of the RGB video signals is increased by the wavelet transform decomposition using the generalized Morse wavelet. The proposed Continuous-Wavelet-Transform-based Sub-Band rPPG method (SB-CWT) is evaluated on the 101 publicly available RGB facial video recordings and corresponding reference blood volume pulse (BVP) signals taken from the MMSE-HR database. The performance of the SB-CWT is compared with the performance of the state-of-the-art Sub-band rPPG (SB).

Results

Median signal-to-noise ratio (SNR) for the proposed SB-CWT ranges from 6.63 to 10.39 dB and for the SB from 4.23 to 6.24 dB. The agreement between the estimated PRs from rPPG pulse signals and the reference signals in terms of the coefficients of determination ranges from 0.81 to 0.91 for SB-CWT and from 0.41 to 0.47 for SB. All the correlation coefficients are statistically significant (p < 0.001). The Bland-Altman plots show that mean difference range from 5.37 to 1.82 BPM for SB-CWT and from 22.18 to 18.80 BPM for SB.

Discussion

The results show that the proposed SB-CWT outperforms SB in terms of SNR and the agreement between the estimated PRs from RGB video signals and PRs from the reference BVP signals.

InPhysible: Camouflage Against Video-Based Physiological Measurement

Imaging photoplethysmography (iPPG) is a powerful set of methods for measuring physiological signals from video. Recent advances have shown that a low-cost webcam can be used to measure heart rate, blood flow, respiration, blood oxygen levels and stress. While these methods have many beneficial applications, the unobtrusive and ubiquitous nature of the sensors risk exposing people to unwanted measurement. We present InPhysible the first camouflage system against video- based physiological measurement. The infra-red system can be embedded into any pair of glasses, or other headwear, and disrupts the measurement of the iPPG signal while being imperceptible by the human eye. Our system is flexible and can simulate realistic pulse signals to hinder heart rate measurement. In this paper we present the design of our prototype and a user study validating its efficacy. Finally, we discuss the limitations and implications for data privacy and security.

Video-based heart rate monitoring across a range of skin pigmentations during an acute hypoxic challenge

The robust monitoring of heart rate from the video-photoplethysmogram (video-PPG) during challenging conditions requires new analysis techniques. The work reported here extends current research in this area by applying a motion tolerant algorithm to extract high quality video-PPGs from a cohort of subjects undergoing marked heart rate changes during a hypoxic challenge, and exhibiting a full range of skin pigmentation types. High uptimes in reported video-based heart rate (HRvid) were targeted, while retaining high accuracy in the results. Ten healthy volunteers were studied during a double desaturation hypoxic challenge. Video-PPGs were generated from the acquired video image stream and processed to generate heart rate. HRvid was compared to the pulse rate posted by a reference pulse oximeter device (HRp). Agreement between video-based heart rate and that provided by the pulse oximeter was as follows: Bias = - 0.21 bpm, RMSD = 2.15 bpm, least squares fit gradient = 1.00 (Pearson R = 0.99, p < 0.0001), with a 98.78% reporting uptime. The difference between the HRvid and HRp exceeded 5 and 10 bpm, for 3.59 and 0.35% of the reporting time respectively, and at no point did these differences exceed 25 bpm. Excellent agreement was found between the HRvid and HRp in a study covering the whole range of skin pigmentation types (Fitzpatrick scales I-VI), using standard room lighting and with moderate subject motion. Although promising, further work should include a larger cohort with multiple subjects per Fitzpatrick class combined with a more rigorous motion and lighting protocol.

Unobtrusive Vital Sign Monitoring in Automotive Environments-A Review

This review provides an overview of unobtrusive monitoring techniques that could be used to monitor some of the human vital signs (i.e., heart activity, breathing activity, temperature and potentially oxygen saturation) in a car seat. It will be shown that many techniques actually measure mechanical displacement, either on the body surface and/or inside the body. However, there are also techniques like capacitive electrocardiogram or bioimpedance that reflect electrical activity or passive electrical properties or thermal properties (infrared thermography). In addition, photopleythysmographic methods depend on optical properties (like scattering and absorption) of biological tissues and-mainly-blood. As all unobtrusive sensing modalities are always fragile and at risk of being contaminated by disturbances (like motion, rapidly changing environmental conditions, triboelectricity), the scope of the paper includes a survey on redundant sensor arrangements. Finally, this review also provides an overview of automotive demonstrators for vital sign monitoring.

Using imaging photoplethysmography for heart rate estimation in non-human primates

For humans and for non-human primates heart rate is a reliable indicator of an individual's current physiological state, with applications ranging from health checks to experimental studies of cognitive and emotional state. In humans, changes in the optical properties of the skin tissue correlated with cardiac cycles (imaging photoplethysmogram, iPPG) allow non-contact estimation of heart rate by its proxy, pulse rate. Yet, there is no established simple and non-invasive technique for pulse rate measurements in awake and behaving animals. Using iPPG, we here demonstrate that pulse rate in rhesus monkeys can be accurately estimated from facial videos. We computed iPPGs from eight color facial videos of four awake head-stabilized rhesus monkeys. Pulse rate estimated from iPPGs was in good agreement with reference data from a contact pulse-oximeter: the error of pulse rate estimation was below 5% of the individual average pulse rate in 83% of the epochs; the error was below 10% for 98% of the epochs. We conclude that iPPG allows non-invasive and non-contact estimation of pulse rate in non-human primates, which is useful for physiological studies and can be used toward welfare-assessment of non-human primates in research.

Improving ROI detection in photoplethysmographic imaging with thermal cameras

Photoplethismographic imaging (PPGi) enables the estimation of heart rate without body contact by analyzing the temporal skin color changes from video recordings. Motion artifacts and atypical facial characteristics cause poor signals and currently limit the applicability of PPGi. We have developed a novel algorithm for locating cheek and forehead region of interests (ROI) with the aim to improve PPGi during challenging situations. The proposed approach is based on the fusion of RGB and far-infrared (FIR) video streams where FIR ROI is used as fall-back when RGB alone fails. We validated and compared the algorithm against the detection based on single sources, using videos from 8 subjects with distinctively different face characteristics. The subject performed three scenarios with incremental motion artifact content (head at rest, intensive head movements, speaking). The results showed that combining the two imaging sources increased the detection rate of cheeks from 75% (RGB) to 92% (RGB+FIR) in the challenging intensive head movement scenario. This work demonstrated that FIR imaging is complementary to simple RGB imaging and when combined, adds robustness to the detection of ROI in PPGi applications.

Remote photoplethysmography with constrained ICA using periodicity and chrominance constraints

BACKGROUND

Remote photoplethysmography (rPPG) has been in the forefront recently for measuring cardiac pulse rates from live or recorded videos. It finds advantages in scenarios requiring remote monitoring, such as medicine and fitness, where contact based monitoring is limiting and cumbersome. The blood volume pulse, defined as the pulsative flow of arterial blood, gives rise to periodic changes in the skin color which are then quantified to estimate a temporal signal. This temporal signal can be analysed using various methods to extract the representative cardiac signal.

METHODS

We present a novel method for measuring rPPG signals using constrained independent component analysis (cICA). We incorporate a priori information into the cICA algorithm to aid in the extraction of the most prominent rPPG signal. This a priori information is implemented using two constraints: first, based on periodicity using autocorrelation, and second, a chrominance-based constraint exploiting the physical characteristics of the skin.

RESULTS AND CONCLUSION

Our method showed improved performances over traditional blind source separation methods like ICA and chrominance based methods with mean absolute errors of 0.62 beats per minute (BPM) and 3.14 BPM for the two datasets in our inhouse video database UBFC-RPPG, and 4.69 BPM for the public MMSE-HR dataset. Its performance was also better in comparison to other state of the art methods in terms of accuracy and robustness. Our UBFC-RPPG database is also made publicly available and is specifically aimed towards testing rPPG measurements.

Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement

Remote camera-based measurement of physiology has great potential for healthcare and affective computing. Recent advances in computer vision and signal processing have enabled photoplethysmography (PPG) measurement using commercially available cameras. However, there remain challenges in recovering accurate noncontact PPG measurements in the presence of rigid head motion. When a subject is moving, their face may be turned away from one camera, be obscured by an object, or move out of the frame resulting in missing observations. As the calculation of pulse rate variability (PRV) requires analysis over a time window of several minutes, the effect of missing observations on such features is deleterious. We present an approach for fusing partial color-channel signals from an array of cameras that enable physiology measurements to be made from moving subjects, even if they leave the frame of one or more cameras, which would not otherwise be possible with only a single camera. We systematically test our method on subjects ( N=25) using a set of six, 5-min tasks (each repeated twice) involving different levels of head motion. This results in validation across 25 h of measurement. We evaluate pulse rate and PRV parameter estimation including statistical, geometric, and frequency-based measures. The median absolute error in pulse rate measurements was 0.57 beats-per-minute (BPM). In all but two tasks with the greatest motion, the median error was within 0.4 BPM of that from a contact PPG device. PRV estimates were significantly improved using our proposed approach compared to an alternative not designed to handle missing values and multiple camera signals; the error was reduced by over 50%. Without our proposed method, errors in pulse rate would be very high, and estimation of PRV parameters would not be feasible due to significant data loss.

Measuring pulse rate variability using long-range, non-contact imaging photoplethysmography

Camera-based measurement of the blood volume pulse via non-contact, imaging photoplethysmography is a very popular approach for measuring pulse rate using a remote imaging sensor. Comparatively less attention has been paid to the usefulness of the method for measuring features of pulse rate variability, and even less focus has been put on the accuracy of any cardiac activity feature that can be achieved at long imager-to-subject distances. In this study, video was recorded from 19 participants, while at rest, at a distance of 25 meters from the imaging sensor. A digital camera was used to record video while cardiovascular measures of both electrical and optical physiological ground truth were recorded. Pulse rate data obtained from the imager using a common blind source separation and periodogram approach were compared to physiological ground truth signals. The quality of the recovered blood volume pulse morphology was sufficient to calculate time-domain measures of pulse rate using inter-pulse interval (IPI) time series. Following this, several features of pulse rate variability were calculated from the IPI time series and compared to those calculated from the corresponding physiological ground truth signals. Use of the time-domain data as compared to the periodogram approach to measure pulse rate reduced the error in the estimate from 1.6 to 0.2 beats per minute. Correlation analysis (r2) between the camera-based measures of pulse rate variability and ECG-derived heart rate variability ranged from 0.779 to 0.973; these results are of comparable outcome to those obtained at imager-to-subject distances of no more than 3 meters. This study demonstrates that pulse rates of less than one beat-per-minute error can be obtained when the recovered blood volume pulse morphology is of adequate quality to resolve systolic onsets for individual cardiac cycles. Further, this approach can yield data of very promising quality for estimating measures of pulse rate variability.

Running wavelet archetype aids the determination of heart rate from the video photoplethysmogram during motion

The extraction of heart rate from a video-based biosignal during motion using a novel wavelet-based ensemble averaging method is described. Running Wavelet Archetyping (RWA) allows for the enhanced extraction of pulse information from the time-frequency representation, from which a video-based heart rate (HRvid) can be derived. This compares favorably to a reference heart rate derived from a pulse oximeter.

Wrist Pulse Rate Monitor Using Self-Injection-Locked Radar Technology

To achieve sensitivity, comfort, and durability in vital sign monitoring, this study explores the use of radar technologies in wearable devices. The study first detected the respiratory rates and heart rates of a subject at a one-meter distance using a self-injection-locked (SIL) radar and a conventional continuous-wave (CW) radar to compare the sensitivity versus power consumption between the two radars. Then, a pulse rate monitor was constructed based on a bistatic SIL radar architecture. This monitor uses an active antenna that is composed of a SIL oscillator (SILO) and a patch antenna. When attached to a band worn on the subject’s wrist, the active antenna can monitor the pulse on the subject’s wrist by modulating the SILO with the associated Doppler signal. Subsequently, the SILO’s output signal is received and demodulated by a remote frequency discriminator to obtain the pulse rate information.

Illumination Variation-Resistant Video-Based Heart Rate Measurement Using Joint Blind Source Separation and Ensemble Empirical Mode Decomposition

Recent studies have demonstrated that heart rate (HR) could be estimated using video data [e.g., exploring human facial regions of interest (ROIs)] under well-controlled conditions. However, in practice, the pulse signals may be contaminated by motions and illumination variations. In this paper, tackling the illumination variation challenge, we propose an illumination-robust framework using joint blind source separation (JBSS) and ensemble empirical mode decomposition (EEMD) to effectively evaluate HR from webcam videos. The framework takes the hypotheses that both facial ROI and background ROI have similar illumination variations. The background ROI is then considered as a noise reference sensor to denoise the facial signals by using the JBSS technique to extract the underlying illumination variation sources. Further, the reconstructed illumination-resisted green channel of the facial ROI is detrended and decomposed into a number of intrinsic mode functions using EEMD to estimate the HR. Experimental results demonstrated that the proposed framework could estimate HR more accurately than the state-of-the-art methods. The Bland-Altman plots showed that it led to better agreement with HR ground truth with the mean bias 1.15 beats/min (bpm), with 95% limits from -15.43 to 17.73 bpm, and the correlation coefficient 0.53. This study provides a promising solution for realistic noncontact and robust HR measurement applications.

Vasomotor assessment by camera-based photoplethysmography

Camera-based photoplethysmography (cbPPG) is a novel technique that allows the contactless acquisition of cardio-respiratory signals. Previous works on cbPPG most often focused on heart rate extraction. This contribution is directed at the assessment of vasomotor activity by means of cameras. In an experimental study, we show that vasodilation and vasoconstriction both lead to significant changes in cbPPG signals. Our findings underline the potential of cbPPG to monitor vasomotor functions in real-life applications.

Video monitoring of oxygen saturation during controlled episodes of acute hypoxia

A method for extracting video photoplethysmographic information from an RGB video stream is tested on data acquired during a porcine model of acute hypoxia. Cardiac pulsatile information was extracted from the acquired signals and processed to determine a continuously reported oxygen saturation (SvidO2). A high degree of correlation was found to exist between the video and a reference from a pulse oximeter. The calculated mean bias and accuracy across all eight desaturation episodes were -0.03% (range: -0.21% to 0.24%) and accuracy 4.90% (range: 3.80% to 6.19%) respectively. The results support the hypothesis that oxygen saturation trending can be evaluated accurately from a video system during acute hypoxia.

A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation

In this paper, we propose a conceptually novel algorithm, namely "Spatial Subspace Rotation" (2SR), that improves the robustness of remote photoplethysmography. Based on the assumption of 1) spatially redundant pixel-sensors of a camera, and 2) a well-defined skin mask, our core idea is to estimate a spatial subspace of skin-pixels and measure its temporal rotation for pulse extraction, which does not require skin-tone or pulse-related priors in contrast to existing algorithms. The proposed algorithm is thoroughly assessed on a benchmark dataset containing 54 videos, which includes challenges of various skin-tones, body-motions in complex illuminance conditions, and pulse-rate recovery after exercise. The experimental results show that given a well-defined skin mask, 2SR outperforms the popular ICA-based approach and two state-of-the-art algorithms (CHROM and PBV). When comparing the pulse frequency spectrum, 2SR improves on average the SNR of ICA by 2.22 dB, CHROM by 1.56 dB, and PBV by 1.95 dB. When comparing the instant pulse-rate, 2SR improves on average the Pearson correlation and precision of ICA by 47% and 65%, CHROM by 22% and 23%, and PBV by 21% and 39%. ANOVA confirms the significant improvement of 2SR in peak-to-peak accuracy. The proposed 2SR algorithm is very simple to use and extend, i.e., the implementation only requires a few lines MATLAB code.