Effects of Video Encoding on Camera-Based Heart Rate Estimation

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

Face2PPG: An Unsupervised Pipeline for Blood Volume Pulse Extraction From Faces

Photoplethysmography (PPG) signals have become a key technology in many fields, such as medicine, well-being, or sports. Our work proposes a set of pipelines to extract remote PPG signals (rPPG) from the face robustly, reliably, and configurably. We identify and evaluate the possible choices in the critical steps of unsupervised rPPG methodologies. We assess a state-of-the-art processing pipeline in six different datasets, incorporating important corrections in the methodology that ensure reproducible and fair comparisons. In addition, we extend the pipeline by proposing three novel ideas; 1) a new method to stabilize the detected face based on a rigid mesh normalization; 2) a new method to dynamically select the different regions in the face that provide the best raw signals, and 3) a new RGB to rPPG transformation method, called Orthogonal Matrix Image Transformation (OMIT) based on QR decomposition, that increases robustness against compression artifacts. We show that all three changes introduce noticeable improvements in retrieving rPPG signals from faces, obtaining state-of-the-art results compared with unsupervised, non-learning-based methodologies and, in some databases, very close to supervised, learning-based methods. We perform a comparative study to quantify the contribution of each proposed idea. In addition, we depict a series of observations that could help in future implementations.

For Heart Rate Assessments from Drone Footage in Disaster Scenarios

The ability to use drones to obtain important vital signs could be very valuable for emergency personnel during mass-casualty incidents. The rapid and robust remote assessment of heart rates could serve as a life-saving decision aid for first-responders. With the flight sensor data of a specialized drone, a pipeline was developed to achieve a robust, non-contact assessment of heart rates through remote photoplethysmography (rPPG). This robust assessment was achieved through adaptive face-aware exposure and comprehensive de-noising of a large number of predicted noise sources. In addition, we performed a proof-of-concept study that involved 18 stationary subjects with clean skin and 36 recordings of their vital signs, using the developed pipeline in outdoor conditions. In this study, we could achieve a single-value heart-rate assessment with an overall root-mean-squared error of 14.3 beats-per-minute, demonstrating the basic feasibility of our approach. However, further research is needed to verify the applicability of our approach in actual disaster situations, where remote photoplethysmography readings could be impacted by other factors, such as blood, dirt, and body positioning.

Unobtrusive Heart Rate Monitoring using Near-Infrared Imaging During Driving

In-vehicle health monitoring allows for continuous vital sign measurement in everyday life. Eventually, this could lead to early detection of cardiovascular diseases. In this work, we propose non-contact heart rate (HR) monitoring utilizing near-infrared (NIR) camera technology. Ten healthy volunteers are monitored in a realistic driving simulator during resting (5 min) and driving (10 min). We synchronously acquire videos using an out-of-the-shelf, low-cost NIR camera and 3-lead electrocardiography (ECG) serves as ground truth. The MediaPipe face detector delivers the region of interest (ROI) and we determine the HR from the peak with maximum amplitude within the power spectrum of skin color changes. We compare video-based with ECG-based HR, resulting in a mean absolute error (MAE) of 7.8 bpm and 13.0 bpm in resting and driving condition, respectively. As we apply only a simple signal processing pipeline without sophisticated filtering, we conclude that NIR camera-based HR measurements enables unobtrusive and non-contact monitoring to a certain extent, but artifacts from subject movement pose a challenge. If these issues can be addressed, continuous vital sign measurement in everyday life could become reality.

Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review

In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.

Adaptive Gaussian Mixture Model Driven Level Set Segmentation for Remote Pulse Rate Detection

This paper presents an approach for pulse rate extraction from videos. The core of the presented approach is a novel method to segment and track a suitable region of interest (ROI). The proposed method combines level sets with subject-individual Gaussian Mixture Models to yield a time varying ROI. The ROI builds up from multiple homogeneous skin areas under constraints regarding the area and contour length of the ROI. Together with state of the art signal processing methods our approach yields an Mean Average Error (MAE) of 2.3 bpm, 1.4 bpm and 2.7 bpm on own data, the PURE database and the UBFC-rPPG database, respectively. Therewith, our method performs equal or better compared to widely used approaches (e.g. the KLT tracker instead of the proposed image processing yields an MAE of 2.6 bpm, 2.6 bpm and 4.4 bpm). Such results and the 2nd place with a MAE of 7.92 bpm in the 1st Challenge on Remote Physiological Signal Sensing prove the applicability of the proposed method. The taken approach, however, bears further potential for optimization in the context of photoplethysmography imaging and should be transferable to other segmentation tasks as well.

A review of wearable and unobtrusive sensing technologies for chronic disease management

With the rapidly increasing number of patients with chronic disease, numerous recent studies have put great efforts into achieving long-term health monitoring and patient management. Specifically, chronic diseases including cardiovascular disease, chronic respiratory disease and brain disease can threaten patients' health conditions over a long period of time, thus effecting their daily lives. Vital health parameters, such as heart rate, respiratory rate, SpO2 and blood pressure, are closely associated with patients’ conditions. Wearable devices and unobtrusive sensing technologies can detect such parameters in a convenient way and provide timely predictions on health condition deterioration by tracking these biomedical signals and health parameters. In this paper, we review current advancements in wearable devices and unobtrusive sensing technologies that can provides possible tools and technological supports for chronic disease management. Current challenges and future directions of related techniques are addressed accordingly.

Systematic analysis of video-based pulse measurement from compressed videos

Camera-based physiological measurement enables vital signs to be captured unobtrusively without contact with the body. Remote, or imaging, photoplethysmography involves recovering peripheral blood flow from subtle variations in video pixel intensities. While the pulse signal might be easy to obtain from high quality uncompressed videos, the signal-to-noise ratio drops dramatically with video bitrate. Uncompressed videos incur large file storage and data transfer costs, making analysis, manipulation and sharing challenging. To help address these challenges, we use compression specific supervised models to mitigate the effect of temporal video compression on heart rate estimates. We perform a systematic evaluation of the performance of state-of-the-art algorithms across different levels, and formats, of compression. We demonstrate that networks trained on compressed videos consistently outperform other benchmark methods, both on stationary videos and videos with significant rigid head motions. By training on videos with the same, or higher compression factor than test videos, we achieve improvements in signal-to-noise ratio (SNR) of up to 3 dB and mean absolute error (MAE) of up to 6 beats per minute (BPM).

MOMBAT: Heart rate monitoring from face video using pulse modeling and Bayesian tracking

A non-invasive yet inexpensive method for heart rate (HR) monitoring is of great importance in many real-world applications including healthcare, psychology understanding, affective computing and biometrics. Face videos are currently utilized for such HR monitoring, but unfortunately this can lead to errors due to the noise introduced by facial expressions, out-of-plane movements, camera parameters (like focus change) and environmental factors. We alleviate these issues by proposing a novel face video based HR monitoring method MOMBAT, that is, MOnitoring using Modeling and BAyesian Tracking. We utilize out-of-plane face movements to define a novel quality estimation mechanism. Subsequently, we introduce a Fourier basis based modeling to reconstruct the cardiovascular pulse signal at the locations containing the poor quality, that is, the locations affected by out-of-plane face movements. Furthermore, we design a Bayesian decision theory based HR tracking mechanism to rectify the spurious HR estimates. Experimental results reveal that our proposed method, MOMBAT outperforms state-of-the-art HR monitoring methods and performs HR monitoring with an average absolute error of 1.329 beats per minute and the Pearson correlation between estimated and actual heart rate is 0.9746. Moreover, it demonstrates that HR monitoring is significantly improved by incorporating the pulse modeling and HR tracking.

Impact of digital video analytics on accuracy of chemobehavioural phenotyping in aquatic toxicology

Chemobehavioural phenotypic analysis using small aquatic model organisms is becoming an important toolbox in aquatic ecotoxicology and neuroactive drug discovery. The analysis of the organisms’ behavior is usually performed by combining digital video recording with animal tracking software. This software detects the organisms in the video frames, and reconstructs their movement trajectory using image processing algorithms. In this work we investigated the impact of video file characteristics, video optimization techniques and differences in animal tracking algorithms on the accuracy of quantitative neurobehavioural endpoints. We employed larval stages of a free-swimming euryhaline crustacean Artemia franciscana,commonly used for marine ecotoxicity testing, as a proxy modelto assess the effects of video analytics on quantitative behavioural parameters. We evaluated parameters such as data processing speed, tracking precision, capability to perform high-throughput batch processing of video files. Using a model toxicant the software algorithms were also finally benchmarked against one another. Our data indicates that variability in video file parameters; such as resolution, frame rate, file containers types, codecs and compression levels, can be a source of experimental biases in behavioural analysis. Similarly, the variability in data outputs between different tracking algorithms should be taken into account when designing standardized behavioral experiments and conducting chemobehavioural phenotyping.