Advancements in noncontact, multiparameter physiological measurements using a webcam

Examination of Potential of Thermopile-Based Contactless Respiratory Gating

To control the spread of coronavirus disease 2019 (COVID-19), it is effective to perform a fast screening of the respiratory rate of the subject at the gate before entering a space to assess the potential risks. In this paper, we examine the potential of a novel yet cost-effective solution, called thermopile-based respiratory gating, to contactlessly screen a subject by measuring their respiratory rate in the scenario with an entrance gate. Based on a customized thermopile array system, we investigate different image and signal processing methods that measure respiratory rate from low-resolution thermal videos, where an automatic region-of-interest selection-based approach obtains a mean absolute error (MAE) of 0.8 breaths per minute. We show the feasibility of thermopile-based respiratory gating and quantify its limitations and boundary conditions in a benchmark (e.g., appearance of face mask, measurement distance and screening time). The technical validation provided by this study is helpful for designing and implementing a respiratory gating solution toward the prevention of the spread of COVID-19 during the pandemic.

Non-Contact Automatic Vital Signs Monitoring of Infants in a Neonatal Intensive Care Unit Based on Neural Networks

Infants with fragile skin are patients who would benefit from non-contact vital sign monitoring due to the avoidance of potentially harmful adhesive electrodes and cables. Non-contact vital signs monitoring has been studied in clinical settings in recent decades. However, studies on infants in the Neonatal Intensive Care Unit (NICU) are still limited. Therefore, we conducted a single-center study to remotely monitor the heart rate (HR) and respiratory rate (RR) of seven infants in NICU using a digital camera. The region of interest (ROI) was automatically selected using a convolutional neural network and signal decomposition was used to minimize the noise artefacts. The experimental results have been validated with the reference data obtained from an ECG monitor. They showed a strong correlation using the Pearson correlation coefficients (PCC) of 0.9864 and 0.9453 for HR and RR, respectively, and a lower error rate with RMSE 2.23 beats/min and 2.69 breaths/min between measured data and reference data. A Bland-Altman analysis of the data also presented a close correlation between measured data and reference data for both HR and RR. Therefore, this technique may be applicable in clinical environments as an economical, non-contact, and easily deployable monitoring system, and it also represents a potential application in home health monitoring.

Non-Contact Heart Rate Estimation via Adaptive RGB/NIR Signal Fusion

We propose a non-contact heart rate (HR) estimation method that is robust to various situations, such as bright, low-light, and varying illumination scenes. We utilize a camera that records red, green, and blue (RGB) and near-infrared (NIR) information to capture the subtle skin color changes induced by the cardiac pulse of a person. The key novelty of our method is the adaptive fusion of RGB and NIR signals for HR estimation based on the analysis of background illumination variations. RGB signals are suitable indicators for HR estimation in bright scenes. Conversely, NIR signals are more reliable than RGB signals in scenes with more complex illumination, as they can be captured independently of the changes in background illumination. By measuring the correlations between the lights reflected from the background and facial regions, we adaptively utilize RGB and NIR observations for HR estimation. The experiments demonstrate the effectiveness of the proposed method.

Non-Invasive Driver Drowsiness Detection System

Drowsiness when in command of a vehicle leads to a decline in cognitive performance that affects driver behavior, potentially causing accidents. Drowsiness-related road accidents lead to severe trauma, economic consequences, impact on others, physical injury and/or even death. Real-time and accurate driver drowsiness detection and warnings systems are necessary schemes to reduce tiredness-related driving accident rates. The research presented here aims at the classification of drowsy and non-drowsy driver states based on respiration rate detection by non-invasive, non-touch, impulsive radio ultra-wideband (IR-UWB) radar. Chest movements of 40 subjects were acquired for 5 m using a lab-placed IR-UWB radar system, and respiration per minute was extracted from the resulting signals. A structured dataset was obtained comprising respiration per minute, age and label (drowsy/non-drowsy). Different machine learning models, namely, Support Vector Machine, Decision Tree, Logistic regression, Gradient Boosting Machine, Extra Tree Classifier and Multilayer Perceptron were trained on the dataset, amongst which the Support Vector Machine shows the best accuracy of 87%. This research provides a ground truth for verification and assessment of UWB to be used effectively for driver drowsiness detection based on respiration.

Effectiveness of consumer-grade contactless vital signs monitors: a systematic review and meta-analysis

The objective of this systematic review and meta-analysis was to analyze the effectiveness of contactless vital sign monitors that utilize a consumer-friendly camera versus medical grade instruments. A multiple database search was conducted from inception to September 2020. Inclusion criteria were as follows: studies that used a consumer-grade camera (smartphone/webcam) to examine contactless vital signs in adults; evaluated the non-contact device against a reference medical device; and used the participants’ face for measurement. Twenty-six studies were included in the review of which 16 were included in Pearson’s correlation and 14 studies were included in the Bland–Altman meta-analysis. Twenty-two studies measured heart rate (HR) (92%), three measured blood pressure (BP) (12%), and respiratory rate (RR) (12%). No study examined blood oxygen saturation (SpO₂). Most studies had a small sample size (≤ 30 participants) and were performed in a laboratory setting. Our meta-analysis found that consumer-grade contactless vital sign monitors were accurate in comparison to a medical device in measuring HR. Current contactless monitors have limitations such as motion, poor lighting, and lack of automatic face tracking. Currently available consumer-friendly contactless monitors measure HR accurately compared to standard medical devices. More studies are needed to assess the accuracy of contactless BP and RR monitors. Implementation of contactless vital sign monitors for clinical use will require validation in a larger population, in a clinical setting, and expanded to encompass other vital signs including BP, RR, and SpO₂.

Development of Nanosensors Based Intelligent Packaging Systems: Food Quality and Medicine

The issue of medication noncompliance has resulted in major risks to public safety and financial loss. The new omnipresent medicine enabled by the Internet of things offers fascinating new possibilities. Additionally, an in-home healthcare station (IHHS), it is necessary to meet the rapidly increasing need for routine nursing and on-site diagnosis and prognosis. This article proposes a universal and preventive strategy to drug management based on intelligent and interactive packaging (I2Pack) and IMedBox. The controlled delamination material (CDM) seals and regulates wireless technologies in novel medicine packaging. As such, wearable biomedical sensors may capture a variety of crucial parameters via wireless communication. On-site treatment and prediction of these critical factors are made possible by high-performance architecture. The user interface is also highlighted to make surgery easier for the elderly, disabled, and patients. Land testing incorporates and validates an approach for prototyping I2Pack and iMedBox. Additionally, sustainability, increased product safety, and quality standards are crucial throughout the life sciences. To achieve these standards, intelligent packaging is also used in the food and pharmaceutical industries. These technologies will continuously monitor the quality of a product and communicate with the user. Data carriers, indications, and sensors are the three most important groups. They are not widely used at the moment, although their potential is well understood. Intelligent packaging should be used in these sectors and the functionality of the systems and the values presented in this analysis.

A Review of Deep Learning-Based Contactless Heart Rate Measurement Methods

The interest in contactless or remote heart rate measurement has been steadily growing in healthcare and sports applications. Contactless methods involve the utilization of a video camera and image processing algorithms. Recently, deep learning methods have been used to improve the performance of conventional contactless methods for heart rate measurement. After providing a review of the related literature, a comparison of the deep learning methods whose codes are publicly available is conducted in this paper. The public domain UBFC dataset is used to compare the performance of these deep learning methods for heart rate measurement. The results obtained show that the deep learning method PhysNet generates the best heart rate measurement outcome among these methods, with a mean absolute error value of 2.57 beats per minute and a mean square error value of 7.56 beats per minute.

Development of a Novel Web Camera-Based Contact-Free Major Depressive Disorder Screening System Using Autonomic Nervous Responses Induced by a Mental Task and Its Clinical Application

Background

To increase the consultation rate of potential major depressive disorder (MDD) patients, we developed a contact-type fingertip photoplethysmography-based MDD screening system. With the outbreak of SARS-CoV-2, we developed an alternative to contact-type fingertip photoplethysmography: a novel web camera-based contact-free MDD screening system (WCF-MSS) for non-contact measurement of autonomic transient responses induced by a mental task.

Methods

The WCF-MSS measures time-series interbeat intervals (IBI) by monitoring color tone changes in the facial region of interest induced by arterial pulsation using a web camera (1920 × 1080 pixels, 30 frames/s). Artifacts caused by body movements and head shakes are reduced. The WCF-MSS evaluates autonomic nervous activation from time-series IBI by calculating LF (0.04-0.15 Hz) components of heart rate variability (HRV) corresponding to sympathetic and parasympathetic nervous activity and HF (0.15-0.4 Hz) components equivalent to parasympathetic activities. The clinical test procedure comprises a pre-rest period (Pre-R; 140 s), mental task period (MT; 100 s), and post-rest period (Post-R; 120 s). The WCF-MSS uses logistic regression analysis to discriminate MDD patients from healthy volunteers via an optimal combination of four explanatory variables determined by a minimum redundancy maximum relevance algorithm: HF during MT (HF _MT ), the percentage change of LF from pre-rest to MT (%ΔLF_(Pre-R⇒MT) ), the percentage change of HF from pre-rest to MT (%ΔHF_(Pre-R⇒MT) ), and the percentage change of HF from MT to post-rest (%ΔHF_{(MT⇒Post-R)} ). To clinically test the WCF-MSS, 26 MDD patients (16 males and 10 females, 20-58 years) were recruited from BESLI Clinic in Tokyo, and 27 healthy volunteers (15 males and 12 females, 18-60 years) were recruited from Tokyo Metropolitan University and RICOH Company, Ltd. Electrocardiography was used to calculate HRV variables as references.

Result

The WCF-MSS achieved 73% sensitivity and 85% specificity on 5-fold cross-validation. IBI correlated significantly with IBI from reference electrocardiography (r = 0.97, p < 0.0001). Logit scores and subjective self-rating depression scale scores correlated significantly (r = 0.43, p < 0.05).

Conclusion

The WCF-MSS seems a promising contact-free MDD screening apparatus. This method enables web camera built-in smartphones to be used as MDD screening systems.

PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography

Remote photoplethysmography (rPPG) is a non-contact technique for measuring cardiac signals from facial videos. High-quality rPPG pulse signals are urgently demanded in many fields, such as health monitoring and emotion recognition. However, most of the existing rPPG methods can only be used to get average heart rate (HR) values due to the limitation of inaccurate pulse signals. In this paper, a new framework based on generative adversarial network, called PulseGAN, is introduced to generate realistic rPPG pulse signals through denoising the chrominance (CHROM) signals. Considering that the cardiac signal is quasi-periodic and has apparent time-frequency characteristics, the error losses defined in time and spectrum domains are both employed with the adversarial loss to enforce the model generating accurate pulse waveforms as its reference. The proposed framework is tested on three public databases. The results show that the PulseGAN framework can effectively improve the waveform quality, thereby enhancing the accuracy of HR, the interbeat interval (IBI) and the related heart rate variability (HRV) features. The proposed method significantly improves the quality of waveforms compared to the input CHROM signals, with the mean absolute error of AVNN (the average of all normal-to-normal intervals) reduced by 41.19%, 40.45%, 41.63%, and the mean absolute error of SDNN (the standard deviation of all NN intervals) reduced by 37.53%, 44.29%, 58.41%, in the cross-database test on the UBFC-RPPG, PURE, and MAHNOB-HCI databases, respectively. This framework can be easily integrated with other existing rPPG methods to further improve the quality of waveforms, thereby obtaining more reliable IBI features and extending the application scope of rPPG techniques.

Contact-Free Pulse Signal Extraction from Human Face Videos: A Review and New Optimized Filtering Approach

In this chapter, we review methods for video-based heart monitoring, from classical signal processing approaches to modern deep learning methods. In addition, we propose a new method for learning an optimal filter that can overcome many of the problems that can affect classical approaches, such as light reflection and subject's movements, at a fraction of the training cost of deep learning approaches. Following the usual procedures for region of interest extraction and tracking, robust skin color estimation and signal pre-processing, we introduce a least-squares error optimal filter, learnt using an established training dataset to estimate the photoplethysmographic (PPG) signal more accurately from the measured color changes over time. This method not only improves the accuracy of heart rate measurement but also resulted in the extraction of a cleaner pulse signal, which could be integrated into many other useful applications such as human biometric recognition or recognition of emotional state. The method was tested on the DEAP dataset and showed improved performance over the best previous classical method on that dataset. The results obtained show that our proposed contact-free heart rate measurement method has significantly improved on existing methods.

Reducing the effects of parallax in camera-based pulse-oximetry

Camera-based pulse-oximetry enables contactless estimation of peripheral oxygen saturation (SpO₂). Because of the lack of readily available and affordable single-optics multi-spectral cameras, custom-made multi-camera setups with different optical filters are currently mostly used. The introduced parallax by these cameras could however jeopardise the SpO₂ algorithm assumptions, especially during subject movement. In this paper we investigate the effect of parallax quantitatively by creating a large dataset consisting of 150 videos with three different parallax settings and with realistic and challenging motion scenarios. We estimate oxygen saturation values with a previously used global frame registration method and with a newly proposed adaptive local registration method to further reduce the parallax-induced image misalignment. We found that the amount of parallax has an important effect on the accuracy of the SpO₂ measurement during movement and that the proposed local image registration reduces the error by more than a factor of 2 for the most common motion scenarios during screening. Extrapolation of the results suggests that the error during the most challenging motion scenario can be reduced to approximately 2 percent when using a parallax-free single-optics camera. This study provides important insights on the possible applications and use cases of remote pulse-oximetry with current affordable and readily available cameras.

Modulation Model of the Photoplethysmography Signal for Vital Sign Extraction

This paper introduces an amplitude and frequency modulation (AM-FM) model to characterize the photoplethysmography (PPG) signal. The model indicates that the PPG signal spectrum contains one dominant frequency component - the heart rate (HR), which is guarded by two weaker frequency components on both sides; the distance from the dominant component to the guard components represents the respiratory rate (RR). Based on this model, an efficient algorithm is proposed to estimate both HR and RR by searching for the dominant frequency component and two guard components. The proposed method is performed in the frequency domain to estimate RR, which is more robust to additive noise than the prior art based on temporal features. Experiments were conducted on two types of PPG signals collected with a contact sensor (an oximeter) and a contactless visible imaging sensor (a color camera), respectively. The PPG signal from the contactless sensor is much noisier than the signal from the contact sensor. The experimental results demonstrate the effectiveness of the proposed algorithm, including under relatively noisy scenarios.

A measurement of illumination variation-resistant noncontact heart rate based on the combination of singular spectrum analysis and sub-band method

BACKGROUND AND OBJECTIVE

The imaging photoplethysmography method is a non-contact and non-invasive measurement method that usually uses surrounding illumination as an illuminant, which can be easily influenced by the surrounding illumination variations. Thus, it has a practical value to develop an efficient method of heart rate measurement that can remove the interference of illumination variations robustly.

METHOD

We propose a novel framework of heart rate measurement that is robust to illumination variations by combining singular spectrum analysis and sub-band method. At first, we extract the blood volume pulse signal by applying the modified sub-band method to the raw facial RGB trace signals. Then the spectra for the interference of illumination variations are extracted from the raw signal obtained from facial regions of interest by using singular spectrum analysis. Finally, we estimate the more robust heart rate through comparison analysis between the spectra of the extracted blood volume pulse signal and the illumination variations.

RESULTS

We compared our method with several state-of-the-art methods through the analysis using the self-collected data and the UBFC-RPPG database. Bland-Altman plots and Pearson correlation coefficients pointed out that the proposed method could measure the heart rate more accurately than the state-of-the-art methods. For the self-collected data and the UBFC-RPPG database, Bland-Altman plots showed that proposed method caused better agreement with 95% limits from -4 bpm to 10 bpm and from -2 bpm to 4 bpm respectively than the other state-of-the-art methods. It also revealed that the highly linear relation was held between the estimated heart rate and ground truth with the correlation coefficients of 0.89 and 0.99, respectively.

CONCLUSION

By extracting illumination variation directly from the facial region of interest rather than from the background region of interest, the proposed method demonstrates that it can overcome the drawbacks of the previous methods due to the illumination variation difference between the background and facial regions of interest. It can be found that the proposed method has a relatively good robustness regardless of whether illumination variation exists or not.

Noncontact Physiological Measurement Using a Camera: A Technical Review and Future Directions

Using a camera as an optical sensor to monitor physiological parameters has garnered considerable research interest in biomedical engineering in recent decades. Researchers have explored the use of a camera for monitoring a variety of physiological waveforms, together with the vital signs carried by these waveforms. Most of the obtained waveforms are related to the human respiratory and cardiovascular systems, and in addition of being indicative of overall health, they can also detect early signs of certain diseases. While using a camera for noncontact physiological signal monitoring offers the advantages of low cost and operational ease, it also has the disadvantages such as vulnerability to motion and lack of burden-free calibration solutions in some use cases. This study presents an overview of the existing camera-based methods that have been reported in recent years. It introduces the physiological principles behind these methods, signal acquisition approaches, various types of acquired signals, data processing algorithms, and application scenarios of these methods. It also discusses the technological gaps between the camera-based methods and traditional medical techniques, which are mostly contact-based. Furthermore, we present the manner in which noncontact physiological signal monitoring use has been extended, particularly over the recent years, to more day-to-day aspects of individuals' lives, so as to go beyond the more conventional use case scenarios. We also report on the development of novel approaches that facilitate easier measurement of less often monitored and recorded physiological signals. These have the potential of ushering a host of new medical and lifestyle applications. We hope this study can provide useful information to the researchers in the noncontact physiological signal measurement community.

Non-Contact HR Monitoring via Smartphone and Webcam During Different Respiratory Maneuvers and Body Movements

As a reliable indicator for individual's healthiness conditions, heart rate (HR) has been widely considered and used. Imaging photoplethysmography (iPPG) is recently highlighted as a promising HR measurement method, due to its non-contact characteristics, by extracting the HR from facial video recordings. In this study, we propose a camera-based HR monitoring technique that estimates HR information from iPPG signals extracted from a video sequence. Videos were recorded using a smartphone or a laptop camera. We adopted the plane-orthogonal-to-skin (POS) method to compute iPPG. The proposed method is evaluated by applying it to extract HR of 9 subjects at rest and during two motion conditions (lateral and frontal) while they were performing several respiratory maneuvers-spontaneous, metronome, and forced. Automatic face detection algorithms were implemented in the proposed method. Our experimental results show that mean values of HR have 0.56% error and 99.4% accuracy when compared to HR calculated from the gold-standard electrocardiography (ECG) reference in diverse conditions of motions and respiratory maneuvers.

Contactless analysis of heart rate variability during cold pressor test using radar interferometry and bidirectional LSTM networks

Contactless measurement of heart rate variability (HRV), which reflects changes of the autonomic nervous system (ANS) and provides crucial information on the health status of a person, would provide great benefits for both patients and doctors during prevention and aftercare. However, gold standard devices to record the HRV, such as the electrocardiograph, have the common disadvantage that they need permanent skin contact with the patient. Being connected to a monitoring device by cable reduces the mobility, comfort, and compliance by patients. Here, we present a contactless approach using a 24 GHz Six-Port-based radar system and an LSTM network for radar heart sound segmentation. The best scores are obtained using a two-layer bidirectional LSTM architecture. To verify the performance of the proposed system not only in a static measurement scenario but also during a dynamic change of HRV parameters, a stimulation of the ANS through a cold pressor test is integrated in the study design. A total of 638 minutes of data is gathered from 25 test subjects and is analysed extensively. High F-scores of over 95% are achieved for heartbeat detection. HRV indices such as HF norm are extracted with relative errors around 5%. Our proposed approach is capable to perform contactless and convenient HRV monitoring and is therefore suitable for long-term recordings in clinical environments and home-care scenarios.

A review of wearable and unobtrusive sensing technologies for chronic disease management

With the rapidly increasing number of patients with chronic disease, numerous recent studies have put great efforts into achieving long-term health monitoring and patient management. Specifically, chronic diseases including cardiovascular disease, chronic respiratory disease and brain disease can threaten patients' health conditions over a long period of time, thus effecting their daily lives. Vital health parameters, such as heart rate, respiratory rate, SpO2 and blood pressure, are closely associated with patients’ conditions. Wearable devices and unobtrusive sensing technologies can detect such parameters in a convenient way and provide timely predictions on health condition deterioration by tracking these biomedical signals and health parameters. In this paper, we review current advancements in wearable devices and unobtrusive sensing technologies that can provides possible tools and technological supports for chronic disease management. Current challenges and future directions of related techniques are addressed accordingly.

HRVCam: robust camera-based measurement of heart rate variability

SIGNIFICANCE

Non-contact, camera-based heart rate variability estimation is desirable in numerous applications, including medical, automotive, and entertainment. Unfortunately, camera-based HRV accuracy and reliability suffer due to two challenges: (a) darker skin tones result in lower SNR and (b) relative motion induces measurement artifacts.

AIM

We propose an algorithm HRVCam that provides sufficient robustness to low SNR and motion-induced artifacts commonly present in imaging photoplethysmography (iPPG) signals.

APPROACH

HRVCam computes camera-based HRV from the instantaneous frequency of the iPPG signal. HRVCam uses automatic adaptive bandwidth filtering along with discrete energy separation to estimate the instantaneous frequency. The parameters of HRVCam use the observed characteristics of HRV and iPPG signals.

RESULTS

We capture a new dataset containing 16 participants with diverse skin tones. We demonstrate that HRVCam reduces the error in camera-based HRV metrics significantly (more than 50% reduction) for videos with dark skin and face motion.

CONCLUSION

HRVCam can be used on top of iPPG estimation algorithms to provide robust HRV measurements making camera-based HRV practical.

Research on the combination of color channels in heart rate measurement based on photoplethysmography imaging

SIGNIFICANCE

The measurement of human vital signs based on photoplethysmography imaging (PPGI) can be severely affected by the interference of various factors in the measurement process; therefore, a lot of complex signal processing techniques are used to remove the influence of the interference.

AIM

We comprehensively analyze several methods for color channel combination in the color spaces currently used in PPGI and determine the combination method that can improve the quality of the pulse signal, which results in a modified plane-orthogonal-to-skin based method (POS).

APPROACH

Based on the analysis of the previous studies, 13 methods for color channel combination in the different color spaces, which can be seen as having potential abilities in measuring vital signs, were compared by employing the average value of signal-to-noise ratio (SNR) and the box-plot in the public databases UBFC-RPPG and PURE. In addition, the pulse signal was extracted through the dual-color space transformation (sRGB → intensity normalized RGB → YCbCr) and fine-tuning on the CbCr plane.

RESULTS

Among the 13 methods for color channel combination, the signal extracted by the Cb+Cr combination in the YCbCr color space includes the most pulse information. Furthermore, the average SNR of the modified POS for all the used databases is improved by 69.3% compared to POS.

CONCLUSIONS

The methods using prior knowledge are not only simple to calculate but can significantly increase the SNR, which will provide a great help in the practical use of vital sign measurements based on PPGI.

An iPPG-Based Device for Pervasive Monitoring of Multi-Dimensional Cardiovascular Hemodynamics

Hemodynamic activities, as an essential measure of physiological and psychological characteristics, can be used for cardiovascular and cerebrovascular disease detection. Photoplethysmography imaging (iPPG) can be applied for such purposes with non-contact advances, however, most cardiovascular hemodynamics of iPPG systems are developed for laboratory research, which limits the application in pervasive healthcare. In this study, a video-based facial iPPG detecting equipment was devised to provide multi-dimensional spatiotemporal hemodynamic pulsations for applications with high portability and self-monitoring requirements. A series of algorithms have also been developed for physiological indices such as heart rate and breath rate extraction, facial region analysis, and visualization of hemodynamic pulsation distribution. Results showed that the new device can provide a reliable measurement of a rich range of cardiovascular hemodynamics. Combined with the advanced computing techniques, the new non-contact iPPG system provides a promising solution for user-friendly pervasive healthcare.

Systematic analysis of video-based pulse measurement from compressed videos

Camera-based physiological measurement enables vital signs to be captured unobtrusively without contact with the body. Remote, or imaging, photoplethysmography involves recovering peripheral blood flow from subtle variations in video pixel intensities. While the pulse signal might be easy to obtain from high quality uncompressed videos, the signal-to-noise ratio drops dramatically with video bitrate. Uncompressed videos incur large file storage and data transfer costs, making analysis, manipulation and sharing challenging. To help address these challenges, we use compression specific supervised models to mitigate the effect of temporal video compression on heart rate estimates. We perform a systematic evaluation of the performance of state-of-the-art algorithms across different levels, and formats, of compression. We demonstrate that networks trained on compressed videos consistently outperform other benchmark methods, both on stationary videos and videos with significant rigid head motions. By training on videos with the same, or higher compression factor than test videos, we achieve improvements in signal-to-noise ratio (SNR) of up to 3 dB and mean absolute error (MAE) of up to 6 beats per minute (BPM).

Multispectral Video Fusion for Non-Contact Monitoring of Respiratory Rate and Apnea

Continuous monitoring of respiratory activity is desirable in many clinical applications to detect respiratory events. Non-contact monitoring of respiration can be achieved with near- and far-infrared spectrum cameras. However, current technologies are not sufficiently robust to be used in clinical applications. For example, they fail to estimate an accurate respiratory rate (RR) during apnea. We present a novel algorithm based on multispectral data fusion that aims at estimating RR also during apnea. The algorithm independently addresses the RR estimation and apnea detection tasks. Respiratory information is extracted from multiple sources and fed into an RR estimator and an apnea detector whose results are fused into a final respiratory activity estimation. We evaluated the system retrospectively using data from 30 healthy adults who performed diverse controlled breathing tasks while lying supine in a dark room and reproduced central and obstructive apneic events. Combining multiple respiratory information from multispectral cameras improved the root mean square error (RMSE) accuracy of the RR estimation from up to 4.64 monospectral data down to 1.60 breaths/min. The median F1 scores for classifying obstructive (0.75 to 0.86) and central apnea (0.75 to 0.93) also improved. Furthermore, the independent consideration of apnea detection led to a more robust system (RMSE of 4.44 vs. 7.96 breaths/min). Our findings may represent a step towards the use of cameras for vital sign monitoring in medical applications.

Advancements in Methods and Camera-Based Sensors for the Quantification of Respiration

Assessment of respiratory function allows early detection of potential disorders in the respiratory system and provides useful information for medical management. There is a wide range of applications for breathing assessment, from measurement systems in a clinical environment to applications involving athletes. Many studies on pulmonary function testing systems and breath monitoring have been conducted over the past few decades, and their results have the potential to broadly impact clinical practice. However, most of these works require physical contact with the patient to produce accurate and reliable measures of the respiratory function. There is still a significant shortcoming of non-contact measuring systems in their ability to fit into the clinical environment. The purpose of this paper is to provide a review of the current advances and systems in respiratory function assessment, particularly camera-based systems. A classification of the applicable research works is presented according to their techniques and recorded/quantified respiration parameters. In addition, the current solutions are discussed with regards to their direct applicability in different settings, such as clinical or home settings, highlighting their specific strengths and limitations in the different environments.

Non-Contact Heart Rate Detection When Face Information Is Missing during Online Learning

Research shows that physiological signals can provide objective data support for the analysis of human emotions. At present, non-contact heart rate data have been employed in the research of medicine, intelligent transportation, smart education, etc. However, it is hard to detect heart rate data using non-contact traditional methods during head rotation, especially when face information is missing in scenarios such as online teaching/learning. Traditional remote photoplethysmography (rPPG) methods require a static, full frontal face within a fixed distance for heart rate detection. These strict requirements make it impractical to measure heart rate data in real-world scenarios, as a lot of videos only partially record the subjects' face information, such as profile, too small distance, and wearing a mask. The current algorithm aims to solve the problem of head deflections between 30 degrees and 45 degrees by employing a symmetry substitution method, which can replace the undetected region of interest (ROI) with the detectable one. When face information is partially missing, our algorithm uses face-eye location to determine ROI. The results show that the method in this paper can solve certain practical problems related to heart rate detection, with a root mean square error (RMSE) under 7.64 bpm.

Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation

Remote photo-plethysmography (rPPG) uses a camera to estimate a person’s heart rate (HR). Similar to how heart rate can provide useful information about a person’s vital signs, insights about the underlying physio/psychological conditions can be obtained from heart rate variability (HRV). HRV is a measure of the fine fluctuations in the intervals between heart beats. However, this measure requires temporally locating heart beats with a high degree of precision. We introduce a refined and efficient real-time rPPG pipeline with novel filtering and motion suppression that not only estimates heart rates, but also extracts the pulse waveform to time heart beats and measure heart rate variability. This unsupervised method requires no rPPG specific training and is able to operate in real-time. We also introduce a new multi-modal video dataset, VicarPPG 2, specifically designed to evaluate rPPG algorithms on HR and HRV estimation. We validate and study our method under various conditions on a comprehensive range of public and self-recorded datasets, showing state-of-the-art results and providing useful insights into some unique aspects. Lastly, we make available CleanerPPG, a collection of human-verified ground truth peak/heart-beat annotations for existing rPPG datasets. These verified annotations should make future evaluations and benchmarking of rPPG algorithms more accurate, standardized and fair.

Detecting Pulse Wave From Unstable Facial Videos Recorded From Consumer-Level Cameras: A Disturbance-Adaptive Orthogonal Matching Pursuit

OBJECTIVE

Modern consumer-level cameras can detect subtle changes in human facial skin color due to varying blood flow; they are beginning to be used as noncontact devices to detect pulse waves. Little, however, do we know about their capacity to perform pulse wave detection when the recorded faces are unstable.

METHODS

Here, we propose a novel method that can extract pulse waves from videos with drastic facial unsteadiness such as head twists and alternating expressions. The method first uses chrominance characteristics in multiple facial sub-regions to construct a raw pulse matrix. Subsequently, it employs a disturbance-adaptive orthogonal matching pursuit (DAOMP) algorithm to recover the underlying pulse matrix corrupted by facial unsteadiness.

RESULTS

To evaluate the efficacy of the method, we perform analyses on two datasets including 268 samples from 67 testing subjects. The results demonstrate that the proposed method outperforms state-of-the-art algorithms, especially in the terrain where drastic facial unsteadiness is present.

CONCLUSION

The proposed framework shows promise to achieve videos-based noncontact pulse wave detection from both steady and unsteady faces recorded by consumer-level cameras.

SIGNIFICANCE

By employing the proposed method, disturbance robustness in noncontact pulse wave detection can be significantly improved.

Comparison of Visible and Infrared Video Plethysmography Captured from Different Regions of the Human Face

Recently, video plethysmography (VPG) - a heart rate estimation technique using a video camera - has gained significant attention. Most studies of VPG have used a visible RGB camera; only a limited number of studies investigating near-infrared light (wavelength 750-2500 nm), which can be used even in a dark environment, have been performed. The purpose of this study was to investigate the differences between VPG data collected using visible light (VPG_VIS) or near-infrared light (VPG_NIR) from four facial areas (forehead, right cheek, left cheek, and nose). An experiment was conducted to obtain both VPG_VIS and VPG_NIR simultaneously by alternately irradiating the face with NIR and VIS lights. Experimental results showed that the root mean squared error of heart rate estimated using VPG_NIR was 1 bpm higher than that of VPG_VIS. However, contrary to our expectations, the power of the heartbeat-related component included in VPG_NIR was not reduced despite the absorbance of hemoglobin in the NIR light range being 1/100 of that in the VIS light range. This result supports the hypothesis that a main factor in the generation of VPG waves was change in the optical properties caused by blood vessels compressing the subcutaneous tissue and the venous bed. Additionally, the accuracy of the heart rate estimation using VPG tended to be high when the nose was set as the ROI. This result was likely associated with the anatomical structure of the nose.

Automated video-based heart rate tracking for the anesthetized and behaving monkey

Heart rate (HR) is extremely valuable in the study of complex behaviours and their physiological correlates in non-human primates. However, collecting this information is often challenging, involving either invasive implants or tedious behavioural training. In the present study, we implement a Eulerian video magnification (EVM) heart tracking method in the macaque monkey combined with wavelet transform. This is based on a measure of image to image fluctuations in skin reflectance due to changes in blood influx. We show a strong temporal coherence and amplitude match between EVM-based heart tracking and ground truth ECG, from both color (RGB) and infrared (IR) videos, in anesthetized macaques, to a level comparable to what can be achieved in humans. We further show that this method allows to identify consistent HR changes following the presentation of conspecific emotional voices or faces. EVM is used to extract HR in humans but has never been applied to non-human primates. Video photoplethysmography allows to extract awake macaques HR from RGB videos. In contrast, our method allows to extract awake macaques HR from both RGB and IR videos and is particularly resilient to the head motion that can be observed in awake behaving monkeys. Overall, we believe that this method can be generalized as a tool to track HR of the awake behaving monkey, for ethological, behavioural, neuroscience or welfare purposes.

PPG3D: Does 3D head tracking improve camera-based PPG estimation?

Over the last few years, camera-based estimation of vital signs referred to as imaging photoplethysmography (iPPG) has garnered significant attention due to the relative simplicity, ease, unobtrusiveness and flexibility offered by such measurements. It is expected that iPPG may be integrated into a host of emerging applications in areas as diverse as autonomous cars, neonatal monitoring, and telemedicine. In spite of this potential, the primary challenge of non-contact camera-based measurements is the relative motion between the camera and the subjects. Current techniques employ 2D feature tracking to reduce the effect of subject and camera motion but they are limited to handling translational and in-plane motion. In this paper, we study, for the first-time, the utility of 3D face tracking to allow iPPG to retain robust performance even in presence of out-of-plane and large relative motions. We use a RGB-D camera to obtain 3D information from the subjects and use the spatial and depth information to fit a 3D face model and track the model over the video frames. This allows us to estimate correspondence over the entire video with pixel-level accuracy, even in the presence of out-of-plane or large motions. We then estimate iPPG from the warped video data that ensures per-pixel correspondence over the entire window-length used for estimation. Our experiments demonstrate improvement in robustness when head motion is large.

Robot Assisted Instantaneous Heart Rate Estimator using Camera based Remote Photoplethysmograpy via Plane-Orthogonal-to-Skin and Finite State Machine

In this paper, we have presented Turtulebot-assisted instantaneous heart rate (HR) estimator using camera based remote photoplethysmography. We used a Turtlebot with a camera to record human face. For the face detection, we used Haar Cascade algorithm. To increase the accuracy of the HR estimation, we combined a plane-orthogonal-to-skin (POS) model with finite state machine (FSM) framework. By combining POS and FSM framework, we achieved 1.08 bpm of MAE, which is the lowest error comparing to the state-of-art methods.

Remote Heart Rate Estimation Based on 3D Facial Landmarks

In this paper, we propose a novel video-based remote heart rate (HR) estimation method based on 3D facial landmarks. The key contributions in our method are twofold: (i) We introduce 3D facial landmarks detection to the video-based HR estimation and (ii) we propose a novel face patch visibility check manner based on the face patch normal in the 3D space. We experimentally demonstrate that, compared with baseline methods using 2D facial landmarks, our proposed method using 3D facial landmarks improves the robustness of HR estimation to head rotations and partial face occlusion. We also demonstrate that our visibility check is effective for selecting sufficiently visible face patches, contributing to the improvement of HR estimation accuracy.

A Review on Biomedical MIMO Radars for Vital Sign Detection and Human Localization

This paper reports a thorough overview on the last developments concerning the vital sign detection and the human localization employing the multiple-input-multiple-output (MIMO) technology. The wireless motion and vital sign detection represents an outstanding research area aimed at monitoring the health conditions of human subjects and at detecting their presence in different environments with minimal concern. MIMO radars exhibit several interesting advantages over conventional single-input-single-output architectures mainly related to their angle detection capabilities and enhanced signal-to-noise ratio. This paper describes the main features and details the operating principles of MIMO technology. Thereafter, it summarizes the state-of-the-art of the available solutions with the purpose of fueling the research activities on this hot topic.

Non-contact imaging of peripheral hemodynamics during cognitive and psychological stressors

Peripheral hemodynamics, measured via the blood volume pulse and vasomotion, provide a valuable way of monitoring physiological state. Camera imaging-based systems can be used to measure these peripheral signals without contact with the body, at distances of multiple meters. While researchers have paid attention to non-contact imaging photoplethysmography, the study of peripheral hemodynamics and the effect of autonomic nervous system activity on these signals has received less attention. Using a method, based on a tissue-like model of the skin, we extract melanin [Formula: see text] and hemoglobin [Formula: see text] concentrations from videos of the hand and face and show that significant decreases in peripheral pulse signal power (by 36% ± 29%) and vasomotion signal power (by 50% ± 26%) occur during periods of cognitive and psychological stress. Via three experiments we show that similar results are achieved across different stimuli and regions of skin (face and hand). While changes in peripheral pulse and vasomotion power were significant the changes in pulse rate variability were less consistent across subjects and tasks.

MOMBAT: Heart rate monitoring from face video using pulse modeling and Bayesian tracking

A non-invasive yet inexpensive method for heart rate (HR) monitoring is of great importance in many real-world applications including healthcare, psychology understanding, affective computing and biometrics. Face videos are currently utilized for such HR monitoring, but unfortunately this can lead to errors due to the noise introduced by facial expressions, out-of-plane movements, camera parameters (like focus change) and environmental factors. We alleviate these issues by proposing a novel face video based HR monitoring method MOMBAT, that is, MOnitoring using Modeling and BAyesian Tracking. We utilize out-of-plane face movements to define a novel quality estimation mechanism. Subsequently, we introduce a Fourier basis based modeling to reconstruct the cardiovascular pulse signal at the locations containing the poor quality, that is, the locations affected by out-of-plane face movements. Furthermore, we design a Bayesian decision theory based HR tracking mechanism to rectify the spurious HR estimates. Experimental results reveal that our proposed method, MOMBAT outperforms state-of-the-art HR monitoring methods and performs HR monitoring with an average absolute error of 1.329 beats per minute and the Pearson correlation between estimated and actual heart rate is 0.9746. Moreover, it demonstrates that HR monitoring is significantly improved by incorporating the pulse modeling and HR tracking.

Identification and Tracking of Physiological Parameters from Skin using Video Photoplethysmography

In recent years, there has been a growing interest in video Photoplethysmography (vPPG), a technique able to estimate cardiovascular parameters from video recordings of the skin. Despite the growing interest in vPPG technology, there are still problems in extracting the correct waveform of blood volume pulse, mainly due to real world artifacts, such as changes in light condition and movement artifacts. Another important issue is the correct definition of skin against background. Therefore, we propose an algorithm of skin detection that is able to recognize skin pixels solid to variations of luminosity. We recorded the signals of interest during an experimental protocol designed to provide thermal stimulation and observe the resulting Autonomic Nervous System changes. Experimental data were gathered from 10 young healthy subjects (age: 21±2 years). Video recordings are processed using a band-pass filter and then an automatic algorithm of peak detection is applied to detect the pulse wave peaks, then used to estimate heart rate variability (HRV). The efficiency and stability of the algorithm are compared against finger-PPG waveforms. Preliminary results show an overall statistical agreement between time and frequency domain indexes. However, further efforts are required to improve the estimation of frequency components, particularly during rest.

Photoplethysmography imaging:camera performance evaluation by means of an optoelectronic skin perfusion phantom

OBJECTIVE

Photoplethysmography imaging (PPGI) is a promising contactless camera-based method of non-invasive cardiovascular diagnostics. To achieve the best results, it is important to choose the most suitable camera for a specific application. The settings of the camera influence the quality of the detected signal.

APPROACH

The standard (European Machine Vision Association 2016 EMVA Standard 1288-Standard for Characterization of Image Sensors and Cameras pp 1-39 (available at: https://www.emva.org/wp-content/uploads/EMVA1288-3.1a.pdf)) for evaluating the imaging performance of machine vision cameras (MVC) helps at the initial decision of the sensor, but the camera should always be tested in terms of usability for a specific application. So far, PPGI lacks a standardized measurement scenario for evaluating the performance of individual cameras. For this, we realized a controllable optoelectronic phantom with artificial silicone skin allowing reproducible tests of cameras to verify their suitability for PPGI. The entire system is housed in a light-tight box. We tested an MVC, a digital single-lens reflex camera (DSLR) camera and a webcam. Each camera varies in used technology and price.

MAIN RESULTS

We simulated real PPGI measurement conditions simulating the ratio of pulse (AC) and non-pulse (DC) component of the photoplethysmographic signal and achieved AC/DC ratios of 0.5 % on average. An additional OLED panel ensures proper DC providing reproducible measurement conditions. We evaluated the signal morphological features, amplitude spectrum, signal-to-noise ratio (SNR) and spatially dependent changes of simulated subcutaneous perfusion. Here, the MVC proved to be the most suitable device. A DSLR is also suitable for PPGI, but a larger smoothing kernel is required to obtain a perfusion map. The webcam, as the weakest contender, proved to be very susceptible to any inhomogeneous illumination of the examined artificial skin surface. However, it is still able to detect cardiac rhythm.

SIGNIFICANCE

The result of our work is an optoelectronic phantom for reproducible testing of PPGI camera performance in terms of signal quality and measurement equipment costs.

Continuous-Spectrum Infrared Illuminator for Camera-PPG in Darkness

Many camera-based remote photoplethysmography (PPG) applications require sensing in near infrared (NIR). The performance of PPG systems benefits from multi-wavelength processing. The illumination source in such system is explored in this paper. We demonstrate that multiple narrow-band LEDs have inferior color homogeneity compared to broadband light sources. Therefore, we consider the broadband option based on phosphor material excited by LEDs. A first prototype was realized and its details are discussed. It was tested within a remote-PPG monitoring scenario in darkness and the full system demonstrates robust pulse-rate measurement. Given its accuracy in pulse rate extraction, the proposed illumination principle is considered a valuable asset for large-scale NIR-PPG applications as it enables multi-wavelength processing, lightweight set-ups with relatively low-power infrared light sources.

StressFoot: Uncovering the Potential of the Foot for Acute Stress Sensing in Sitting Posture

Stress is a naturally occurring psychological response and identifiable by several body signs. We propose a novel way to discriminate acute stress and relaxation, using movement and posture characteristics of the foot. Based on data collected from 23 participants performing tasks that induced stress and relaxation, we developed several machine learning models to construct the validity of our method. We tested our models in another study with 11 additional participants. The results demonstrated replicability with an overall accuracy of 87%. To also demonstrate external validity, we conducted a field study with 10 participants, performing their usual everyday office tasks over a working day. The results showed substantial robustness. We describe ten significant features in detail to enable an easy replication of our models.

Video-Based Pulse Rate Variability Measurement Using Periodic Variance Maximization and Adaptive Two-Window Peak Detection

Many previous studies have shown that the remote photoplethysmography (rPPG) can measure the Heart Rate (HR) signal with very high accuracy. The remote measurement of the Pulse Rate Variability (PRV) signal is also possible, but this is much more complicated because it is then necessary to detect the peaks on the temporal rPPG signal, which is usually quite noisy and has a lower temporal resolution than PPG signals obtained by contact equipment. Since the PRV signal is vital for various applications such as remote recognition of stress and emotion, the improvement of PRV measurement by rPPG is a critical task. Contact based PRV measurement has already been investigated, but the research on remotely measured PRV is very limited. In this paper, we propose to use the Periodic Variance Maximization (PVM) method to extract the rPPG signal and event-related Two-Window algorithm to improve the peak detection for PRV measurement. We have made several contributions. Firstly, we show that the newly proposed PVM method and Two-Window algorithm can be used for PRV measurement in the non-contact scenario. Secondly, we propose a method to adaptively determine the parameters of the Two-Window method. Thirdly, we compare the algorithm with other attempts for improving the non-contact PRV measurement such as the Slope Sum Function (SSF) method and the Local Maximum method. We calculated several features and compared the accuracy based on the ground truth provided by contact equipment. Our experiments showed that this algorithm performed the best of all the algorithms.

Infection Screening System Using Thermography and CCD Camera with Good Stability and Swiftness for Non-contact Vital-Signs Measurement by Feature Matching and MUSIC Algorithm

Screening systems for infectious diseases based on fever have been implemented at international airports to prevent the spread of infection for over a decade. Currently, only Infrared Thermography (IRT) is used for screening and measuring facial skin temperature, which is one of clinical indicators of potential infection. Aiming at higher accuracy in screening, our group adopted heart rate (HR) and respiration rate (RR) for the first time as the new screening parameters. In our previous study, we proposed a screening system based on dual image sensors, which include IRT and a charged-coupled devices (CCD) camera. The sensors can measure three vital signs simultaneously, namely HR, RR, and facial skin temperature. For the measurement of RR in this system, stability and swiftness must be applied for application in airports. In this study, we introduce feature matching and multiple signal classification (MUSIC) algorithm in this system. Feature matching between thermal images and RGB images captured by a CCD camera and IRT, respectively, is used to detect the nose and mouth in IRT, which helps extract respiration signals corresponding to airflow from breathing. In addition, the MUSIC algorithm improves the accuracy of RR frequency estimations in limited time respiration signal and achieves swiftness. The proposed method improves stability by simultaneously detecting the nose and mouth in thermal images, and enhances the accuracy of estimated RR using the MUSIC algorithm. By using this system, we evaluate the accuracy of the estimated vital signs. The performance of this screening system was evaluated using data obtained from 12 influenza patients and 13 healthy subjects at a clinical facility in Fukushima, Japan.