A Pulse Rate Estimation Algorithm Using PPG and Smartphone Camera

Your smartphone could act as a pulse-oximeter and as a single-lead ECG

In the post-covid19 era, every new wave of the pandemic causes an increased concern/interest among the masses to learn more about their state of well-being. Therefore, it is the need of the hour to come up with ubiquitous, low-cost, non-invasive tools for rapid and continuous monitoring of body vitals that reflect the status of one's overall health. In this backdrop, this work proposes a deep learning approach to turn a smartphone-the popular hand-held personal gadget-into a diagnostic tool to measure/monitor the three most important body vitals, i.e., pulse rate (PR), blood oxygen saturation level (aka SpO2), and respiratory rate (RR). Furthermore, we propose another method that could extract a single-lead electrocardiograph (ECG) of the subject. The proposed methods include the following core steps: subject records a small video of his/her fingertip by placing his/her finger on the rear camera of the smartphone, and the recorded video is pre-processed to extract the filtered and/or detrended video-photoplethysmography (vPPG) signal, which is then fed to custom-built convolutional neural networks (CNN), which eventually spit-out the vitals (PR, SpO2, and RR) as well as a single-lead ECG of the subject. To be precise, the contribution of this paper is twofold: (1) estimation of the three body vitals (PR, SpO2, RR) from the vPPG data using custom-built CNNs, vision transformer, and most importantly by CLIP model (a popular image-caption-generator model); (2) a novel discrete cosine transform+feedforward neural network-based method that translates the recorded video-PPG signal to a single-lead ECG signal. The significance of this work is twofold: (i) it allows rapid self-testing of body vitals (e.g., self-monitoring for covid19 symptoms), (ii) it enables rapid self-acquisition of a single-lead ECG, and thus allows early detection of atrial fibrillation (abormal heart beat or arrhythmia), which in turn could enable early intervention in response to a range of cardiovascular diseases, and could help save many precious lives. Our work could help reduce the burden on healthcare facilities and could lead to reduction in health insurance costs.

Brno University of Technology Smartphone PPG Database (BUT PPG): Annotated Dataset for PPG Quality Assessment and Heart Rate Estimation

To the best of our knowledge, there is no annotated database of PPG signals recorded by smartphone publicly available. This article introduces Brno University of Technology Smartphone PPG Database (BUT PPG) which is an original database created by the cardiology team at the Department of Biomedical Engineering, Brno University of Technology, for the purpose of evaluating photoplethysmographic (PPG) signal quality and estimation of heart rate (HR). The data comprises 48 10-second recordings of PPGs and associated electrocardiographic (ECG) signals used for determination of reference HR. The data were collected from 12 subjects (6 female, 6 male) aged between 21 and 61. PPG data were collected by smartphone Xiaomi Mi9 with sampling frequency of 30 Hz. Reference ECG signals were recorded using a mobile ECG recorder (Bittium Faros 360) with a sampling frequency of 1,000 Hz. Each PPG signal includes annotation of quality created manually by biomedical experts and reference HR. PPG signal quality is indicated binary: 1 indicates good quality for HR estimation, 0 indicates signals where HR cannot be detected reliably, and thus, these signals are unsuitable for further analysis. As the only available database containing PPG signals recorded by smartphone, BUT PPG is a unique tool for the development of smart, user-friendly, cheap, on-the-spot, self-home-monitoring of heart rate with the potential of widespread using.

Ambulatory monitoring promises equitable personalized healthcare delivery in underrepresented patients

The pandemic has brought to everybody's attention the apparent need of remote monitoring, highlighting hitherto unseen challenges in healthcare. Today, mobile monitoring and real-time data collection, processing and decision-making, can drastically improve the cardiorespiratory-haemodynamic health diagnosis and care, not only in the rural communities, but urban ones with limited healthcare access as well. Disparities in socioeconomic status and geographic variances resulting in regional inequity in access to healthcare delivery, and significant differences in mortality rates between rural and urban communities have been a growing concern. Evolution of wireless devices and smartphones has initiated a new era in medicine. Mobile health technologies have a promising role in equitable delivery of personalized medicine and are becoming essential components in the delivery of healthcare to patients with limited access to in-hospital services. Yet, the utility of portable health monitoring devices has been suboptimal due to the lack of user-friendly and computationally efficient physiological data collection and analysis platforms. We present a comprehensive review of the current cardiac, pulmonary, and haemodynamic telemonitoring technologies. We also propose a novel low-cost smartphone-based system capable of providing complete cardiorespiratory assessment using a single platform for arrhythmia prediction along with detection of underlying ischaemia and sleep apnoea; we believe this system holds significant potential in aiding the diagnosis and treatment of cardiorespiratory diseases, particularly in underserved populations.

Validation of Smartphone Based Heart Rate Tracking for Remote Treatment of Panic Attacks

Panic attacks are an impairing mental health problem that affects 11% of adults every year [1]. Those who suffer from panic attacks often do not seek psychological treatment, citing the inability to receive care during their attacks as a contributing factor. A digital medicine solution which provides an accessible, real-time mobile health (mHealth) biofeedback intervention for panic attacks may address this problem. Critical to this approach are methods for capturing physiological arousal during an attack. Herein, we validate an algorithm for capturing physiological arousal using smartphone video of the fingertip. Results demonstrate that the algorithm is able to estimate heart rates that are highly correlated with ECG-derived values (r > 0.99), effectively reject low-quality data often captured outside of controlled laboratory environments (AUC > 0.90), and resolve the physiological arousal experienced during a panic attack. Moreover, patient reported measures indicate that this measurement modality is feasible during panic attacks, and the act of taking the measurement may stop the attack. These results point toward the need for future development and clinical evaluation of this mHealth intervention for preventing panic attacks.

Utilization of Smartphone Depth Mapping Cameras for App-Based Grading of Facial Movement Disorders: Development and Feasibility Study

BACKGROUND

For the classification of facial paresis, various systems of description and evaluation in the form of clinician-graded or software-based scoring systems are available. They serve the purpose of scientific and clinical assessment of the spontaneous course of the disease or monitoring therapeutic interventions. Nevertheless, none have been able to achieve universal acceptance in everyday clinical practice. Hence, a quick and precise tool for assessing the functional status of the facial nerve would be desirable. In this context, the possibilities that the TrueDepth camera of recent iPhone models offer have sparked our interest.

OBJECTIVE

This paper describes the utilization of the iPhone's TrueDepth camera via a specially developed app prototype for quick, objective, and reproducible quantification of facial asymmetries.

METHODS

After conceptual and user interface design, a native app prototype for iOS was programmed that accesses and processes the data of the TrueDepth camera. Using a special algorithm, a new index for the grading of unilateral facial paresis ranging from 0% to 100% was developed. The algorithm was adapted to the well-established Stennert index by weighting the individual facial regions based on functional and cosmetic aspects. Test measurements with healthy subjects using the app were performed in order to prove the reliability of the system.

RESULTS

After the development process, the app prototype had no runtime or buildtime errors and also worked under suboptimal conditions such as different measurement angles, so it met our criteria for a safe and reliable app. The newly defined index expresses the result of the measurements as a generally understandable percentage value for each half of the face. The measurements that correctly rated the facial expressions of healthy individuals as symmetrical in all cases were reproducible and showed no statistically significant intertest variability.

CONCLUSIONS

Based on the experience with the app prototype assessing healthy subjects, the use of the TrueDepth camera should have considerable potential for app-based grading of facial movement disorders. The app and its algorithm, which is based on theoretical considerations, should be evaluated in a prospective clinical study and correlated with common facial scores.

Smartphone App (2kmFIT-App) for Measuring Cardiorespiratory Fitness: Validity and Reliability Study

BACKGROUND

There is strong evidence suggesting that higher levels of cardiorespiratory fitness (CRF) are associated with a healthier metabolic profile, and that CRF can serve as a powerful predictor of morbidity and mortality. In this context, a smartphone app based on the 2-km walk test (UKK test) would provide the possibility to assess CRF remotely in individuals geographically distributed around a country or continent, and even between continents, with minimal equipment and low costs.

OBJECTIVE

The overall aim of this study was to evaluate the validity and reliability of 2kmFIT-App developed for Android and iOS mobile operating systems to estimate maximum oxygen consumption (VO2max) as an indicator of CRF. The specific aims of the study were to determine the validity of 2kmFIT-App to track distance and calculate heart rate (HR).

METHODS

Twenty participants were included for field-testing validation and reliability analysis. The participants completed the UKK test twice using 2kmFIT-App. Distance and HR were measured with the app as well as with accurate methods, and VO2max was estimated using the UKK test equation.

RESULTS

The validity results showed the following mean differences (app minus criterion): distance (-70.40, SD 51.47 meters), time (-0.59, SD 0.45 minutes), HR (-16.75, SD 9.96 beats/minute), and VO2max (3.59, SD 2.01 ml/kg/min). There was moderate validity found for HR (intraclass correlation coefficient [ICC] 0.731, 95% CI -0.211 to 0.942) and good validity found for VO2max (ICC 0.878, 95% CI -0.125 to 0.972). The reliability results showed the following mean differences (retest minus test): app distance (25.99, SD 43.21 meters), app time (-0.15, SD 0.94 seconds), pace (-0.18, SD 0.33 min/km), app HR (-4.5, 13.44 beats/minute), and app VO2max (0.92, SD 3.04 ml/kg/min). There was good reliability for app HR (ICC 0.897, 95% CI 0.742-0.959) and excellent validity for app VO2max (ICC 0.932, 95% CI 0.830-0.973). All of these findings were observed when using the app with an Android operating system, whereas validity was poor when the app was used with iOS.

CONCLUSIONS

This study shows that 2kmFIT-App is a new, scientifically valid and reliable tool able to objectively and remotely estimate CRF, HR, and distance with an Android but not iOS mobile operating system. However, certain limitations such as the time required by 2kmFIT-App to calculate HR or the temperature environment should be considered when using the app.

A validity and reliability study of Conditional Entropy Measures of Pulse Rate Variability

In this work, we present the feasibility to use a simpler methodological approach for the assessment of the short-term complexity of Heart Rate Variability (HRV). Specifically, we propose to exploit Pulse Rate Variability (PRV) recorded through photoplethysmography in place of HRV measured from the ECG, and to compute complexity via a linear Gaussian approximation in place of the standard model-free methods (e.g., nearest neighbor entropy estimates) usually applied to HRV. Linear PRV-based and model-free HRV-based complexity measures were compared via statistical tests, correlation analysis and Bland-Altman plots, demonstrating an overall good agreement. These results support the applicability of the simpler proposed approach, which is faster and easier-to-implement, making our approach eligible for portable/wearable devices and thus broadening the out-of-lab accessibility of autonomic indexes.

Detail-preserving pulse wave extraction from facial videos using consumer-level camera

With the popularity of smart phones, non-contact video-based vital sign monitoring using a camera has gained increased attention over recent years. Especially, imaging photoplethysmography (IPPG), a technique for extracting pulse waves from videos, conduces to monitor physiological information on a daily basis, including heart rate, respiration rate, blood oxygen saturation, and so on. The main challenge for accurate pulse wave extraction from facial videos is that the facial color intensity change due to cardiovascular activities is subtle and is often badly disturbed by noise, such as illumination variation, facial expression changes, and head movements. Even a tiny interference could bring a big obstacle for pulse wave extraction and reduce the accuracy of the calculated vital signs. In recent years, many novel approaches have been proposed to eliminate noise such as filter banks, adaptive filters, Distance-PPG, and machine learning, but these methods mainly focus on heart rate detection and neglect the retention of useful details of pulse wave. For example, the pulse wave extracted by the filter bank method has no dicrotic wave and approaching sine wave, but dicrotic waves are essential for calculating vital signs like blood viscosity and blood pressure. Therefore, a new framework is proposed to achieve accurate pulse wave extraction that contains mainly two steps: 1) preprocessing procedure to remove baseline offset and high frequency random noise; and 2) a self-adaptive singular spectrum analysis algorithm to obtain cyclical components and remove aperiodic irregular noise. Experimental results show that the proposed method can extract detail-preserved pulse waves from facial videos under realistic situations and outperforms state-of-the-art methods in terms of detail-preserving and real time heart rate estimation. Furthermore, the pulse wave extracted by our approach enabled the non-contact estimation of atrial fibrillation, heart rate variability, blood pressure, as well as other physiological indices that require standard pulse wave.

Enhancing the Robustness of Smartphone Photoplethysmography: A Signal Quality Index Approach

Heart rate variability (HRV) provides essential health information such as the risks of heart attacks and mental disorders. However, inconvenience related to the accurate detection of HRV limits its potential applications. The ubiquitous use of smartphones makes them an excellent choice for regular and portable health monitoring. Following this trend, smartphone photoplethysmography (PPG) has recently garnered prominence; however, the lack of robustness has prevented both researchers and practitioners from embracing this technology. This study aimed to bridge the gap in the literature by developing a novel smartphone PPG quality index (SPQI) that can filter corrupted data. A total of 226 participants joined the study, and results from 1343 samples were used to validate the proposed sinusoidal function-based model. In both the correlation coefficient and Bland–Altman analyses, the agreement between HRV measurements generated by both the smartphone PPG and the reference electrocardiogram improved when data were filtered through the SPQI. Our results support not only the proposed approach but also the general value of using smartphone PPG in HRV analysis.

Reliability of Short-Term Heart Rate Variability Indexes Assessed through Photoplethysmography

The gold standard method to monitor heart rate variability (HRV) comprises measuring the time series of interbeat interval durations from electrocardiographic (ECG) recordings. However, due to the widespread use, simplicity and usability of photoplethysmographic (PPG) techniques, monitoring pulse rate variability (PRV) from pulse wave recordings has become a viable alternative to standard HRV analysis. The present study investigates the accuracy of PRV, measured as a surrogate of HRV, for the quantification of descriptive indexes computed in the time domain (mean, variance), frequency domain (low-to-high frequency power ratio LF/HF, HF band central frequency) and information domain (entropy, conditional entropy). We analyze short time series (300 intervals) of HRV measured from the ECG and of PRV acquired from Finometer device in 76 subjects monitored in the resting supine position (SU) and in the upright position during head-up tilt (HUT). Time, frequency and information domain indexes are computed for each HRV and PRV series and, for each index, the comparison between the two approaches is performed through statistical comparison of the distributions across subjects, robust linear regression, and Bland-Altman plots. Results of the comparison indicate an overall good agreement between PRV-based and HRV-based indexes, with an accuracy that is slightly lower during HUT than during SU, and for the band-power ratio and conditional entropy. These results suggest the feasibility of PRV-based assessment of HRV descriptive indexes, and suggest to further investigate the agreement in conditions of physiological stress.

Passive Sensing of Health Outcomes Through Smartphones: Systematic Review of Current Solutions and Possible Limitations

BACKGROUND

Technological advancements, together with the decrease in both price and size of a large variety of sensors, has expanded the role and capabilities of regular mobile phones, turning them into powerful yet ubiquitous monitoring systems. At present, smartphones have the potential to continuously collect information about the users, monitor their activities and behaviors in real time, and provide them with feedback and recommendations.

OBJECTIVE

This systematic review aimed to identify recent scientific studies that explored the passive use of smartphones for generating health- and well-being-related outcomes. In addition, it explores users' engagement and possible challenges in using such self-monitoring systems.

METHODS

A systematic review was conducted, following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, to identify recent publications that explore the use of smartphones as ubiquitous health monitoring systems. We ran reproducible search queries on PubMed, IEEE Xplore, ACM Digital Library, and Scopus online databases and aimed to find answers to the following questions: (1) What is the study focus of the selected papers? (2) What smartphone sensing technologies and data are used to gather health-related input? (3) How are the developed systems validated? and (4) What are the limitations and challenges when using such sensing systems?

RESULTS

Our bibliographic research returned 7404 unique publications. Of these, 118 met the predefined inclusion criteria, which considered publication dates from 2014 onward, English language, and relevance for the topic of this review. The selected papers highlight that smartphones are already being used in multiple health-related scenarios. Of those, physical activity (29.6%; 35/118) and mental health (27.9; 33/118) are 2 of the most studied applications. Accelerometers (57.7%; 67/118) and global positioning systems (GPS; 40.6%; 48/118) are 2 of the most used sensors in smartphones for collecting data from which the health status or well-being of its users can be inferred.

CONCLUSIONS

One relevant outcome of this systematic review is that although smartphones present many advantages for the passive monitoring of users' health and well-being, there is a lack of correlation between smartphone-generated outcomes and clinical knowledge. Moreover, user engagement and motivation are not always modeled as prerequisites, which directly affects user adherence and full validation of such systems.

Smartphone Sensors for Health Monitoring and Diagnosis

Over the past few decades, we have witnessed a dramatic rise in life expectancy owing to significant advances in medical science and technology, medicine as well as increased awareness about nutrition, education, and environmental and personal hygiene. Consequently, the elderly population in many countries are expected to rise rapidly in the coming years. A rapidly rising elderly demographics is expected to adversely affect the socioeconomic systems of many nations in terms of costs associated with their healthcare and wellbeing. In addition, diseases related to the cardiovascular system, eye, respiratory system, skin and mental health are widespread globally. However, most of these diseases can be avoided and/or properly managed through continuous monitoring. In order to enable continuous health monitoring as well as to serve growing healthcare needs; affordable, non-invasive and easy-to-use healthcare solutions are critical. The ever-increasing penetration of smartphones, coupled with embedded sensors and modern communication technologies, make it an attractive technology for enabling continuous and remote monitoring of an individual’s health and wellbeing with negligible additional costs. In this paper, we present a comprehensive review of the state-of-the-art research and developments in smartphone-sensor based healthcare technologies. A discussion on regulatory policies for medical devices and their implications in smartphone-based healthcare systems is presented. Finally, some future research perspectives and concerns regarding smartphone-based healthcare systems are described.

Comparison of short-term heart rate variability indexes evaluated through electrocardiographic and continuous blood pressure monitoring

Heart rate variability (HRV) analysis represents an important tool for the characterization of complex cardiovascular control. HRV indexes are usually calculated from electrocardiographic (ECG) recordings after measuring the time duration between consecutive R peaks, and this is considered the gold standard. An alternative method consists of assessing the pulse rate variability (PRV) from signals acquired through photoplethysmography, a technique also employed for the continuous noninvasive monitoring of blood pressure. In this work, we carry out a thorough analysis and comparison of short-term variability indexes computed from HRV time series obtained from the ECG and from PRV time series obtained from continuous blood pressure (CBP) signals, in order to evaluate the reliability of using CBP-based recordings in place of standard ECG tracks. The analysis has been carried out on short time series (300 beats) of HRV and PRV in 76 subjects studied in different conditions: resting in the supine position, postural stress during 45° head-up tilt, and mental stress during computation of arithmetic test. Nine different indexes have been taken into account, computed in the time domain (mean, variance, root mean square of the successive differences), frequency domain (low-to-high frequency power ratio LF/HF, HF spectral power, and central frequency), and information domain (entropy, conditional entropy, self entropy). Thorough validation has been performed using comparison of the HRV and PRV distributions, robust linear regression, and Bland-Altman plots. Results demonstrate the feasibility of extracting HRV indexes from CBP-based data, showing an overall relatively good agreement of time-, frequency-, and information-domain measures. The agreement decreased during postural and mental arithmetic stress, especially with regard to band-power ratio, conditional, and self-entropy. This finding suggests to use caution in adopting PRV as a surrogate of HRV during stress conditions.

A Novel Personalized Motion and Noise Artifact (MNA) Detection Method for Smartphone Photoplethysmograph (PPG) Signals

Photoplethysmography (PPG) is a technique to detect blood volume changes in an optical way. Representative PPG applications are the measurements of oxygen saturation, heart rate, and respiratory rate. However, PPG signals are sensitive to motion and noise artifacts (MNAs) especially when they are obtained from smartphone cameras. Moreover, PPG signals are different among users and each individual's PPG signal has a unique characteristic. Hence, an effective MNA detection and reduction method for smartphone PPG signals, which adapts itself to each user in a personalized way, is highly demanded. Here, a concept of the probabilistic neural network (PNN) is introduced to be used with the proposed extracted parameters. The signal amplitude, standard deviation of peak to peak time intervals and amplitudes, along with the mean of moving standard deviation, signal slope changes, and the optimal autoregressive (AR) model order are proposed for effective MNA detection. Accordingly, the performance of the proposed personalized algorithm is compared with conventional MNA detection algorithms. As performance metrics, we considered accuracy, sensitivity, and specificity. The results show that the overall performance of the personalized MNA detection is enhanced compared to the generalized algorithm. The average values of the accuracy, sensitivity and specificity of the personalized one are 98.07%, 92.6%, and 99.78%, respectively, while these are 89.92%, 84.21%, and 93.63% for the general one.

Analysis of a Pulse Rate Variability Measurement Using a Smartphone Camera

Background

Heart rate variability (HRV) provides information about the activity of the autonomic nervous system. Because of the small amount of data collected, the importance of HRV has not yet been proven in clinical practice. To collect population-level data, smartphone applications leveraging photoplethysmography (PPG) and some medical knowledge could provide the means for it.

Objective

To assess the capabilities of our smartphone application, we compared PPG (pulse rate variability (PRV)) with ECG (HRV). To have a baseline, we also compared the differences among ECG channels.

Method

We took fifty parallel measurements using iPhone 6 at a 240 Hz sampling frequency and Cardiax PC-ECG devices. The correspondence between the PRV and HRV indices was investigated using correlation, linear regression, and Bland-Altman analysis.

Results

High PPG accuracy: the deviation of PPG-ECG is comparable to that of ECG channels. Mean deviation between PPG-ECG and two ECG channels: RR: 0.01 ms–0.06 ms, SDNN: 0.78 ms–0.46 ms, RMSSD: 1.79 ms–1.21 ms, and pNN50: 2.43%–1.63%.

Conclusions

Our iPhone application yielded good results on PPG-based PRV indices compared to ECG-based HRV indices and to differences among ECG channels. We plan to extend our results on the PPG-ECG correspondence with a deeper analysis of the different ECG channels.

Smartphone-based photoplethysmographic imaging for heart rate monitoring

The purpose of this study is to make use of visible light reflected mode photoplethysmographic (PPG) imaging for heart rate (HR) monitoring via smartphones. The system uses the built-in camera feature in mobile phones to capture video from the subject's index fingertip. The video is processed, and then the PPG signal resulting from the video stream processing is used to calculate the subject's heart rate. Records from 19 subjects were used to evaluate the system's performance. The HR values obtained by the proposed method were compared with the actual HR. The obtained results show an accuracy of 99.7% and a maximum absolute error of 0.4 beats/min where most of the absolute errors lay in the range of 0.04-0.3 beats/min. Given the encouraging results, this type of HR measurement can be adopted with great benefit, especially in the conditions of personal use or home-based care. The proposed method represents an efficient portable solution for HR accurate detection and recording.