The Role of Heart-Rate Variability Parameters in Activity Recognition and Energy-Expenditure Estimation Using Wearable Sensors

Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry

Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

A contactless monitoring system for accurately predicting energy expenditure during treadmill walking based on an ensemble neural network

The monitoring of treadmill walking energy expenditure (EE) plays an important role in health evaluations and management, particularly in older individuals and those with chronic diseases. However, universal and highly accurate prediction methods for walking EE are still lacking. In this paper, we propose an ensemble neural network (ENN) model that predicts the treadmill walking EE of younger and older adults and stroke survivors with high precision based on easy-to-obtain features. Compared with previous studies, the proposed model reduced the estimation error by 13.95% and 66.20% for stroke survivors and younger adults, respectively. Furthermore, a contactless monitoring system was developed based on Kinect, mm-wave radar, and ENN algorithms, and the treadmill walking EE was monitored in real time. This ENN model and monitoring system can be combined with smart devices and treadmill, making them suitable for evaluating, monitoring, and tracking changes in health during exercise and in rehabilitation environments.

Validity of Apple Watch 6 and Polar A370 for monitoring energy expenditure while resting or performing light to vigorous physical activity

OBJECTIVES

The purpose of this study was to assess the validity of energy expenditure measurements taken by the Apple Watch 6 and Polar A370 during resting, light- to vigorous-intensity running on the treadmill and ground, and post-exercise of each speed.

DESIGN

Randomized cross-over trial.

METHODS

This study included 11 male adults (22.5 ± 1.8 years old). Participants were asked to wear the Apple Watch 6 and the Polar A370 simultaneously to measure energy expenditure under various intensities of physical activities on the treadmill and ground, which were then compared with results from a gas metabolism analysis system.

RESULTS

Monitoring energy expenditure for both treadmill and ground, the Apple Watch 6 (range: 13.40-50.34 %) had a higher mean absolute percent error than the Polar A370 (range: 12.74-27.70 %) in resting and running state, while the mean absolute percent error of the Apple Watch 6 (range: 9.71-32.81 %) is smaller than that of the Polar A370 (15.79-43.51 %) in post-exercise. The Apple Watch 6 tends to overestimate energy expenditure, with a mean percent error ranging from -6.61 % to 53.24 %, while the Polar A370 tends to underestimate energy expenditure, with a mean percent error ranging from -3.51 % to 11.33 %. No estimated energy expenditure of both devices fell in the equivalence testing zone.

CONCLUSIONS

For young adult males, the Apple Watch 6 and Polar A370 provide similar levels of accuracy in monitoring energy expenditure during treadmill and ground running exercise. However, both devices are still inadequate in this regard.

Multi-dimensional task recognition for human-robot teaming: literature review

Human-robot teams collaborating to achieve tasks under various conditions, especially in unstructured, dynamic environments will require robots to adapt autonomously to a human teammate's state. An important element of such adaptation is the robot's ability to infer the human teammate's tasks. Environmentally embedded sensors (e.g., motion capture and cameras) are infeasible in such environments for task recognition, but wearable sensors are a viable task recognition alternative. Human-robot teams will perform a wide variety of composite and atomic tasks, involving multiple activity components (i.e., gross motor, fine-grained motor, tactile, visual, cognitive, speech and auditory) that may occur concurrently. A robot's ability to recognize the human's composite, concurrent tasks is a key requirement for realizing successful teaming. Over a hundred task recognition algorithms across multiple activity components are evaluated based on six criteria: sensitivity, suitability, generalizability, composite factor, concurrency and anomaly awareness. The majority of the reviewed task recognition algorithms are not viable for human-robot teams in unstructured, dynamic environments, as they only detect tasks from a subset of activity components, incorporate non-wearable sensors, and rarely detect composite, concurrent tasks across multiple activity components.

Heart Rate Variability Code: Does It Exist and Can We Hack It?

A code is generally defined as a system of signals or symbols for communication. Experimental evidence is synthesized for the presence and utility of such communication in heart rate variability (HRV) with particular attention to fetal HRV: HRV contains signatures of information flow between the organs and of response to physiological or pathophysiological stimuli as signatures of states (or syndromes). HRV exhibits features of time structure, phase space structure, specificity with respect to (organ) target and pathophysiological syndromes, and universality with respect to species independence. Together, these features form a spatiotemporal structure, a phase space, that can be conceived of as a manifold of a yet-to-be-fully understood dynamic complexity. The objective of this article is to synthesize physiological evidence supporting the existence of HRV code: hereby, the process-specific subsets of HRV measures indirectly map the phase space traversal reflecting the specific information contained in the code required for the body to regulate the physiological responses to those processes. The following physiological examples of HRV code are reviewed, which are reflected in specific changes to HRV properties across the signal-analytical domains and across physiological states and conditions: the fetal systemic inflammatory response, organ-specific inflammatory responses (brain and gut), chronic hypoxia and intrinsic (heart) HRV (iHRV), allostatic load (physiological stress due to surgery), and vagotomy (bilateral cervical denervation). Future studies are proposed to test these observations in more depth, and the author refers the interested reader to the referenced publications for a detailed study of the HRV measures involved. While being exemplified mostly in the studies of fetal HRV, the presented framework promises more specific fetal, postnatal, and adult HRV biomarkers of health and disease, which can be obtained non-invasively and continuously.

Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions

Children with autism face challenges in various skills (e.g., communication and social) and they exhibit challenging behaviours. These challenging behaviours represent a challenge to their families, therapists, and caregivers, especially during therapy sessions. In this study, we have investigated several machine learning techniques and data modalities acquired using wearable sensors from children with autism during their interactions with social robots and toys in their potential to detect challenging behaviours. Each child wore a wearable device that collected data. Video annotations of the sessions were used to identify the occurrence of challenging behaviours. Extracted time features (i.e., mean, standard deviation, min, and max) in conjunction with four machine learning techniques were considered to detect challenging behaviors. The heart rate variability (HRV) changes have also been investigated in this study. The XGBoost algorithm has achieved the best performance (i.e., an accuracy of 99%). Additionally, physiological features outperformed the kinetic ones, with the heart rate being the main contributing feature in the prediction performance. One HRV parameter (i.e., RMSSD) was found to correlate with the occurrence of challenging behaviours. This work highlights the importance of developing the tools and methods to detect challenging behaviors among children with autism during aided sessions with social robots.

DDM-HSA: Dual Deterministic Model-Based Heart Sound Analysis for Daily Life Monitoring

A sudden cardiac event in patients with heart disease can lead to a heart attack in extreme cases. Therefore, prompt interventions for the particular heart situation and periodic monitoring are critical. This study focuses on a heart sound analysis method that can be monitored daily using multimodal signals acquired with wearable devices. The dual deterministic model-based heart sound analysis is designed in a parallel structure that uses two bio-signals (PCG and PPG signals) related to the heartbeat, enabling more accurate heart sound identification. The experimental results show promising performance of the proposed Model III (DDM-HSA with window and envelope filter), which had the highest performance, and S1 and S2 showed average accuracy (unit: %) of 95.39 (±2.14) and 92.55 (±3.74), respectively. The findings of this study are anticipated to provide improved technology to detect heart sounds and analyze cardiac activities using only bio-signals that can be measured using wearable devices in a mobile environment.

Are Machine Learning Models on Wrist Accelerometry Robust against Differences in Physical Performance among Older Adults?

Sufficient physical activity (PA) reduces the risk of a myriad of diseases and preserves physical capabilities in later life. While there have been significant achievements in mapping accelerations to real-life movements using machine learning (ML), errors continue to be common, particularly for wrist-worn devices. It remains unknown whether ML models are robust for estimating age-related loss of physical function. In this study, we evaluated the performance of ML models (XGBoost and LASSO) to estimate the hallmark measures of PA in low physical performance (LPP) and high physical performance (HPP) groups. Our models were built to recognize PA types and intensities, identify each individual activity, and estimate energy expenditure (EE) using wrist-worn accelerometer data (33 activities per participant) from a large sample of participants (n = 247, 57% females, aged 60+ years). Results indicated that the ML models were accurate in recognizing PA by type and intensity while also estimating EE accurately. However, the models built to recognize individual activities were less robust. Across all tasks, XGBoost outperformed LASSO. XGBoost obtained F1-Scores for sedentary (0.932 ± 0.005), locomotion (0.946 ± 0.003), lifestyle (0.927 ± 0.006), and strength flexibility exercise (0.915 ± 0.017) activity type recognition tasks. The F1-Scores for recognizing low, light, and moderate activity intensity were (0.932 ± 0.005), (0.840 ± 0.004), and (0.869 ± 0.005), respectively. The root mean square error for EE estimation was 0.836 ± 0.059 METs. There was no evidence showing that splitting the participants into the LPP and HPP groups improved the models’ performance on estimating the hallmark measures of physical activities. In conclusion, using features derived from wrist-worn accelerometer data, machine learning models can accurately recognize PA types and intensities and estimate EE for older adults with high and low physical function.

Deep Multi-Branch Two-Stage Regression Network for Accurate Energy Expenditure Estimation with ECG and IMU Data

OBJECTIVE

Energy Expenditure (EE) estimation plays an important role in objectively evaluating physical activity and its impact on human health. EE during activity can be affected by many factors, including activity intensity, individual physical and physiological characteristics, environment, etc. However, current studies only use very limited information, such as heart rate and step count, to estimate EE, which leads to a low estimation accuracy.

METHODS

In this study, we proposed a deep multi-branch two-stage regression network (DMTRN) to effectively fuse a variety of related information including motion information, physiological characteristics, and human physical information, which significantly improved the EE estimation accuracy. The proposed DMTRN consists of two main modules: a multi-branch convolutional neural network module which is used to extract multi-scale context features from inertial measurement unit (IMU) data and electrocardiogram (ECG) data and a two-stage regression module which aggregated the extracted multi-scale context features containing the physiological and motion information and the anthropometric features to accurately estimate EE.

RESULTS

Experiments performed on 33 participants show that our proposed method is more accurate and the average root mean square error (RMSE) is reduced by 22.8% compared with previous works.

CONCLUSION

The EE estimation accuracy was improved by the proposed DMTRN model with a well-designed network structure and new input signal ECG.

SIGNIFICANCE

This study verified that ECG was much more effective than HR for EE estimation and cast light on EE estimation using the deep learning method.

SensorHub: Multimodal Sensing in Real-Life Enables Home-Based Studies

Observational studies are an important tool for determining whether the findings from controlled experiments can be transferred into scenarios that are closer to subjects’ real-life circumstances. A rigorous approach to observational studies involves collecting data from different sensors to comprehensively capture the situation of the subject. However, this leads to technical difficulties especially if the sensors are from different manufacturers, as multiple data collection tools have to run simultaneously. We present SensorHub, a system that can collect data from various wearable devices from different manufacturers, such as inertial measurement units, portable electrocardiographs, portable electroencephalographs, portable photoplethysmographs, and sensors for electrodermal activity. Additionally, our tool offers the possibility to include ecological momentary assessments (EMAs) in studies. Hence, SensorHub enables multimodal sensor data collection under real-world conditions and allows direct user feedback to be collected through questionnaires, enabling studies at home. In a first study with 11 participants, we successfully used SensorHub to record multiple signals with different devices and collected additional information with the help of EMAs. In addition, we evaluated SensorHub’s technical capabilities in several trials with up to 21 participants recording simultaneously using multiple sensors with sampling frequencies as high as 1000 Hz. We could show that although there is a theoretical limitation to the transmissible data rate, in practice this limitation is not an issue and data loss is rare. We conclude that with modern communication protocols and with the increasingly powerful smartphones and wearables, a system like our SensorHub establishes an interoperability framework to adequately combine consumer-grade sensing hardware which enables observational studies in real life.

Towards Human Stress and Activity Recognition: A Review and a First Approach Based on Low-Cost Wearables

Detecting stress when performing physical activities is an interesting field that has received relatively little research interest to date. In this paper, we took a first step towards redressing this, through a comprehensive review and the design of a low-cost body area network (BAN) made of a set of wearables that allow physiological signals and human movements to be captured simultaneously. We used four different wearables: OpenBCI and three other open-hardware custom-made designs that communicate via bluetooth low energy (BLE) to an external computer—following the edge-computingconcept—hosting applications for data synchronization and storage. We obtained a large number of physiological signals (electroencephalography (EEG), electrocardiography (ECG), breathing rate (BR), electrodermal activity (EDA), and skin temperature (ST)) with which we analyzed internal states in general, but with a focus on stress. The findings show the reliability and feasibility of the proposed body area network (BAN) according to battery lifetime (greater than 15 h), packet loss rate (0% for our custom-made designs), and signal quality (signal-noise ratio (SNR) of 9.8 dB for the ECG circuit, and 61.6 dB for the EDA). Moreover, we conducted a preliminary experiment to gauge the main ECG features for stress detection during rest.

Human Activity Recognition: A Comparative Study to Assess the Contribution Level of Accelerometer, ECG, and PPG Signals

Inertial sensors are widely used in the field of human activity recognition (HAR), since this source of information is the most informative time series among non-visual datasets. HAR researchers are actively exploring other approaches and different sources of signals to improve the performance of HAR systems. In this study, we investigate the impact of combining bio-signals with a dataset acquired from inertial sensors on recognizing human daily activities. To achieve this aim, we used the PPG-DaLiA dataset consisting of 3D-accelerometer (3D-ACC), electrocardiogram (ECG), photoplethysmogram (PPG) signals acquired from 15 individuals while performing daily activities. We extracted hand-crafted time and frequency domain features, then, we applied a correlation-based feature selection approach to reduce the feature-set dimensionality. After introducing early fusion scenarios, we trained and tested random forest models with subject-dependent and subject-independent setups. Our results indicate that combining features extracted from the 3D-ACC signal with the ECG signal improves the classifier’s performance F1-scores by 2.72% and 3.00% (from 94.07% to 96.80%, and 83.16% to 86.17%) for subject-dependent and subject-independent approaches, respectively.

Depth-Camera Based Energy Expenditure Estimation System for Physical Activity Using Posture Classification Algorithm

Insufficient physical activity is common in modern society. By estimating the energy expenditure (EE) of different physical activities, people can develop suitable exercise plans to improve their lifestyle quality. However, several limitations still exist in the related works. Therefore, the aim of this study is to propose an accurate EE estimation model based on depth camera data with physical activity classification to solve the limitations in the previous research. To decide the best location and amount of cameras of the EE estimation, three depth cameras were set at three locations, namely the side, rear side, and rear views, to obtain the kinematic data and EE estimation. Support vector machine was used for physical activity classification. Three EE estimation models, namely linear regression, multilayer perceptron (MLP), and convolutional neural network (CNN) models, were compared and determined the model with optimal performance in different experimental settings. The results have shown that if only one depth camera is available, optimal EE estimation can be obtained using the side view and MLP model. The mean absolute error (MAE), mean square error (MSE), and root MSE (RMSE) of the classification results under the aforementioned settings were 0.55, 0.66, and 0.81, respectively. If higher accuracy is required, two depth cameras can be set at the side and rear views, the CNN model can be used for light-to-moderate activities, and the MLP model can be used for vigorous activities. The RMSEs for estimating the EEs of standing, walking, and running were 0.19, 0.57, and 0.96, respectively. By applying the different models on different amounts of cameras, the optimal performance can be obtained, and this is also the first study to discuss the issue.

Reliability and validity of the physical activity monitor for assessing energy expenditures in sedentary, regularly exercising, non-endurance athlete, and endurance athlete adults

Background

Inertial sensors, such as accelerometers, serve as convenient devices to predict the energy expenditures (EEs) during physical activities by a predictive equation. Although the accuracy of estimate EEs especially matter to athletes receive physical training, most EE predictive equations adopted in accelerometers are based on the general population, not athletes. This study included the heart rate reserve (HRR) as a compensatory parameter for physical intensity and derived new equations customized for sedentary, regularly exercising, non-endurance athlete, and endurance athlete adults.

Methods

With indirect calorimetry as the criterion measure (CM), the EEs of participants on a treadmill were measured, and vector magnitudes (VM), as well as HRR, were simultaneously recorded by a waist-worn accelerometer with a heart rate monitor. Participants comprised a sedentary group (SG), an exercise-habit group (EHG), a non-endurance group (NEG), and an endurance group (EG), with 30 adults in each group.

Results

EE predictive equations were revised using linear regression with cross-validation on VM, HRR, and body mass (BM). The modified model demonstrates valid and reliable predictions across four populations (Pearson correlation coefficient, r: 0.922 to 0.932; intraclass correlation coefficient, ICC: 0.919 to 0.930).

Conclusion

Using accelerometers with a heart rate monitorcan accurately predict EEs of athletes and non-athletes with an optimized predictive equation integrating the VM, HRR, and BM parameters.

Feasibility of the Energy Expenditure Prediction for Athletes and Non-Athletes from Ankle-Mounted Accelerometer and Heart Rate Monitor

Due to the nature of micro-electromechanical systems, the vector magnitude (VM) activity of accelerometers varies depending on the wearing position and does not identify different levels of physical fitness. Without an appropriate energy expenditure (EE) estimation equation, bias can occur in the estimated values. We aimed to amend the EE estimation equation using heart rate reserve (HRR) parameters as the correction factor, which could be applied to athletes and non-athletes who primarily use ankle-mounted devices. Indirect calorimetry was used as the criterion measure with an accelerometer (ankle-mounted) equipped with a heart rate monitor to synchronously measure the EE of 120 healthy adults on a treadmill in four groups. Compared with ankle-mounted accelerometer outputs, when the traditional equation was modified using linear regression by combining VM with body weight and/or HRR parameters (modified models: Model A, without HRR; Model B, with HRR), both Model A (r: 0.931 to 0.972; ICC: 0.913 to 0.954) and Model B (r: 0.933 to 0.975; ICC: 0.930 to 0.959) showed the valid and reliable predictive ability for the four groups. With respect to the simplest and most reasonable mode, Model A seems to be a good choice for predicting EE when using an ankle-mounted device.

Cardio-Diagnostic Assisting Computer System

The mathematical analysis and the assessment of heart rate variability (HRV) based on computer systems can assist the diagnostic process with determining the cardiac status of patients. The new cardio-diagnostic assisting computer system created uses the classic Time-Domain, Frequency-Domain, and Time-Frequency analysis indices, as well as the nonlinear methods (Poincaré plot, Recurrence plot, Hurst R/S method, Detrended Fluctuation Analysis (DFA), Multi-Fractal DFA, Approximate Entropy and Sample Entropy). To test the feasibility of the software developed, 24-hour Holter recordings of four groups of people were analysed: healthy subjects and patients with arrhythmia, heart failure and syncope. Time-Domain (SDNN < 50 ms, SDANN < 100 ms, RMSSD < 17 ms) and Frequency-Domain (the spectrum of HRV in the LF < 550 ms², and HF < 540 ms²) parameter values decreased in the cardiovascular disease groups compared to the control group as a result of lower HRV due to decreased parasympathetic and increased sympathetic activity. The results of the nonlinear analysis showed low values of (SD1 < 56 ms, SD2 < 110 ms) at Poincaré plot (Alpha < 90 ms) at DFA in patients with diseases. Significantly reducing these parameters are markers of cardiac dysfunction. The examined groups of patients showed an increase in the parameters (DET% > 95, REC% > 38, ENTR > 3.2) at the Recurrence plot. This is evidence of a pathological change in the regulation of heart rhythm. The system created can be useful in making the diagnosis by the cardiologist and in bringing greater accuracy and objectivity to the treatment.

Innovative Use of Wrist-Worn Wearable Devices in the Sports Domain: A Systematic Review

Wrist wearables are becoming more and more popular, and its use is widespread in sports, both professional and amateur. However, at present, they do not seem to exploit all their potential. The objective of this study is to explore innovative proposals for the use of wearable wrist technology in the field of sports, to understand its potential and identify new challenges and lines of future research related to this technology. A systematic review of the scientific literature, collected in 4 major repositories, was carried out to locate research initiatives where wrist wearables were introduced to address some sports-related challenges. Those works that were limited to evaluating sensor performance in sports activities and those in which wrist wearable devices did not play a significant role were excluded. 26 articles were eventually selected for full-text analysis that discuss the introduction of wrist-worn wearables to address some innovative use in the sports field. This study showcases relevant proposals in 10 different sports. The research initiatives identified are oriented to the use of wearable wrist technology (i) for the comprehensive monitoring of sportspeople’s behavior in activities not supported by the vendors, (ii) to identify specific types of movements or actions in specific sports, and (iii) to prevent injuries. There are, however, open issues that should be tackled in the future, such as the incorporation of these devices in sports activities not currently addressed, or the provision of specific recommendation services for sport practitioners.

Stress Resilience Measurement With Heart-Rate Variability During Mental And Physical Stress

Stress management is particularly important for healthcare of modern people. In stress research, heart-rate variability (HRV), indicating the change of time intervals in successive heart beats, significantly contributed due to its close relationship with autonomic nervous system. However, the adaptive response to stress, also known as stress resilience, has not been studied much yet. We collected electrocardiogram during mental and physical stress, experimentally designed by mental arithmetic tasks and physical activities for 14 healthy subjects. As a result, we found that resting HRV parameters, particularly associated with the parasympathetic activity, had significant positive correlations with reactivity and recovery from mental and physical stress. These HRV parameters can be used as a measure of stress resilience quantitatively. Our findings suggest that these parameters can help one's stress management by enabling to predict the adaptive response to upcoming stressful events.

Current Status and Prospects of Health-Related Sensing Technology in Wearable Devices

The healthcare-related functions of wearable devices are very useful for continuous monitoring of biological information. Wearable devices equipped with communication function can be used for additional healthcare services. Among the wearable devices, the wristband type is most suitable for acquiring biological signals, and the wear preference of the user is high, so it is highly likely to be used more in the future. In this paper, the health-related functions of wristband were investigated and the technical limitations and prospects were also reviewed. Most current wristband-type devices are equipped with the combination of accelerometer, optical sensor, and electrodes for their health functions, and continuously measured data are expanding the possibility of discovering new medical meanings. The blood pressure measurement function without using cuff is the most useful and expected function among the health-related functions expected to be mounted on the wrist wearable device, in spite of its technical limits and difficulties.