Development and evaluation of an ambulatory stress monitor based on wearable sensors

Stress classification with in-ear heartbeat sounds

BACKGROUND

Although stress plays a key role in tinnitus and decreased sound tolerance, conventional hearing devices used to manage these conditions are not currently capable of monitoring the wearer's stress level. The aim of this study was to assess the feasibility of stress monitoring with an in-ear device.

METHOD

In-ear heartbeat sounds and clinical-grade electrocardiography (ECG) signals were simultaneously recorded while 30 healthy young adults underwent a stress protocol. Heart rate variability features were extracted from both signals to train classification algorithms to predict stress vs. rest.

RESULTS

Models trained and tested using in-ear heartbeat sounds appeared to perform better than the models trained and tested using the ECG signals. However, further analyses comparing heart rate variability features extracted from ECG and the in-ear heartbeat sounds suggest that the improvement in stress prediction performance was driven by the increased presence of artifacts (e.g. movement or speech) during the stress tasks, rather than physiologically meaningful changes in the heartbeat signals that would be indicative of stress in real-world settings. To address this difference in error between rest and stress conditions, a data augmentation method was proposed to balance the error.

CONCLUSIONS

The final system demonstrates the viability of robust stress recognition with only in-ear heartbeat sounds, which could expand the capabilities of hearing devices used to address conditions related to stress and noise. The proposed data augmentation method effectively identified and addressed artifact-related biases, which could broadly be applied to improve robustness of biosignal monitoring with machine learning.

Exploring physiological responses through electrodermal activity (EDA) for evaluating the impact of universal design features in a hotel environment

Purpose: The aim of this study was to explore the feasibility of using electrodermal activity (EDA) to detect changes in physiological arousal linked to perceptions of accommodations, focusing on universal design (UD) features. In environments like hotels, designers must consider wellness, social integration, and cultural appropriateness to effectively implement UD. Challenges exist with implementing and evaluating UD to accommodate diverse user needs due to conflicting definitions and application issues. To meet the need for post-design evaluation discerning accommodations by features and user groups, EDA measures offer a way to capture individual reactions to external stimuli.

Materials and Methods: In this study, 22 adults (14 young, 8 older) completed an independent hotel walkthrough while expressing their perceptions. EDA was measured using a wristband, and participants' perceived stress and usability were assessed through questionnaires. Phasic EDA was extracted to represent discrete event-evoked changes in arousal.

Results: Findings demonstrated the potential of EDA to identify physiological response variations based on age and location within the hotel. Older adults displayed significantly higher levels of arousal and more favorable usability ratings (4.61 out of 5) compared to young adults, with peak arousal in the corridor and public restroom. Younger adults showed the highest arousal in the bathroom, often with negative associations. The groups differed in their reactions to the bathroom and reception areas.

Conclusions: Divergences between physiological responses and subjective outcomes highlighted the complexity of translating arousal measures into meaningful insights. EDA, combined with commentary, enhanced our understanding of user reactions to design elements to fill gaps left by subjective methods.

Recent Progress in Biosensors for Depression Monitoring-Advancing Personalized Treatment

Depression is currently a major contributor to unnatural deaths and the healthcare burden globally, and a patient's battle with depression is often a long one. Because the causes, symptoms, and effects of medications are complex and highly individualized, early identification and personalized treatment of depression are key to improving treatment outcomes. The development of wearable electronics, machine learning, and other technologies in recent years has provided more possibilities for the realization of this goal. Conducting regular monitoring through biosensing technology allows for a more comprehensive and objective analysis than previous self-evaluations. This includes identifying depressive episodes, distinguishing somatization symptoms, analyzing etiology, and evaluating the effectiveness of treatment programs. This review summarizes recent research on biosensing technologies for depression. Special attention is given to technologies that can be portable or wearable, with the potential to enable patient use outside of the hospital, for long periods.

Mitigating data quality challenges in ambulatory wrist-worn wearable monitoring through analytical and practical approaches

Chronic disease management and follow-up are vital for realizing sustained patient well-being and optimal health outcomes. Recent advancements in wearable technologies, particularly wrist-worn devices, offer promising solutions for longitudinal patient monitoring, replacing subjective, intermittent self-reporting with objective, continuous monitoring. However, collecting and analyzing data from wearables presents several challenges, such as data entry errors, non-wear periods, missing data, and wearable artifacts. In this work, we explore these data analysis challenges using two real-world datasets (mBrain21 and ETRI lifelog2020). We introduce practical countermeasures, including participant compliance visualizations, interaction-triggered questionnaires to assess personal bias, and an optimized pipeline for detecting non-wear periods. Additionally, we propose a visualization-oriented approach to validate processing pipelines using scalable tools such as tsflex and Plotly-Resampler. Lastly, we present a bootstrapping methodology to evaluate the variability of wearable-derived features in the presence of partially missing data segments. Prioritizing transparency and reproducibility, we provide open access to our detailed code examples, facilitating adaptation in future wearable research. In conclusion, our contributions provide actionable approaches for improving wearable data collection and analysis.

Advancements in Polymer-Assisted Layer-by-Layer Fabrication of Wearable Sensors for Health Monitoring

Recent advancements in polymer-assisted layer-by-layer (LbL) fabrication have revolutionized the development of wearable sensors for health monitoring. LbL self-assembly has emerged as a powerful and versatile technique for creating conformal, flexible, and multi-functional films on various substrates, making it particularly suitable for fabricating wearable sensors. The incorporation of polymers, both natural and synthetic, has played a crucial role in enhancing the performance, stability, and biocompatibility of these sensors. This review provides a comprehensive overview of the principles of LbL self-assembly, the role of polymers in sensor fabrication, and the various types of LbL-fabricated wearable sensors for physical, chemical, and biological sensing. The applications of these sensors in continuous health monitoring, disease diagnosis, and management are discussed in detail, highlighting their potential to revolutionize personalized healthcare. Despite significant progress, challenges related to long-term stability, biocompatibility, data acquisition, and large-scale manufacturing are still to be addressed, providing insights into future research directions. With continued advancements in polymer-assisted LbL fabrication and related fields, wearable sensors are poised to improve the quality of life for individuals worldwide.

Noninvasive monitoring technologies to identify discomfort and distressing symptoms in persons with limited communication at the end of life: a scoping review

BACKGROUND

Discomfort and distressing symptoms are common at the end of life, while people in this stage are often no longer able to express themselves. Technologies may aid clinicians in detecting and treating these symptoms to improve end-of-life care. This review provides an overview of noninvasive monitoring technologies that may be applied to persons with limited communication at the end of life to identify discomfort.

METHODS

A systematic search was performed in nine databases, and experts were consulted. Manuscripts were included if they were written in English, Dutch, German, French, Japanese or Chinese, if the monitoring technology measured discomfort or distressing symptoms, was noninvasive, could be continuously administered for 4 hours and was potentially applicable for bed-ridden people. The screening was performed by two researchers independently. Information about the technology, its clinimetrics (validity, reliability, sensitivity, specificity, responsiveness), acceptability, and feasibility were extracted.

RESULTS

Of the 3,414 identified manuscripts, 229 met the eligibility criteria. A variety of monitoring technologies were identified, including actigraphy, brain activity monitoring, electrocardiography, electrodermal activity monitoring, surface electromyography, incontinence sensors, multimodal systems, and noncontact monitoring systems. The main indicators of discomfort monitored by these technologies were sleep, level of consciousness, risk of pressure ulcers, urinary incontinence, agitation, and pain. For the end-of-life phase, brain activity monitors could be helpful and acceptable to monitor the level of consciousness during palliative sedation. However, no manuscripts have reported on the clinimetrics, feasibility, and acceptability of the other technologies for the end-of-life phase.

CONCLUSIONS

Noninvasive monitoring technologies are available to measure common symptoms at the end of life. Future research should evaluate the quality of evidence provided by existing studies and investigate the feasibility, acceptability, and usefulness of these technologies in the end-of-life setting. Guidelines for studies on healthcare technologies should be better implemented and further developed.

Detection of arousal and valence from facial expressions and physiological responses evoked by different types of stressors

Automatically detecting mental state such as stress from video images of the face could support evaluating stress responses in applicants for high risk jobs or contribute to timely stress detection in challenging operational settings (e.g., aircrew, command center operators). Challenges in automatically estimating mental state include the generalization of models across contexts and across participants. We here aim to create robust models by training them using data from different contexts and including physiological features. Fifty-one participants were exposed to different types of stressors (cognitive, social evaluative and startle) and baseline variants of the stressors. Video, electrocardiogram (ECG), electrodermal activity (EDA) and self-reports (arousal and valence) were recorded. Logistic regression models aimed to classify between high and low arousal and valence across participants, where "high" and "low" were defined relative to the center of the rating scale. Accuracy scores of different models were evaluated: models trained and tested within a specific context (either a baseline or stressor variant of a task), intermediate context (baseline and stressor variant of a task), or general context (all conditions together). Furthermore, for these different model variants, only the video data was included, only the physiological data, or both video and physiological data. We found that all (video, physiological and video-physio) models could successfully distinguish between high- and low-rated arousal and valence, though performance tended to be better for (1) arousal than valence, (2) specific context than intermediate and general contexts, (3) video-physio data than video or physiological data alone. Automatic feature selection resulted in inclusion of 3-20 features, where the models based on video-physio data usually included features from video, ECG and EDA. Still, performance of video-only models approached the performance of video-physio models. Arousal and valence ratings by three experienced human observers scores based on part of the video data did not match with self-reports. In sum, we showed that it is possible to automatically monitor arousal and valence even in relatively general contexts and better than humans can (in the given circumstances), and that non-contact video images of faces capture an important part of the information, which has practical advantages.

Stress Monitoring in Free-Living Environments

Stress monitoring is an important area of research with significant implications for individuals' physical and mental health. We present a data-driven approach for stress detection based on convolutional neural networks while addressing the problems of the best sensor channel and the lack of knowledge about stress episodes. Our work is the first to present an analysis of stress-related sensor data collected in real-world conditions from individuals diagnosed with Alcohol Use Disorder (AUD) and undergoing treatment to abstain from alcohol. We developed polynomial-time sensor channel selection algorithms to determine the best sensor modality for a machine learning task. We model the time variation in stress labels expressed by the participants as the subjective effects of stress. We addressed the subjective nature of stress by determining the optimal input length around stress events with an iterative search algorithm. We found the skin conductance modality to be most indicative of stress, and the segment length of 60 seconds around user-reported stress labels resulted in top stress detection performance. We used both majority undersampling and minority oversampling to balance our dataset. With majority undersampling, the binary stress classification model achieved an average accuracy of 99% and an f1-score of 0.99 on the training and test sets after 5-fold cross-validation. With minority oversampling, the performance on the test set dropped to an average accuracy of 76.25% and an f1-score of 0.68, highlighting the challenges of working with real-world datasets.

Artificial Intelligence-Powered Electronic Skin

Skin-interfaced electronics is gradually changing medical practices by enabling continuous and noninvasive tracking of physiological and biochemical information. With the rise of big data and digital medicine, next-generation electronic skin (e-skin) will be able to use artificial intelligence (AI) to optimize its design as well as uncover user-personalized health profiles. Recent multimodal e-skin platforms have already employed machine learning (ML) algorithms for autonomous data analytics. Unfortunately, there is a lack of appropriate AI protocols and guidelines for e-skin devices, resulting in overly complex models and non-reproducible conclusions for simple applications. This review aims to present AI technologies in e-skin hardware and assess their potential for new inspired integrated platform solutions. We outline recent breakthroughs in AI strategies and their applications in engineering e-skins as well as understanding health information collected by e-skins, highlighting the transformative deployment of AI in robotics, prosthetics, virtual reality, and personalized healthcare. We also discuss the challenges and prospects of AI-powered e-skins as well as predictions for the future trajectory of smart e-skins.

The Use of Wearable Technology in Providing Assistive Solutions for Mental Well-Being

The main goal of this manuscript is to provide an extensive literature review and analysis of certain biomarkers, which are frequently used to identify stress, anxiety, and other emotions, leading to potential solutions for the monitoring of mental wellness using wearable technologies. It is possible to see the impacts of several biomarkers in detecting stress levels and their effectiveness with an investigation into the literature on this subject. Biofeedback training has demonstrated some psychological effects, such as a reduction in anxiety and self-control enhancement. This survey demonstrates backed up by evidence that wearable devices are assistive in providing health and mental wellness solutions. Because physical activity tracing would reduce the stress stressors, which affect the subject's body, therefore, it would also affect the mental activity and would lead to a reduction in cognitive mental load.

AI-Enabled Smart Wristband Providing Real-Time Vital Signs and Stress Monitoring

This work introduces the design, architecture, implementation, and testing of a low-cost and machine-learning-enabled device to be worn on the wrist. The suggested wearable device has been developed for use during emergency incidents of large passenger ship evacuations, and enables the real-time monitoring of the passengers' physiological state, and stress detection. Based on a properly preprocessed PPG signal, the device provides essential biometric data (pulse rate and oxygen saturation level) and an efficient unimodal machine learning pipeline. The stress detecting machine learning pipeline is based on ultra-short-term pulse rate variability, and has been successfully integrated into the microcontroller of the developed embedded device. As a result, the presented smart wristband is able to provide real-time stress detection. The stress detection system has been trained with the use of the publicly available WESAD dataset, and its performance has been tested through a two-stage process. Initially, evaluation of the lightweight machine learning pipeline on a previously unseen subset of the WESAD dataset was performed, reaching an accuracy score equal to 91%. Subsequently, external validation was conducted, through a dedicated laboratory study of 15 volunteers subjected to well-acknowledged cognitive stressors while wearing the smart wristband, which yielded an accuracy score equal to 76%.

Comparison between Two Time Synchronization and Data Alignment Methods for Multi-Channel Wearable Biosensor Systems Using BLE Protocol

Wireless wearable sensor systems for biomedical signal acquisition have developed rapidly in recent years. Multiple sensors are often deployed for monitoring common bioelectric signals, such as EEG (electroencephalogram), ECG (electrocardiogram), and EMG (electromyogram). Compared with ZigBee and low-power Wi-Fi, Bluetooth Low Energy (BLE) can be a more suitable wireless protocol for such systems. However, current time synchronization methods for BLE multi-channel systems, via either BLE beacon transmissions or additional hardware, cannot satisfy the requirements of high throughput with low latency, transferability between commercial devices, and low energy consumption. We developed a time synchronization and simple data alignment (SDA) algorithm, which was implemented in the BLE application layer without the need for additional hardware. We further developed a linear interpolation data alignment (LIDA) algorithm to improve upon SDA. We tested our algorithms using sinusoidal input signals at different frequencies (10 to 210 Hz in increments of 20 Hz-frequencies spanning much of the relevant range of EEG, ECG, and EMG signals) on Texas Instruments (TI) CC26XX family devices, with two peripheral nodes communicating with one central node. The analysis was performed offline. The lowest average (±standard deviation) absolute time alignment error between the two peripheral nodes achieved by the SDA algorithm was 384.3 ± 386.5 μs, while that of the LIDA algorithm was 189.9 ± 204.7 μs. For all sinusoidal frequencies tested, the performance of LIDA was always statistically better than that of SDA. These average alignment errors were quite low-well below one sample period for commonly acquired bioelectric signals.

Stress Detection Using Frequency Spectrum Analysis of Wrist-Measured Electrodermal Activity

The article deals with the detection of stress using the electrodermal activity (EDA) signal measured at the wrist. We present an approach for feature extraction from EDA. The approach uses frequency spectrum analysis in multiple frequency bands. We evaluate the proposed approach using the 4 Hz EDA signal measured at the wrist in the publicly available Wearable Stress and Affect Detection (WESAD) dataset. Seven existing approaches to stress detection using EDA signals measured by wrist-worn sensors are analysed and the reported results are compared with ours. The proposed approach represents an improvement in accuracy over the other techniques studied. Moreover, we focus on time to detection (TTD) and show that our approach is able to outperform competing techniques, with fewer data points. The proposed feature extraction is computationally inexpensive, thus the presented approach is suitable for use in real-world wearable applications where both short response times and high detection performance are important. We report both binary (stress vs. no stress) as well as three-class (baseline/stress/amusement) results.

From Monitoring to Assisting: A Systematic Review towards Healthier Workplaces

Long-term stress is associated with a decline in global health, affecting social, intellectual, and economic development alike. Although comprehensive action plans have been implemented to provide people access to mental health services and promote mental well-being, employees' mental health generally takes second place to productivity and profit in business settings. This review paper offers an overview of the current interactive approaches used for relieving work-related stress associated with mental health. Results from the 38 included studies show that affective computing is used mainly for monitoring purposes and is usually combined with tangible interfaces that collect workers' physiological changes. Although the ability to sense and predict employees' affective states can potentially improve mental health in the workplace, there is a substantial disparity between monitoring one's health and the delivery of practical interventions to mitigate stress found in the surveyed studies. Designing systems that capitalize on embodied interaction principles is paramount, especially in the post-pandemic context, as the concepts of physical and mental safety take on new meanings that must be consciously and carefully addressed, particularly in workplace settings. Finally, this paper highlights the main design implications for the effective implementation of interfaces to help mitigate stress in the workplace.

Stress Monitoring Using Wearable Sensors: A Pilot Study and Stress-Predict Dataset

With the recent advancements in the field of wearable technologies, the opportunity to monitor stress continuously using different physiological variables has gained significant interest. The early detection of stress can help improve healthcare and minimizes the negative impact of long-term stress. This paper reports outcomes of a pilot study and associated stress-monitoring dataset, named the "Stress-Predict Dataset", created by collecting physiological signals from healthy subjects using wrist-worn watches with a photoplethysmogram (PPG) sensor. While wearing these watches, 35 healthy volunteers underwent a series of tasks (i.e., Stroop color test, Trier Social Stress Test and Hyperventilation Provocation Test), along with a rest period in-between each task. They also answered questionnaires designed to induce stress levels compatible with daily life. The changes in the blood volume pulse (BVP) and heart rate were recorded by the watch and were labelled as occurring during stress-inducing tasks or a rest period (no stress). Additionally, respiratory rate was estimated using the BVP signal. Statistical models and personalised adaptive reference ranges were used to determine the utility of the proposed stressors and the extracted variables (heart rate and respiratory rate). The analysis showed that the interview session was the most significant stress stimulus, causing a significant variation in heart rate of 27 (77%) participants and respiratory rate of 28 (80%) participants out of 35. The outcomes of this study contribute to the understanding the role of stressors and their association with physiological response and provide a dataset to help develop new wearable solutions for more reliable, valid, and sensitive physio-logical stress monitoring.

Stressalyzer: Convolutional Neural Network Framework for Personalized Stress Classification

Stress detection and monitoring is an active area of research with important implications for an individual's personal, professional, and social health. Current approaches for stress classification use traditional machine learning algorithms trained on features computed from multiple sensor modalities. These methods are data and computation-intensive, rely on hand-crafted features, and lack reproducibility. These limitations impede the practical use of stress detection and classification systems in the real world. To overcome these shortcomings, we propose Stressalyzer, a novel stress classification and personalization framework from single-modality sensor data without feature computation and selection. Stressalyzer uses only Electrodermal activity (EDA) sensor data while providing competitive results compared to the state-of-the-art techniques that use multiple sensor modalities and are computationally expensive due to the calculation of large number of features. Using the dataset collected in a laboratory setting from 15 subjects, our single-channel neural network-based model achieves a classification accuracy of 92.9% and an f1 score of 0.89 for binary stress classification. Our leave-one-subject-out analysis establishes the subjective nature of stress and shows that personalizing stress models using Stressalyzer significantly improves the model performance. Without model personalization, we found a performance decline in 40% of the subjects, suggesting the need for model personalization.

Personalized PPG Normalization Based on Subject Heartbeat in Resting State Condition

Physiological responses are currently widely used to recognize the affective state of subjects in real-life scenarios. However, these data are intrinsically subject-dependent, making machine learning techniques for data classification not easily applicable due to inter-subject variability. In this work, the reduction of inter-subject heterogeneity was considered in the case of Photoplethysmography (PPG), which was successfully used to detect stress and evaluate experienced cognitive load. To face the inter-subject heterogeneity, a novel personalized PPG normalization is herein proposed. A subject-normalized discrete domain where the PPG signals are properly re-scaled is introduced, considering the subject’s heartbeat frequency in resting state conditions. The effectiveness of the proposed normalization was evaluated in comparison to other normalization procedures in a binary classification task, where cognitive load and relaxed state were considered. The results obtained on two different datasets available in the literature confirmed that applying the proposed normalization strategy permitted increasing the classification performance.

Photoplethysmogram Analysis and Applications: An Integrative Review

Beyond its use in a clinical environment, photoplethysmogram (PPG) is increasingly used for measuring the physiological state of an individual in daily life. This review aims to examine existing research on photoplethysmogram concerning its generation mechanisms, measurement principles, clinical applications, noise definition, pre-processing techniques, feature detection techniques, and post-processing techniques for photoplethysmogram processing, especially from an engineering point of view. We performed an extensive search with the PubMed, Google Scholar, Institute of Electrical and Electronics Engineers (IEEE), ScienceDirect, and Web of Science databases. Exclusion conditions did not include the year of publication, but articles not published in English were excluded. Based on 118 articles, we identified four main topics of enabling PPG: (A) PPG waveform, (B) PPG features and clinical applications including basic features based on the original PPG waveform, combined features of PPG, and derivative features of PPG, (C) PPG noise including motion artifact baseline wandering and hypoperfusion, and (D) PPG signal processing including PPG preprocessing, PPG peak detection, and signal quality index. The application field of photoplethysmogram has been extending from the clinical to the mobile environment. Although there is no standardized pre-processing pipeline for PPG signal processing, as PPG data are acquired and accumulated in various ways, the recently proposed machine learning-based method is expected to offer a promising solution.

CN-waterfall: a deep convolutional neural network for multimodal physiological affect detection

Affective computing solutions, in the literature, mainly rely on machine learning methods designed to accurately detect human affective states. Nevertheless, many of the proposed methods are based on handcrafted features, requiring sufficient expert knowledge in the realm of signal processing. With the advent of deep learning methods, attention has turned toward reduced feature engineering and more end-to-end machine learning. However, most of the proposed models rely on late fusion in a multimodal context. Meanwhile, addressing interrelations between modalities for intermediate-level data representation has been largely neglected. In this paper, we propose a novel deep convolutional neural network, called CN-Waterfall, consisting of two modules: Base and General. While the Base module focuses on the low-level representation of data from each single modality, the General module provides further information, indicating relations between modalities in the intermediate- and high-level data representations. The latter module has been designed based on theoretically grounded concepts in the Explainable AI (XAI) domain, consisting of four different fusions. These fusions are mainly tailored to correlation- and non-correlation-based modalities. To validate our model, we conduct an exhaustive experiment on WESAD and MAHNOB-HCI, two publicly and academically available datasets in the context of multimodal affective computing. We demonstrate that our proposed model significantly improves the performance of physiological-based multimodal affect detection.

Towards a Contactless Stress Classification Using Thermal Imaging

Thermal cameras capture the infrared radiation emitted from a body in a contactless manner and can provide an indirect estimation of the autonomic nervous system (ANS) dynamics through the regulation of the skin temperature. This study investigates the contribution given by thermal imaging for an effective automatic stress detection with the perspective of a contactless stress recognition system. To this aim, we recorded both ANS correlates (cardiac, electrodermal, and respiratory activity) and thermal images from 25 volunteers under acute stress induced by the Stroop test. We conducted a statistical analysis on the features extracted from each signal, and we implemented subject-independent classifications based on the support vector machine model with an embedded recursive feature elimination algorithm. Particularly, we trained three classifiers using different feature sets: the full set of features, only those derived from the peripheral autonomic correlates, and only those derived from the thermal images. Classification accuracy and feature selection results confirmed the relevant contribution provided by the thermal features in the acute stress detection task. Indeed, a combination of ANS correlates and thermal features achieved 97.37% of accuracy. Moreover, using only thermal features we could still successfully detect stress with an accuracy of 86.84% in a contact-free manner.

Measuring posture change to detect emotional conditions for workers: A scoping review

BACKGROUND

The emotional management of workers can not only increase the efficiency of work, but also contribute to the improvement of the productivity of a company.

OBJECTIVE

This scoping review surveyed the literature to identify the relationship between postural expression and emotion during sedentary tasks.

METHODS

We searched relevant literature published up to December 1, 2019 using seven electronic databases (PubMed, CINAHL, Embase, Web of Science, PsycINFO, IEEE Xplore, and MEDLINE Complete).

RESULTS

A total of 14 publications were included in this scoping review. It was found that the application of pressure sensor and camera-based measurement equipment was effective. Additionally, it was proposed to predict the emotional state of the worker by using forward and backward movements as the main variable as opposed to left and right movements. The information-based analysis technique was able to further increase the accuracy of workers' emotion prediction.

CONCLUSIONS

The emotion prediction of workers based on sitting posture could be confirmed for certain movements, and the information-based technical method could further increase the accuracy of prediction. Expansion of information-based technical research will further increase the possibility of predicting the emotions of workers based on posture, and this will in turn promote safer and more efficient work performance.

A Wearable Head Mounted Display Bio-Signals Pad System for Emotion Recognition

Multimodal bio-signals acquisition based on wearable devices and using virtual reality (VR) as stimulus source are promising techniques in emotion recognition research field. Numerous studies have shown that emotional states can be better evoked through Immersive Virtual Environments (IVE). The main goal of this paper is to provide researchers with a system for emotion recognition in VR environments. In this paper, we present a wearable forehead bio-signals acquisition pad which is attached to Head-Mounted Displays (HMD), termed HMD Bio Pad. This system can simultaneously record emotion-related two-channel electroencephalography (EEG), one-channel electrodermal activity (EDA), photoplethysmograph (PPG) and skin temperature (SKT) signals. In addition, we develop a human-computer interaction (HCI) interface which researchers can carry out emotion recognition research using VR HMD as stimulus presentation device. To evaluate the performance of the proposed system, we conducted different experiments to validate the multimodal bio-signals quality, respectively. To validate EEG signal, we have assessed the performance in terms of EEG eyes-blink task and eyes-open and eyes-closed task. The EEG eyes-blink task indicates that the proposed system can achieve comparable EEG signal quality in comparison to the dedicated bio-signals measuring device. The eyes-open and eyes-closed task proves that the proposed system can efficiently record alpha rhythm. Then we used signal-to-noise ratio (SNR) and Skin Conductance Reaction (SCR) signal to validate the performance for EDA acquisition system. A filtered EDA signal, with a high mean SNR of 28.52 dB, is plotted on HCI interface. Moreover, the SCR signal related to stimulus response can be correctly extracted from EDA signal. The SKT acquisition system has been validated effectively by the temperature change experiment when subjects are in unpleasant emotion. The pulse rate (PR) estimated from PPG signal achieved the low mean average absolute error (AAE), which is 1.12 beats per minute (BPM) over 8 recordings. In summary, the proposed HMD Bio Pad offers a portable, comfortable and easy-to-wear device for recording bio-signals. The proposed system could contribute to emotion recognition research in VR environments.

Automated Detection of Stressful Conversations Using Wearable Physiological and Inertial Sensors

Stressful conversation is a frequently occurring stressor in our daily life. Stressors not only adversely affect our physical and mental health but also our relationships with family, friends, and coworkers. In this paper, we present a model to automatically detect stressful conversations using wearable physiological and inertial sensors. We conducted a lab and a field study with cohabiting couples to collect ecologically valid sensor data with temporally-precise labels of stressors. We introduce the concept of stress cycles, i.e., the physiological arousal and recovery, within a stress event. We identify several novel features from stress cycles and show that they exhibit distinguishing patterns during stressful conversations when compared to physiological response due to other stressors. We observe that hand gestures also show a distinct pattern when stress occurs due to stressful conversations. We train and test our model using field data collected from 38 participants. Our model can determine whether a detected stress event is due to a stressful conversation with an F1-score of 0.83, using features obtained from only one stress cycle, facilitating intervention delivery within 3.9 minutes since the start of a stressful conversation.

Evaluating the Reproducibility of Physiological Stress Detection Models

Recent advances in wearable sensor technologies have led to a variety of approaches for detecting physiological stress. Even with over a decade of research in the domain, there still exist many significant challenges, including a near-total lack of reproducibility across studies. Researchers often use some physiological sensors (custom-made or off-the-shelf), conduct a study to collect data, and build machine-learning models to detect stress. There is little effort to test the applicability of the model with similar physiological data collected from different devices, or the efficacy of the model on data collected from different studies, populations, or demographics. This paper takes the first step towards testing reproducibility and validity of methods and machine-learning models for stress detection. To this end, we analyzed data from 90 participants, from four independent controlled studies, using two different types of sensors, with different study protocols and research goals. We started by evaluating the performance of models built using data from one study and tested on data from other studies. Next, we evaluated new methods to improve the performance of stress-detection models and found that our methods led to a consistent increase in performance across all studies, irrespective of the device type, sensor type, or the type of stressor. Finally, we developed and evaluated a clustering approach to determine the stressed/not-stressed classification when applying models on data from different studies, and found that our approach performed better than selecting a threshold based on training data. This paper's thorough exploration of reproducibility in a controlled environment provides a critical foundation for deeper study of such methods, and is a prerequisite for tackling reproducibility in free-living conditions.

GIS-Based Emotional Computing: A Review of Quantitative Approaches to Measure the Emotion Layer of Human–Environment Relationships

In recent years, with the growing accessibility of abundant contextual emotion information, which is benefited by the numerous georeferenced user-generated content and the maturity of artificial intelligence (AI)-based emotional computing technics, the emotion layer of human–environment relationship is proposed for enriching traditional methods of various related disciplines such as urban planning. This paper proposes the geographic information system (GIS)-based emotional computing concept, which is a novel framework for applying GIS methods to collective human emotion. The methodology presented in this paper consists of three key steps: (1) collecting georeferenced data containing emotion and environment information such as social media and official sites, (2) detecting emotions using AI-based emotional computing technics such as natural language processing (NLP) and computer vision (CV), and (3) visualizing and analyzing the spatiotemporal patterns with GIS tools. This methodology is a great synergy of multidisciplinary cutting-edge techniques, such as GIScience, sociology, and computer science. Moreover, it can effectively and deeply explore the connection between people and their surroundings with the help of GIS methods. Generally, the framework provides a standard workflow to calculate and analyze the new information layer for researchers, in which a measured human-centric perspective onto the environment is possible.

Stress detection using ECG and EMG signals: A comprehensive study

BACKGROUND AND OBJECTIVE

In recent years, stress and mental health have been considered as important worldwide concerns. Stress detection using physiological signals such as electrocardiogram (ECG), skin conductance (SC), electromyogram (EMG) and electroencephalogram (EEG) is a traditional approach. However, the effect of stress on the EMG signal of different muscles and the efficacy of combination of the EMG and other biological signals for stress detection have not been taken into account yet. This paper presents a comprehensive review of the EMG signal of the right and left trapezius and right and left erector spinae muscles for multi-level stress recognition. Also, the ECG signal was employed to evaluate the efficacy of EMG signals for stress detection.

METHODS

Both EMG and ECG signals were acquired simultaneously from 34 healthy students (23 females and 11 males, aged 20-37 years). Mental arithmetic, Stroop color-word test, time pressure, and stressful environment were employed to induce stress in the laboratory.

RESULTS

The accuracies of stress recognition in two, three and four levels were 100%, 97.6%, and 96.2%, respectively, obtained from the distinct combination of feature selection and machine learning algorithms.

CONCLUSIONS

The comparison of stress detection accuracies resulted from EMG and ECG indicators demonstrated the strong ability and the effectiveness of EMG signal for multi-level stress detection.

Combining Bioimpedance and Myographic Signals for the Assessment of COPD During Loaded Breathing

Chronic Obstructive Pulmonary Disease (COPD) is one of the most common chronic conditions. The current assessment of COPD requires a maximal maneuver during a spirometry test to quantify airflow limitations of patients. Other less invasive measurements such as thoracic bioimpedance and myographic signals have been studied as an alternative to classical methods as they provide information about respiration. Particularly, strong correlations have been shown between thoracic bioimpedance and respiratory volume. The main objective of this study is to investigate bioimpedance and its combination with myographic parameters in COPD patients to assess the applicability in respiratory disease monitoring. We measured bioimpedance, surface electromyography and surface mechanomyography in forty-three COPD patients during an incremental inspiratory threshold loading protocol. We introduced two novel features that can be used to assess COPD condition derived from the variation of bioimpedance and the electrical and mechanical activity during each respiratory cycle. These features demonstrate significant differences between mild and severe patients, indicating a lower inspiratory contribution of the inspiratory muscles to global respiratory ventilation in the severest COPD patients. In conclusion, the combination of bioimpedance and myographic signals provides useful indices to noninvasively assess the breathing of COPD patients.

Deep Support Vector Machines for the Identification of Stress Condition from Electrodermal Activity

Early detection of stress condition is beneficial to prevent long-term mental illness like depression and anxiety. This paper introduces an accurate identification of stress/calm condition from electrodermal activity (EDA) signals. The acquisition of EDA signals from a commercial wearable as well as their storage and processing are presented. Several time-domain, frequency-domain and morphological features are extracted over the skin conductance response of the EDA signals. Afterwards, a classification is undergone by using several classical support vector machines (SVMs) and deep support vector machines (D-SVMs). In addition, several binary classifiers are also compared with SVMs in the stress/calm identification task. Moreover, a series of video clips evoking calm and stress conditions have been viewed by 147 volunteers in order to validate the classification results. The highest F1-score obtained for SVMs and D-SVMs are 83% and 92%, respectively. These results demonstrate that not only classical SVMs are appropriate for classification of biomarker signals, but D-SVMs are very competitive in comparison to other classification techniques. In addition, the results have enabled drawing useful considerations for the future use of SVMs and D-SVMs in the specific case of stress/calm identification.

StressFoot: Uncovering the Potential of the Foot for Acute Stress Sensing in Sitting Posture

Stress is a naturally occurring psychological response and identifiable by several body signs. We propose a novel way to discriminate acute stress and relaxation, using movement and posture characteristics of the foot. Based on data collected from 23 participants performing tasks that induced stress and relaxation, we developed several machine learning models to construct the validity of our method. We tested our models in another study with 11 additional participants. The results demonstrated replicability with an overall accuracy of 87%. To also demonstrate external validity, we conducted a field study with 10 participants, performing their usual everyday office tasks over a working day. The results showed substantial robustness. We describe ten significant features in detail to enable an easy replication of our models.

Emotional State Recognition from Peripheral Physiological Signals Using Fused Nonlinear Features and Team-Collaboration Identification Strategy

Emotion recognition realizing human inner perception has a very important application prospect in human-computer interaction. In order to improve the accuracy of emotion recognition, a novel method combining fused nonlinear features and team-collaboration identification strategy was proposed for emotion recognition using physiological signals. Four nonlinear features, namely approximate entropy (ApEn), sample entropy (SaEn), fuzzy entropy (FuEn) and wavelet packet entropy (WpEn) are employed to reflect emotional states deeply with each type of physiological signal. Then the features of different physiological signals are fused to represent the emotional states from multiple perspectives. Each classifier has its own advantages and disadvantages. In order to make full use of the advantages of other classifiers and avoid the limitation of single classifier, the team-collaboration model is built and the team-collaboration decision-making mechanism is designed according to the proposed team-collaboration identification strategy which is based on the fusion of support vector machine (SVM), decision tree (DT) and extreme learning machine (ELM). Through analysis, SVM is selected as the main classifier with DT and ELM as auxiliary classifiers. According to the designed decision-making mechanism, the proposed team-collaboration identification strategy can effectively employ different classification methods to make decision based on the characteristics of the samples through SVM classification. For samples which are easy to be identified by SVM, SVM directly determines the identification results, whereas SVM-DT-ELM collaboratively determines the identification results, which can effectively utilize the characteristics of each classifier and improve the classification accuracy. The effectiveness and universality of the proposed method are verified by Augsburg database and database for emotion analysis using physiological (DEAP) signals. The experimental results uniformly indicated that the proposed method combining fused nonlinear features and team-collaboration identification strategy presents better performance than the existing methods.

The uulmMAC Database-A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional and cognitive load states. It consists of six experimental sequences, inducing Interest, Overload, Normal, Easy, Underload, and Frustration. Each sequence is followed by subjective feedbacks to validate the induction, a respiration baseline to level off the physiological reactions, and a summary of results. Further, prior to the experiment, three questionnaires related to emotion regulation (ERQ), emotional control (TEIQue-SF), and personality traits (TIPI) were collected from each subject to evaluate the stability of the induction paradigm. Based on this HCI scenario, the University of Ulm Multimodal Affective Corpus (uulmMAC), consisting of two homogenous samples of 60 participants and 100 recording sessions was generated. We recorded 16 sensor modalities including 4 × video, 3 × audio, and 7 × biophysiological, depth, and pose streams. Further, additional labels and annotations were also collected. After recording, all data were post-processed and checked for technical and signal quality, resulting in the final uulmMAC dataset of 57 subjects and 95 recording sessions. The evaluation of the reported subjective feedbacks shows significant differences between the sequences, well consistent with the induced states, and the analysis of the questionnaires shows stable results. In summary, our uulmMAC database is a valuable contribution for the field of affective computing and multimodal data analysis: Acquired in a mobile interactive scenario close to real HCI, it consists of a large number of subjects and allows transtemporal investigations. Validated via subjective feedbacks and checked for quality issues, it can be used for affective computing and machine learning applications.

Continuous Detection of Physiological Stress with Commodity Hardware

Timely detection of an individual's stress level has the potential to improve stress management, thereby reducing the risk of adverse health consequences that may arise due to mismanagement of stress. Recent advances in wearable sensing have resulted in multiple approaches to detect and monitor stress with varying levels of accuracy. The most accurate methods, however, rely on clinical-grade sensors to measure physiological signals; they are often bulky, custom made, and expensive, hence limiting their adoption by researchers and the general public. In this article, we explore the viability of commercially available off-the-shelf sensors for stress monitoring. The idea is to be able to use cheap, nonclinical sensors to capture physiological signals and make inferences about the wearer's stress level based on that data. We describe a system involving a popular off-the-shelf heart rate monitor, the Polar H7; we evaluated our system with 26 participants in both a controlled lab setting with three well-validated stress-inducing stimuli and in free-living field conditions. Our analysis shows that using the off-the-shelf sensor alone, we were able to detect stressful events with an F1-score of up to 0.87 in the lab and 0.66 in the field, on par with clinical-grade sensors.

Stress Detection via Keyboard Typing Behaviors by Using Smartphone Sensors and Machine Learning Techniques

Stress is one of the biggest problems in modern society. It may not be possible for people to perceive if they are under high stress or not. It is important to detect stress early and unobtrusively. In this context, stress detection can be considered as a classification problem. In this study, it was investigated the effects of stress by using accelerometer and gyroscope sensor data of the writing behavior on a smartphone touchscreen panel. For this purpose, smartphone data including two states (stress and calm) were collected from 46 participants. The obtained sensor signals were divided into 5, 10 and 15 s interval windows to create three different data sets and 112 different features were defined from the raw data. To obtain more effective feature subsets, these features were ranked by using Gain Ratio feature selection algorithm. Afterwards, writing behaviors were classified by C4.5 Decision Trees, Bayesian Networks and k-Nearest Neighbor methods. As a result of the experiments, 74.26%, 67.86%, and 87.56% accuracy classification results were obtained respectively.

Human Emotion Recognition: Review of Sensors and Methods

Automated emotion recognition (AEE) is an important issue in various fields of activities which use human emotional reactions as a signal for marketing, technical equipment, or human-robot interaction. This paper analyzes scientific research and technical papers for sensor use analysis, among various methods implemented or researched. This paper covers a few classes of sensors, using contactless methods as well as contact and skin-penetrating electrodes for human emotion detection and the measurement of their intensity. The results of the analysis performed in this paper present applicable methods for each type of emotion and their intensity and propose their classification. The classification of emotion sensors is presented to reveal area of application and expected outcomes from each method, as well as their limitations. This paper should be relevant for researchers using human emotion evaluation and analysis, when there is a need to choose a proper method for their purposes or to find alternative decisions. Based on the analyzed human emotion recognition sensors and methods, we developed some practical applications for humanizing the Internet of Things (IoT) and affective computing systems.

A State-Space Approach for Detecting Stress from Electrodermal Activity

The human body responds to neurocognitive stress in multiple ways through its autonomic nervous system. Increases in heart rate, salivary cortisol and skin conductance level are often observed accompanying high levels of stress. Stress can also take on different forms including emotional, cognitive and motivational. While a precise definition for stress is lacking, a pertinent issue is to quantify the state of psychological stress manifested in the nervous system. State-space models have previously been applied to estimate an unobserved neural state (e.g. learning, consciousness) from physiological signal measurements and data collected during behavioral experiments. In this paper, we relate stress to the probability that a phasic driver impulse occurs in skin conductance signals. We apply state-space modeling to extracted binary measures to continuously track a stress level across episodes of cognitive and emotional stress as well as relaxation. Results demonstrate a promising approach for tracking stress through wearable devices.

Wearable-Based Affect Recognition-A Review

Affect recognition is an interdisciplinary research field bringing together researchers from natural and social sciences. Affect recognition research aims to detect the affective state of a person based on observables, with the goal to, for example, provide reasoning for the person's decision making or to support mental wellbeing (e.g., stress monitoring). Recently, beside of approaches based on audio, visual or text information, solutions relying on wearable sensors as observables, recording mainly physiological and inertial parameters, have received increasing attention. Wearable systems enable an ideal platform for long-term affect recognition applications due to their rich functionality and form factor, while providing valuable insights during everyday life through integrated sensors. However, existing literature surveys lack a comprehensive overview of state-of-the-art research in wearable-based affect recognition. Therefore, the aim of this paper is to provide a broad overview and in-depth understanding of the theoretical background, methods and best practices of wearable affect and stress recognition. Following a summary of different psychological models, we detail the influence of affective states on the human physiology and the sensors commonly employed to measure physiological changes. Then, we outline lab protocols eliciting affective states and provide guidelines for ground truth generation in field studies. We also describe the standard data processing chain and review common approaches related to the preprocessing, feature extraction and classification steps. By providing a comprehensive summary of the state-of-the-art and guidelines to various aspects, we would like to enable other researchers in the field to conduct and evaluate user studies and develop wearable systems.

micro-Stress EMA: A Passive Sensing Framework for Detecting in-the-wild Stress in Pregnant Mothers

High levels of stress during pregnancy increase the chances of having a premature or low-birthweight baby. Perceived self-reported stress does not often capture or align with the physiological and behavioral response. But what if there was a self-report measure that could better capture the physiological response? Current perceived stress self-report assessments require users to answer multi-item scales at different time points of the day. Reducing it to one question, using microinteraction-based ecological momentary assessment (micro-EMA, collecting a single in situ self-report to assess behaviors) allows us to identify smaller or more subtle changes in physiology. It also allows for more frequent responses to capture perceived stress while at the same time reducing burden on the participant. We propose a framework for selecting the optimal micro-EMA that combines unbiased feature selection and unsupervised Agglomerative clustering. We test our framework in 18 women performing 16 activities in-lab wearing a Biostamp, a NeuLog, and a Polar chest strap. We validated our results in 17 pregnant women in real-world settings. Our framework shows that the question "How worried were you?" results in the highest accuracy when using a physiological model. Our results provide further in-depth exposure to the challenges of evaluating stress models in real-world situations.

A Novel Wearable EEG and ECG Recording System for Stress Assessment

Suffering from continuous stress can lead to serious psychological and even physical disorders. Objective stress assessment methods using noninvasive physiological responses such as heart rate variability (HRV) and electroencephalograms (EEG) have therefore been proposed for effective stress management. In this study, a novel wearable device that can measure electrocardiograms (ECG) and EEG simultaneously was designed to enable continuous stress monitoring in daily life. The developed system is easily worn by hanging from both ears, is lightweight (i.e., 42.5 g), and exhibits an excellent noise performance of 0.12 μVrms. Significant time and frequency features of HRV and EEG were found in two different stressors, namely the Stroop color word and mental arithmetic tests, using 14 young subjects. Stressor situations were classified using various HRV and EEG feature selections and a support vector machine technique. The five-fold cross-validation results obtained when using both EEG and HRV features showed the best performance with an accuracy of 87.5%, which demonstrated the requirement for simultaneous HRV and EEG measurements.

Predicting Emotion and Engagement of Workers in Order Picking Based on Behavior and Pulse Waves Acquired by Wearable Devices

Many logistics companies adopt a manual order picking system. In related research, the effect of emotion and engagement on work efficiency and human errors was verified. However, related research has not established a method to predict emotion and engagement during work with high exercise intensity. Therefore, important variables for predicting the emotion and engagement during work with high exercise intensity are not clear. In this study, to clarify the mechanism of occurrence of emotion and engagement during order picking. Then, we clarify the explanatory variables which are important in predicting the emotion and engagement during work with high exercise intensity. We conducted verification experiments. We compared the accuracy of estimating human emotion and engagement by inputting pulse wave, eye movements, and movements to deep neural networks. We showed that emotion and engagement during order picking can be predicted from the behavior of the worker with an accuracy of error rate of 0.12 or less. Moreover, we have constructed a psychological model based on the questionnaire results and show that the work efficiency of workers is improved by giving them clear targets.

ECG-Based Concentration Recognition With Multi-Task Regression

OBJECTIVE

Recognition of human activities and mental states using wearable sensors and smartphones has attracted considerable attention recently. In particular, prediction of the stress level of a subject using an electrocardiogram sensor has been studied extensively. In this paper, we attempt to predict the degree of concentration by using heart-rate features. However, due to strong diversity in individuals and high sampling costs, building an accurate prediction model is still highly challenging.

METHOD

To overcome these difficulties, we propose to use a multitask learning (MTL) technique for effectively sharing information among similar individuals.

RESULT

Through experiments with 18 healthy subjects performing daily office works, such as writing reports, we demonstrate that the proposed method significantly improves the accuracy of concentration prediction in small sample situations.

CONCLUSION

The performance of the MTL method is shown to be stable across different subjects, which is an important advantage over conventional models.

SIGNIFICANCE

This improvement has significant impact in real-world concentration recognition because the data collection burden of each user can be drastically mitigated.

Quantitative Analysis of the Effect of an Ectopic Beat on the Heart Rate Variability in the Resting Condition

The purpose of this study is to quantitatively analyze the effect of an ectopic beat on heart rate variability (HRV) in the time domain, frequency domain, and in a non-linear analysis. A quantitative analysis was carried out by generating artificial ectopic beats that probabilistically contained a missed beat or a false-detected beat, and the statistical significance was evaluated though a comparison with an ectopic-free HRV by increasing the ratio of the ectopic beat in 0.1% increments from 0 to 50%. The effect of the interpolation on the ectopic HRV was also investigated by applying nearest-neighbor interpolation, linear interpolation, and cubic spline interpolation. The results confirmed a statistically significant difference (P < 0.05) even in the less-than-1% ectopic HRV in every domain. When interpolation was applied, there were differences according to the interpolation method used, but statistical significance was secured for an ectopic beat ratio from 1 to 2% to several tens of a percent. In the effect, linear interpolation, and spline interpolation were confirmed to have a higher effect on the high-frequency related HRV variables, and nearest-neighbor interpolation had a higher effect on low-frequency related variables.

A Distributed Model for Stressors Monitoring Based on Environmental Smart Sensors

Nowadays, in many countries, stress is becoming a problem that increasingly affects the health of people. Suffering stress continuously can lead to serious behavioral disorders such as anxiety or depression. Every person, in his daily routine, can face many factors which can contribute to increase his stress level. This paper describes a flexible and distributed model to monitor environmental variables associated with stress, which provides adaptability to any environment in an agile way. This model was designed to transform stress environmental variables in value added information (key stress indicator) and to provide it to external systems, in both proactive and reactive mode. Thus, this value-added information will assist organizations and users in a personalized way helping in the detection and prevention of acute stress cases. Our proposed model is supported by an architecture that achieves the features above mentioned, in addition to interoperability, robustness, scalability, autonomy, efficient, low cost and consumption, and information availability in real time. Finally, a prototype of the system was implemented, allowing the validation of the proposal in different environments at the University of Alicante.

Interpretation of physiological indicators of motivation: Caveats and recommendations

Motivation scientists employing physiological measures to gather information about motivation-related states are at risk of committing two fundamental errors: overstating the inferences that can be drawn from their physiological measures and circular reasoning. We critically discuss two complementary approaches, Cacioppo and colleagues' model of psychophysiological relations and construct validation theory, to highlight the conditions under which these errors are committed and provide guidance on how to avoid them. In particular, we demonstrate that the direct inference from changes in a physiological measure to changes in a motivation-related state requires the demonstration that the measure is not related to other relevant psychological states. We also point out that circular reasoning can be avoided by separating the definition of the motivation-related state from the hypotheses that are empirically tested.