micro-Stress EMA: A Passive Sensing Framework for Detecting in-the-wild Stress in Pregnant Mothers

Wearable network for multilevel physical fatigue prediction in manufacturing workers

Manufacturing workers face prolonged strenuous physical activities, impacting both financial aspects and their health due to work-related fatigue. Continuously monitoring physical fatigue and providing meaningful feedback is crucial to mitigating human and monetary losses in manufacturing workplaces. This study introduces a novel application of multimodal wearable sensors and machine learning techniques to quantify physical fatigue and tackle the challenges of real-time monitoring on the factory floor. Unlike past studies that view fatigue as a dichotomous variable, our central formulation revolves around the ability to predict multilevel fatigue, providing a more nuanced understanding of the subject's physical state. Our multimodal sensing framework is designed for continuous monitoring of vital signs, including heart rate, heart rate variability, skin temperature, and more, as well as locomotive signs by employing inertial motion units strategically placed at six locations on the upper body. This comprehensive sensor placement allows us to capture detailed data from both the torso and arms, surpassing the capabilities of single-point data collection methods. We developed an innovative asymmetric loss function for our machine learning model, which enhances prediction accuracy for numerical fatigue levels and supports real-time inference. We collected data on 43 subjects following an authentic manufacturing protocol and logged their self-reported fatigue. Based on the analysis, we provide insights into our multilevel fatigue monitoring system and discuss results from an in-the-wild evaluation of actual operators on the factory floor. This study demonstrates our system's practical applicability and contributes a valuable open-access database for future research.

Recent Progress in Biosensors for Depression Monitoring-Advancing Personalized Treatment

Depression is currently a major contributor to unnatural deaths and the healthcare burden globally, and a patient's battle with depression is often a long one. Because the causes, symptoms, and effects of medications are complex and highly individualized, early identification and personalized treatment of depression are key to improving treatment outcomes. The development of wearable electronics, machine learning, and other technologies in recent years has provided more possibilities for the realization of this goal. Conducting regular monitoring through biosensing technology allows for a more comprehensive and objective analysis than previous self-evaluations. This includes identifying depressive episodes, distinguishing somatization symptoms, analyzing etiology, and evaluating the effectiveness of treatment programs. This review summarizes recent research on biosensing technologies for depression. Special attention is given to technologies that can be portable or wearable, with the potential to enable patient use outside of the hospital, for long periods.

Investigating Receptivity and Affect Using Machine Learning: Ecological Momentary Assessment and Wearable Sensing Study

BACKGROUND

As mobile health (mHealth) studies become increasingly productive owing to the advancements in wearable and mobile sensor technology, our ability to monitor and model human behavior will be constrained by participant receptivity. Many health constructs are dependent on subjective responses, and without such responses, researchers are left with little to no ground truth to accompany our ever-growing biobehavioral data. This issue can significantly impact the quality of a study, particularly for populations known to exhibit lower compliance rates. To address this challenge, researchers have proposed innovative approaches that use machine learning (ML) and sensor data to modify the timing and delivery of surveys. However, an overarching concern is the potential introduction of biases or unintended influences on participants' responses when implementing new survey delivery methods.

OBJECTIVE

This study aims to demonstrate the potential impact of an ML-based ecological momentary assessment (EMA) delivery system (using receptivity as the predictor variable) on the participants' reported emotional state. We examine the factors that affect participants' receptivity to EMAs in a 10-day wearable and EMA-based emotional state-sensing mHealth study. We study the physiological relationships indicative of receptivity and affect while also analyzing the interaction between the 2 constructs.

METHODS

We collected data from 45 healthy participants wearing 2 devices measuring electrodermal activity, accelerometer, electrocardiography, and skin temperature while answering 10 EMAs daily, containing questions about perceived mood. Owing to the nature of our constructs, we can only obtain ground truth measures for both affect and receptivity during responses. Therefore, we used unsupervised and supervised ML methods to infer affect when a participant did not respond. Our unsupervised method used k-means clustering to determine the relationship between physiology and receptivity and then inferred the emotional state during nonresponses. For the supervised learning method, we primarily used random forest and neural networks to predict the affect of unlabeled data points as well as receptivity.

RESULTS

Our findings showed that using a receptivity model to trigger EMAs decreased the reported negative affect by >3 points or 0.29 SDs in our self-reported affect measure, scored between 13 and 91. The findings also showed a bimodal distribution of our predicted affect during nonresponses. This indicates that this system initiates EMAs more commonly during states of higher positive emotions.

CONCLUSIONS

Our results showed a clear relationship between affect and receptivity. This relationship can affect the efficacy of an mHealth study, particularly those that use an ML algorithm to trigger EMAs. Therefore, we propose that future work should focus on a smart trigger that promotes EMA receptivity without influencing affect during sampled time points.

K-EmoPhone: A Mobile and Wearable Dataset with In-Situ Emotion, Stress, and Attention Labels

With the popularization of low-cost mobile and wearable sensors, several studies have used them to track and analyze mental well-being, productivity, and behavioral patterns. However, there is still a lack of open datasets collected in real-world contexts with affective and cognitive state labels such as emotion, stress, and attention; the lack of such datasets limits research advances in affective computing and human-computer interaction. This study presents K-EmoPhone, a real-world multimodal dataset collected from 77 students over seven days. This dataset contains (1) continuous probing of peripheral physiological signals and mobility data measured by commercial off-the-shelf devices, (2) context and interaction data collected from individuals' smartphones, and (3) 5,582 self-reported affect states, including emotions, stress, attention, and task disturbance, acquired by the experience sampling method. We anticipate the dataset will contribute to advancements in affective computing, emotion intelligence technologies, and attention management based on mobile and wearable sensor data.

Perception of Whole Day Workload as a Mediator Between Activity Engagement and Stress in Workers with Type 1 Diabetes

Associations between various forms of activity engagement (e.g. work, leisure) and the experience of stress in workers have been widely documented. The mechanisms underlying these effects, however, are not fully understood. Our goal was to investigate if perceived whole day workload accounted for the relationships between daily frequencies of activities (i.e. work hours and leisure/rest) and daily stress. We analyzed data from 56 workers with type 1 diabetes (T1D) who completed approximately two weeks of intensive longitudinal assessments. Daily whole day workload was measured with an adapted version of the National Aeronautics and Space Administration Task Load Index (NASA-TLX). A variety of occupations were reported including lawyer, housekeeper, and teacher. In multilevel path analyses, day-to-day changes in whole day workload mediated 67% (p<.001), 61% (p<.001), 38% (p<.001), and 55% (p<.001) of the within-person relationships between stress and work hours, rest frequency, active leisure frequency, and day of week, respectively. Our results provided evidence that whole day workload perception may contribute to the processes linking daily activities with daily stress in workers with T1D. Perceived whole day workload may deserve greater attention as a possible stress intervention target, ones that perhaps ergonomists would be especially suited to address.

A psycho-educational intervention programme for parents with SGA foetuses supported by an adaptive mHealth system: design, proof of concept and usability assessment

Background

Technology-based approaches during pregnancy can facilitate the self-reporting of emotional health issues and improve well-being. There is evidence to suggest that stress during pregnancy can affect the foetus and result in restricted growth and preterm birth. Although a number of mobile health (mHealth) approaches are designed to monitor pregnancy and provide information about a specific aspect, no proposal specifically addresses the interventions in parents at risk of having small-for-gestational-age (SGA) or premature babies. Very few studies, however, follow any design and usability guidelines which aim to ensure end-user satisfaction when using these systems.

Results

We have developed an interactive, adaptable mHealth system to support a psycho-educational intervention programme for parents with SGA foetuses. The relevant results include a metamodel to support the task of modelling current or new intervention programmes, an mHealth system model with runtime adaptation to changes in the programme, the design of a usable app (called VivEmbarazo) and an architectural design and prototype implementation. The developed mHealth system has also enabled us to conduct a proof of concept based on the use of the mHealth systems and this includes data analysis and assesses usability and acceptance.

Conclusions

The proof of concept confirms that parents are satisfied and that they are enthusiastic about the mHealth-supported intervention programme. It helps to technically validate the results obtained in the other stages relating to the development of the solution. The data analysis resulting from the proof of concept confirms that the stress experienced by parents who followed the mHealth-supported intervention programme was significantly lower than among those who did not follow it. This implies an improvement in the emotional health not only of the parents but also of their child. In fact, the babies of couples who followed the mHealth-supported programme weigh more than the babies of couples under traditional care. In terms of user acceptance and usability, the analysis confirms that mothers place greater value on the app design, usefulness and ease of use and are generally more satisfied than their partners. Although these results are promising in comparison with more traditional and other more recent technology-based approaches.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-02036-9.

Feasibility of Passive ECG Bio-sensing and EMA Emotion Reporting Technologies and Acceptability of Just-in-Time Content in a Well-being Intervention, Considerations for Scalability and Improved Uptake

Researchers increasingly use passive sensing data and frequent self-report to implement personalized mobile health (mHealth) interventions. Yet, we know that certain populations may find these technical protocols burdensome and intervention uptake as well as treatment efficacy may be affected as a result. In the present study, we predicted feasibility (participant adherence to protocol) and acceptability (participant engagement with intervention content) as a function of baseline sociodemographic, mental health, and well-being characteristics of 99 women randomized in the personalized preventive intervention Wellness-for-Two (W-4-2), a randomized trial evaluating stress-related alterations during pregnancy and their effect on infant neurodevelopmental trajectories. The W-4-2 study used ecological momentary assessment (EMA) and wearable electrocardiograph (ECG) sensors to detect physiological stress and personalize the intervention. Participant adherence to protocols was 67% for EMAs and 52% for ECG bio-sensors. Higher baseline negative affect significantly predicted lower adherence to both protocols. Women assigned to the intervention group engaged on average with 42% of content they received. Women with higher annual household income were more likely to engage with more of the intervention content. Researchers should carefully consider tailoring of the intensity of technical intervention protocols to reduce fatigue, especially among participants with higher baseline negative affect, which may improve intervention uptake and efficacy findings at scale.

Predicting the Next-Day Perceived and Physiological Stress of Pregnant Women by Using Machine Learning and Explainability: Algorithm Development and Validation

Background

Cognitive behavioral therapy–based interventions are effective in reducing prenatal stress, which can have severe adverse health effects on mothers and newborns if unaddressed. Predicting next-day physiological or perceived stress can help to inform and enable pre-emptive interventions for a likely physiologically and perceptibly stressful day. Machine learning models are useful tools that can be developed to predict next-day physiological and perceived stress by using data collected from the previous day. Such models can improve our understanding of the specific factors that predict physiological and perceived stress and allow researchers to develop systems that collect selected features for assessment in clinical trials to minimize the burden of data collection.

Objective

The aim of this study was to build and evaluate a machine-learned model that predicts next-day physiological and perceived stress by using sensor-based, ecological momentary assessment (EMA)–based, and intervention-based features and to explain the prediction results.

Methods

We enrolled pregnant women into a prospective proof-of-concept study and collected electrocardiography, EMA, and cognitive behavioral therapy intervention data over 12 weeks. We used the data to train and evaluate 6 machine learning models to predict next-day physiological and perceived stress. After selecting the best performing model, Shapley Additive Explanations were used to identify the feature importance and explainability of each feature.

Results

A total of 16 pregnant women enrolled in the study. Overall, 4157.18 hours of data were collected, and participants answered 2838 EMAs. After applying feature selection, 8 and 10 features were found to positively predict next-day physiological and perceived stress, respectively. A random forest classifier performed the best in predicting next-day physiological stress (F1 score of 0.84) and next-day perceived stress (F1 score of 0.74) by using all features. Although any subset of sensor-based, EMA-based, or intervention-based features could reliably predict next-day physiological stress, EMA-based features were necessary to predict next-day perceived stress. The analysis of explainability metrics showed that the prolonged duration of physiological stress was highly predictive of next-day physiological stress and that physiological stress and perceived stress were temporally divergent.

Conclusions

In this study, we were able to build interpretable machine learning models to predict next-day physiological and perceived stress, and we identified unique features that were highly predictive of next-day stress that can help to reduce the burden of data collection.

Sensing and Artificial Intelligent Maternal-Infant Health Care Systems: A Review

Currently, information and communication technology (ICT) allows health institutions to reach disadvantaged groups in rural areas using sensing and artificial intelligence (AI) technologies. Applications of these technologies are even more essential for maternal and infant health, since maternal and infant health is vital for a healthy society. Over the last few years, researchers have delved into sensing and artificially intelligent healthcare systems for maternal and infant health. Sensors are exploited to gauge health parameters, and machine learning techniques are investigated to predict the health conditions of patients to assist medical practitioners. Since these healthcare systems deal with large amounts of data, significant development is also noted in the computing platforms. The relevant literature reports the potential impact of ICT-enabled systems for improving maternal and infant health. This article reviews wearable sensors and AI algorithms based on existing systems designed to predict the risk factors during and after pregnancy for both mothers and infants. This review covers sensors and AI algorithms used in these systems and analyzes each approach with its features, outcomes, and novel aspects in chronological order. It also includes discussion on datasets used and extends challenges as well as future work directions for researchers.

Stress prediction using micro-EMA and machine learning during COVID-19 social isolation

Accurately predicting users' perceived stress is beneficial to aid early intervention and prevent both mental illness and physical disease during the COVID-19 pandemic. However, the existing perceived stress predicting system needs to collect a large amount of previous data for training but has a limited prediction range (i.e., next 1-2 days). Therefore, we propose a perceived stress prediction system based on the history data of micro-EMA for identifying risks 7 days earlier. Specifically, we first select and deliver an optimal set of micro-EMA questions to users every Monday, Wednesday, and Friday for reducing the burden. Then, we extract time-series features from the past micro-EMA responses and apply an Elastic net regularization model to discard redundant features. After that, selected features are fed to an ensemble prediction model for forecasting fine-grained perceived stress in the next 7 days. Experiment results show that our proposed prediction system can achieve around 4.26 (10.65% of the scale) mean absolute error for predicting the next 7 day's PSS scores, and higher than 81% accuracy for predicting the next 7 day's stress labels.

Contextual Biases in Microinteraction Ecological Momentary Assessment (μEMA) Non-response

Ecological momentary assessment (EMA) is used to gather in-situ self-report on behaviors using mobile devices. Microinteraction EMA (μEMA), is a type of EMA where each survey is only one single question that can be answered with a glanceable microinteraction on a smartwatch. Prior work shows that even when μEMA interrupts far more frequently than smartphone-EMA, μEMA yields higher response rates with lower burden. We examined the contextual biases associated with non-response of μEMA prompts on a smartwatch. Based on prior work on EMA non-response and smartwatch use, we identified 10 potential contextual biases from three categories: temporal (time of the day, parts of waking day, day of the week, and days in study), device use (screen state, charging status, battery mode, and phone usage), and activity (wrist motion and location). We used data from a longitudinal study where 131 participants (Mean age 22.9 years, SD = 3.0) responded to μEMA surveys on a smartwatch for at least six months. Using mixed-effects logistic regression, we found that all temporal, activity/mobility, and device use variables had a statistically significant (p<0.001) association with momentary μEMA non-response. We discuss the implication of these results for future use of context-aware μEMA methodology.

Intensive Longitudinal Data Collection Using Microinteraction Ecological Momentary Assessment: Pilot and Preliminary Results

Background

Ecological momentary assessment (EMA) uses mobile technology to enable in situ self-report data collection on behaviors and states. In a typical EMA study, participants are prompted several times a day to answer sets of multiple-choice questions. Although the repeated nature of EMA reduces recall bias, it may induce participation burden. There is a need to explore complementary approaches to collecting in situ self-report data that are less burdensome yet provide comprehensive information on an individual’s behaviors and states. A new approach, microinteraction EMA (μEMA), restricts EMA items to single, cognitively simple questions answered on a smartwatch with single-tap assessments using a quick, glanceable microinteraction. However, the viability of using μEMA to capture behaviors and states in a large-scale longitudinal study has not yet been demonstrated.

Objective

This paper describes the μEMA protocol currently used in the Temporal Influences on Movement & Exercise (TIME) Study conducted with young adults, the interface of the μEMA app used to gather self-report responses on a smartwatch, qualitative feedback from participants after a pilot study of the μEMA app, changes made to the main TIME Study μEMA protocol and app based on the pilot feedback, and preliminary μEMA results from a subset of active participants in the TIME Study.

Methods

The TIME Study involves data collection on behaviors and states from 246 individuals; measurements include passive sensing from a smartwatch and smartphone and intensive smartphone-based hourly EMA, with 4-day EMA bursts every 2 weeks. Every day, participants also answer a nightly EMA survey. On non–EMA burst days, participants answer μEMA questions on the smartwatch, assessing momentary states such as physical activity, sedentary behavior, and affect. At the end of the study, participants describe their experience with EMA and μEMA in a semistructured interview. A pilot study was used to test and refine the μEMA protocol before the main study.

Results

Changes made to the μEMA study protocol based on pilot feedback included adjusting the single-question selection method and smartwatch vibrotactile prompting. We also added sensor-triggered questions for physical activity and sedentary behavior. As of June 2021, a total of 81 participants had completed at least 6 months of data collection in the main study. For 662,397 μEMA questions delivered, the compliance rate was 67.6% (SD 24.4%) and the completion rate was 79% (SD 22.2%).

Conclusions

The TIME Study provides opportunities to explore a novel approach for collecting temporally dense intensive longitudinal self-report data in a sustainable manner. Data suggest that μEMA may be valuable for understanding behaviors and states at the individual level, thus possibly supporting future longitudinal interventions that require within-day, temporally dense self-report data as people go about their lives.

Comparing stress prediction models using smartwatch physiological signals and participant self-reports

Recent advances in wearable technology have facilitated the non-obtrusive monitoring of physiological signals, creating opportunities to monitor and predict stress. Researchers have utilized machine learning methods using these physiological signals to develop stress prediction models. Many of these prediction models have utilized objective stressor tasks (e.g., a public speaking task or solving math problems). Alternatively, the subjective user responses with self-reports have also been used for measuring stress. In this paper, we describe a methodological approach (a) to compare the prediction performance of models developed using objective markers of stress using participant-reported subjective markers of stress from self-reports; and (b) to develop personalized stress models by accounting for inter-individual differences. Towards this end, we conducted a laboratory-based study with 32 healthy volunteers. Participants completed a series of stressor tasks-social, cognitive and physical-wearing an instrumented commercial smartwatch that collected physiological signals and participant responses using timed self-reports. After extensive data preprocessing using a combination of signal processing techniques, we developed two types of models: objective stress models using the stressor tasks as labels; and subjective stress models using participant responses to each task as the label for that stress task. We trained and tested several machine learning algorithms-support vector machine (SVM), random forest (RF), gradient boosted trees (GBT), AdaBoost, and Logistic Regression (LR)-and evaluated their performance. SVM had the best performance for the models using the objective stressor (i.e., stressor tasks) with an AUROC of 0.790 and an F-1 score of 0.623. SVM also had the highest performance for the models using the subjective stress (i.e., participant self-reports) with an AUROC of 0.719 and an F-1 score of 0.520. Model performance improved with a personalized threshold model to an AUROC of 0.751 and an F-1 score of 0.599. The performance of the stress models using an instrumented commercial smartwatch was comparable to similar models from other state-of-the-art laboratory-based studies. However, the subjective stress models had a lower performance, indicating the need for further research on the use of self-reports for stress-related studies. The improvement in performance with the personalized threshold-based models provide new directions for building stress prediction models.

Measuring Criterion Validity of Microinteraction Ecological Momentary Assessment (Micro-EMA): Exploratory Pilot Study With Physical Activity Measurement

Background

Ecological momentary assessment (EMA) is an in situ method of gathering self-report on behaviors using mobile devices. In typical phone-based EMAs, participants are prompted repeatedly with multiple-choice questions, often causing participation burden. Alternatively, microinteraction EMA (micro-EMA or μEMA) is a type of EMA where all the self-report prompts are single-question surveys that can be answered using a 1-tap glanceable microinteraction conveniently on a smartwatch. Prior work suggests that μEMA may permit a substantially higher prompting rate than EMA, yielding higher response rates and lower participation burden. This is achieved by ensuring μEMA prompt questions are quick and cognitively simple to answer. However, the validity of participant responses from μEMA self-report has not yet been formally assessed.

Objective

In this pilot study, we explored the criterion validity of μEMA self-report on a smartwatch, using physical activity (PA) assessment as an example behavior of interest.

Methods

A total of 17 participants answered 72 μEMA prompts each day for 1 week using a custom-built μEMA smartwatch app. At each prompt, they self-reported whether they were doing sedentary, light/standing, moderate/walking, or vigorous activities by tapping on the smartwatch screen. Responses were compared with a research-grade activity monitor worn on the dominant ankle simultaneously (and continuously) measuring PA.

Results

Participants had an 87.01% (5226/6006) μEMA completion rate and a 74.00% (5226/7062) compliance rate taking an average of only 5.4 (SD 1.5) seconds to answer a prompt. When comparing μEMA responses with the activity monitor, we observed significantly higher (P<.001) momentary PA levels on the activity monitor when participants self-reported engaging in moderate+vigorous activities compared with sedentary or light/standing activities. The same comparison did not yield any significant differences in momentary PA levels as recorded by the activity monitor when the μEMA responses were randomly generated (ie, simulating careless taps on the smartwatch).

Conclusions

For PA measurement, high-frequency μEMA self-report could be used to capture information that appears consistent with that of a research-grade continuous sensor for sedentary, light, and moderate+vigorous activity, suggesting criterion validity. The preliminary results show that participants were not carelessly answering μEMA prompts by randomly tapping on the smartwatch but were reporting their true behavior at that moment. However, more research is needed to examine the criterion validity of μEMA when measuring vigorous activities.

Evaluating the Reproducibility of Physiological Stress Detection Models

Recent advances in wearable sensor technologies have led to a variety of approaches for detecting physiological stress. Even with over a decade of research in the domain, there still exist many significant challenges, including a near-total lack of reproducibility across studies. Researchers often use some physiological sensors (custom-made or off-the-shelf), conduct a study to collect data, and build machine-learning models to detect stress. There is little effort to test the applicability of the model with similar physiological data collected from different devices, or the efficacy of the model on data collected from different studies, populations, or demographics. This paper takes the first step towards testing reproducibility and validity of methods and machine-learning models for stress detection. To this end, we analyzed data from 90 participants, from four independent controlled studies, using two different types of sensors, with different study protocols and research goals. We started by evaluating the performance of models built using data from one study and tested on data from other studies. Next, we evaluated new methods to improve the performance of stress-detection models and found that our methods led to a consistent increase in performance across all studies, irrespective of the device type, sensor type, or the type of stressor. Finally, we developed and evaluated a clustering approach to determine the stressed/not-stressed classification when applying models on data from different studies, and found that our approach performed better than selecting a threshold based on training data. This paper's thorough exploration of reproducibility in a controlled environment provides a critical foundation for deeper study of such methods, and is a prerequisite for tackling reproducibility in free-living conditions.

Moving the dial on prenatal stress mechanisms of neurodevelopmental vulnerability to mental health problems: A personalized prevention proof of concept

Prenatal stress exposure increases vulnerability to virtually all forms of psychopathology. Based on this robust evidence base, we propose a "Mental Health, Earlier" paradigm shift for prenatal stress research, which moves from the documentation of stress-related outcomes to their prevention, with a focus on infant neurodevelopmental indicators of vulnerability to subsequent mental health problems. Achieving this requires an expansive team science approach. As an exemplar, we introduce the Promoting Healthy Brain Project (PHBP), a randomized trial testing the impact of the Wellness-4-2 personalized prenatal stress-reduction intervention on stress-related alterations in infant neurodevelopmental trajectories in the first year of life. Wellness-4-2 utilizes bio-integrated stress monitoring for just-in-time adaptive intervention. We highlight unique challenges and opportunities this novel team science approach presents in synergizing expertise across predictive analytics, bioengineering, health information technology, prevention science, maternal-fetal medicine, neonatology, pediatrics, and neurodevelopmental science. We discuss how innovations across many areas of study facilitate this personalized preventive approach, using developmentally sensitive brain and behavioral methods to investigate whether altering children's adverse gestational exposures, i.e., maternal stress in the womb, can improve their mental health outlooks. In so doing, we seek to propel developmental SEED research towards preventive applications with the potential to reduce the pernicious effect of prenatal stress on neurodevelopment, mental health, and wellbeing.