Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator

Psychological, economic, and ethical factors in human feedback for a chatbot-based smoking cessation intervention

Integrating human support with chatbot-based behavior change interventions raises three challenges: (1) attuning the support to an individual's state (e.g., motivation) for enhanced engagement, (2) limiting the use of the concerning human resources for enhanced efficiency, and (3) optimizing outcomes on ethical aspects (e.g., fairness). Therefore, we conducted a study in which 679 smokers and vapers had a 20% chance of receiving human feedback between five chatbot sessions. We find that having received feedback increases retention and effort spent on preparatory activities. However, analyzing a reinforcement learning (RL) model fit on the data shows there are also states where not providing feedback is better. Even this "standard" benefit-maximizing RL model is value-laden. It not only prioritizes people who would benefit most, but also those who are already doing well and want feedback. We show how four other ethical principles can be incorporated to favor other smoker subgroups, yet, interdependencies exist.

Methods in dynamic treatment regimens using observational healthcare data: A systematic review

We present a systematic review of methods used to estimate Dynamic Treatment Regimens (DTR) using observational healthcare data and provide a brief summary of their strengths and weaknesses, evaluation metrics, and suitable research problem settings. We considered all observational studies identified in PubMed or EMBASE between January 1950 until January 2022, including only studies that evaluated medical treatments or interventions as exposure and/or outcome in patients and where DTRs were estimated. 83 studies met our inclusion criteria; 44.6% estimating DTR utilizing reinforcement learning, 18.1% utilizing counterfactual-based models, 12.1% utilizing classification-based methods, and 9.6% utilized g-methods. Among the studies analyzed, 28.9% aimed to replicate human expert DTRs, while 71.1% aimed to refine and improve existing DTRs. Approximately two-thirds of studies (65.1%) reported the assumptions required for their applied methods, such as exchangeability, positivity, consistency, and Markov property. Most of the studies (83.1%) estimated DTRs with more than two treatment options; 50.6% mentioned time-varying confounders, only a few estimated conditional average treatment effects (7.2%). Most (85.5%) validated their methods, with 32.5% using expected outcomes (e.g., survival rates), 26.5% employing simulated data, and 25.3% conducting direct comparisons with observational data.

Factors driving the use of mobile health app: insights from a survey

Background

Mobile health (mHealth) offers easy accessibility to healthcare information and services, promoting positive behaviour change. However, user engagement to mHealth diminishes over time, resulting in significant dropout rates. This study aims to investigate the factors contributing to the discontinuation of mHealth use and examine how persuasive elements influence users' intention to continue using mHealth. It also seeks to identify the key motivators and barriers affecting mHealth engagement.

Methods

A survey was conducted to assess persuasive elements, motivators, and barriers related to mHealth usage. The survey included measures to evaluate users' perceived persuasiveness of mHealth, the factors influencing their intention to continue using it, and both the motivators and barriers to its sustained use.

Results

The analysis revealed that unobtrusiveness had the strongest positive correlation with the intention to continue using mHealth. Additionally, a positive association was found between users' perception of mHealth's persuasiveness and their intention to continue using it. The study also identified key motivators that encourage mHealth adoption and several barriers that hinder long-term engagement.

Conclusions

These findings highlight the importance of developing strategies to enhance the long-term adoption of mHealth solutions and reduce dropout rates. Future research is needed to explore effective interventions for sustaining mHealth usage and addressing the barriers that lead to disengagement.

Machine Learning Methods to Personalize Persuasive Strategies in mHealth Interventions That Promote Physical Activity: Scoping Review and Categorization Overview

BACKGROUND

Although physical activity (PA) has positive effects on health and well-being, physical inactivity is a worldwide problem. Mobile health interventions have been shown to be effective in promoting PA. Personalizing persuasive strategies improves intervention success and can be conducted using machine learning (ML). For PA, several studies have addressed personalized persuasive strategies without ML, whereas others have included personalization using ML without focusing on persuasive strategies. An overview of studies discussing ML to personalize persuasive strategies in PA-promoting interventions and corresponding categorizations could be helpful for such interventions to be designed in the future but is still missing.

OBJECTIVE

First, we aimed to provide an overview of implemented ML techniques to personalize persuasive strategies in mobile health interventions promoting PA. Moreover, we aimed to present a categorization overview as a starting point for applying ML techniques in this field.

METHODS

A scoping review was conducted based on the framework by Arksey and O'Malley and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) criteria. Scopus, Web of Science, and PubMed were searched for studies that included ML to personalize persuasive strategies in interventions promoting PA. Papers were screened using the ASReview software. From the included papers, categorized by the research project they belonged to, we extracted data regarding general study information, target group, PA intervention, implemented technology, and study details. On the basis of the analysis of these data, a categorization overview was given.

RESULTS

In total, 40 papers belonging to 27 different projects were included. These papers could be categorized in 4 groups based on their dimension of personalization. Then, for each dimension, 1 or 2 persuasive strategy categories were found together with a type of ML. The overview resulted in a categorization consisting of 3 levels: dimension of personalization, persuasive strategy, and type of ML. When personalizing the timing of the messages, most projects implemented reinforcement learning to personalize the timing of reminders and supervised learning (SL) to personalize the timing of feedback, monitoring, and goal-setting messages. Regarding the content of the messages, most projects implemented SL to personalize PA suggestions and feedback or educational messages. For personalizing PA suggestions, SL can be implemented either alone or combined with a recommender system. Finally, reinforcement learning was mostly used to personalize the type of feedback messages.

CONCLUSIONS

The overview of all implemented persuasive strategies and their corresponding ML methods is insightful for this interdisciplinary field. Moreover, it led to a categorization overview that provides insights into the design and development of personalized persuasive strategies to promote PA. In future papers, the categorization overview might be expanded with additional layers to specify ML methods or additional dimensions of personalization and persuasive strategies.

Just-in-Time Adaptive Interventions for Behavior Change in Physiological Health Outcomes and the Use Case for Knee Osteoarthritis: Systematic Review

BACKGROUND

The prevalence of knee osteoarthritis (KOA) in the adult population is high and patients profit from individualized therapy approaches. Just-in-time adaptive interventions (JITAIs) are upcoming digital interventions for behavior change.

OBJECTIVE

This systematic review summarizes the features and effectiveness of existing JITAIs regarding important physiological health outcomes and derives the most promising features for the use case of KOA.

METHODS

The electronic databases PubMed, Web of Science, Scopus, and EBSCO were searched using keywords related to JITAIs, physical activity (PA), sedentary behavior (SB), physical function, quality of life, pain, and stiffness. JITAIs for adults that focused on the effectiveness of at least 1 of the selected outcomes were included and synthesized qualitatively. Study quality was assessed with the Quality Assessment Tool Effective Public Health Practice Project.

RESULTS

A total of 45 studies with mainly weak overall quality were included in this review. The studies were mostly focused on PA and SB and no study examined stiffness. The design of JITAIs varied, with a frequency of decision points from a minute to a day, device-based measured and self-reported tailoring variables, intervention options including audible or vibration prompts and tailored feedback, and decision rules from simple if-then conditions based on 1 variable to more complex algorithms including contextual variables.

CONCLUSIONS

The use of frequent decision points, device-based measured tailoring variables accompanied by user input, intervention options tailored to user preferences, and simple decision rules showed the most promising results in previous studies. This can be transferred to a JITAI for the use case of KOA by using target variables that include breaks in SB and an optimum of PA considering individual knee load for the health benefits of patients.

Just-in-Time Adaptive Intervention for Stabilizing Sleep Hours of Japanese Workers: Microrandomized Trial

BACKGROUND

Sleep disturbance is a major contributor to future health and occupational issues. Mobile health can provide interventions that address adverse health behaviors for individuals in a vulnerable health state in real-world settings (just-in-time adaptive intervention).

OBJECTIVE

This study aims to identify a subpopulation with vulnerable sleep state in daily life (study 1) and, immediately afterward, to test whether providing mobile health intervention improved habitual sleep behaviors and psychological wellness in real-world settings by conducting a microrandomized trial (study 2).

METHODS

Japanese workers (n=182) were instructed to collect data on their habitual sleep behaviors and momentary symptoms (including depressive mood, anxiety, and subjective sleep quality) using digital devices in a real-world setting. In study 1, we calculated intraindividual mean and variability of sleep hours, midpoint of sleep, and sleep efficiency to characterize their habitual sleep behaviors. In study 2, we designed and conducted a sleep just-in-time adaptive intervention, which delivered objective push-type sleep feedback messages to improve their sleep hours for a subset of participants in study 1 (n=81). The feedback messages were generated based on their sleep data measured on previous nights and were randomly sent to participants with a 50% chance for each day (microrandomization).

RESULTS

In study 1, we applied hierarchical clustering to dichotomize the population into 2 clusters (group A and group B) and found that group B was characterized by unstable habitual sleep behaviors (large intraindividual variabilities). In addition, linear mixed-effect models showed that the interindividual variability of sleep hours was significantly associated with depressive mood (β=3.83; P=.004), anxiety (β=5.70; P=.03), and subjective sleep quality (β=-3.37; P=.03). In study 2, we found that providing sleep feedback prolonged subsequent sleep hours (increasing up to 40 min; P=.01), and this effect lasted for up to 7 days. Overall, the stability of sleep hours in study 2 was significantly improved among participants in group B compared with the participants in study 1 (P=.001).

CONCLUSIONS

This is the first study to demonstrate that providing sleep feedback can benefit the modification of habitual sleep behaviors in a microrandomized trial. The findings of this study encourage the use of digitalized health intervention that uses real-time health monitoring and personalized feedback.

Mixed-Response State-Space Model for Analyzing Multi-Dimensional Digital Phenotypes

Digital technologies (e.g., mobile phones) can be used to obtain objective, frequent, and real-world digital phenotypes from individuals. However, modeling these data poses substantial challenges since observational data are subject to confounding and various sources of variabilities. For example, signals on patients' underlying health status and treatment effects are mixed with variation due to the living environment and measurement noises. The digital phenotype data thus shows extensive variabilities between- and within-patient as well as across different health domains (e.g., motor, cognitive, and speaking). Motivated by a mobile health study of Parkinson's disease (PD), we develop a mixed-response state-space (MRSS) model to jointly capture multi-dimensional, multi-modal digital phenotypes and their measurement processes by a finite number of latent state time series. These latent states reflect the dynamic health status and personalized time-varying treatment effects and can be used to adjust for informative measurements. For computation, we use the Kalman filter for Gaussian phenotypes and importance sampling with Laplace approximation for non-Gaussian phenotypes. We conduct comprehensive simulation studies and demonstrate the advantage of MRSS in modeling a mobile health study that remotely collects real-time digital phenotypes from PD patients.

Patient Acceptability of Home Monitoring for Neovascular Age-Related Macular Degeneration Reactivation: A Qualitative Study

Neovascular age-related macular degeneration (nAMD) is a chronic, progressive condition and the commonest cause of visual disability in older adults. This study formed part of a diagnostic test accuracy study to quantify the ability of three index home monitoring (HM) tests (one paper-based and two digital tests) to identify reactivation in nAMD. The aim of this qualitative research was to investigate patients' or participants' views about acceptability and explore adherence to weekly HM. Semi-structured interviews were held with 78/297 participants (26%), with close family members (n = 11) and with healthcare professionals involved in training participants in HM procedures (n = 9) (n = 98 in total). A directed thematic analytical approach was applied to the data using a deductive and inductive coding framework informed by theories of technology acceptance. Five themes emerged related to: 1. The role of HM; 2. Suitability of procedures and instruments; 3. Experience of HM; 4. Feasibility of HM in usual practice; and 5. Impediments to patient acceptability of HM. Various factors influenced acceptability including a patient's understanding about the purpose of monitoring. While initial training and ongoing support were regarded as essential for overcoming unfamiliarity with use of digital technology, patients viewed HM as relatively straightforward and non-burdensome. There is a need for further research about how use of performance feedback, level of support and nature of tailoring might facilitate further the implementation of routinely conducted HM. Home monitoring was acceptable to patients and they recognised its potential to reduce clinic visits during non-active treatment phases. Findings have implications for implementation of digital HM in the care of older people with nAMD and other long-term conditions.

Sensor Technology and Intelligent Systems in Anorexia Nervosa: Providing Smarter Healthcare Delivery Systems

Ubiquitous technology, big data, more efficient electronic health records, and predictive analytics are now at the core of smart healthcare systems supported by artificial intelligence. In the present narrative review, we focus on sensing technologies for the healthcare of Anorexia Nervosa (AN). We employed a framework inspired by the Interpersonal Neurobiology Theory (IPNB), which posits that human experience is characterized by a flow of energy and information both within us (within our whole body), and between us (in the connections we have with others and with nature). In line with this framework, we focused on sensors designed to evaluate bodily processes (body sensors such as implantable sensors, epidermal sensors, and wearable and portable sensors), human social interaction (sociometric sensors), and the physical environment (indoor and outdoor ambient sensors). There is a myriad of man-made sensors as well as nature-based sensors such as plants that can be used to design and deploy intelligent systems for human monitoring and healthcare. In conclusion, sensing technologies and intelligent systems can be employed for smarter healthcare of AN and help to relieve the burden of health professionals. However, there are technical, ethical, and environmental sustainability issues that must be considered prior to implementing these systems. A joint collaboration of professionals and other members of the society involved in the healthcare of individuals with AN can help in the development of these systems. The evolution of cyberphysical systems should also be considered in these collaborations.

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

Just-in-Time Prompts for Running, Walking, and Performing Strength Exercises in the Built Environment: 4-Week Randomized Feasibility Study

BACKGROUND

App-based mobile health exercise interventions can motivate individuals to engage in more physical activity (PA). According to the Fogg Behavior Model, it is important that the individual receive prompts at the right time to be successfully persuaded into PA. These are referred to as just-in-time (JIT) interventions. The Playful Active Urban Living (PAUL) app is among the first to include 2 types of JIT prompts: JIT adaptive reminder messages to initiate a run or walk and JIT strength exercise prompts during a walk or run (containing location-based instruction videos). This paper reports on the feasibility of the PAUL app and its JIT prompts.

OBJECTIVE

The main objective of this study was to examine user experience, app engagement, and users' perceptions and opinions regarding the PAUL app and its JIT prompts and to explore changes in the PA behavior, intrinsic motivation, and the perceived capability of the PA behavior of the participants.

METHODS

In total, 2 versions of the closed-beta version of the PAUL app were evaluated: a basic version (Basic PAUL) and a JIT adaptive version (Smart PAUL). Both apps send JIT exercise prompts, but the versions differ in that the Smart PAUL app sends JIT adaptive reminder messages to initiate running or walking behavior, whereas the Basic PAUL app sends reminder messages at randomized times. A total of 23 participants were randomized into 1 of the 2 intervention arms. PA behavior (accelerometer-measured), intrinsic motivation, and the perceived capability of PA behavior were measured before and after the intervention. After the intervention, participants were also asked to complete a questionnaire on user experience, and they were invited for an exit interview to assess user perceptions and opinions of the app in depth.

RESULTS

No differences in PA behavior were observed (Z=-1.433; P=.08), but intrinsic motivation for running and walking and for performing strength exercises significantly increased (Z=-3.342; P<.001 and Z=-1.821; P=.04, respectively). Furthermore, participants increased their perceived capability to perform strength exercises (Z=2.231; P=.01) but not to walk or run (Z=-1.221; P=.12). The interviews indicated that the participants were enthusiastic about the strength exercise prompts. These were perceived as personal, fun, and relevant to their health. The reminders were perceived as important initiators for PA, but participants from both app groups explained that the reminder messages were often not sent at times they could exercise. Although the participants were enthusiastic about the functionalities of the app, technical issues resulted in a low user experience.

CONCLUSIONS

The preliminary findings suggest that the PAUL apps are promising and innovative interventions for promoting PA. Users perceived the strength exercise prompts as a valuable addition to exercise apps. However, to be a feasible intervention, the app must be more stable.

Mitigating feelings of loneliness and depression by means of web-based or print-based physical activity interventions: Pooled analysis of two community-based intervention trials (Preprint)

Background

Physical activity (PA) is associated with benefits, such as fewer depressive symptoms and loneliness. Web- and print-based PA interventions can help older individuals accordingly.

Objective

We aimed to test the following research questions: Do PA interventions delivered in a web- or print-based mode improve self-reported PA stage of change, social-cognitive determinants of PA, loneliness, and symptoms of depression? Is subjective age a mediator and stage of change a moderator of this effect?

Methods

Overall, 831 adults aged ≥60 years were recruited and either allocated to a print-based or web-based intervention group or assigned to a wait-list control group (WLCG) in 2 community-based PA intervention trials over 10 weeks. Missing value imputation using an expectation-maximization algorithm was applied. Frequency analyses, multivariate analyses of variance, and moderated mediation analyses were conducted.

Results

The web-based intervention outperformed (47/59, 80% of initially inactive individuals being adopters, and 396/411, 96.4% of initially active individuals being maintainers of the recommended PA behavior) the print-based intervention (20/25, 80% of adopters, and 63/69, 91% of maintainers) and the WLCG (5/7, 71% of adopters; 141/150, 94% of maintainers). The pattern regarding adopters was statistically significant (web vs print Z=–1.94; P=.02; WLCG vs web Z=3.8367; P=.01). The pattern was replicated with stages (χ²₄=79.1; P<.001; contingency coefficient 0.314; P<.001); in the WLCG, 40.1% (63/157) of the study participants moved to or remained in action stage. This number was higher in the groups receiving web-based (357/470, 76%) or print-based interventions (64/94, 68.1%). A significant difference was observed favoring the 2 intervention groups over and above the WLCG (F_{19, 701}=4.778; P<.001; η²=0.098) and a significant interaction of time and group (F_{19, 701}=2.778; P<.001; η²=0.070) for predictors of behavior. The effects of the interventions on subjective age, loneliness, and depression revealed that both between-group effects (F_{3, 717}=8.668; P<.001; η²=0.018) and the interaction between group and time were significant (F_{3, 717}=6.101; P<.001; η²=0.025). In a moderated mediation model, both interventions had a significant direct effect on depression in comparison with the WLCG (web-based: c′ path −0.86, 95% CI −1.58 to −0.13, SE 0.38; print-based: c′ path −1.96, 95% CI −2.99 to −0.92, SE 0.53). Furthermore, subjective age was positively related to depression (b path 0.14, 95% CI 0.05-0.23; SE 0.05). An indirect effect of the intervention on depression via subjective age was only present for participants who were in actor stage and received the web-based intervention (ab path −0.14, 95% CI −0.34 to −0.01; SE 0.09).

Conclusions

Web-based interventions appear to be as effective as print-based interventions. Both modes might help older individuals remain or become active and experience fewer depression symptoms, especially if they feel younger.

Trial Registration

German Registry of Clinical Trials DRKS00010052 (PROMOTE 1); https://tinyurl.com/nnzarpsu and DRKS00016073 (PROMOTE 2); https://tinyurl.com/4fhcvkwy

International Registered Report Identifier (IRRID)

RR2-10.2196/15168