Behavioral Modeling in Weight Loss Interventions

Reinforcement Learning Interventions on Boundedly Rational Human Agents in Frictionful Tasks

Many important behavior changes are frictionful; they require individuals to expend effort over a long period with little immediate gratification. Here, an artificial intelligence (AI) agent can provide personalized interventions to help individuals stick to their goals. In these settings, the AI agent must personalize rapidly (before the individual disengages) and interpretably, to help us understand the behavioral interventions. In this paper, we introduce Behavior Model Reinforcement Learning (BMRL), a framework in which an AI agent intervenes on the parameters of a Markov Decision Process (MDP) belonging to a boundedly rational human agent. Our formulation of the human decision-maker as a planning agent allows us to attribute undesirable human policies (ones that do not lead to the goal) to their maladapted MDP parameters, such as an extremely low discount factor. Furthermore, we propose a class of tractable human models that captures fundamental behaviors in frictionful tasks. Introducing a notion of MDP equivalence specific to BMRL, we theoretically and empirically show that AI planning with our human models can lead to helpful policies on a wide range of more complex, ground-truth humans.

Enhancing digital health services: A machine learning approach to personalized exercise goal setting

Background

The utilization of digital health has increased recently, and these services provide extensive guidance to encourage users to exercise frequently by setting daily exercise goals to promote a healthy lifestyle. These comprehensive guides evolved from the consideration of various personalized behavioral factors. Nevertheless, existing approaches frequently neglect the users' dynamic behavior and the changing in their health conditions.

Objective

This study aims to fill this gap by developing a machine learning algorithm that dynamically updates auto-suggestion exercise goals using retrospective data and realistic behavior trajectory.

Methods

We conducted a methodological study by designing a deep reinforcement learning algorithm to evaluate exercise performance, considering fitness-fatigue effects. The deep reinforcement learning algorithm combines deep learning techniques to analyze time series data and infer user's exercise behavior. In addition, we use the asynchronous advantage actor-critic algorithm for reinforcement learning to determine the optimal exercise intensity through exploration and exploitation. The personalized exercise data and biometric data used in this study were collected from publicly available datasets, encompassing walking, sports logs, and running.

Results

In our study, we conducted the statistical analyses/inferential tests to compare the effectiveness of machine learning approach in exercise goal setting across different exercise goal-setting strategies. The 95% confidence intervals demonstrated the robustness of these findings, emphasizing the superior outcomes of the machine learning approach.

Conclusions

Our study demonstrates the adaptability of machine learning algorithm to users' exercise preferences and behaviors in exercise goal setting, emphasizing the substantial influence of goal design on service effectiveness.

Behavioral Analytics for Myopic Agents

Many multi-agent systems have a single coordinator providing incentives to a large number of agents. Two challenges faced by the coordinator are a finite budget from which to allocate incentives, and an initial lack of knowledge about the utility function of the agents. Here, we present a behavioral analytics approach for solving the coordinator's problem when the agents make decisions by maximizing utility functions that depend on prior system states, inputs, and other parameters that are initially unknown. Our behavioral analytics framework involves three steps: first, we develop a model that describes the decision-making process of an agent; second, we use data to estimate the model parameters for each agent and predict their future decisions; and third, we use these predictions to optimize a set of incentives that will be provided to each agent. The framework and approaches we propose in this paper can then adapt incentives as new information is collected. Furthermore, we prove that the incentives computed by this approach are asymptotically optimal with respect to a loss function that describes the coordinator's objective. We optimize incentives with a decomposition scheme, where each sub-problem solves the coordinator's problem for a single agent, and the master problem is a pure integer program. We conclude with a simulation study to evaluate the effectiveness of our approach for designing a personalized weight loss program. The results show that our approach maintains efficacy of the program while reducing its costs by up to 60%, while adaptive heuristics provide substantially less savings.

The potential of artificial intelligence in enhancing adult weight loss: a scoping review

OBJECTIVE

To present an overview of how artificial intelligence (AI) could be used to regulate eating and dietary behaviours, exercise behaviours and weight loss.

DESIGN

A scoping review of global literature published from inception to 15 December 2020 was conducted according to Arksey and O'Malley's five-step framework. Eight databases (CINAHL, Cochrane-Central, Embase, IEEE Xplore, PsycINFO, PubMed, Scopus and Web of Science) were searched. Included studies were independently screened for eligibility by two reviewers with good interrater reliability (k = 0·96).

RESULTS

Sixty-six out of 5573 potential studies were included, representing more than 2031 participants. Three tenets of self-regulation were identified - self-monitoring (n 66, 100 %), optimisation of goal setting (n 10, 15·2 %) and self-control (n 10, 15·2 %). Articles were also categorised into three AI applications, namely machine perception (n 50), predictive analytics only (n 6) and real-time analytics with personalised micro-interventions (n 10). Machine perception focused on recognising food items, eating behaviours, physical activities and estimating energy balance. Predictive analytics focused on predicting weight loss, intervention adherence, dietary lapses and emotional eating. Studies on the last theme focused on evaluating AI-assisted weight management interventions that instantaneously collected behavioural data, optimised prediction models for behavioural lapse events and enhance behavioural self-control through adaptive and personalised nudges/prompts. Only six studies reported average weight losses (2·4-4·7 %) of which two were statistically significant.

CONCLUSION

The use of AI for weight loss is still undeveloped. Based on the current study findings, we proposed a framework on the applicability of AI for weight loss but cautioned its contingency upon engagement and contextualisation.

Applying machine learning to predict future adherence to physical activity programs

BACKGROUND

Identifying individuals who are unlikely to adhere to a physical exercise regime has potential to improve physical activity interventions. The aim of this paper is to develop and test adherence prediction models using objectively measured physical activity data in the Mobile Phone-Based Physical Activity Education program (mPED) trial. To the best of our knowledge, this is the first to apply Machine Learning methods to predict exercise relapse using accelerometer-recorded physical activity data.

METHODS

We use logistic regression and support vector machine methods to design two versions of a Discontinuation Prediction Score (DiPS), which uses objectively measured past data (e.g., steps and goal achievement) to provide a numerical quantity indicating the likelihood of exercise relapse in the upcoming week. The respective prediction accuracy of these two versions of DiPS are compared, and then numerical simulation is performed to explore the potential of using DiPS to selectively allocate financial incentives to participants to encourage them to increase physical activity.

RESULTS

we had access to a physical activity trial data that were continuously collected every 60 sec every day for 9 months in 210 participants. By using the first 15 weeks of data as training and test on weeks 16-30, we show that both versions of DiPS have a test AUC of 0.9 with high sensitivity and specificity in predicting the probability of exercise adherence. Simulation results assuming different intervention regimes suggest the potential benefit of using DiPS as a score to allocate resources in physical activity intervention programs in reducing costs over other allocation schemes.

CONCLUSIONS

DiPS is capable of making accurate and robust predictions for future weeks. The most predictive features are steps and physical activity intensity. Furthermore, the use of DiPS scores can be a promising approach to determine when or if to provide just-in-time messages and step goal adjustments to improve compliance. Further studies on the use of DiPS in the design of physical activity promotion programs are warranted.

TRIAL REGISTRATION

ClinicalTrials.gov NCT01280812 Registered on January 21, 2011.

Personalizing Mobile Fitness Apps using Reinforcement Learning

Despite the vast number of mobile fitness applications (apps) and their potential advantages in promoting physical activity, many existing apps lack behavior-change features and are not able to maintain behavior change motivation. This paper describes a novel fitness app called CalFit, which implements important behavior-change features like dynamic goal setting and self-monitoring. CalFit uses a reinforcement learning algorithm to generate personalized daily step goals that are challenging but attainable. We conducted the Mobile Student Activity Reinforcement (mSTAR) study with 13 college students to evaluate the efficacy of the CalFit app. The control group (receiving goals of 10,000 steps/day) had a decrease in daily step count of 1,520 (SD ± 740) between baseline and 10-weeks, compared to an increase of 700 (SD ± 830) in the intervention group (receiving personalized step goals). The difference in daily steps between the two groups was 2,220, with a statistically significant p = 0.039.