Evaluation of the impact of extrinsic rewards on user engagement in a health promotion context

The Effectiveness of Strategies to Improve User Engagement With Digital Health Interventions Targeting Nutrition, Physical Activity, and Overweight and Obesity: Systematic Review and Meta-Analysis

BACKGROUND

Digital health interventions (DHIs) are effective in improving poor nutrition, physical inactivity, overweight and obesity. There is evidence suggesting that the impact of DHIs may be enhanced by improving user engagement. However, little is known about the overall effectiveness of strategies on engagement with DHIs.

OBJECTIVE

This study aims to assess the overall effectiveness of strategies to improve engagement with DHIs targeting nutrition, physical activity, and overweight or obesity and explore associations between strategies and engagement outcomes. The secondary aim was to explore the impact of these strategies on health risk outcomes.

METHODS

The MEDLINE, Embase, PsycINFO, CINAHL, CENTRAL, Scopus, and Academic Source Complete databases were searched up to July 24, 2023. Eligible studies were randomized controlled trials that evaluated strategies to improve engagement with DHIs and reported on outcomes related to DHI engagement (use or user experience). Strategies were classified according to behavior change techniques (BCTs) and design features (eg, supplementary emails). Multiple-variable meta-analyses of the primary outcomes (usage and user experience) were undertaken to assess the overall effectiveness of strategies. Meta-regressions were conducted to assess associations between strategies and use and user experience outcomes. Synthesis of secondary outcomes followed the "Synthesis Without Meta-Analysis" guidelines. The methodological quality and evidence was assessed using the Cochrane risk-of-bias tool, and the Grading of Recommendations Assessment, Development, and Evaluation tool respectively.

RESULTS

Overall, 54 studies (across 62 publications) were included. Pooled analysis found very low-certainty evidence of a small-to-moderate positive effect of the use of strategies to improve DHI use (standardized mean difference=0.33, 95% CI 0.20-0.46; P<.001) and very low-certainty evidence of a small-to-moderate positive effect on user experience (standardized mean difference=0.29, 95% CI 0.07-0.52; P=.01). A significant positive association was found between the BCTs social support (effect size [ES]=0.40, 95% CI 0.14-0.66; P<.001) and shaping knowledge (ES=0.39, 95% CI 0.03-0.74; P=.03) and DHI use. A significant positive association was found among the BCTs social support (ES=0.70, 95% CI 0.18-1.22; P=.01), repetition and substitution (ES=0.29, 95% CI 0.05-0.53; P=.03), and natural consequences (ES=0.29, 95% CI 0.05-0.53; P=.02); the design features email (ES=0.29, 95% CI 0.05-0.53; P=.02) and SMS text messages (ES=0.34, 95% CI 0.11-0.57; P=.01); and DHI user experience. For secondary outcomes, 47% (7/15) of nutrition-related, 73% (24/33) of physical activity-related, and 41% (14/34) of overweight- and obesity-related outcomes reported an improvement in health outcomes.

CONCLUSIONS

Although findings suggest that the use of strategies may improve engagement with DHIs targeting such health outcomes, the true effect is unknown because of the low quality of evidence. Future research exploring whether specific forms of social support, repetition and substitution, natural consequences, emails, and SMS text messages have a greater impact on DHI engagement is warranted.

TRIAL REGISTRATION

PROSPERO CRD42018077333; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=77333.

Design Features Associated With Engagement in Mobile Health Physical Activity Interventions Among Youth: Systematic Review of Qualitative and Quantitative Studies

BACKGROUND

Globally, 81% of youth do not meet the physical activity (PA) guidelines. Youth of families with a low socioeconomic position are less likely to meet the recommended PA guidelines. Mobile health (mHealth) interventions are preferred by youth over traditional in-person approaches and are in line with their media preferences. Despite the promise of mHealth interventions in promoting PA, a common challenge is to engage users in the long term or effectively. Earlier reviews highlighted the association of different design features (eg, notifications and rewards) with engagement among adults. However, little is known about which design features are important for increasing engagement among youth.

OBJECTIVE

To inform the design process of future mHealth tools, it is important to investigate the design features that can yield effective user engagement. This systematic review aimed to identify which design features are associated with engagement in mHealth PA interventions among youth who were aged between 4 and 18 years.

METHODS

A systematic search was conducted in EBSCOhost (MEDLINE, APA PsycINFO, and Psychology & Behavioral Sciences Collection) and Scopus. Qualitative and quantitative studies were included if they documented design features associated with engagement. Design features and related behavior change techniques and engagement measures were extracted. Study quality was assessed according to the Mixed Method Assessment Tool, and one-third of all screening and data extraction were double coded by a second reviewer.

RESULTS

Studies (n=21) showed that various features were associated with engagement, such as a clear interface, rewards, multiplayer game mode, social interaction, variety of challenges with personalized difficulty level, self-monitoring, and variety of customization options among others, including self-set goals, personalized feedback, progress, and a narrative. In contrast, various features need to be carefully considered while designing mHealth PA interventions, such as sounds, competition, instructions, notifications, virtual maps, or self-monitoring, facilitated by manual input. In addition, technical functionality can be considered as a prerequisite for engagement. Research addressing youth from low socioeconomic position families is very limited with regard to engagement in mHealth apps.

CONCLUSIONS

Mismatches between different design features in terms of target group, study design, and content translation from behavior change techniques to design features are highlighted and set up in a design guideline and future research agenda.

TRIAL REGISTRATION

PROSPERO CRD42021254989; https://tinyurl.com/5n6ppz24.

Pilot Evaluation of the Impact of Lottery-Based Incentives on Engagement Levels of Male Low SES Vocational Students With an mHealth App

In general, individuals with lower socioeconomic status (SES) are less physically active and adhere to poorer diets than higher SES individuals. To promote healthier lifestyles in lower SES populations, we hosted a digital health promotion program among male vocational students at a school in The Netherlands. In a pilot study, we evaluated whether this target audience could be engaged with an mHealth app using lottery-based incentives that trigger feelings of anticipated regret. Especially, we studied the social and interpersonal aspects of regret lotteries in a within-subject experimental design. In this design, subjects either participated in a social variant (i.e., with students competing against their peers for a chance at a regret lottery), or an individual variant (i.e., with subjects solely individually engaged in a lottery). Additionally, we studied the impact of different payout schedules in a between-subject experimental design. In this design, participants were assigned to either a short-term, low-value payout schedule, or a long-term, high-value payout schedule. From a population of 72 male students, only half voluntarily participated in our 10-week program. From interviews, we learned that the main reason for neglecting the program was not related to the lottery-based incentives, nor to the prizes that were awarded. Instead, non-enrolled subjects did not join the program, because their peers were not joining. Paradoxically, it was suggested that students withheld their active participation until a larger portion of the sample was actively participating. From the subjects that enrolled in the program (N = 36, males, between 15 and 25 years of age), we found that a large proportion stopped interacting with the program over time (e.g., after roughly 4 weeks). Our results also indicated that students performed significantly more health-related activities when assigned to the social regret lottery, as opposed to the individual variant. This result was supported by interview responses from active participants: They mainly participated to compete against their peers, and not so much for the prizes. Hence, from this study, we obtained initial evidence on the impact of social and competitive aspects in lottery-based incentives to stimulate engagement levels in lower SES students with an mHealth app.

Preadolescent Students' Engagement With an mHealth Intervention Fostering Social Comparison for Health Behavior Change: Crossover Experimental Study

Background

Contemporary mobile health (mHealth) interventions use various behavior change techniques to promote healthier lifestyles. Social comparison is one of the techniques that is consensually agreed to be effective in engaging the general population in mHealth interventions. However, it is unclear how this strategy can be best used to engage preadolescents. Nevertheless, this strategy has great potential for this target audience, as they are particularly developing their social skills.

Objective

This study aims to evaluate how social comparison drives preadolescents’ engagement with an mHealth app.

Methods

We designed a 12-week crossover experiment in which we studied 3 approaches to implementing behavior change via social comparison. This study was hosted in a school environment to leverage naturally existing social structures among preadolescents. During the experiment, students and teachers used an mHealth tool that awarded points for performing healthy activities. Participants could read their aggregated scores on a leaderboard and compare their performance with others. In particular, these leaderboards were tweaked to implement 3 approaches of the social comparison technique. The first approach focused on intragroup comparison (ie, students and teachers competing against each other to obtain the most points), whereas the other two approaches focused on intergroup comparison (ie, classes of students and their mentoring teachers collaborating to compete against other classes). Additionally, in the third approach, the performance of teachers was highlighted to further increase students’ engagement through teachers’ natural exemplary function. To obtain our results, we used linear modeling techniques to analyze the dropout rates and engagement levels for the different approaches. In such analyses, we also considered individual participant traits.

Results

Our sample included 313 participants—290 students (92.7%) and 23 teachers (7.3%). It was found that student engagement levels dropped over time and declined during holidays. However, students seemed to monitor the intergroup competitions more closely than the intragroup competitions, as they, on average, checked the mHealth app more often when they were engaged in team-based comparisons. Students, on average, performed the most unique activities when they were engaged in the second intergroup setting, perhaps because their teachers were most active in this setting. Moreover, teachers seemed to play an important role in engaging their students, as their relationship with their students influenced the engagement of the students.

Conclusions

When using social comparison to engage preadolescents with an mHealth tool, an intergroup setting, rather than an intragroup competition, motivated them to engage with the app but did not necessarily motivate them to perform more activities. It seems that the number of unique activities that preadolescents perform depends on the activeness of a role model. Moreover, this effect is amplified by preadolescents’ perceptions of closeness to that role model.

Promoting Occupational Health through Gamification and E-Coaching: A 5-Month User Engagement Study

Social gamification systems have shown potential for promoting healthy lifestyles, but applying them to occupational settings faces unique design challenges. While occupational settings offer natural communities for social interaction, fairness issues due to heterogeneous personal goals and privacy concerns increase the difficulty of designing engaging games. We explored a two-level game-design, where the first level related to achieving personal goals and the second level was a privacy-protected social competition to maximize goal compliance among colleagues. The solution was strengthened by employing occupational physicians who personalized users' goals and coached them remotely. The design was evaluated in a 5-month study with 53 employees from a Dutch university. Results suggested that the application helped half of the participants to improve their lifestyles, and most appreciated the role of the physician in goal-setting. However, long-term user engagement was undermined by the scalability-motivated design choice of one-way communication between employees and their physician. Implications for social gamification design in occupational health are discussed.