1
|
Oldham M, Dina LM, Loebenberg G, Perski O, Brown J, Angus C, Beard E, Burton R, Field M, Greaves F, Hickman M, Kaner E, Michie S, Munafò MR, Pizzo E, Garnett C. Evaluating the Acceptability of the Drink Less App and the National Health Service Alcohol Advice Web Page: Qualitative Interview Process Evaluation. J Med Internet Res 2024; 26:e42319. [PMID: 39024575 PMCID: PMC11294780 DOI: 10.2196/42319] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 03/08/2024] [Accepted: 05/14/2024] [Indexed: 07/20/2024] Open
Abstract
BACKGROUND The extent to which interventions are perceived as acceptable to users impacts engagement and efficacy. OBJECTIVE In this study, we evaluated the acceptability of (1) the smartphone app Drink Less (intervention) and (2) the National Health Service (NHS) alcohol advice web page (usual digital care and comparator) among adult drinkers in the United Kingdom participating in a randomized controlled trial evaluating the effectiveness of the Drink Less app. METHODS A subsample of 26 increasing- and higher-risk drinkers (Alcohol Use Disorders Identification Test score≥8) assigned to the intervention group (Drink Less; n=14, 54%; female: n=10, 71%; age: 22-72 years; White: n=9, 64%) or usual digital care group (NHS alcohol advice web page; n=12, 46%; female: n=5, 42%; age: 23-68 years: White: n=9, 75%) took part in semistructured interviews. The interview questions were mapped on to the 7 facets of acceptability according to the Theoretical Framework of Acceptability: affective attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs, and self-efficacy. Alongside these constructs, we also included a question on perceived personal relevance, which previous research has linked to acceptability and engagement. Framework and thematic analysis of data was undertaken. RESULTS The Drink Less app was perceived as being ethical, easy, user-friendly, and effective for the period the app was used. Participants reported particularly liking the tracking and feedback sections of the app, which they reported increased personal relevance and which resulted in positive affect when achieving their goals. They reported no opportunity cost. Factors such as negative affect when not meeting goals and boredom led to disengagement in the longer term for some participants. The NHS alcohol advice web page was rated as being easy and user-friendly with no opportunity costs. However, the information presented was not perceived as being personally relevant or effective in changing drinking behavior. Most participants reported neutral or negative affect, most participants thought the alcohol advice web page was accessible, and some participants reported ethical concerns around the availability of suggested resources. Some participants reported that it had acted as a starting point or a signpost to other resources. Participants in both groups discussed motivation to change and contextual factors such as COVID-19 lockdowns, which influenced their perceived self-efficacy regardless of their assigned intervention. CONCLUSIONS Drink Less appears to be an acceptable digital intervention among the recruited sample. The NHS alcohol advice web page was generally considered unacceptable as a stand-alone intervention among the recruited sample, although it may signpost and help people access other resources and interventions.
Collapse
Affiliation(s)
- Melissa Oldham
- Behavioural Science and Health, University College London, London, United Kingdom
| | - Larisa-Maria Dina
- Behavioural Science and Health, University College London, London, United Kingdom
| | - Gemma Loebenberg
- Behavioural Science and Health, University College London, London, United Kingdom
| | - Olga Perski
- Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, San Diego, CA, United States
| | - Jamie Brown
- Behavioural Science and Health, University College London, London, United Kingdom
| | - Colin Angus
- School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
| | - Emma Beard
- Behavioural Science and Health, University College London, London, United Kingdom
| | - Robyn Burton
- Addictions Directorate, Office for Health Improvement and Disparities, London, United Kingdom
- Institute of Psychiatry, Psychology and Neuroscience, Kings College, London, United Kingdom
| | - Matt Field
- Department of Psychology, University of Sheffield, Sheffield, United Kingdom
| | - Felix Greaves
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
- National Institute for Health and Care Excellence, London, United Kingdom
| | - Matthew Hickman
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom
| | - Eileen Kaner
- Population Health Sciences Institute, Newcastle University, Newcastle upon Tyne, United Kingdom
| | - Susan Michie
- Centre for Behaviour Change, University College London, London, United Kingdom
| | - Marcus R Munafò
- School of Psychological Science, University of Bristol, Bristol, United Kingdom
- Medical Research Council Integrative Epidemiology Unit, University of Bristol, Bristol, United Kingdom
| | - Elena Pizzo
- Department of Applied Health Research, University College London, London, United Kingdom
| | - Claire Garnett
- Behavioural Science and Health, University College London, London, United Kingdom
- School of Psychological Science, University of Bristol, Bristol, United Kingdom
| |
Collapse
|
2
|
Oldham M, Dinu L, Loebenberg G, Field M, Hickman M, Michie S, Brown J, Garnett C. Methodological Insights on Recruitment and Retention From a Remote Randomized Controlled Trial Examining the Effectiveness of an Alcohol Reduction App: Descriptive Analysis Study. JMIR Form Res 2024; 8:e51839. [PMID: 38180802 PMCID: PMC10799283 DOI: 10.2196/51839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Revised: 11/07/2023] [Accepted: 11/24/2023] [Indexed: 01/06/2024] Open
Abstract
BACKGROUND Randomized controlled trials (RCTs) with no in-person contact (ie, remote) between researchers and participants offer savings in terms of cost and time but present unique challenges. OBJECTIVE The goal of this study is to examine the differences between different forms of remote recruitment (eg, National Health Service [NHS] website, social media, and radio advertising) in the proportion of participants recruited, demographic diversity, follow-up rates, and cost. We also examine the cost per participant of sequential methods of follow-up (emails, phone calls, postal surveys, and postcards). Finally, our experience with broader issues around study advertising and participant deception is discussed. METHODS We conducted a descriptive analysis of 5602 increasing-and-higher-risk drinkers (Alcohol Use Disorders Identification Test score ≥8), taking part in a 2-arm, parallel group, remote RCT with a 1:1 allocation, comparing the intervention (Drink Less app) with usual digital care (NHS alcohol advice web page). Participants were recruited between July 2020 and March 2022 and compensated with gift vouchers of up to £36 (a currency exchange rate of £1=US $1.26988 is applicable) for completing follow-up surveys, with 4 stages of follow-up: email reminders, phone calls, postal survey, and postcard. RESULTS The three main recruitment methods were advertisements on (1) social media (2483/5602, 44.32%), (2) the NHS website (1961/5602, 35.01%), and (3) radio and newspapers (745/5602, 13.3%), with the remaining methods of recruitment accounting 7.37% (413/5602) of the sample. The overall recruitment cost per participant varied from £0 to £11.01. Costs were greater when recruiting participants who were men (£0-£28.85), from an ethnic minority group (£0-£303.81), and more disadvantaged (£0-£49.12). Targeted approaches were useful for recruiting more men but less useful in achieving diversity in ethnicity and socioeconomic status. Follow-up at 6 months was 79.58% (4458/5602). Of those who responded, 92.4% (4119/4458) responded by email. Each additional stage of follow-up resulted in an additional 2-3 percentage points of the overall sample being followed up, although phone calls, postal surveys, and postcards were more resource intensive than email reminders. CONCLUSIONS For remote RCTs, researchers could benefit from using a range of recruitment methods and cost-targeted approaches to achieve demographic diversity. Automated emails with substantial financial incentives for prompt completion can achieve good follow-up rates, and sequential, offline follow-up options, such as phone calls and postal surveys, can further increase follow-up rates but are comparatively expensive. We also make broader recommendations focused on striking the right balance when designing remote RCTs. Careful planning, ongoing maintenance, and dynamic decision-making are required throughout a trial to balance the competing demands of participation among those eligible, deceptive participation among those who are not eligible, and ensuring no postrandomization bias is introduced by data-checking protocols.
Collapse
Affiliation(s)
- Melissa Oldham
- University College London, London, London, United Kingdom
| | - Larisa Dinu
- University College London, London, London, United Kingdom
| | | | - Matt Field
- Department of Psychology, University of Sheffield, Sheffield, United Kingdom
| | - Matthew Hickman
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom
| | - Susan Michie
- Centre for Behaviour Change, University College London, London, United Kingdom
| | - Jamie Brown
- University College London, London, London, United Kingdom
| | - Claire Garnett
- University College London, London, London, United Kingdom
| |
Collapse
|
3
|
Loebenberg G, Oldham M, Brown J, Dinu L, Michie S, Field M, Greaves F, Garnett C. Bot or Not? Detecting and Managing Participant Deception When Conducting Digital Research Remotely: Case Study of a Randomized Controlled Trial. J Med Internet Res 2023; 25:e46523. [PMID: 37707943 PMCID: PMC10540014 DOI: 10.2196/46523] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 06/19/2023] [Accepted: 06/28/2023] [Indexed: 09/15/2023] Open
Abstract
BACKGROUND Evaluating digital interventions using remote methods enables the recruitment of large numbers of participants relatively conveniently and cheaply compared with in-person methods. However, conducting research remotely based on participant self-report with little verification is open to automated "bots" and participant deception. OBJECTIVE This paper uses a case study of a remotely conducted trial of an alcohol reduction app to highlight and discuss (1) the issues with participant deception affecting remote research trials with financial compensation; and (2) the importance of rigorous data management to detect and address these issues. METHODS We recruited participants on the internet from July 2020 to March 2022 for a randomized controlled trial (n=5602) evaluating the effectiveness of an alcohol reduction app, Drink Less. Follow-up occurred at 3 time points, with financial compensation offered (up to £36 [US $39.23]). Address authentication and telephone verification were used to detect 2 kinds of deception: "bots," that is, automated responses generated in clusters; and manual participant deception, that is, participants providing false information. RESULTS Of the 1142 participants who enrolled in the first 2 months of recruitment, 75.6% (n=863) of them were identified as bots during data screening. As a result, a CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart) was added, and after this, no more bots were identified. Manual participant deception occurred throughout the study. Of the 5956 participants (excluding bots) who enrolled in the study, 298 (5%) were identified as false participants. The extent of this decreased from 110 in November 2020, to a negligible level by February 2022 including a number of months with 0. The decline occurred after we added further screening questions such as attention checks, removed the prominence of financial compensation from social media advertising, and added an additional requirement to provide a mobile phone number for identity verification. CONCLUSIONS Data management protocols are necessary to detect automated bots and manual participant deception in remotely conducted trials. Bots and manual deception can be minimized by adding a CAPTCHA, attention checks, a requirement to provide a phone number for identity verification, and not prominently advertising financial compensation on social media. TRIAL REGISTRATION ISRCTN Number ISRCTN64052601; https://doi.org/10.1186/ISRCTN64052601.
Collapse
Affiliation(s)
- Gemma Loebenberg
- UCL Tobacco and Alcohol Research Group, University College London, London, United Kingdom
| | - Melissa Oldham
- UCL Tobacco and Alcohol Research Group, University College London, London, United Kingdom
| | - Jamie Brown
- UCL Tobacco and Alcohol Research Group, University College London, London, United Kingdom
| | - Larisa Dinu
- UCL Tobacco and Alcohol Research Group, University College London, London, United Kingdom
| | - Susan Michie
- Clinical Educational and Health Psychology, University College London, London, United Kingdom
| | - Matt Field
- Department of Psychology, University of Sheffield, Sheffield, United Kingdom
| | - Felix Greaves
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | - Claire Garnett
- UCL Tobacco and Alcohol Research Group, University College London, London, United Kingdom
| |
Collapse
|
4
|
Oldham M, Perski O, Loebenberg G, Brown J, Garnett C. The Effect of the First UK COVID-19 Lockdown on Users of the Drink Less App: Interrupted Time Series Analysis of Sociodemographic Characteristics, Engagement, and Alcohol Reduction. J Med Internet Res 2022; 24:e42320. [PMID: 36240461 PMCID: PMC9693718 DOI: 10.2196/42320] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 10/10/2022] [Accepted: 10/13/2022] [Indexed: 11/05/2022] Open
Abstract
BACKGROUND The first UK COVID-19 lockdown had a polarizing impact on drinking behavior and may have impacted engagement with digital interventions to reduce alcohol consumption. OBJECTIVE We examined the effect of lockdown on engagement, alcohol reduction, and the sociodemographic characteristics of users of the popular and widely available alcohol reduction app Drink Less. METHODS This was a natural experiment. The study period spanned 468 days between March 24, 2019, and July 3, 2020, with the introduction of UK lockdown measures beginning on March 24, 2020. Users were 18 years or older, based in the United Kingdom, and interested in drinking less. Interrupted time series analyses using generalized additive mixed models (GAMMs) were conducted for each outcome variable (ie, sociodemographic characteristics, app downloads and engagement levels, alcohol consumption, and extent of alcohol reduction) for existing (downloaded the app prelockdown) and new (downloaded the app during the lockdown) users of the app. RESULTS Among existing users of the Drink Less app, there were increases in the time spent on the app per day (B=0.01, P=.01), mean units of alcohol recorded per day (B>0.00 P=.02), and mean heavy drinking (>6 units) days (B>0.00, P=.02) during the lockdown. Previous declines in new app downloads plateaued during the lockdown (incidence rate ratio [IRR]=1.00, P=.18). Among new app users, there was an increase in the proportion of female users (B>0.00, P=.04) and those at risk of alcohol dependence (B>0.00, P=.01) and a decrease in the proportion of nonmanual workers (B>-0.00, P=.04). Among new app users, there were step increases in the mean number of alcohol units per day (B=20.12, P=.03), heavy-drinking days (B=1.38, P=.01), and the number of days the app was used (B=2.05, P=.02), alongside a step decrease in the percentage of available screens viewed (B=-0.03, P=.04), indicating users were using less of the intervention components within the app. CONCLUSIONS Following the first UK lockdown, there was evidence of increases in engagement and alcohol consumption among new and existing users of the Drink Less app.
Collapse
Affiliation(s)
- Melissa Oldham
- Department of Behavioural Science and Health, University College London, London, United Kingdom
| | - Olga Perski
- Department of Behavioural Science and Health, University College London, London, United Kingdom
| | - Gemma Loebenberg
- Department of Behavioural Science and Health, University College London, London, United Kingdom
| | - Jamie Brown
- Department of Behavioural Science and Health, University College London, London, United Kingdom
| | - Claire Garnett
- Department of Behavioural Science and Health, University College London, London, United Kingdom
| |
Collapse
|