Statistical issues in pragmatic trials of behavioral economic interventions

Meta-research on pragmatism of randomized trials: rationale and design of the PragMeta database

BACKGROUND

Pragmatic trials provide decision-oriented, real-world evidence that is highly applicable and generalizable. The interest in real-world evidence is fueled by the assumption that effects in the "real-world" are different to effects obtained under artificial, controlled, research conditions as often used for traditional explanatory trials. However, it is unknown which features of pragmatism, generalizability, and applicability would be responsible for such differences. There is a need to provide empirical evidence and promote meta-research to answer these fundamental questions on the pragmatism of randomized trials and real-world evidence. Here, we describe the rationale and design of the PragMeta database which pursues this goal ( www.PragMeta.org ).

METHODS

PragMeta is a non-commercial, open data platform and infrastructure to facilitate research on pragmatic trials. It collects and shares data from published randomized trials that either have a specific design feature or other characteristic related to pragmatism or they form clusters of trials addressing the same research question but having different aspects of pragmatism. This lays the foundation to determine the relationship of various features of pragmatism, generalizability, and applicability with intervention effects or other trial characteristics. The database contains trial data actively collected for PragMeta but also allows to import and link existing datasets of trials collected for other purposes, forming a large-scale meta-database. PragMeta captures data on (1) trial and design characteristics (e.g., sample size, population, intervention/comparison, outcome, longitudinal structure, blinding), (2) effects estimates, and (3) various determinants of pragmatism (e.g., the use of routinely collected data) and ratings from established tools used to determine pragmatism (e.g., the PRagmatic-Explanatory Continuum Indicator Summary 2; PRECIS-2). PragMeta is continuously provided online, inviting the meta-research community to collaborate, contribute, and/or use the database. As of April 2023, PragMeta contains data from > 700 trials, mostly with assessments on pragmatism.

CONCLUSIONS

PragMeta will inform a better understanding of pragmatism and the generation and interpretation of real-world evidence.

A first step towards best practice recommendations for the design and statistical analyses of pragmatic clinical trials: a modified Delphi approach

AIM

Pragmatic clinical trials (PCTs) are randomised trials implemented through routine clinical practice, where design parameters of traditional randomised controlled trials are modified to increase generalizability. However, this may introduce statistical challenges. We aimed to identify these challenges and discuss possible solutions leading to best practice recommendations for the design and analysis of PCTs.

METHODS

A modified Delphi method was used to reach consensus among a panel of 11 experts in clinical trials and statistics. Statistical issues were identified in a focused literature review and aggregated with insights and possible solutions from expert collected through a series of survey iterations. Issues were ranked according to their importance.

RESULTS

27 articles were included and combined with experts' insight to generate a list of issues categorized into: participants; recruiting sites; randomisation, blinding and intervention; outcome (selection and measurement); and data analysis. Consensus was reached about the most important issues: risk of participants' attrition; heterogeneity of "usual care" across sites; absence of blinding; use of a subjective endpoint; and data analysis aligned with the trial estimand. Potential issues should be anticipated and preferably be addressed in the trial protocol. The experts provided solutions regarding data collection and data analysis, which were considered of equal importance.

DISCUSSION

A set of important statistical issues in PCTs was identified and approaches were suggested to anticipate and/or minimize these through data analysis. Any impact of choosing a pragmatic design feature should be gauged in the light of the trial estimand.

Leave me out: Patients' characteristics and reasons for opting out of a pragmatic clinical trial involving medication adherence

Opt-out procedures are sometimes used instead of standard consent practices to enable patients to exercise their autonomous preferences regarding research participation while reducing patient and researcher burden. However, little is known about the characteristics of patients who opt-out of research and their reasons for doing so. We gathered such information in a large pragmatic clinical trial (PCT) evaluating the effect of theory informed text messages on medication adherence.Eligible patients, identified through electronic health records, were sent information about the study and provided with an opportunity to opt-out. Those opting out were asked to complete a voluntary survey regarding their reasons for doing so. Demographic data were compared among patients opting-out vs those included in the study using chi-squared tests and a log binomial regression model.Of 9046 patients receiving study packets, 906 (10.0%) patients returned opt-out forms. Of those, 451 (49.8%) returned the opt-out survey. Patients who opted out were more likely to be older, white, and nonHispanic than those who were included in the PCT. Survey respondents expressed high levels of trust in their health care providers, research, and system. Nearly half (46.6%) reported concerns about time as a reason to opt-out.In this PCT, 10% of patients receiving packets opted out, with significant differences in age, race, gender, and ethnicity compared to those included. Future trials should further investigate representativeness and reasons patients choose to opt-out of participating in research.

Effectiveness of an Integrated Engagement Support System to Facilitate Patient Use of Digital Diabetes Prevention Programs: Protocol for a Randomized Controlled Trial

Background

Digital diabetes prevention programs (dDPPs) are effective behavior change tools to prevent disease progression in patients at risk for diabetes. At present, these programs are poorly integrated into existing health information technology infrastructure and clinical workflows, resulting in barriers to provider-level knowledge of, interaction with, and support of patients who use dDPPs. Tools that can facilitate patient-provider interaction around dDPPs may contribute to improved patient engagement and adherence to these programs and improved health outcomes.

Objective

This study aims to use a rigorous, user-centered design (UCD) methodology to develop a theory-driven system that supports patient engagement with dDPPs and their primary care providers with their care.

Methods

This study will be conducted in 3 phases. In phase 1, we will use systematic UCD, Agile software development, and qualitative research methods to identify key user (patients, providers, clinical staff, digital health technologists, and content experts) requirements, constraints, and prioritization of high-impact features to design, develop, and refine a viable intervention prototype for the engagement system. In phase 2, we will conduct a single-arm feasibility pilot of the engagement system among patients with prediabetes and their primary care providers. In phase 3, we will conduct a 2-arm randomized controlled trial using the engagement system. Primary outcomes will be weight, BMI, and A_1c at 6 and 12 months. Secondary outcomes will be patient engagement (use and activity) in the dDPP. The mediator variables (self-efficacy, digital health literacy, and patient-provider relationship) will be measured.

Results

The project was initiated in 2018 and funded in September 2019. Enrollment and data collection for phase 1 began in September 2019 under an Institutional Review Board quality improvement waiver granted in July 2019. As of December 2020, 27 patients have been enrolled and first results are expected to be submitted for publication in early 2021. The study received Institutional Review Board approval for phases 2 and 3 in December 2020, and phase 2 enrollment is expected to begin in early 2021.

Conclusions

Our findings will provide guidance for the design and development of technology to integrate dDPP platforms into existing clinical workflows. This will facilitate patient engagement in digital behavior change interventions and provider engagement in patients’ use of dDPPs. Integrated clinical tools that can facilitate patient-provider interaction around dDPPs may contribute to improved patient adherence to these programs and improved health outcomes by addressing barriers faced by both patients and providers. Further evaluation with pilot testing and a clinical trial will assess the effectiveness and implementation of these tools.

Trial Registration

ClinicalTrials.gov NCT04049500; https://clinicaltrials.gov/ct2/show/NCT04049500

International Registered Report Identifier (IRRID)

DERR1-10.2196/26750

Effectiveness of the e-Tabac Info Service application for smoking cessation: a pragmatic randomised controlled trial

OBJECTIVE

To compare the effectiveness of the mobile e-Tabac Info Service (e-TIS) application (app) for helping adult smokers quit smoking with current practices.

DESIGN

Pragmatic randomised controlled trial with a 1-year follow-up (2017-2018).

SETTING

France, population-wide level.

PARTICIPANTS

2806 adult smokers who wished to quit smoking were recruited via the website of the French National Mandatory Health Insurance fund. Of them, 1400 were randomised to the e-TIS app arm and 1406 were randomised to the current practices arm (control).

INTERVENTION

The app involved personalised interactive contacts that included questionnaires, advice, activities and text messages. All contacts were individually tailored and based on each smoker's progress.In the control group, recommended practices for quitting smoking were described on a non-interactive website.

PRIMARY AND SECONDARY OUTCOMES MEASURES

The primary outcome was 7-day point prevalence abstinence (PPA) at 6 months. The secondary outcomes included continuous abstinence rates at 6 and 12 months, minimum 24-hour point abstinence at 3 months, minimum 30-day point abstinence at 12 months and number and duration of quit attempts.

RESULTS

There was no difference between the e-TIS and control arms for the primary outcome (12.6% vs 13.7% for 7-day PPA at 6 months, p=0.3949, intention-to-treat analysis). However, e-TIS participants with high levels of exposure to the app, which was defined by the completion of at least eight activities or questionnaires, showed higher rates of smoking cessation than the control participants (17.6% vs 12.9% for 7-day PPA at 6 months, p=0.0169, per-protocol analysis).

CONCLUSION

Use of the e-TIS app was not associated with a higher rate of smoking cessation. However, high level of exposure to the e-TIS app may have been more effective than current practices.

TRIAL REGISTRATION NUMBER

NCT02841683.

Innovative Implementation Strategies for Hypertension Control in Low- and Middle-Income Countries: a Narrative Review

PURPOSE OF REVIEW

This review summarizes the most recent and innovative implementation strategies for hypertension control in low- and middle-income countries (LMICs).

RECENT FINDINGS

Implementation strategies from Latin America, Africa, and Asia were organized across three levels: community, health system, and policy/population. Multicomponent interventions involving task-shifting strategies, with or without mobile health tools, had the most supporting evidence, with policy or population-level interventions having the least, focused only on salt reduction with mixed results. More research is needed to better understand how context affects intervention implementation. There is an emerging evidence base for implementation strategies for hypertension control and CVD risk reduction in LMICs at the community and health system levels, but further research is needed to determine the most effective policy and population-level strategies. How to best account for local context in adapting and implementing these evidence-based interventions in LMICs still remains largely unknown. Accelerating the translation of this implementation research into policy and practice is imperative to improve health and save lives globally.

Administrative claims data to support pragmatic clinical trial outcome ascertainment on cardiovascular health

BACKGROUND/AIMS

Health plan administrative claims data present a cost-effective complement to traditional trial-specific ascertainment of clinical events typically conducted through patient report or a single health system electronic health record. We aim to demonstrate the value of health plan claims data in improving the capture of endpoints in longitudinal pragmatic clinical trials.

METHODS

This retrospective cohort study paralleled the design of the ADAPTABLE (Aspirin Dosing: A Patient-centric Trial Assessing Benefits and Long-Term Effectiveness) trial designed to compare the effectiveness of two doses of aspirin. We applied the ADAPTABLE identification query in claims data from Anthem, an American health insurance company, and identified health plan members who met the ADAPTABLE trial criteria. Among the ADAPTABLE eligible members, we selected overlapping members with PCORnet Clinical Data Research Networks in the 2 years prior to the index date (1 April 2014). PCORnet Clinical Data Research Networks consist of network partners (or healthcare systems) that store their electronic health record data in the same format to support multi-institutional research. ADAPTABLE outcome events-cardiovascular hospitalizations including admissions for myocardial infarction, stroke, or cardiac procedures; hospitalizations for major bleeding; and in-hospital deaths-were evaluated for a 2-year follow-up period. Events were classified as within or outside PCORnet Clinical Data Research Networks using facility identifiers affiliated with each hospital stay. Patient characteristics were examined with descriptive statistics, and incidence rates were reported for available Clinical Data Research Networks and claims data.

RESULTS

Among 884,311 ADAPTABLE eligible health plan members, 11,101 patients overlapped with PCORnet Clinical Data Research Networks. Average age was 70 years, 71% were male, and average follow-up was 20.7 months. Patients had 1521 cardiovascular hospitalizations (571 (37.5%) occurred outside PCORnet Clinical Data Research Networks), 710 for major bleeding (296 (41.7%) outside PCORnet Clinical Data Research Networks), and 196 in-hospital deaths (67 (34.2%) outside PCORnet Clinical Data Research Networks). Incidence rates (events per1000 patient-months) differed between available network partners and claims data: cardiovascular hospitalizations, 4.1 (95% confidence interval: 3.9, 4.4) versus 6.6 (95% confidence interval: 6.3, 7.0), major bleeding, 1.8 (95% confidence interval: 1.6, 2.0) versus 3.1 (95% confidence interval: 2.9, 3.3), and in-hospital death, 0.56 (95% confidence interval: 0.47, 0.67) versus 0.85 (95% confidence interval: 0.74, 0.98), respectively.

CONCLUSION

This study demonstrated the value of supplementing longitudinal site-based clinical studies with administrative claims data. Our results suggest that claims data together with network partner electronic health record data constitute an effective vehicle to capture patient outcomes since >30% of patients have non-fatal and fatal events outside of enrolling sites.

Feasibility and acceptability of two incentive-based implementation strategies for mental health therapists implementing cognitive-behavioral therapy: a pilot study to inform a randomized controlled trial

BACKGROUND

Informed by our prior work indicating that therapists do not feel recognized or rewarded for implementation of evidence-based practices, we tested the feasibility and acceptability of two incentive-based implementation strategies that seek to improve therapist adherence to cognitive-behavioral therapy for youth, an evidence-based practice.

METHODS

This study was conducted over 6 weeks in two community mental health agencies with therapists (n = 11) and leaders (n = 4). Therapists were randomized to receive either a financial or social incentive if they achieved a predetermined criterion on adherence to cognitive-behavioral therapy. In the first intervention period (block 1; 2 weeks), therapists received the reward they were initially randomized to if they achieved criterion. In the second intervention period (block 2; 2 weeks), therapists received both rewards if they achieved criterion. Therapists recorded 41 sessions across 15 unique clients over the project period. Primary outcomes included feasibility and acceptability. Feasibility was assessed quantitatively. Fifteen semi-structured interviews were conducted with therapists and leaders to assess acceptability. Difference in therapist adherence by condition was examined as an exploratory outcome. Adherence ratings were ascertained using an established and validated observational coding system of cognitive-behavioral therapy.

RESULTS

Both implementation strategies were feasible and acceptable-however, modifications to study design for the larger trial will be necessary based on participant feedback. With respect to our exploratory analysis, we found a trend suggesting the financial reward may have had a more robust effect on therapist adherence than the social reward.

CONCLUSIONS

Incentive-based implementation strategies can be feasibly administered in community mental health agencies with good acceptability, although iterative pilot work is essential. Larger, fully powered trials are needed to compare the effectiveness of implementation strategies to incentivize and enhance therapists' adherence to evidence-based practices such as cognitive-behavioral therapy.