Recruitment and implementation challenges were common in stepped-wedge cluster randomized trials: results from a methodological review

A review of current practice in the design and analysis of extremely small stepped-wedge cluster randomized trials

BACKGROUND/AIMS

Stepped-wedge cluster randomized trials tend to require fewer clusters than standard parallel-arm designs due to the switches between control and intervention conditions, but there are no recommendations for the minimum number of clusters. Trials randomizing an extremely small number of clusters are not uncommon, but the justification for small numbers of clusters is often unclear and appropriate analysis is often lacking. In addition, stepped-wedge cluster randomized trials are methodologically more complex due to their longitudinal correlation structure, and ignoring the distinct within- and between-period intracluster correlations can underestimate the sample size in small stepped-wedge cluster randomized trials. We conducted a review of published small stepped-wedge cluster randomized trials to understand how and why they are used, and to characterize approaches used in their design and analysis.

METHODS

Electronic searches were used to identify primary reports of full-scale stepped-wedge cluster randomized trials published during the period 2016-2022; the subset that randomized two to six clusters was identified. Two reviewers independently extracted information from each report and any available protocol. Disagreements were resolved through discussion.

RESULTS

We identified 61 stepped-wedge cluster randomized trials that randomized two to six clusters: median sample size (Q1-Q3) 1426 (420-7553) participants. Twelve (19.7%) gave some indication that the evaluation was considered a "preliminary" evaluation and 16 (26.2%) recognized the small number of clusters as a limitation. Sixteen (26.2%) provided an explanation for the limited number of clusters: the need to minimize contamination (e.g. by merging adjacent units), limited availability of clusters, and logistical considerations were common explanations. Majority (51, 83.6%) presented sample size or power calculations, but only one assumed distinct within- and between-period intracluster correlations. Few (10, 16.4%) utilized restricted randomization methods; more than half (34, 55.7%) identified baseline imbalances. The most common statistical method for analysis was the generalized linear mixed model (44, 72.1%). Only four trials (6.6%) reported statistical analyses considering small numbers of clusters: one used generalized estimating equations with small-sample correction, two used generalized linear mixed model with small-sample correction, and one used Bayesian analysis. Another eight (13.1%) used fixed-effects regression, the performance of which requires further evaluation under stepped-wedge cluster randomized trials with small numbers of clusters. None used permutation tests or cluster-period level analysis.

CONCLUSION

Methods appropriate for the design and analysis of small stepped-wedge cluster randomized trials have not been widely adopted in practice. Greater awareness is required that the use of standard sample size calculation methods can provide spuriously low numbers of required clusters. Methods such as generalized estimating equations or generalized linear mixed models with small-sample corrections, Bayesian approaches, and permutation tests may be more appropriate for the analysis of small stepped-wedge cluster randomized trials. Future research is needed to establish best practices for stepped-wedge cluster randomized trials with a small number of clusters.

A systematic review of stepped wedge cluster randomized trials in high impact journals: assessing the design, rationale, and analysis

OBJECTIVES

Stepped wedge cluster randomized trials (SW-CRTs) are an appealing study design because they enable sequential roll out of an intervention across clusters, bringing logistical advantages. This review aimed to evaluate the design rationale, design features, stepped wedge diagram, and analytical approaches of SW-CRTs published in high-impact medical journals from 2020 to 2023, focusing particularly on adherence to key guidelines from the Consolidated Standards of Reporting Trials extension to SW-CRTs.

STUDY DESIGN AND SETTING

We conducted searches across PubMed and Cochrane Central Registry of Controlled Trials databases for SW-CRTs published between January 2020 and December 2023 in eight high-impact journals. Eligibility criteria included peer-reviewed publications of randomized SW-CRTs involving human participants, published in English.

RESULTS

Of the 23 SW-CRTs included in the review, 70% had "stepped wedge" explicitly mentioned in their titles. Most studies (96%) included a stepped wedge diagram, but only 65% of these diagrams clearly communicated the duration of each time period. There was considerable variability in design features, including number of sequences (median of 7, range 3-20) and clusters (median of 15, range 9-19). The majority of trials (78%) provided robust justifications for selecting a SW-CRT design, for example, citing practical or logistical constraints. However, 22% of the studies offered less convincing rationales. Generalized linear mixed models were the most frequent analysis method employed.

CONCLUSION

Our review has highlighted areas for improvement in the presentation of SW-CRTs, particularly in clearly indicating the duration of time periods within diagrams and providing robust justifications for selecting a SW-CRT design.

PLAIN LANGUAGE SUMMARY

The stepped wedge cluster randomized trial (SW-CRT) is a type of study design that introduces interventions to different groups at different times. This review examined reports of SW-CRTs published in top medical journals from 2020 to 2023 to see if they followed certain guidelines such as including the word "stepped wedge" in their title. A total of 23 SW-CRTs were included in the review, with 70% mentioning "stepped wedge" in the title. Most (96%) included diagrams, but only 65% showed the duration of each time period clearly. There was variability in design, such as variations in the number of sequences and groups. 78% gave valid reasons for using SW-CRTs, citing practical benefits, whereas 22% did not give convincing reasons. This review suggests that improvements can be made in the presentation of stepped wedge diagrams and in the reporting of SW-CRTs. Researchers should clearly report the length of time periods and provide strong justifications for their design choice.

Overcoming challenges in real-world evidence generation: An example from an Adult Medical Care Coordination program

The Adult Medical Care Coordination program ("the program") was implemented at Mayo Clinic to promote patient self-management and improve 30-day unplanned readmission for patients with high risk for readmission after hospital discharge. This study aimed to evaluate the impact of the program compared to usual care using a pragmatic, stepped wedge cluster randomized trial ("stepped wedge trial"). However, several challenges arose including large differences between the study arms. Our goal is to describe the challenges and present lessons learned on how to overcome such challenges and generate evidence to support practice decisions. We describe the challenges encountered during the trial, the approach to addressing these challenges, and lessons learned for other learning health system researchers facing similar challenges. The trial experienced several challenges in implementation including several clinics dropping from the study and care disruptions due to COVID-19. Additionally, there were large differences in the patient population between the program and usual care arms. For example, the mean age was 76.8 for the program and 68.1 for usual care. Due to these differences, we adapted the methods using the propensity score matching approach that is traditionally applied to observational designs and adjusted for differences in observable characteristics. When conducting pragmatic research, researchers will encounter factors beyond their control that may introduce bias. The lessons learned include the need to weigh the tradeoffs of pragmatic design elements and the potential value of adaptive designs for pragmatic trials. Applying these lessons would promote the successful generation of evidence that informs practice decisions.

Adherence to key recommendations for design and analysis of stepped-wedge cluster randomized trials: A review of trials published 2016-2022

BACKGROUND/AIMS

The stepped-wedge cluster randomized trial (SW-CRT), in which clusters are randomized to a time at which they will transition to the intervention condition - rather than a trial arm - is a relatively new design. SW-CRTs have additional design and analytical considerations compared to conventional parallel arm trials. To inform future methodological development, including guidance for trialists and the selection of parameters for statistical simulation studies, we conducted a review of recently published SW-CRTs. Specific objectives were to describe (1) the types of designs used in practice, (2) adherence to key requirements for statistical analysis, and (3) practices around covariate adjustment. We also examined changes in adherence over time and by journal impact factor.

METHODS

We used electronic searches to identify primary reports of SW-CRTs published 2016-2022. Two reviewers extracted information from each trial report and its protocol, if available, and resolved disagreements through discussion.

RESULTS

We identified 160 eligible trials, randomizing a median (Q1-Q3) of 11 (8-18) clusters to 5 (4-7) sequences. The majority (122, 76%) were cross-sectional (almost all with continuous recruitment), 23 (14%) were closed cohorts and 15 (9%) open cohorts. Many trials had complex design features such as multiple or multivariate primary outcomes (50, 31%) or time-dependent repeated measures (27, 22%). The most common type of primary outcome was binary (51%); continuous outcomes were less common (26%). The most frequently used method of analysis was a generalized linear mixed model (112, 70%); generalized estimating equations were used less frequently (12, 8%). Among 142 trials with fewer than 40 clusters, only 9 (6%) reported using methods appropriate for a small number of clusters. Statistical analyses clearly adjusted for time effects in 119 (74%), for within-cluster correlations in 132 (83%), and for distinct between-period correlations in 13 (8%). Covariates were included in the primary analysis of the primary outcome in 82 (51%) and were most often individual-level covariates; however, clear and complete pre-specification of covariates was uncommon. Adherence to some key methodological requirements (adjusting for time effects, accounting for within-period correlation) was higher among trials published in higher versus lower impact factor journals. Substantial improvements over time were not observed although a slight improvement was observed in the proportion accounting for a distinct between-period correlation.

CONCLUSIONS

Future methods development should prioritize methods for SW-CRTs with binary or time-to-event outcomes, small numbers of clusters, continuous recruitment designs, multivariate outcomes, or time-dependent repeated measures. Trialists, journal editors, and peer reviewers should be aware that SW-CRTs have additional methodological requirements over parallel arm designs including the need to account for period effects as well as complex intracluster correlations.

What type of cluster randomized trial for which setting?

The cluster randomized trial allows a randomized evaluation when it is either not possible to randomize the individual or randomizing individuals would put the trial at high risk of contamination across treatment arms. There are many variations of the cluster randomized design, including the parallel design with or without baseline measures, the cluster randomized cross-over design, the stepped-wedge cluster randomized design, and more recently-developed variants such as the batched stepped-wedge design and the staircase design. Once it has been clearly established that there is a need for cluster randomization, one ever important question is which form the cluster design should take. If a design in which time is split into multiple trial periods is to be adopted (e.g. as in a stepped-wedge), researchers must decide whether the same participants should be measured in multiple trial periods (cohort sampling); or if different participants should be measured in each period (continual recruitment or cross-sectional sampling). Here we outline the different possible options and weigh up the pros and cons of the different design choices, which revolve around statistical efficiency, study logistics and the assumptions required.

Practical and methodological challenges when conducting a cluster randomized trial: Examples and recommendations

The use of cluster randomized trial design to answer research questions is increasing. This design and associated variants such as the cluster randomized crossover and stepped wedge are useful to assess complex interventions in a pragmatic way but when adopting such designs, one may face specific implementation challenges. This article summarizes common challenges faced when conducting cluster randomized trials, cluster randomized crossover trials, and stepped wedge trials, and provides recommendations.

Efficacy of a digitally supported regional systems intervention for suicide prevention (SUPREMOCOL) in Noord-Brabant, the Netherlands

OBJECTIVE

We evaluated the effect of a digitally supported systems intervention for suicide prevention (SUPREMOCOL) in Noord-Brabant, the Netherlands.

METHOD

Non-randomized stepped wedge trial design (SWTD). Stepwise implementation in the five subregions of the systems intervention. Pre-post analysis for the whole province (Exact Rate Ratio Test, Poisson count). SWTD Hazard Ratios of suicides per person-years for subregional analysis of control versus intervention conditions over five times three months. Sensitivity analysis.

RESULTS

Suicide rates dropped 17.8% (p = .013) from 14.4 suicides per 100,000 before the start of implementation of the systems intervention (2017), to 11.9 (2018) and 11.8 (2019) per 100, during implementation; a significant reduction (p = .043) compared to no changes in the rest of the Netherlands. Suicide rates dropped further by 21.5% (p = .002) to 11.3 suicides per 100,000 during sustained implementation in 2021. Sensitivity analysis confirmed the reduction (p = .02). The SWTD analysis over 15 months in 2018-2019 did not show a significant association of this reduction with implementation per subregional level, probably due to insufficient power given the short SWTD timeframe for implementation and low suicide rates per subregion.

CONCLUSIONS

During the SUPREMOCOL systems intervention, over four years, there was a sustained and significant reduction of suicides in Noord-Brabant.

POWER Up-Improving pre-exposure prophylaxis (PrEP) uptake among Black cisgender women in the Southern United States: Protocol for a stepped-wedge cluster randomized trial (SW-CRT)

BACKGROUND

HIV disproportionately affects Black/African American cisgender women (hereafter women) in the United States. Despite its proven effectiveness, pre-exposure prophylaxis (PrEP) for HIV prevention remains vastly under-prescribed to women based on their need. Increasing PrEP uptake and persistence among women is crucial to reducing HIV transmission; however, there have been few studies designed specifically for women. This article describes the study protocol used to assess the feasibility, acceptability, and effectiveness of implementation strategies to improve PrEP uptake and persistence among Black women in the Midwest and South.

METHODS

PrEP Optimization among Women to Enhance Retention and Uptake (POWER Up) is an evidence-based, woman-focused set of five implementation science strategies that addresses barriers of PrEP utilization at the provider, patient, and clinic levels. POWER Up includes 1) routine PrEP education for patients, 2) standardized provider training, 3) electronic medical record (EMR) optimization, 4) PrEP navigation, and 5) PrEP clinical champions. These strategies will be adapted to specific clinics for implementation, tested via a stepped-wedge trial, and, if effective, packaged for further dissemination.

DISCUSSION

We will utilize a stepped-wedge cluster randomized trial (SW-CRT) to measure change in PrEP utilization across diverse geographic areas. Preparation for adapting and implementing the bundle of strategies is needed to determine how to tailor them to specific clinics. Implementation challenges will include adapting strategies with the available resources at each site, maintaining stakeholder involvement and staff buy-in, adjusting the study protocol and planned procedures as needed, and ensuring minimal crossover. Additionally, strengths and limitations of each strategy must be examined before, during, and after the adaptation and implementation processes. Finally, the implementation outcomes of the strategies must be evaluated to determine the real-world success of the strategies. This study is an important step toward addressing the inequity in PrEP service delivery and increasing PrEP utilization among Black women in the U.S.

A scoping review described diversity in methods of randomization and reporting of baseline balance in stepped-wedge cluster randomized trials

OBJECTIVES

In stepped-wedge cluster randomized trials (SW-CRTs), clusters are randomized not to treatment and control arms but to sequences dictating the times of crossing from control to intervention conditions. Randomization is an essential feature of this design but application of standard methods to promote and report on balance at baseline is not straightforward. We aimed to describe current methods of randomization and reporting of balance at baseline in SW-CRTs.

STUDY DESIGN AND SETTING

We used electronic searches to identify primary reports of SW-CRTs published between 2016 and 2022.

RESULTS

Across 160 identified trials, the median number of clusters randomized was 11 (Q1-Q3: 8-18). Sixty-three (39%) used restricted randomization-most often stratification based on a single cluster-level covariate; 12 (19%) of these adjusted for the covariate(s) in the primary analysis. Overall, 50 (31%) and 134 (84%) reported on balance at baseline on cluster- and individual-level characteristics, respectively. Balance on individual-level characteristics was most often reported by condition in cross-sectional designs and by sequence in cohort designs. Authors reported baseline imbalances in 72 (45%) trials.

CONCLUSION

SW-CRTs often randomize a small number of clusters using unrestricted allocation. Investigators need guidance on appropriate methods of randomization and assessment and reporting of balance at baseline.