Systematic review showed that stepped-wedge cluster randomized trials often did not reach their planned sample size

A systematic review of stepped wedge cluster randomized trials in high impact journals: assessing the design, rationale, and analysis

OBJECTIVES

Stepped wedge cluster randomized trials (SW-CRTs) are an appealing study design because they enable sequential roll out of an intervention across clusters, bringing logistical advantages. This review aimed to evaluate the design rationale, design features, stepped wedge diagram, and analytical approaches of SW-CRTs published in high-impact medical journals from 2020 to 2023, focusing particularly on adherence to key guidelines from the Consolidated Standards of Reporting Trials extension to SW-CRTs.

STUDY DESIGN AND SETTING

We conducted searches across PubMed and Cochrane Central Registry of Controlled Trials databases for SW-CRTs published between January 2020 and December 2023 in eight high-impact journals. Eligibility criteria included peer-reviewed publications of randomized SW-CRTs involving human participants, published in English.

RESULTS

Of the 23 SW-CRTs included in the review, 70% had "stepped wedge" explicitly mentioned in their titles. Most studies (96%) included a stepped wedge diagram, but only 65% of these diagrams clearly communicated the duration of each time period. There was considerable variability in design features, including number of sequences (median of 7, range 3-20) and clusters (median of 15, range 9-19). The majority of trials (78%) provided robust justifications for selecting a SW-CRT design, for example, citing practical or logistical constraints. However, 22% of the studies offered less convincing rationales. Generalized linear mixed models were the most frequent analysis method employed.

CONCLUSION

Our review has highlighted areas for improvement in the presentation of SW-CRTs, particularly in clearly indicating the duration of time periods within diagrams and providing robust justifications for selecting a SW-CRT design.

PLAIN LANGUAGE SUMMARY

The stepped wedge cluster randomized trial (SW-CRT) is a type of study design that introduces interventions to different groups at different times. This review examined reports of SW-CRTs published in top medical journals from 2020 to 2023 to see if they followed certain guidelines such as including the word "stepped wedge" in their title. A total of 23 SW-CRTs were included in the review, with 70% mentioning "stepped wedge" in the title. Most (96%) included diagrams, but only 65% showed the duration of each time period clearly. There was variability in design, such as variations in the number of sequences and groups. 78% gave valid reasons for using SW-CRTs, citing practical benefits, whereas 22% did not give convincing reasons. This review suggests that improvements can be made in the presentation of stepped wedge diagrams and in the reporting of SW-CRTs. Researchers should clearly report the length of time periods and provide strong justifications for their design choice.

Adherence to key recommendations for design and analysis of stepped-wedge cluster randomized trials: A review of trials published 2016-2022

BACKGROUND/AIMS

The stepped-wedge cluster randomized trial (SW-CRT), in which clusters are randomized to a time at which they will transition to the intervention condition - rather than a trial arm - is a relatively new design. SW-CRTs have additional design and analytical considerations compared to conventional parallel arm trials. To inform future methodological development, including guidance for trialists and the selection of parameters for statistical simulation studies, we conducted a review of recently published SW-CRTs. Specific objectives were to describe (1) the types of designs used in practice, (2) adherence to key requirements for statistical analysis, and (3) practices around covariate adjustment. We also examined changes in adherence over time and by journal impact factor.

METHODS

We used electronic searches to identify primary reports of SW-CRTs published 2016-2022. Two reviewers extracted information from each trial report and its protocol, if available, and resolved disagreements through discussion.

RESULTS

We identified 160 eligible trials, randomizing a median (Q1-Q3) of 11 (8-18) clusters to 5 (4-7) sequences. The majority (122, 76%) were cross-sectional (almost all with continuous recruitment), 23 (14%) were closed cohorts and 15 (9%) open cohorts. Many trials had complex design features such as multiple or multivariate primary outcomes (50, 31%) or time-dependent repeated measures (27, 22%). The most common type of primary outcome was binary (51%); continuous outcomes were less common (26%). The most frequently used method of analysis was a generalized linear mixed model (112, 70%); generalized estimating equations were used less frequently (12, 8%). Among 142 trials with fewer than 40 clusters, only 9 (6%) reported using methods appropriate for a small number of clusters. Statistical analyses clearly adjusted for time effects in 119 (74%), for within-cluster correlations in 132 (83%), and for distinct between-period correlations in 13 (8%). Covariates were included in the primary analysis of the primary outcome in 82 (51%) and were most often individual-level covariates; however, clear and complete pre-specification of covariates was uncommon. Adherence to some key methodological requirements (adjusting for time effects, accounting for within-period correlation) was higher among trials published in higher versus lower impact factor journals. Substantial improvements over time were not observed although a slight improvement was observed in the proportion accounting for a distinct between-period correlation.

CONCLUSIONS

Future methods development should prioritize methods for SW-CRTs with binary or time-to-event outcomes, small numbers of clusters, continuous recruitment designs, multivariate outcomes, or time-dependent repeated measures. Trialists, journal editors, and peer reviewers should be aware that SW-CRTs have additional methodological requirements over parallel arm designs including the need to account for period effects as well as complex intracluster correlations.

What type of cluster randomized trial for which setting?

The cluster randomized trial allows a randomized evaluation when it is either not possible to randomize the individual or randomizing individuals would put the trial at high risk of contamination across treatment arms. There are many variations of the cluster randomized design, including the parallel design with or without baseline measures, the cluster randomized cross-over design, the stepped-wedge cluster randomized design, and more recently-developed variants such as the batched stepped-wedge design and the staircase design. Once it has been clearly established that there is a need for cluster randomization, one ever important question is which form the cluster design should take. If a design in which time is split into multiple trial periods is to be adopted (e.g. as in a stepped-wedge), researchers must decide whether the same participants should be measured in multiple trial periods (cohort sampling); or if different participants should be measured in each period (continual recruitment or cross-sectional sampling). Here we outline the different possible options and weigh up the pros and cons of the different design choices, which revolve around statistical efficiency, study logistics and the assumptions required.

Protocol-driven primary care and community linkage to reduce all-cause mortality in rural Zambia: a stepped-wedge cluster randomized trial

Introduction

While tremendous progress has been made in recent years to improve the health of people living in low- and middle-income countries (LMIC), significant challenges remain. Chief among these are poor health systems, which are often ill-equipped to respond to current challenges. It remains unclear whether intensive intervention at the health system level will result in improved outcomes, as there have been few rigorously designed comparative studies. We present results of a complex health system intervention that was implemented in Zambia using a cluster randomized design.

Methods

BHOMA was a complex health system intervention comprising intensive clinical training and quality improvement measures, support for commodities procurement, improved community outreach, and district level management support. The intervention was introduced as a stepped wedge cluster-randomized trial in 42 predominately rural health centers and their surrounding communities in Lusaka Province, Zambia. Baseline survey was conducted between January-May 2011, mid-line survey was conducted February-November, 2013 and Endline survey, February-November 2015.The primary outcome was all-cause mortality among those between 28 days and 60 years of age and assessed through community-based mortality surveys. Secondary outcomes included post-neonatal under-five mortality and service coverage scores. Service coverage scores were calculated across five domains (child preventative services; child treatment services; family planning; maternal health services, and adult health services). We fit Cox proportional hazards model with shared frailty at the cluster level for the primary analysis. Mortality rates were age-standardized using the WHO World Standard Population.

Results

Mortality declined substantially from 3.9 per 1,000 person-years in the pre-intervention period, to 1.5 per 1,000 person-years in the post intervention period. When we compared intervention and control periods, there were 174 deaths in 49,230 person years (age-standardized rate = 4.4 per 1,000 person-years) in the control phase and 277 deaths in 74,519 person years (age-standardized rate = 4.6 per 1,000 person-years) in the intervention phase. Overall, there was no evidence for an effect of the intervention in minimally-adjusted [hazard ratio (HR) = 1.18; 95% confidence interval (CI): 0.88, 1.56; value of p = 0.265], or adjusted (HR = 1.12; 95% CI: 0.84, 1.49; value of p = 0.443) analyses.Coverage scores that showed some evidence of changing with time since the cluster joined the intervention were: an increasing proportion of children sleeping under insecticide treated bed-net (value of p < 0.001); an increasing proportion of febrile children who received appropriate anti-malarial drugs (value of p = 0.039); and an increasing proportion of ever hypertensive adults with currently controlled hypertension (value of p = 0.047). No adjustments were made for multiple-testing and the overall coverage score showed no statistical evidence for a change over time (value of p = 0.308).

Conclusion

We noted an overall reduction in post-neonatal under 60 mortality in the study communities during the period of our study, but this could not be attributed to the BHOMA intervention. Some improvements in service coverage scores were observed.

Clinical Trial Registration

clinicaltrials.gov, Identifier NCT01942278.

Influential methods reports for group-randomized trials and related designs

BACKGROUND

This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial.

METHODS

PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article.

RESULTS

In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were "first reports" on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design.

CONCLUSIONS

Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

A first step towards best practice recommendations for the design and statistical analyses of pragmatic clinical trials: a modified Delphi approach

AIM

Pragmatic clinical trials (PCTs) are randomised trials implemented through routine clinical practice, where design parameters of traditional randomised controlled trials are modified to increase generalizability. However, this may introduce statistical challenges. We aimed to identify these challenges and discuss possible solutions leading to best practice recommendations for the design and analysis of PCTs.

METHODS

A modified Delphi method was used to reach consensus among a panel of 11 experts in clinical trials and statistics. Statistical issues were identified in a focused literature review and aggregated with insights and possible solutions from expert collected through a series of survey iterations. Issues were ranked according to their importance.

RESULTS

27 articles were included and combined with experts' insight to generate a list of issues categorized into: participants; recruiting sites; randomisation, blinding and intervention; outcome (selection and measurement); and data analysis. Consensus was reached about the most important issues: risk of participants' attrition; heterogeneity of "usual care" across sites; absence of blinding; use of a subjective endpoint; and data analysis aligned with the trial estimand. Potential issues should be anticipated and preferably be addressed in the trial protocol. The experts provided solutions regarding data collection and data analysis, which were considered of equal importance.

DISCUSSION

A set of important statistical issues in PCTs was identified and approaches were suggested to anticipate and/or minimize these through data analysis. Any impact of choosing a pragmatic design feature should be gauged in the light of the trial estimand.

Recruitment and implementation challenges were common in stepped-wedge cluster randomized trials: results from a methodological review

OBJECTIVE

To explore challenges in recruitment and intervention implementation in recent stepped-wedge cluster randomized trials (SW-CRTs).

STUDY DESIGN AND SETTING

We searched PubMed to identify primary reports of SW-CRTs (2019-2020). Two reviewers independently screened studies and extracted data from each report. Recruitment challenge was defined as planned number of clusters or participants not achieved, or any reported changes made to the design to address recruitment difficulties. Implementation challenge was defined as early, late or no implementation of the intervention in at least one cluster.

RESULTS

Of 55 SW-CRTs, 18 (33%) had a recruitment challenge, 23 (42%) had none, and for 14 (26%) it was impossible to judge. At least one implementation challenge was present in 24 (44%), 8 (15%) had none, and for 23 (42%) it was impossible to judge. Of the 35 (64%) trials with recruitment or implementation challenges, 18 (72%) had one or more modifications of their design, most often a modification of the trial duration.

CONCLUSION

Investigators must be aware of the risks of recruitment or implementation challenges when considering use of a SW-CRT design. Mitigating strategies should be adopted when planning the trial. More transparent reporting of planned and actual design features is required.

An individualized decision aid for physicians and patients for total knee replacement in osteoarthritis (Value-based TKR study): study protocol for a multi-center, stepped wedge, cluster randomized controlled trial

Background

Total knee replacement (TKR) is one of the most commonly performed routine procedures in the world. Prognostic studies indicate that the number of TKR will further increase constituting growing burden on healthcare systems. There is also substantial regional heterogeneity in TKR rates within and between countries. Despite the known therapeutic effects, a subset of patients undergoing TKR does not benefit from the procedure as intended. To improve the appropriateness of TKR indication, the EKIT initiative (“evidence and consensus based indication critera for total arthroplasty”) developed a clinical guideline for Germany on the indication of TKR. This guideline is the basis for a digital medical decision aid (EKIT tool) to facilitate shared decision making (SDM) in order to improve decision quality for elective surgery. The aim of this cluster randomized trial is to investigate the effectiveness of the EKIT tool on decision quality.

Methods

The Value-based TKR study is a prospective pragmatic multi-center, stepped wedge, cluster randomized controlled trial (SW-RCT). The EKIT tool provides (1) a systematic presentation of individual patient and disease-specific information (symptoms, expectations), (2) the fulfillment of the indication criteria and (3) health information about safety and effectiveness of TKR. All study sites will follow routine care as control clusters until the start of the intervention. In total, there will be 10 clusters (study sites) and 6 sequential steps over 16 month, with clusters receiving the intervention with a minimum 2 months of standard routine care. The primary outcome is patients’ decision quality measured with the Decision Quality Instrument (DQI)-Knee Osteoarthritis questionnaire. Furthermore, we will collect information on global patient satisfaction, patient reported outcome measures and the fulfilment of the individual expectations 12 months after SDM. The power calculation yielded an estimated power of 89% using robust Poisson regression under the following assumptions: 10 study sites with a total of N=1,080 patients (including a dropout rate of 11%), a 10% increase in decision quality due to the use of the EKIT tool, and a significance level of 5%.

Discussion

There is a high potential for transferring the intervention into routine practice if the evaluation is positive.

Trial registration

ClinicalTrials.gov: NCT04837053. Registered on 08/04/2021.

The role and challenges of cluster randomised trials for global health

Evaluating whether an intervention works when trialled in groups of individuals can pose complex challenges for clinical research. Cluster randomised controlled trials involve the random allocation of groups or clusters of individuals to receive an intervention, and they are commonly used in global health research. In this paper, we describe the potential reasons for the increasing popularity of cluster trials in low-income and middle-income countries. We also draw on key areas of global health research for an assessment of common trial planning practices, and we address their methodological shortcomings and pitfalls. Lastly, we discuss alternative approaches for population-level intervention trials that could be useful for research undertaken in low-income and middle-income countries for situations in which the use of cluster randomisation might not be appropriate.

Sample size and power calculations for open cohort longitudinal cluster randomized trials

When calculating sample size or power for stepped wedge or other types of longitudinal cluster randomized trials, it is critical that the planned sampling structure be accurately specified. One common assumption is that participants will provide measurements in each trial period, that is, a closed cohort, and another is that each participant provides only one measurement during the course of the trial. However some studies have an "open cohort" sampling structure, where participants may provide measurements in variable numbers of periods. To date, sample size calculations for longitudinal cluster randomized trials have not accommodated open cohorts. Feldman and McKinlay (1994) provided some guidance, stating that the participant-level autocorrelation could be varied to account for the degree of overlap in different periods of the study, but did not indicate precisely how to do so. We present sample size and power formulas that allow for open cohorts and discuss the impact of the degree of "openness" on sample size and power. We consider designs where the number of participants in each cluster will be maintained throughout the trial, but individual participants may provide differing numbers of measurements. Our results are a unification of closed cohort and repeated cross-sectional sample results of Hooper et al (2016), and indicate precisely how participant autocorrelation of Feldman and McKinlay should be varied to account for an open cohort sampling structure. We discuss different types of open cohort sampling schemes and how open cohort sampling structure impacts on power in the presence of decaying within-cluster correlations and autoregressive participant-level errors.

Solar-powered oxygen delivery for the treatment of children with hypoxemia: protocol for a cluster-randomized stepped-wedge controlled trial in Uganda

BACKGROUND

Child mortality due to pneumonia is a major global health problem and is associated with hypoxemia. Access to safe and continuous oxygen therapy can reduce mortality; however, low-income countries may lack the necessary resources for oxygen delivery. We have previously demonstrated proof-of-concept that solar-powered oxygen (SPO2) delivery can reliably provide medical oxygen remote settings with minimal access to electricity. This study aims to demonstrate the efficacy of SPO2 in children hospitalized with acute hypoxemic respiratory illness across Uganda.

METHODS

Objectives: Demonstrate efficacy of SPO2 in children hospitalized with acute hypoxemic respiratory illness.

STUDY DESIGN

Multi-center, stepped-wedge cluster-randomized trial.

SETTING

Twenty health facilities across Uganda, a low-income, high-burden country for pediatric pneumonia. Site selection: Facilities with pediatric inpatient services lacking consistent O2 supply on pediatric wards.

PARTICIPANTS

Children aged < 5 years hospitalized with hypoxemia (saturation < 92%) warranting hospital admission based on clinical judgement. Randomization methods: Random installation order generated a priori with allocation concealment. Study procedure: Patients receive standard of care within pediatric wards with or without SPO2 system installed.

OUTCOME MEASURES

Primary: 48-h mortality. Secondary: safety, efficacy, SPO2 system functionality, operating costs, nursing knowledge, skills, and retention for oxygen administration. Statistical analysis of primary outcome: Linear mixed effects logistic regression model with 48-h mortality (dependent variable) as a function of SPO2 treatment (before versus after installation), while adjusting for confounding effects of calendar time (fixed effect) and site (random effect).

SAMPLE SIZE

2400 patients across 20 health facilities, predicted to provide 80% power to detect a 35% reduction in mortality after introduction of SPO2, based on a computer simulation of > 5000 trials.

DISCUSSION

Overall, our study aims to demonstrate mortality benefit of SPO2 relative to standard (unreliable) oxygen delivery. The innovative trial design (stepped-wedge, cluster-randomized) is supported by a computer simulation. Capacity building for nursing care and oxygen therapy is a non-scientific objective of the study. If successful, SPO2 could be scaled across a variety of resource-constrained remote or rural settings in sub-Saharan Africa and beyond.

TRIAL REGISTRATION

Clinicaltrials.gov, NCT03851783. Registered on 22 February 2019.

Delaying and withholding interventions: ethics and the stepped wedge trial

Ethics has been identified as a central reason for choosing the stepped wedge trial over other kinds of trial designs. The potential advantage of the stepped wedge design is that it provides all arms of the trial with the active intervention over the course of the study. Some groups receive it later than others, but the study intervention is not withheld from any group. This feature of the stepped wedge design seems particularly ethically advantageous in two instances: (1) when the study intervention appears especially likely to be effective and (2) when the consequences of not receiving the intervention may be dire. But despite an increase in the use of the stepped wedge design and appeals to its ethical superiority as the motivation for its selection, there has been limited attention to the stepped wedge trial in the ethics literature. In the following, I examine whether there are persuasive ethical reasons to prefer or to require a stepped wedge trial. I argue that while the stepped wedge design is ethically permissible, it is not morally superior to other kinds of trials. To this end, I examine the ethical justification for providing, withholding, and delaying interventions in research.

Study protocol for a stepped-wedge randomized cookstove intervention in rural Honduras: household air pollution and cardiometabolic health

BACKGROUND

Growing evidence links household air pollution exposure from biomass-burning cookstoves to cardiometabolic disease risk. Few randomized controlled interventions of cookstoves (biomass or otherwise) have quantitatively characterized changes in exposure and indicators of cardiometabolic health, a growing and understudied burden in low- and middle-income countries (LMICs). Ideally, the solution is to transition households to clean cooking, such as with electric or liquefied petroleum gas stoves; however, those unable to afford or to access these options will continue to burn biomass for the foreseeable future. Wood-burning cookstove designs such as the Justa (incorporating an engineered combustion zone and chimney) have the potential to substantially reduce air pollution exposures. Previous cookstove intervention studies have been limited by stove types that did not substantially reduce exposures and/or by low cookstove adoption and sustained use, and few studies have incorporated community-engaged approaches to enhance the intervention.

METHODS/DESIGN

We conducted an individual-level, stepped-wedge randomized controlled trial with the Justa cookstove intervention in rural Honduras. We enrolled 230 female primary cooks who were not pregnant, non-smoking, aged 24-59 years old, and used traditional wood-burning cookstoves at baseline. A community advisory board guided survey development and communication with participants, including recruitment and retention strategies. Over a 3-year study period, participants completed 6 study visits approximately 6 months apart. Half of the women received the Justa after visit 2 and half after visit 4. At each visit, we measured 24-h gravimetric personal and kitchen fine particulate matter (PM2.5) concentrations, qualitative and quantitative cookstove use and adoption metrics, and indicators of cardiometabolic health. The primary health endpoints were blood pressure, C-reactive protein, and glycated hemoglobin. Overall study goals are to explore barriers and enablers of new cookstove adoption and sustained use, compare health endpoints by assigned cookstove type, and explore the exposure-response associations between PM2.5 and indicators of cardiometabolic health.

DISCUSSION

This trial, utilizing an economically feasible, community-vetted cookstove and evaluating endpoints relevant for the major causes of morbidity and mortality in LMICs, will provide critical information for household air pollution stakeholders globally.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier NCT02658383 , posted January 18, 2016, field work completed May 2018. Official title, "Community-Based Participatory Research: A Tool to Advance Cookstove Interventions." Principal Investigator Maggie L. Clark, Ph.D. Last update posted July 12, 2018.