Sample size requirements for detecting treatment effect heterogeneity in cluster randomized trials

Using Power Analysis to Choose the Unit of Randomization, Outcome, and Approach for Subgroup Analysis for a Multilevel Randomized Controlled Clinical Trial to Reduce Disparities in Cardiovascular Health

We give examples of three features in the design of randomized controlled clinical trials which can increase power and thus decrease sample size and costs. We consider an example multilevel trial with several levels of clustering. For a fixed number of independent sampling units, we show that power can vary widely with the choice of the level of randomization. We demonstrate that power and interpretability can improve by testing a multivariate outcome rather than an unweighted composite outcome. Finally, we show that using a pooled analytic approach, which analyzes data for all subgroups in a single model, improves power for testing the intervention effect compared to a stratified analysis, which analyzes data for each subgroup in a separate model. The power results are computed for a proposed prevention research study. The trial plans to randomize adults to either telehealth (intervention) or in-person treatment (control) to reduce cardiovascular risk factors. The trial outcomes will be measures of the Essential Eight, a set of scores for cardiovascular health developed by the American Heart Association which can be combined into a single composite score. The proposed trial is a multilevel study, with outcomes measured on participants, participants treated by the same provider, providers nested within clinics, and clinics nested within hospitals. Investigators suspect that the intervention effect will be greater in rural participants, who live farther from clinics than urban participants. The results use published, exact analytic methods for power calculations with continuous outcomes. We provide example code for power analyses using validated software.

Assessing treatment effect heterogeneity in the presence of missing effect modifier data in cluster-randomized trials

Understanding whether and how treatment effects vary across subgroups is crucial to inform clinical practice and recommendations. Accordingly, the assessment of heterogeneous treatment effects based on pre-specified potential effect modifiers has become a common goal in modern randomized trials. However, when one or more potential effect modifiers are missing, complete-case analysis may lead to bias and under-coverage. While statistical methods for handling missing data have been proposed and compared for individually randomized trials with missing effect modifier data, few guidelines exist for the cluster-randomized setting, where intracluster correlations in the effect modifiers, outcomes, or even missingness mechanisms may introduce further threats to accurate assessment of heterogeneous treatment effect. In this article, the performance of several missing data methods are compared through a simulation study of cluster-randomized trials with continuous outcome and missing binary effect modifier data, and further illustrated using real data from the Work, Family, and Health Study. Our results suggest that multilevel multiple imputation and Bayesian multilevel multiple imputation have better performance than other available methods, and that Bayesian multilevel multiple imputation has lower bias and closer to nominal coverage than standard multilevel multiple imputation when there are model specification or compatibility issues.

Sample size and power calculation for testing treatment effect heterogeneity in cluster randomized crossover designs

The cluster randomized crossover design has been proposed to improve efficiency over the traditional parallel-arm cluster randomized design. While statistical methods have been developed for designing cluster randomized crossover trials, they have exclusively focused on testing the overall average treatment effect, with little attention to differential treatment effects across subpopulations. Recently, interest has grown in understanding whether treatment effects may vary across pre-specified patient subpopulations, such as those defined by demographic or clinical characteristics. In this article, we consider the two-treatment two-period cluster randomized crossover design under either a cross-sectional or closed-cohort sampling scheme, where it is of interest to detect the heterogeneity of treatment effect via an interaction test. Assuming a patterned correlation structure for both the covariate and the outcome, we derive new sample size formulas for testing the heterogeneity of treatment effect with continuous outcomes based on linear mixed models. Our formulas also address unequal cluster sizes and therefore allow us to analytically assess the impact of unequal cluster sizes on the power of the interaction test in cluster randomized crossover designs. We conduct simulations to confirm the accuracy of the proposed methods, and illustrate their application in two real cluster randomized crossover trials.

Power calculation for detecting interaction effect in cross-sectional stepped-wedge cluster randomized trials: an important tool for disparity research

BACKGROUND

The stepped-wedge cluster randomized trial (SW-CRT) design has become popular in healthcare research. It is an appealing alternative to traditional cluster randomized trials (CRTs) since the burden of logistical issues and ethical problems can be reduced. Several approaches for sample size determination for the overall treatment effect in the SW-CRT have been proposed. However, in certain situations we are interested in examining the heterogeneity in treatment effect (HTE) between groups instead. This is equivalent to testing the interaction effect. An important example includes the aim to reduce racial disparities through healthcare delivery interventions, where the focus is the interaction between the intervention and race. Sample size determination and power calculation for detecting an interaction effect between the intervention status variable and a key covariate in the SW-CRT study has not been proposed yet for binary outcomes.

METHODS

We utilize the generalized estimating equation (GEE) method for detecting the heterogeneity in treatment effect (HTE). The variance of the estimated interaction effect is approximated based on the GEE method for the marginal models. The power is calculated based on the two-sided Wald test. The Kauermann and Carroll (KC) and the Mancl and DeRouen (MD) methods along with GEE (GEE-KC and GEE-MD) are considered as bias-correction methods.

RESULTS

Among three approaches, GEE has the largest simulated power and GEE-MD has the smallest simulated power. Given cluster size of 120, GEE has over 80% statistical power. When we have a balanced binary covariate (50%), simulated power increases compared to an unbalanced binary covariate (30%). With intermediate effect size of HTE, only cluster sizes of 100 and 120 have more than 80% power using GEE for both correlation structures. With large effect size of HTE, when cluster size is at least 60, all three approaches have more than 80% power. When we compare an increase in cluster size and increase in the number of clusters based on simulated power, the latter has a slight gain in power. When the cluster size changes from 20 to 40 with 20 clusters, power increases from 53.1% to 82.1% for GEE; 50.6% to 79.7% for GEE-KC; and 48.1% to 77.1% for GEE-MD. When the number of clusters changes from 20 to 40 with cluster size of 20, power increases from 53.1% to 82.1% for GEE; 50.6% to 81% for GEE-KC; and 48.1% to 79.8% for GEE-MD.

CONCLUSIONS

We propose three approaches for cluster size determination given the number of clusters for detecting the interaction effect in SW-CRT. GEE and GEE-KC have reasonable operating characteristics for both intermediate and large effect size of HTE.

Planning stepped wedge cluster randomized trials to detect treatment effect heterogeneity

Stepped wedge design is a popular research design that enables a rigorous evaluation of candidate interventions by using a staggered cluster randomization strategy. While analytical methods were developed for designing stepped wedge trials, the prior focus has been solely on testing for the average treatment effect. With a growing interest on formal evaluation of the heterogeneity of treatment effects across patient subpopulations, trial planning efforts need appropriate methods to accurately identify sample sizes or design configurations that can generate evidence for both the average treatment effect and variations in subgroup treatment effects. To fill in that important gap, this article derives novel variance formulas for confirmatory analyses of treatment effect heterogeneity, that are applicable to both cross-sectional and closed-cohort stepped wedge designs. We additionally point out that the same framework can be used for more efficient average treatment effect analyses via covariate adjustment, and allows the use of familiar power formulas for average treatment effect analyses to proceed. Our results further sheds light on optimal design allocations of clusters to maximize the weighted precision for assessing both the average and heterogeneous treatment effects. We apply the new methods to the Lumbar Imaging with Reporting of Epidemiology Trial, and carry out a simulation study to validate our new methods.

Hierarchical Bayesian modeling of heterogeneous outcome variance in cluster randomized trials

BACKGROUND

Heterogeneous outcome correlations across treatment arms and clusters have been increasingly acknowledged in cluster randomized trials with binary endpoints, where analytical methods have been developed to study such heterogeneity. However, cluster-specific outcome variances and correlations have yet to be studied for cluster randomized trials with continuous outcomes.

METHODS

This article proposes models fitted in the Bayesian setting with hierarchical variance structure to quantify heterogeneous variances across clusters and explain it with cluster-level covariates when the outcome is continuous. The models can also be extended to analyzing heterogeneous variances in individually randomized group treatment trials, with arm-specific cluster-level covariates, or in partially nested designs. Simulation studies are carried out to validate the performance of the newly introduced models across different settings.

RESULTS

Simulations showed that overall the newly introduced models have good performance, reporting low bias and approximately 95% coverage for the intraclass correlation coefficients and regression parameters in the variance model. When variances are heterogeneous, our proposed models had improved model fit over models with homogeneous variances. When used to analyze data from the Kerala Diabetes Prevention Program study, our models identified heterogeneous variances and intraclass correlation coefficients across clusters and examined cluster-level characteristics associated with such heterogeneity.

CONCLUSION

We proposed new hierarchical Bayesian variance models to accommodate cluster-specific variances in cluster randomized trials. The newly developed methods inform the understanding of how an intervention strategy is implemented and disseminated differently across clusters and can help improve future trial design.

Sample size requirements for testing treatment effect heterogeneity in cluster randomized trials with binary outcomes

Cluster randomized trials (CRTs) refer to a popular class of experiments in which randomization is carried out at the group level. While methods have been developed for planning CRTs to study the average treatment effect, and more recently, to study the heterogeneous treatment effect, the development for the latter objective has currently been limited to a continuous outcome. Despite the prevalence of binary outcomes in CRTs, determining the necessary sample size and statistical power for detecting differential treatment effects in CRTs with a binary outcome remain unclear. To address this methodological gap, we develop sample size procedures for testing treatment effect heterogeneity in two-level CRTs under a generalized linear mixed model. Closed-form sample size expressions are derived for a binary effect modifier, and in addition, a computationally efficient Monte Carlo approach is developed for a continuous effect modifier. Extensions to multiple effect modifiers are also discussed. We conduct simulations to examine the accuracy of the proposed sample size methods. We present several numerical illustrations to elucidate features of the proposed formulas and to compare our method to the approximate sample size calculation under a linear mixed model. Finally, we use data from the Strategies and Opportunities to Stop Colon Cancer in Priority Populations (STOP CRC) CRT to illustrate the proposed sample size procedure for testing treatment effect heterogeneity.

Designing three-level cluster randomized trials to assess treatment effect heterogeneity

Cluster randomized trials often exhibit a three-level structure with participants nested in subclusters such as health care providers, and subclusters nested in clusters such as clinics. While the average treatment effect has been the primary focus in planning three-level randomized trials, interest is growing in understanding whether the treatment effect varies among prespecified patient subpopulations, such as those defined by demographics or baseline clinical characteristics. In this article, we derive novel analytical design formulas based on the asymptotic covariance matrix for powering confirmatory analyses of treatment effect heterogeneity in three-level trials, that are broadly applicable to the evaluation of cluster-level, subcluster-level, and participant-level effect modifiers and to designs where randomization can be carried out at any level. We characterize a nested exchangeable correlation structure for both the effect modifier and the outcome conditional on the effect modifier, and generate new insights from a study design perspective for conducting analyses of treatment effect heterogeneity based on a linear mixed analysis of covariance model. A simulation study is conducted to validate our new methods and two real-world trial examples are used for illustrations.

Sample Size Requirements to Test Subgroup-Specific Treatment Effects in Cluster-Randomized Trials

Cluster-randomized trials (CRTs) often allocate intact clusters of participants to treatment or control conditions and are increasingly used to evaluate healthcare delivery interventions. While previous studies have developed sample size methods for testing confirmatory hypotheses of treatment effect heterogeneity in CRTs (i.e., targeting the difference between subgroup-specific treatment effects), sample size methods for testing the subgroup-specific treatment effects themselves have not received adequate attention-despite a rising interest in health equity considerations in CRTs. In this article, we develop formal methods for sample size and power analyses for testing subgroup-specific treatment effects in parallel-arm CRTs with a continuous outcome and a binary subgroup variable. We point out that the variances of the subgroup-specific treatment effect estimators and their covariance are given by weighted averages of the variance of the overall average treatment effect estimator and the variance of the heterogeneous treatment effect estimator. This analytical insight facilitates an explicit characterization of the requirements for both the omnibus test and the intersection-union test to achieve the desired level of power. Generalizations to allow for subgroup-specific variance structures are also discussed. We report on a simulation study to validate the proposed sample size methods and demonstrate that the empirical power corresponds well with the predicted power for both tests. The design and setting of the Umea Dementia and Exercise (UMDEX) CRT in older adults are used to illustrate our sample size methods.

Estimating intra-cluster correlation coefficients for planning longitudinal cluster randomized trials: a tutorial

It is well-known that designing a cluster randomized trial (CRT) requires an advance estimate of the intra-cluster correlation coefficient (ICC). In the case of longitudinal CRTs, where outcomes are assessed repeatedly in each cluster over time, estimates for more complex correlation structures are required. Three common types of correlation structures for longitudinal CRTs are exchangeable, nested/block exchangeable and exponential decay correlations-the latter two allow the strength of the correlation to weaken over time. Determining sample sizes under these latter two structures requires advance specification of the within-period ICC and cluster autocorrelation coefficient as well as the intra-individual autocorrelation coefficient in the case of a cohort design. How to estimate these coefficients is a common challenge for investigators. When appropriate estimates from previously published longitudinal CRTs are not available, one possibility is to re-analyse data from an available trial dataset or to access observational data to estimate these parameters in advance of a trial. In this tutorial, we demonstrate how to estimate correlation parameters under these correlation structures for continuous and binary outcomes. We first introduce the correlation structures and their underlying model assumptions under a mixed-effects regression framework. With practical advice for implementation, we then demonstrate how the correlation parameters can be estimated using examples and we provide programming code in R, SAS, and Stata. An Rshiny app is available that allows investigators to upload an existing dataset and obtain the estimated correlation parameters. We conclude by identifying some gaps in the literature.

Maximin optimal cluster randomized designs for assessing treatment effect heterogeneity

Cluster randomized trials (CRTs) are studies where treatment is randomized at the cluster level but outcomes are typically collected at the individual level. When CRTs are employed in pragmatic settings, baseline population characteristics may moderate treatment effects, leading to what is known as heterogeneous treatment effects (HTEs). Pre-specified, hypothesis-driven HTE analyses in CRTs can enable an understanding of how interventions may impact subpopulation outcomes. While closed-form sample size formulas have recently been proposed, assuming known intracluster correlation coefficients (ICCs) for both the covariate and outcome, guidance on optimal cluster randomized designs to ensure maximum power with pre-specified HTE analyses has not yet been developed. We derive new design formulas to determine the cluster size and number of clusters to achieve the locally optimal design (LOD) that minimizes variance for estimating the HTE parameter given a budget constraint. Given the LODs are based on covariate and outcome-ICC values that are usually unknown, we further develop the maximin design for assessing HTE, identifying the combination of design resources that maximize the relative efficiency of the HTE analysis in the worst case scenario. In addition, given the analysis of the average treatment effect is often of primary interest, we also establish optimal designs to accommodate multiple objectives by combining considerations for studying both the average and heterogeneous treatment effects. We illustrate our methods using the context of the Kerala Diabetes Prevention Program CRT, and provide an R Shiny app to facilitate calculation of optimal designs under a wide range of design parameters.

Sample size considerations for assessing treatment effect heterogeneity in randomized trials with heterogeneous intracluster correlations and variances

An important consideration in the design and analysis of randomized trials is the need to account for outcome observations being positively correlated within groups or clusters. Two notable types of designs with this consideration are individually randomized group treatment trials and cluster randomized trials. While sample size methods for testing the average treatment effect are available for both types of designs, methods for detecting treatment effect modification are relatively limited. In this article, we present new sample size formulas for testing treatment effect modification based on either a univariate or multivariate effect modifier in both individually randomized group treatment and cluster randomized trials with a continuous outcome but any types of effect modifier, while accounting for differences across study arms in the outcome variance, outcome intracluster correlation coefficient (ICC) and the cluster size. We consider cases where the effect modifier can be measured at either the individual level or cluster level, and with a univariate effect modifier, our closed-form sample size expressions provide insights into the optimal allocation of groups or clusters to maximize design efficiency. Overall, our results show that the required sample size for testing treatment effect heterogeneity with an individual-level effect modifier can be affected by unequal ICCs and variances between arms, and accounting for such between-arm heterogeneity can lead to more accurate sample size determination. We use simulations to validate our sample size formulas and illustrate their application in the context of two real trials: an individually randomized group treatment trial (the AWARE study) and a cluster randomized trial (the K-DPP study).

Accounting for expected attrition in the planning of cluster randomized trials for assessing treatment effect heterogeneity

BACKGROUND

Detecting treatment effect heterogeneity is an important objective in cluster randomized trials and implementation research. While sample size procedures for testing the average treatment effect accounting for participant attrition assuming missing completely at random or missing at random have been previously developed, the impact of attrition on the power for detecting heterogeneous treatment effects in cluster randomized trials remains unknown.

METHODS

We provide a sample size formula for testing for a heterogeneous treatment effect assuming the outcome is missing completely at random. We also propose an efficient Monte Carlo sample size procedure for assessing heterogeneous treatment effect assuming covariate-dependent outcome missingness (missing at random). We compare our sample size methods with the direct inflation method that divides the estimated sample size by the mean follow-up rate. We also evaluate our methods through simulation studies and illustrate them with a real-world example.

RESULTS

Simulation results show that our proposed sample size methods under both missing completely at random and missing at random provide sufficient power for assessing heterogeneous treatment effect. The proposed sample size methods lead to more accurate sample size estimates than the direct inflation method when the missingness rate is high (e.g., ≥ 30%). Moreover, sample size estimation under both missing completely at random and missing at random is sensitive to the missingness rate, but not sensitive to the intracluster correlation coefficient among the missingness indicators.

CONCLUSION

Our new sample size methods can assist in planning cluster randomized trials that plan to assess a heterogeneous treatment effect and participant attrition is expected to occur.

Estimands in cluster-randomized trials: choosing analyses that answer the right question

BACKGROUND

Cluster-randomized trials (CRTs) involve randomizing groups of individuals (e.g. hospitals, schools or villages) to different interventions. Various approaches exist for analysing CRTs but there has been little discussion around the treatment effects (estimands) targeted by each.

METHODS

We describe the different estimands that can be addressed through CRTs and demonstrate how choices between different analytic approaches can impact the interpretation of results by fundamentally changing the question being asked, or, equivalently, the target estimand.

RESULTS

CRTs can address either the participant-average treatment effect (the average treatment effect across participants) or the cluster-average treatment effect (the average treatment effect across clusters). These two estimands can differ when participant outcomes or the treatment effect depends on the cluster size (referred to as 'informative cluster size'), which can occur for reasons such as differences in staffing levels or types of participants between small and large clusters. Furthermore, common estimators, such as mixed-effects models or generalized estimating equations with an exchangeable working correlation structure, can produce biased estimates for both the participant-average and cluster-average treatment effects when cluster size is informative. We describe alternative estimators (independence estimating equations and cluster-level analyses) that are unbiased for CRTs even when informative cluster size is present.

CONCLUSION

We conclude that careful specification of the estimand at the outset can ensure that the study question being addressed is clear and relevant, and, in turn, that the selected estimator provides an unbiased estimate of the desired quantity.

Power Analysis for Moderator Effects in Longitudinal Cluster Randomized Designs

Cluster randomized control trials often incorporate a longitudinal component where, for example, students are followed over time and student outcomes are measured repeatedly. Besides examining how intervention effects induce changes in outcomes, researchers are sometimes also interested in exploring whether intervention effects on outcomes are modified by moderator variables at the individual (e.g., gender, race/ethnicity) and/or the cluster level (e.g., school urbanicity) over time. This study provides methods for statistical power analysis of moderator effects in two- and three-level longitudinal cluster randomized designs. Power computations take into account clustering effects, the number of measurement occasions, the impact of sample sizes at different levels, covariates effects, and the variance of the moderator variable. Illustrative examples are offered to demonstrate the applicability of the methods.

Health equity considerations in pragmatic trials in Alzheimer's and dementia disease: Results from a methodological review

Introduction

To improve dementia care delivery for persons across all backgrounds, it is imperative that health equity is integrated into pragmatic trials.

Methods

We reviewed 62 pragmatic trials of people with dementia published 2014 to 2019. We assessed health equity in the objectives; design, conduct, analysis; and reporting using PROGRESS-Plus which stands for Place of residence, Race/ethnicity, Occupation, Gender/sex, Religion, Education, Socioeconomic status, Social capital, and other factors such as age and disability.

Results

Two (3.2%) trials incorporated equity considerations into their objectives; nine (14.5%) engaged with communities; 4 (6.5%) described steps to increase enrollment from equity-relevant groups. Almost all trials (59, 95.2%) assessed baseline balance for at least one PROGRESS-Plus characteristic, but only 10 (16.1%) presented subgroup analyses across such characteristics. Differential recruitment, attrition, implementation, adherence, and applicability across PROGRESS-Plus were seldom discussed.

Discussion

Ongoing and future pragmatic trials should more rigorously integrate equity considerations in their design, conduct, and reporting.

Highlights

Few pragmatic trials are explicitly designed to inform equity-relevant objectives.Few pragmatic trials take steps to increase enrollment from equity-relevant groups.Disaggregated results across equity-relevant groups are seldom reported.Adherence to existing tools (e.g., IMPACT Best Practices, CONSORT-Equity) is key.

Rethinking children's physical activity interventions at school: A new context-specific approach

Physical activity is important for children's health. However, evidence suggests that many children and adults do not meet international physical activity recommendations. Current school-based interventions have had limited effect on physical activity and alternative approaches are needed. Context, which includes school setting, ethos, staff, and sociodemographic factors, is a key and largely ignored contributing factor to school-based physical activity intervention effectiveness, impacting in several interacting ways.

Conceptualization

Current programs focus on tightly-constructed content that ignores the context in which the program will be delivered, thereby limiting effectiveness. We propose a move away from uniform interventions that maximize internal validity toward a flexible approach that enables schools to tailor content to their specific context.

Evaluation designs

Evaluation of context-specific interventions should explicitly consider context. This is challenging in cluster randomized controlled trial designs. Thus, alternative designs such as natural experiment and stepped-wedge designs warrant further consideration.

Primary outcome

A collective focus on average minutes of moderate-to-vigorous intensity physical activity may not always be the most appropriate choice. A wider range of outcomes may improve children's physical activity and health in the long-term. In this paper, we argue that greater consideration of school context is key in the design and analysis of school-based physical activity interventions and may help overcome existing limitations in the design of effective interventions and thus progress the field. While this focus on context-specific interventions and evaluation is untested, we hope to stimulate debate of the key issues to improve future physical activity intervention development and implementation.

Sample size calculation in hierarchical 2×2 factorial trials with unequal cluster sizes

Motivated by a suicide prevention trial with hierarchical treatment allocation (cluster-level and individual-level treatments), we address the sample size requirements for testing the treatment effects as well as their interaction. We assume a linear mixed model, within which two types of treatment effect estimands (controlled effect and marginal effect) are defined. For each null hypothesis corresponding to an estimand, we derive sample size formulas based on large-sample z-approximation, and provide finite-sample modifications based on a t-approximation. We relax the equal cluster size assumption and express the sample size formulas as functions of the mean and coefficient of variation of cluster sizes. We show that the sample size requirement for testing the controlled effect of the cluster-level treatment is more sensitive to cluster size variability than that for testing the controlled effect of the individual-level treatment; the same observation holds for testing the marginal effects. In addition, we show that the sample size for testing the interaction effect is proportional to that for testing the controlled or the marginal effect of the individual-level treatment. We conduct extensive simulations to validate the proposed sample size formulas, and find the empirical power agrees well with the predicted power for each test. Furthermore, the t-approximations often provide better control of type I error rate with a small number of clusters. Finally, we illustrate our sample size formulas to design the motivating suicide prevention factorial trial. The proposed methods are implemented in the R package H2x2Factorial.

Accounting for unequal cluster sizes in designing cluster randomized trials to detect treatment effect heterogeneity

Unequal cluster sizes are common in cluster randomized trials (CRTs). While there are a number of previous investigations studying the impact of unequal cluster sizes on the power for testing the average treatment effect in CRTs, little is known about the impact of unequal cluster sizes on the power for testing the heterogeneous treatment effect (HTE) in CRTs. In this work, we expand the sample size procedures for studying HTE in CRTs to accommodate cluster size variation under the linear mixed model framework. Through analytical derivation and graphical exploration, we show that the sample size for the HTE with an individual-level effect modifier is less affected by unequal cluster sizes than with a cluster-level effect modifier. The impact of cluster size variability jointly depends on the mean and coefficient of variation of cluster sizes, covariate intraclass correlation coefficient (ICC) and the conditional outcome ICC. In addition, we demonstrate that the HTE-motivated analysis of covariance framework can be used for analyzing the average treatment effect, and offer a more efficient sample size procedure for studying the average treatment effect adjusting for the effect modifier. We use simulations to confirm the accuracy of the proposed sample size procedures for both the average treatment effect and HTE in CRTs. Extensions to multivariate effect modifiers are provided and our procedure is illustrated in the context of the Strategies to Reduce Injuries and Develop Confidence in Elders trial.