Influential methods reports for group-randomized trials and related designs

Transformer-Based Language Models for Group Randomized Trial Classification in Biomedical Literature: Model Development and Validation

Background

For the public health community, monitoring recently published articles is crucial for staying informed about the latest research developments. However, identifying publications about studies with specific research designs from the extensive body of public health publications is a challenge with the currently available methods.

Objective

Our objective is to develop a fine-tuned pretrained language model that can accurately identify publications from clinical trials that use a group- or cluster-randomized trial (GRT), individually randomized group-treatment trial (IRGT), or stepped wedge group- or cluster-randomized trial (SWGRT) design within the biomedical literature.

Methods

We fine-tuned the BioMedBERT language model using a dataset of biomedical literature from the Office of Disease Prevention at the National Institute of Health. The model was trained to classify publications into three categories of clinical trials that use nested designs. The model performance was evaluated on unseen data and demonstrated high sensitivity and specificity for each class.

Results

When our proposed model was tested for generalizability with unseen data, it delivered high sensitivity and specificity for each class as follows: negatives (0.95 and 0.93), GRTs (0.94 and 0.90), IRGTs (0.81 and 0.97), and SWGRTs (0.96 and 0.99), respectively.

Conclusions

Our work demonstrates the potential of fine-tuned, domain-specific language models to accurately identify publications reporting on complex and specialized study designs, addressing a critical need in the public health research community. This model offers a valuable tool for the public health community to directly identify publications from clinical trials that use one of the three classes of nested designs.

Intracluster correlation coefficients from cluster randomized trials conducted within the NCI Community Oncology Research Program (NCORP)

The intracluster correlation coefficient (ICC) measures the correlation of observations within clusters and is a key parameter for power and sample size calculations for cluster randomized trials (CRTs). To facilitate the design of future CRTs within the National Cancer Institute Community Oncology Research Program (NCORP), all studies from the NCORP website were reviewed to identify completed CRTs. ICCs for primary and secondary outcomes (when available) were ascertained from these trials and summarized in this article as a resource for future trial development. Although ICCs are relatively small for many outcome cluster combinations, that is not always the case, so consideration should always be given to the specific outcome of interest, trial design, and type of cluster when estimating an ICC to facilitate trial development.

Guidelines for the content of statistical analysis plans in clinical trials: protocol for an extension to cluster randomized trials

BACKGROUND

Guidance exists to inform the content of statistical analysis plans in clinical trials. Though not explicitly stated, this guidance is generally focused on clinical trials in which the randomization units are individual patients and not groups of patients. There are critical considerations for the analysis of cluster randomized trials, such as accounting for clustering, the risk of imbalances between the arms due to post-randomization recruitment, and the need to use small sample corrections when the number of clusters is small.

METHODS

This paper outlines the protocol for the development of a set of reporting guidelines for the content of statistical analysis plans for cluster randomized trials (including variations such as the stepped wedge cluster randomized trial and other cluster cross-over designs) by extending the minimum reporting analysis requirements as previously defined for individually randomized trials to cluster randomized trials. The guideline will be developed using a consensus-based approach, modifying existing reporting items from the guideline for individually randomized trials and extending to include new items.

DISCUSSION

The guideline will be developed so it can be used independently of the guideline for individually randomized designs. The consensus guidelines will be published in an open-access journal, including key guidance as well as exploration and elaboration.

Review of the quality of reporting of statistical analysis plans for cluster randomized trials

BACKGROUND AND OBJECTIVES

The guideline for the content of Statistical Analysis Plans (SAPs) outlines recommendations for items to be included in SAPs. As yet there is no specific tailoring of this guideline for Cluster Randomized Trials (CRTs). There has also been no assessment of reporting quality of SAPs against this guideline. Our intention is to identify how well a sample of SAPs for CRTs are adhering to the reporting of key items in the current guidelines, as well as additional analysis aspects considered to be important in CRTs.

METHODS

We include (i) fully published standalone SAPs identified via Ovid-MEDLINE and (ii) SAPs published as supplementary material or appendices to the final published report identified by searching an existing database of nearly 800 CRTs.

RESULTS

The search identified 85 unique SAPs: 26 were published in standalone format and 59 were supplementary material to the full trial report. There was mixed clarity in reporting of items related to the current guideline (eg, most (61/85, 72%) reported what covariates will be included in any adjustment; but fewer (26/85, 31%) reported what method will be used to estimate the absolute measure of effect). Considering additional aspects important for CRTs, the majority (79/85, 93%) included a plan to allow for clustering in the analysis; but fewer (10/40, 25%) reported how a small number of clusters would be accommodated (this was only considered relevant for the subset of CRTs with fewer than 40 clusters). Few (5/85, 6%) reported how the intracluster correlation would be estimated. Few clearly reported statistical targets of inference: in only two SAPs (2/85, 2%) it was clear whether the objectives were related to the individual or cluster-level average; in trials where relevant, only three (3/70, 4%) clearly reported whether the objectives were related to the marginal or cluster-specific effect.

CONCLUSION

This review has identified specific areas of poor quality of reporting that might need additional consideration when developing the guidance for the reporting of SAPs for CRTs.

Statistical analysis plan for a pragmatic randomised controlled trial comparing enhanced acceptance and commitment therapy plus ( +) added to usual aftercare versus usual aftercare only, in patients living with or beyond cancer: SUrvivors' Rehabilitation Evaluation after CANcer (SURECAN) trial

BACKGROUND

The aim of the SURECAN trial is to evaluate a person-centred intervention, based on Acceptance and Commitment Therapy (ACT Plus ( +)), for people who have completed treatment for cancer with curative intent, but are experiencing poor quality of life. We present the statistical analysis plan for assessing the effectiveness and cost-effectiveness of the intervention in improving quality of life 1 year post randomisation.

METHODS AND DESIGN

SURECAN is a multi-centre, pragmatic, two-arm, partially clustered randomised controlled superiority trial comparing the effectiveness of ACT + added to usual care with usual aftercare. The target sample size is 344 (172 per arm), randomised centrally in a 1:1 ratio.

RESULTS

The primary outcome is the total score of the Functional Assessment of Cancer Therapy scale-General (FACT-G) at 52 weeks, analysed using a partially nested mixed-effects model with heteroskedastic error terms. Secondary outcomes include scores at 16 and 52 weeks: FACT-G subscales; Fear of Cancer Recurrence Inventory (FCR4); positive and negative Impact of Cancer scales (IOCv2); Hospital Anxiety and Depression scale (HADS); Chalder Fatigue Scale (CFQ); and physical activity, measured on a modified version of the Godin scale. Health economic analyses will determine the incremental cost-effectiveness ratio (ICER) in terms of quality-adjusted life years (QALYs) derived from the Euroqol 5-Dimension 5-Level (EQ-5D-5L) compared to usual care at 52 weeks.

DISCUSSION

This manuscript is the statistical analysis plan (SAP) and economic evaluation for the SURECAN trial. Any exploratory or post hoc analyses will be identified as such in the respective analysis report.

TRIAL REGISTRATION

The trial was prospectively registered.

ISRCTN

ISRCTN67900293. Registered on 09 December 2019.

Reconsidering stepped wedge cluster randomized trial designs with implementation periods: Fewer sequences or the parallel-group design with baseline and implementation periods are potentially more efficient

BACKGROUND/AIMS

When designing a cluster randomized trial, advantages and disadvantages of tentative designs must be weighed. The stepped wedge design is popular for multiple reasons, including its potential to increase power via improved efficiency relative to a parallel-group design. In many realistic settings, it will take time for clusters to fully implement the intervention. When designing the HEALing (Helping to End Addiction Long-termSM) Communities Study, implementation time was a major consideration, and we examined the efficiency and practicality of three designs. Specifically, a three-sequence stepped wedge design with implementation periods, a corresponding two-sequence modified design that is created by removing the middle sequence, and a parallel-group design with baseline and implementation periods. In this article, we study the relative efficiencies of these specific designs. More generally, we study the relative efficiencies of modified designs when the stepped wedge design with implementation periods has three or more sequences. We also consider different correlation structures.

METHODS

We compare efficiencies of stepped wedge designs with implementation periods consisting of three to nine sequences with a variety of corresponding designs. The three-sequence design is compared to the two-sequence modified design and to the parallel-group design with baseline and implementation periods analysed via analysis of covariance. Stepped wedge designs with implementation periods consisting of four or more sequences are compared to modified designs that remove all or a subset of 'middle' sequences. Efficiencies are based on the use of linear mixed effects models.

RESULTS

In the studied settings, the modified design is more efficient than the three-sequence stepped wedge design with implementation periods. The parallel-group design with baseline and implementation periods with analysis of covariance-based analysis is often more efficient than the three-sequence design. With respect to stepped wedge designs with implementation periods that are comprised of more sequences, there are often corresponding modified designs that improve efficiency. However, use of only the first and last sequences has the potential to be either relatively efficient or inefficient. Relative efficiency is impacted by the strength of the statistical correlation among outcomes from the same cluster; for example, the relative efficiencies of modified designs tend to be greater for smaller cluster auto-correlation values.

CONCLUSION

If a three-sequence stepped wedge design with implementation periods is being considered for a future cluster randomized trial, then a corresponding modified design using only the first and last sequences should be considered if sole focus is on efficiency. However, a parallel-group design with baseline and implementation periods and analysis of covariance-based analysis can be a practical, efficient alternative. For stepped wedge designs with implementation periods and a larger number of sequences, modified versions that remove 'middle' sequences should be considered. Due to the potential sensitivity of design efficiencies, statistical correlation should be carefully considered.

School-level intra-cluster correlation coefficients and autocorrelations for children's accelerometer-measured physical activity in England by age and gender

BACKGROUND

Randomised, cluster-based study designs in schools are commonly used to evaluate children's physical activity interventions. Sample size estimation relies on accurate estimation of the intra-cluster correlation coefficient (ICC), but published estimates, especially using accelerometry-measured physical activity, are few and vary depending on physical activity outcome and participant age. Less commonly-used cluster-based designs, such as stepped wedge designs, also need to account for correlations over time, e.g. cluster autocorrelation (CAC) and individual autocorrelation (IAC), but no estimates are currently available. This paper estimates the school-level ICC, CAC and IAC for England children's accelerometer-measured physical activity outcomes by age group and gender, to inform the design of future school-based cluster trials.

METHODS

Data were pooled from seven large English datasets of accelerometer-measured physical activity data between 2002-18 (> 13,500 pupils, 540 primary and secondary schools). Linear mixed effect models estimated ICCs for weekday and whole week for minutes spent in moderate-to-vigorous physical activity (MVPA) and being sedentary for different age groups, stratified by gender. The CAC (1,252 schools) and IAC (34,923 pupils) were estimated by length of follow-up from pooled longitudinal data.

RESULTS

School-level ICCs for weekday MVPA were higher in primary schools (from 0.07 (95% CI: 0.05, 0.10) to 0.08 (95% CI: 0.06, 0.11)) compared to secondary (from 0.04 (95% CI: 0.03, 0.07) to (95% CI: 0.04, 0.10)). Girls' ICCs were similar for primary and secondary schools, but boys' were lower in secondary. For all ages, combined the CAC was 0.60 (95% CI: 0.44-0.72), and the IAC was 0.46 (95% CI: 0.42-0.49), irrespective of follow-up time. Estimates were higher for MVPA vs sedentary time, and for weekdays vs the whole week.

CONCLUSIONS

Adequately powered studies are important to evidence effective physical activity strategies. Our estimates of the ICC, CAC and IAC may be used to plan future school-based physical activity evaluations and were fairly consistent across a range of ages and settings, suggesting that results may be applied to other high income countries with similar school physical activity provision. It is important to use estimates appropriate to the study design, and that match the intended study population as closely as possible.

What type of cluster randomized trial for which setting?

The cluster randomized trial allows a randomized evaluation when it is either not possible to randomize the individual or randomizing individuals would put the trial at high risk of contamination across treatment arms. There are many variations of the cluster randomized design, including the parallel design with or without baseline measures, the cluster randomized cross-over design, the stepped-wedge cluster randomized design, and more recently-developed variants such as the batched stepped-wedge design and the staircase design. Once it has been clearly established that there is a need for cluster randomization, one ever important question is which form the cluster design should take. If a design in which time is split into multiple trial periods is to be adopted (e.g. as in a stepped-wedge), researchers must decide whether the same participants should be measured in multiple trial periods (cohort sampling); or if different participants should be measured in each period (continual recruitment or cross-sectional sampling). Here we outline the different possible options and weigh up the pros and cons of the different design choices, which revolve around statistical efficiency, study logistics and the assumptions required.

Does it decay? Obtaining decaying correlation parameter values from previously analysed cluster randomised trials

A frequently applied assumption in the analysis of data from cluster randomised trials is that the outcomes from all participants within a cluster are equally correlated. That is, the intracluster correlation, which describes the degree of dependence between outcomes from participants in the same cluster, is the same for each pair of participants in a cluster. However, recent work has discussed the importance of allowing for this correlation to decay as the time between the measurement of participants in a cluster increases. Incorrect omission of such a decay can lead to under-powered studies, and confidence intervals for estimated treatment effects can be too narrow or too wide, depending on the characteristics of the design. When planning studies, researchers often rely on previously reported analyses of trials to inform their choice of intracluster correlation. However, most reported analyses of clustered data do not incorporate a correlation decay. Thus, often all that is available are estimates of intracluster correlations obtained under the potentially incorrect assumption of no decay. In this article, we show that it is possible to use intracluster correlation values obtained from models that incorrectly omit a decay to inform plausible choices of decaying correlations. Our focus is on intracluster correlation estimates for continuous outcomes obtained by fitting linear mixed models with exchangeable or block-exchangeable correlation structures. We describe how plausible values for decaying correlations may be obtained given these estimated intracluster correlations. An online app is presented that allows users to obtain plausible values of the decay, which can be used at the trial planning stage to assess the sensitivity of sample size and power calculations to decaying correlation structures.

Key considerations for designing, conducting and analysing a cluster randomized trial

Not only do cluster randomized trials require a larger sample size than individually randomized trials, they also face many additional complexities. The potential for contamination is the most commonly used justification for using cluster randomization, but the risk of contamination should be carefully weighed against the more serious problem of questionable scientific validity in settings with post-randomization identification or recruitment of participants unblinded to the treatment allocation. In this paper we provide some simple guidelines to help researchers conduct cluster trials in a way that minimizes potential biases and maximizes statistical efficiency. The overarching theme of this guidance is that methods that apply to individually randomized trials rarely apply to cluster randomized trials. We recommend that cluster randomization be only used when necessary-balancing the benefits of cluster randomization with its increased risks of bias and increased sample size. Researchers should also randomize at the lowest possible level-balancing the risks of contamination with ensuring an adequate number of randomization units-as well as exploring other options for statistically efficient designs. Clustering should always be allowed for in the sample size calculation; and the use of restricted randomization (and adjustment in the analysis for covariates used in the randomization) should be considered. Where possible, participants should be recruited before randomizing clusters and, when recruiting (or identifying) participants post-randomization, recruiters should be masked to the allocation. In the analysis, the target of inference should align with the research question, and adjustment for clustering and small sample corrections should be used when the trial includes less than about 40 clusters.

Sample size determination for external pilot cluster randomised trials with binary feasibility outcomes: a tutorial

Justifying sample size for a pilot trial is a reporting requirement, but few pilot trials report a clear rationale for their chosen sample size. Unlike full-scale trials, pilot trials should not be designed to test effectiveness, and so, conventional sample size justification approaches do not apply. Rather, pilot trials typically specify a range of primary and secondary feasibility objectives. Often, these objectives relate to estimation of parameters that inform the sample size justification for the full-scale trial, many of which are binary. These binary outcomes are referred to as "feasibility outcomes" and include expected prevalence of the primary trial outcome, primary outcome availability, or recruitment or retention proportions.For pilot cluster trials, sample size calculations depend on the number of clusters, the cluster sizes, the anticipated intra-cluster correlation coefficient for the feasibility outcome and the anticipated proportion for that outcome. Of key importance is the intra-cluster correlation coefficient for the feasibility outcome. It has been suggested that correlations for feasibility outcomes are larger than for clinical outcomes measuring effectiveness. Yet, there is a dearth of information on realised values for these correlations.In this tutorial, we demonstrate how to justify sample size in external pilot cluster trials where the objective is to estimate a binary feasibility outcome. We provide sample size calculation formulae for a variety of scenarios, make available an R Shiny app for implementation, and compile a report of intra-cluster correlations for feasibility outcomes from a convenience sample. We demonstrate that unless correlations are very low, external pilot cluster trials can be made more efficient by including more clusters and fewer observations per cluster.

Reporting quality of randomized controlled trials evaluating non-vitamin K oral anticoagulants in atrial fibrillation: a systematic review

BACKGROUND

Randomized controlled trials (RCTs) are subject to bias if they lack methodological quality. Furthermore, optimal and transparent reporting of RCT findings aids their critical appraisal and interpretation. This study aimed to comprehensively evaluate the report quality of RCTs of non-vitamin K oral anticoagulants (NOACs) for the treatment of atrial fibrillation (AF) and to analyze the factors influencing the quality.

METHODS

By searching PubMed, Embase, Web of Science, and Cochrane Library databases RCTs published from inception to 2022 evaluating the efficacy of NOACs on AF were collected. By using the 2010 Consolidated Standards for Reporting Tests (CONSORT) statement, the overall quality of each report was assessed.

RESULTS

Sixty-two RCTs were retrieved in this study. The median of overall quality score in 2010 was 14 (range: 8.5-20). The extent of compliance with the Consolidated Standards of Reporting Trials reporting guideline differed substantially across items: 9 items were reported adequately (more than 90%), and 3 were reported adequately in less than 10% of trials. Multivariate linear regression analysis showed that the higher reporting scores were associated with higher journal impact factor (P = 0.01), international collaboration (P < 0.01), and Sources of trial funding (P = 0.02).

CONCLUSIONS

Although a large number of randomized controlled trials of NOACs for the treatment of AF were published after the CONSORT statement in 2010, the overall quality is still not satisfactory, thus weakening their potential utility and may mislead clinical decisions. This survey provides the first hint for researchers conducting trials of NOACs for AF to improve the quality of reports and to actively apply the CONSORT statement.