Influence of selection bias on the test decision. A simulation study

A roadmap to using randomization in clinical trials

BACKGROUND

Randomization is the foundation of any clinical trial involving treatment comparison. It helps mitigate selection bias, promotes similarity of treatment groups with respect to important known and unknown confounders, and contributes to the validity of statistical tests. Various restricted randomization procedures with different probabilistic structures and different statistical properties are available. The goal of this paper is to present a systematic roadmap for the choice and application of a restricted randomization procedure in a clinical trial.

METHODS

We survey available restricted randomization procedures for sequential allocation of subjects in a randomized, comparative, parallel group clinical trial with equal (1:1) allocation. We explore statistical properties of these procedures, including balance/randomness tradeoff, type I error rate and power. We perform head-to-head comparisons of different procedures through simulation under various experimental scenarios, including cases when common model assumptions are violated. We also provide some real-life clinical trial examples to illustrate the thinking process for selecting a randomization procedure for implementation in practice.

RESULTS

Restricted randomization procedures targeting 1:1 allocation vary in the degree of balance/randomness they induce, and more importantly, they vary in terms of validity and efficiency of statistical inference when common model assumptions are violated (e.g. when outcomes are affected by a linear time trend; measurement error distribution is misspecified; or selection bias is introduced in the experiment). Some procedures are more robust than others. Covariate-adjusted analysis may be essential to ensure validity of the results. Special considerations are required when selecting a randomization procedure for a clinical trial with very small sample size.

CONCLUSIONS

The choice of randomization design, data analytic technique (parametric or nonparametric), and analysis strategy (randomization-based or population model-based) are all very important considerations. Randomization-based tests are robust and valid alternatives to likelihood-based tests and should be considered more frequently by clinical investigators.

Altmetrics Attention Scores for Randomized Controlled Trials in Total Joint Arthroplasty Are Reflective of High Scientific Quality: An Altmetrics-Based Methodological Quality and Bias Analysis

The Altmetric Attention Score (AAS) has been associated with citation rates across medical and surgical disciplines. However, factors that drive high AAS remain poorly understood and there remains multiple pitfalls to correlating these metrics alone with the quality of a study. The purpose of the current study was to determine the relationship between methodologic and study biases and the AAS in randomized controlled trials (RCTs) published in total joint arthroplasty journals.

Design and analysis of stratified clinical trials in the presence of bias

BACKGROUND

Among various design aspects, the choice of randomization procedure have to be agreed on, when planning a clinical trial stratified by center. The aim of the paper is to present a methodological approach to evaluate whether a randomization procedure mitigates the impact of bias on the test decision in clinical trial stratified by center.

METHODS

We use the weighted t test to analyze the data from a clinical trial stratified by center with a two-arm parallel group design, an intended 1:1 allocation ratio, aiming to prove a superiority hypothesis with a continuous normal endpoint without interim analysis and no adaptation in the randomization process. The derivation is based on the weighted t test under misclassification, i.e. ignoring bias. An additive bias model combing selection bias and time-trend bias is linked to different stratified randomization procedures.

RESULTS

Various aspects to formulate stratified versions of randomization procedures are discussed. A formula for sample size calculation of the weighted t test is derived and used to specify the tolerated imbalance allowed by some randomization procedures. The distribution of the weighted t test under misclassification is deduced, taking the sequence of patient allocation to treatment, i.e. the randomization sequence into account. An additive bias model combining selection bias and time-trend bias at strata level linked to the applied randomization sequence is proposed. With these before mentioned components, the potential impact of bias on the type one error probability depending on the selected randomization sequence and thus the randomization procedure is formally derived and exemplarily calculated within a numerical evaluation study.

CONCLUSION

The proposed biasing policy and test distribution are necessary to conduct an evaluation of the comparative performance of (stratified) randomization procedure in multi-center clinical trials with a two-arm parallel group design. It enables the choice of the best practice procedure. The evaluation stimulates the discussion about the level of evidence resulting in those kind of clinical trials.

Randomization in survival studies: An evaluation method that takes into account selection and chronological bias

The random allocation of patients to treatments is a crucial step in the design and conduct of a randomized controlled trial. For this purpose, a variety of randomization procedures is available. In the case of imperfect blinding, the extent to which a randomization procedure forces balanced group sizes throughout the allocation process affects the predictability of allocations. As a result, some randomization procedures perform superior with respect to selection bias, whereas others are less susceptible to chronological bias. The choice of a suitable randomization procedure therefore depends on the expected risk for selection and chronological bias within the particular study in question. To enable a sound comparison of different randomization procedures, we introduce a model for the combined effect of selection and chronological bias in randomized studies with a survival outcome. We present an evaluation method to quantify the influence of bias on the test decision of the log-rank test in a randomized parallel group trial with a survival outcome. The effect of selection and chronological bias and the dependence on the study setting are illustrated in a sensitivity analysis. We conclude with a case study to showcase the application of our model for comparing different randomization procedures in consideration of the expected type I error probability.

Implementing Optimal Designs for Dose-Response Studies Through Adaptive Randomization for a Small Population Group

In dose-response studies with censored time-to-event outcomes, D-optimal designs depend on the true model and the amount of censored data. In practice, such designs can be implemented adaptively, by performing dose assignments according to updated knowledge of the dose-response curve at interim analysis. It is also essential that treatment allocation involves randomization-to mitigate various experimental biases and enable valid statistical inference at the end of the trial. In this work, we perform a comparison of several adaptive randomization procedures that can be used for implementing D-optimal designs for dose-response studies with time-to-event outcomes with small to moderate sample sizes. We consider single-stage, two-stage, and multi-stage adaptive designs. We also explore robustness of the designs to experimental (chronological and selection) biases. Simulation studies provide evidence that both the choice of an allocation design and a randomization procedure to implement the target allocation impact the quality of dose-response estimation, especially for small samples. For best performance, a multi-stage adaptive design with small cohort sizes should be implemented using a randomization procedure that closely attains the targeted D-optimal design at each stage. The results of the current work should help clinical investigators select an appropriate randomization procedure for their dose-response study.

Lessons learned from IDeAl - 33 recommendations from the IDeAl-net about design and analysis of small population clinical trials

BACKGROUND

IDeAl (Integrated designs and analysis of small population clinical trials) is an EU funded project developing new statistical design and analysis methodologies for clinical trials in small population groups. Here we provide an overview of IDeAl findings and give recommendations to applied researchers.

METHOD

The description of the findings is broken down by the nine scientific IDeAl work packages and summarizes results from the project's more than 60 publications to date in peer reviewed journals. In addition, we applied text mining to evaluate the publications and the IDeAl work packages' output in relation to the design and analysis terms derived from in the IRDiRC task force report on small population clinical trials.

RESULTS

The results are summarized, describing the developments from an applied viewpoint. The main result presented here are 33 practical recommendations drawn from the work, giving researchers a comprehensive guidance to the improved methodology. In particular, the findings will help design and analyse efficient clinical trials in rare diseases with limited number of patients available. We developed a network representation relating the hot topics developed by the IRDiRC task force on small population clinical trials to IDeAl's work as well as relating important methodologies by IDeAl's definition necessary to consider in design and analysis of small-population clinical trials. These network representation establish a new perspective on design and analysis of small-population clinical trials.

CONCLUSION

IDeAl has provided a huge number of options to refine the statistical methodology for small-population clinical trials from various perspectives. A total of 33 recommendations developed and related to the work packages help the researcher to design small population clinical trial. The route to improvements is displayed in IDeAl-network representing important statistical methodological skills necessary to design and analysis of small-population clinical trials. The methods are ready for use.

The impact of selection bias in randomized multi-arm parallel group clinical trials

The impact of selection bias on the results of clinical trials has been analyzed extensively for trials of two treatments, yet its impact in multi-arm trials is still unknown. In this paper, we investigate selection bias in multi-arm trials by its impact on the type I error probability. We propose two models for selection bias, so-called biasing policies, that both extend the classic guessing strategy by Blackwell and Hodges. We derive the distribution of the F-test statistic under the misspecified outcome model and provide a formula for the type I error probability under selection bias. We apply the presented approach to quantify the influence of selection bias in multi-arm trials with increasing number of treatment groups using a permuted block design for different assumptions and different biasing strategies. Our results confirm previous findings that smaller block sizes lead to a higher proportion of sequences with inflated type I error probability. Astonishingly, our results also show that the proportion of sequences with inflated type I error probability remains constant when the number of treatment groups is increased. Realizing that the impact of selection bias cannot be completely eliminated, we propose a bias adjusted statistical model and show that the power of the statistical test is only slightly deflated for larger block sizes.

Chronological Bias in Randomized Clinical Trials Arising from Different Types of Unobserved Time Trends

Background: In clinical trials patients are commonly recruited sequentially over time incurring the risk of chronological bias due to (unobserved) time trends. To minimize the risk of chronological bias, a suitable randomization procedure should be chosen.

Objectives: Considering different time trend scenarios, we aim at a detailed evaluation of the extent of chronological bias under permuted block randomization in order to provide recommendations regarding the choice of randomization at the design stage of a clinical trial and to assess the maximum extent of bias for a realized sequence in the analysis stage.

Methods: For the assessment of chronological bias we consider linear, logarithmic and stepwise trends illustrating typical changes during recruitment in clinical practice. Bias and variance of the treatment effect estimator as well as the empirical type I error rate when applying the t-test are investigated. Different sample sizes, block sizes and strengths of time trends are considered.

Results: Using large block sizes, a notable bias exists in the estimate of the treatment effect for specific sequences. This results in a heavily inflated type I error for realized worst-case sequences and an enlarged mean squared error of the treatment effect estimator. Decreasing the block size restricts these effects of time trends. Already applying permuted block randomization with two blocks instead of the random allocation rule achieves a good reduction of the mean squared error and of the inflated type I error. Averaged over all sequences, the type I error of the t-test is far below the nominal significance level due to an overestimated variance.

Conclusions: Unobserved time trends can induce a strong bias in the treatment effect estimate and in the test decision. Therefore, already in the design stage of a clinical trial a suitable randomization procedure should be chosen. According to our results, small block sizes should be preferred, but also medium block sizes are sufficient to restrict chronological bias to an acceptable extent if other contrary aspects have to be considered (e.g. serious risk of selection bias). Regardless of the block size, a blocked ANOVA should be used because the t-test is far too conservative, even for weak time trends.

ERDO - a framework to select an appropriate randomization procedure for clinical trials

BACKGROUND

Randomization is considered to be a key feature to protect against bias in randomized clinical trials. Randomization induces comparability with respect to known and unknown covariates, mitigates selection bias, and provides a basis for inference. Although various randomization procedures have been proposed, no single procedure performs uniformly best. In the design phase of a clinical trial, the scientist has to decide which randomization procedure to use, taking into account the practical setting of the trial with respect to the potential of bias. Less emphasis has been placed on this important design decision than on analysis, and less support has been available to guide the scientist in making this decision.

METHODS

We propose a framework that weights the properties of the randomization procedure with respect to practical needs of the research question to be answered by the clinical trial. In particular, the framework assesses the impact of chronological and selection bias on the probability of a type I error. The framework is applied to a case study with a 2-arm parallel group, single center randomized clinical trial with continuous endpoint, with no-interim analysis, 1:1 allocation and no adaptation in the randomization process.

RESULTS

In so doing, we derive scientific arguments for the selection of an appropriate randomization procedure and develop a template which is illustrated in parallel by a case study. Possible extensions are discussed.

CONCLUSION

The proposed ERDO framework guides the investigator through a template for the choice of a randomization procedure, and provides easy to use tools for the assessment. The barriers for the thorough reporting and assessment of randomization procedures could be further reduced in the future when regulators and pharmaceutical companies employ similar, standardized frameworks for the choice of a randomization procedure.

Assessing the impact of selection bias on test decisions in trials with a time-to-event outcome

If past treatment assignments are unmasked, selection bias may arise even in randomized controlled trials. The impact of such bias can be measured by considering the type I error probability. In case of a normally distributed outcome, there already exists a model accounting for selection bias that permits calculating the corresponding type I error probabilities. To model selection bias for trials with a time‐to‐event outcome, we introduce a new biasing policy for exponentially distributed data. Using this biasing policy, we derive an exact formula to compute type I error probabilities whenever an F‐test is performed and no observations are censored. Two exemplary settings, with and without random censoring, are considered in order to illustrate how our results can be applied to compare distinct randomization procedures with respect to their performance in the presence of selection bias. © 2017 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Mechanisms and direction of allocation bias in randomised clinical trials

BACKGROUND

Selective allocation of patients into the compared groups of a randomised trial may cause allocation bias, but the mechanisms behind the bias and its directionality are incompletely understood. We therefore analysed the mechanisms and directionality of allocation bias in randomised clinical trials.

METHODS

Two systematic reviews and a theoretical analysis. We conducted one systematic review of empirical studies of motives/methods for deciphering patient allocation sequences; and another review of methods publications commenting on allocation bias. We theoretically analysed the mechanisms of allocation bias and hypothesised which main factors predicts its direction.

RESULTS

Three empirical studies addressed motives/methods for deciphering allocation sequences. Main motives included ensuring best care for patients and ensuring best outcome for the trial. Main methods included various manipulations with randomisation envelopes. Out of 57 methods publications 11 (19 %) mentioned explicitly that allocation bias can go in either direction. We hypothesised that the direction of allocation bias is mainly decided by the interaction between the patient allocators' motives and treatment preference.

CONCLUSION

Inadequate allocation concealment may exaggerate treatment effects in some trials while underestimate effects in others. Our hypothesis provides a theoretical overview of the main factors responsible for the direction of allocation bias.

Compression Stockings for Preventing the Postthrombotic Syndrome in Patients with Deep Vein Thrombosis

OBJECTIVE

We conducted a systematic review and meta-analysis to address benefits and harms of using elastic compression stockings after lower-extremity deep vein thrombosis.

METHODS

We searched 7 electronic databases through January 15, 2015, including randomized controlled trials (RCTs)/quasi-randomized trials reporting on elastic compression stocking efficacy on postthrombotic syndrome incidence, recurrent venous thromboembolism, mortality, and acute pain after deep vein thrombosis. Two reviewers independently screened records, extracted data, assessed risk of bias, and assessed confidence in effect estimates using Grading of Recommendations Assessment, Development, and Evaluation methodology. We applied random-effects meta-analysis models.

RESULTS

We included 5 RCTs (n = 1418) reporting on postthrombotic syndrome. The hazard ratio (HR) for postthrombotic syndrome with elastic compression stockings was 0.69 (95% confidence interval [CI], 0.47-1.02). We have very low confidence in this estimate due to heterogeneity and inclusion of unblinded studies at high risk of bias. Excluding high risk of bias studies, a single large RCT at low risk of bias provided moderate-quality evidence of no effect on postthrombotic syndrome (HR 1.00; 95% CI, 0.81-1.24). Moderate-quality evidence including all 5 studies suggests no effect of elastic compression stockings on recurrent venous thromboembolism (relative risk [RR] 0.88; 95% CI, 0.63-1.24) or mortality (RR 1.00; 95% CI, 0.73-1.37, 5 studies). Moderate-quality evidence from one large RCT does not suggest effect on acute pain after deep vein thrombosis.

CONCLUSIONS

The highest-quality evidence available suggests no effect of elastic compression stockings on postthrombotic syndrome or pain relief, from a single large RCT. However, results for preventing postthrombotic syndrome differ substantially across studies, and future guideline updates should reflect uncertainty about treatment effects. Elastic compression stockings are unlikely to prevent death or recurrent venous thromboembolism.

Risk of selection bias in randomised trials

BACKGROUND

Selection bias occurs when recruiters selectively enrol patients into the trial based on what the next treatment allocation is likely to be. This can occur even if appropriate allocation concealment is used if recruiters can guess the next treatment assignment with some degree of accuracy. This typically occurs in unblinded trials when restricted randomisation is implemented to force the number of patients in each arm or within each centre to be the same. Several methods to reduce the risk of selection bias have been suggested; however, it is unclear how often these techniques are used in practice.

METHODS

We performed a review of published trials which were not blinded to assess whether they utilised methods for reducing the risk of selection bias. We assessed the following techniques: (a) blinding of recruiters; (b) use of simple randomisation; (c) avoidance of stratification by site when restricted randomisation is used; (d) avoidance of permuted blocks if stratification by site is used; and (e) incorporation of prognostic covariates into the randomisation procedure when restricted randomisation is used. We included parallel group, individually randomised phase III trials published in four general medical journals (BMJ, Journal of the American Medical Association, The Lancet, and New England Journal of Medicine) in 2010.

RESULTS

We identified 152 eligible trials. Most trials (98%) provided no information on whether recruiters were blind to previous treatment allocations. Only 3% of trials used simple randomisation; 63% used some form of restricted randomisation, and 35% did not state the method of randomisation. Overall, 44% of trials were stratified by site of recruitment; 27% were not, and 29% did not report this information. Most trials that did stratify by site of recruitment used permuted blocks (58%), and only 15% reported using random block sizes. Many trials that used restricted randomisation also included prognostic covariates in the randomisation procedure (56%).

CONCLUSIONS

The risk of selection bias could not be ascertained for most trials due to poor reporting. Many trials which did provide details on the randomisation procedure were at risk of selection bias due to a poorly chosen randomisation methods. Techniques to reduce the risk of selection bias should be more widely implemented.