Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

Statistically significant differences versus convincing evidence of real treatment effects: an analysis of the false positive risk for single-centre trials in anaesthesia

BACKGROUND

The American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant.

METHODS

We reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses.

RESULTS

For equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%).

CONCLUSIONS

Our findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect.

CLINICAL TRIAL REGISTRATION

PROSPERO (CRD42023350783).

Using the Delphi process to determine the minimum clinically important effect size for the Balanced-2 randomised controlled trial

BACKGROUND

The sample size calculation is an important step in designing randomised controlled trials. For a trial comparing a control and an intervention group, where the outcome is binary, the sample size calculation requires choosing values for the anticipated event rates in both the control and intervention groups (the effect size), and the error rates. The Difference ELicitation in TriAls guidance recommends that the effect size should be both realistic, and clinically important to stakeholder groups. Overestimating the effect size leads to sample sizes that are too small to reliably detect the true population effect size, which in turn results in low achieved power. In this study, we use the Delphi approach to gain consensus on what the minimum clinically important effect size is for Balanced-2, a randomised controlled trial comparing processed electroencephalogram-guided 'light' to 'deep' general anaesthesia on the incidence of postoperative delirium in older adults undergoing major surgery.

METHODS

Delphi rounds were conducted using electronic surveys. Surveys were administered to two stakeholder groups: specialist anaesthetists from a general adult department in Auckland City Hospital, New Zealand (Group 1), and specialist anaesthetists with expertise in clinical research, identified from the Australian and New Zealand College of Anaesthetist's Clinical Trials Network (Group 2). A total of 187 anaesthetists were invited to participate (81 from Group 1 and 106 from Group 2). Results from each Delphi round were summarised and presented in subsequent rounds until consensus was reached (>70% agreement).

RESULTS

The overall response rate for the first Delphi survey was 47% (88/187). The median minimum clinically important effect size was 5.0% (interquartile range: 5.0-10.0) for both stakeholder groups. The overall response rate for the second Delphi survey was 51% (95/187). Consensus was reached after the second round, as 74% of respondents in Group 1 and 82% of respondents in Group 2 agreed with the median effect size. The combined minimum clinically important effect size across both groups was 5.0% (interquartile range: 3.0-6.5).

CONCLUSIONS

This study demonstrates that surveying stakeholder groups using a Delphi process is a simple way of defining a minimum clinically important effect size, which aids the sample size calculation and determines whether a randomised study is feasible.

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are "significant" (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are "not significant," we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect.In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%.Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing.Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the "significance problem" based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.