How to use likelihood ratios to interpret evidence from randomized trials

Statistically significant differences versus convincing evidence of real treatment effects: an analysis of the false positive risk for single-centre trials in anaesthesia

BACKGROUND

The American Statistical Association has highlighted problems with null hypothesis significance testing and outlined alternative approaches that may 'supplement or even replace P-values'. One alternative is to report the false positive risk (FPR), which quantifies the chance the null hypothesis is true when the result is statistically significant.

METHODS

We reviewed single-centre, randomised trials in 10 anaesthesia journals over 6 yr where differences in a primary binary outcome were statistically significant. We calculated a Bayes factor by two methods (Gunel, Kass). From the Bayes factor we calculated the FPR for different prior beliefs for a real treatment effect. Prior beliefs were quantified by assigning pretest probabilities to the null and alternative hypotheses.

RESULTS

For equal pretest probabilities of 0.5, the median (inter-quartile range [IQR]) FPR was 6% (1-22%) by the Gunel method and 6% (1-19%) by the Kass method. One in five trials had an FPR ≥20%. For trials reporting P-values 0.01-0.05, the median (IQR) FPR was 25% (16-30%) by the Gunel method and 20% (16-25%) by the Kass method. More than 90% of trials reporting P-values 0.01-0.05 required a pretest probability >0.5 to achieve an FPR of 5%. The median (IQR) difference in the FPR calculated by the two methods was 0% (0-2%).

CONCLUSIONS

Our findings suggest that a substantial proportion of single-centre trials in anaesthesia reporting statistically significant differences provide limited evidence of real treatment effects, or, alternatively, required an implausibly high prior belief in a real treatment effect.

CLINICAL TRIAL REGISTRATION

PROSPERO (CRD42023350783).

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are "significant" (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are "not significant," we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect.In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%.Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing.Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the "significance problem" based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.

Evidence of Lack of Treatment Efficacy Derived From Statistically Nonsignificant Results of Randomized Clinical Trials

Importance

Many randomized clinical trials yield statistically nonsignificant results. Such results are difficult to interpret within the dominant statistical framework.

Objective

To estimate the strength of evidence in favor of the null hypothesis of no effect vs the prespecified effectiveness hypothesis among nonsignificant primary outcome results of randomized clinical trials by application of the likelihood ratio.

Design, Setting, and Participants

Cross-sectional study of statistically nonsignificant results for primary outcomes of randomized clinical trials published in 6 leading general medical journals in 2021.

Outcome measures

The likelihood ratio for the null hypothesis of no effect vs the effectiveness hypothesis stated in the trial protocol (alternate hypothesis). The likelihood ratio quantifies the support that the data provide to one hypothesis vs the other.

Results

In 130 articles that reported 169 statistically nonsignificant results for primary outcomes, 15 results (8.9%) favored the alternate hypothesis (likelihood ratio, <1), and 154 (91.1%) favored the null hypothesis of no effect (likelihood ratio, >1). For 117 (69.2%), the likelihood ratio exceeded 10; for 88 (52.1%), it exceeded 100; and for 50 (29.6%), it exceeded 1000. Likelihood ratios were only weakly correlated with P values (Spearman r, 0.16; P = .045).

Conclusions

A large proportion of statistically nonsignificant primary outcome results of randomized clinical trials provided strong support for the hypothesis of no effect vs the alternate hypothesis of clinical efficacy stated a priori. Reporting the likelihood ratio may improve the interpretation of clinical trials, particularly when observed differences in the primary outcome are statistically nonsignificant.

Assessing the Utility of the Montreal Cognitive Assessment in Screening for Cognitive Impairment in Patients With Systemic Lupus Erythematosus

OBJECTIVE

Screening for cognitive impairment (CI) in systemic lupus erythematosus (SLE) relies on the American College of Rheumatology (ACR) neuropsychological battery (NB). By studying the concurrent criterion validity, our goal was to assess the Montreal Cognitive Assessment (MoCA) as a screening tool for CI compared to the ACR-NB and to evaluate the added value of the MoCA to the Automated Neuropsychological Assessment Metrics (ANAM).

METHODS

A total of 285 adult SLE patients were administered the ACR-NB, MoCA, and ANAM. For the ACR-NB, patients were classified as having CI if there was a Z score of ≤-1.5 in ≥2 domains. The area under the curve (AUC) and sensitivities/specificities were determined. A discriminant function analysis was applied to assess the ability of the MoCA to differentiate between CI, undetermined CI, and non-CI patients.

RESULTS

CI was not accurately identified by the MoCA compared to the ACR-NB (AUC of 0.66). Sensitivity and specificity were poor at 50% and 69%, respectively, for the cutoff of 26, and 80% and 45%, respectively, for the cutoff of 28. The MoCA had a low ability to identify CI status. The addition of the MoCA to the ANAM led to improvement on the AUC by only 2.5%.

CONCLUSION

The MoCA does not have adequate concurrent criterion validity to accurately identify CI in patients with SLE. The low specificity of the MoCA may lead to overdiagnosis and concern among patients. Adding the MoCA to the ANAM does not substantially improve the accuracy of the ANAM. These results do not support using the MoCA as a screening tool for CI in patients with SLE.

Likelihood Ratios: An Important Concept for Palliative Physicians to Understand

Palliative care has several tools and questionnaires which are commonly used for patient-related outcomes and prognosis. As an example, the Surprise Question (I would or would not be surprised that this person would have died in a year) has been used as a screen for palliative care referral but also used as a prognostic tool. Diagnostic tests, prognostic tools, and tools for gauging outcomes have certain sensitivity and specificity in predicting a diagnosis or outcome. Clinicians often use positive and negative predictive values in judging the merits of a diagnostic tool or questionnaire. However positive and negative predictive values are highly dependent on the prevalence of disease or outcome in a population and thus are not portable across studies. Likelihood ratios are both portable across populations but also provide the strength of the diagnostic or predictive measure of a test or questionnaire. In this article, we review the value and limitations of likelihood ratios and illustrate the value of using likelihood ratios using 3 studies centered on the Surprise Question published in 2022.