The fragility index should not be used for sample size calculations in clinical trials

The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials

Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R² = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.

Statistical robustness of RCTs in high-impact journals has improved, but was low across medical specialties

OBJECTIVES

To determine whether the statistical fragility of RCTs in high-impact journals has improved in the last decade, and to perform an umbrella review of all published data on the Fragility Index (FI) across medical specialties.

STUDY DESIGN AND SETTING

The FI was calculated for all eligible RCTs published 2014-2021 in the New England Journal of Medicine, The Lancet, the Journal of the American Medical Association, the British Medical Journal and the Annals of Internal Medicine. Trials reporting dichotomous, statistically significant, superiority results were eligible. All previously published systematic reviews on the FI were included in the umbrella review and analyzed by medical (sub-)specialty.

RESULTS

Out of 2544 screened RCTs, 643 were eligible for the FI analysis. These had a median sample size of 625 [IQR: 265-2056], a median FI of 12 [IQR: 3-28] and a median Fragility Quotient of 0.015 [IQR: 0.004-0.045]. This is an improvement compared with the median FI of 8 [IQR: 3-18] of RCTs published a decade earlier in the same five journals (p<0.001). The umbrella review included 57 publications across 15 different medical specialties, with a total of between 10 and 692 RCTs for each specialty. The median FI ranged between two and four for all disciplines.

CONCLUSION

In the last decade, the median statistical robustness of RCTs published in high-impact journals has improved, yet the unchanged lower bound of the interquartile range reveals that statistical significance in 25% of trials is still dependent on three or less events. The umbrella review revealed that statistical fragility is prevalent across all medical specialties. The FI is an easy to understand metric that can be used to supplement reported P-values and help readers look beyond merely reaching statistical significance.