Fragility of results from randomized controlled trials supporting the guidelines for the treatment of osteoporosis: a retrospective analysis

The fragility index of randomized controlled trials in advanced/metastatic renal cell cancer

PURPOSE

The fragility index (FI) has been applied as a supplement to the noncomprehensive P-values to assess the robustness of randomized controlled trials (RCTs). The objective of this study is to evaluate the statistical robustness of RCTs of advanced/metastatic renal cell cancer (a/mRCC) using the FI.

MATERIALS AND METHODS

RCTs related to a/mRCC published in the 4 highest-impact general medical journals and the 25 highest-impact urological journals between January 1, 2000, and December 31, 2023, were identified from PubMed database. The FI was calculated by using Fisher's exact test. Spearman's correlation analysis was conducted to assess potential correlates regarding FI.

RESULTS

16 eligible RCTs were screened with a median total sample size of 654.5 (IQR, 461-847) and a median patients lost to follow-up of 14 (IQR, 3-23). The median FI was 12.5 (IQR, 8.5-27), suggesting that a switch in outcomes in only 13 patients would have reversed the significance of the trials. The number of patients lost to follow-up exceeded or equaled to the FI in 7 (44%) RCTs. P-values were negatively associated with the FI, while the number of patients lost to follow-up and patients enrolled were not statistically significant.

CONCLUSION

Not all RCTs associated with a/mRCC are as statistically robust as previously considered and should therefore be construed carefully. We suggest that additional reporting of FI in urological RCTs as a supplement to the P-value to assist readers in concluding reliably by considering the fragility of the outcomes.

Statistical fragility of randomized clinical trials pertaining to femoral neck fractures

OBJECTIVE

P values were frequently misused and misinterpreted, the fragility index (FI) has been utilized to evaluate the robustness of randomized controlled trials (RCTs) as a complement to p-values. This study aimed to assess the statistical robustness of RCTs for femoral neck fractures through the utilization of the FI.

METHODS

We systematically reviewed PubMed, Cochrane Library, and Embase database to identify RCTs pertaining to femoral neck fractures published in the top 25 highest-impact orthopaedic journals and 4 high-impact general medical journals from January 1, 2000, to December 31, 2022. The FI was calculated for the dichotomous, categorical study outcomes in the identified RCTs using the Fisher exact test, with previously published methods. Spearman correlation analyses were used to evaluate potential associated factors associated with the FI.

RESULTS

We identified 10 eligible RCTs with a median total sample size of 101 (IQR, 79.5 to 174.75) and a number of patients lost to follow-up of 19.5 (IQR, 4.5 to 28). The median FI was 3.5 (IQR, 1 to 14.25), implying that reversal of outcome in only 4 patients was sufficient to alter trial significance. The FI was less than the number of patients lost to follow-up in seven (70%) RCTs. P values were negatively associated with the FI, while the number of patients lost to follow-up and patients enroled were not statistically significantly associated with the FI.

CONCLUSIONS

The RCTs pertaining to femoral neck fractures were not as statistically robust as previously thought and should be interpreted with caution. We recommend that the orthopaedic RCT report FI as a supplement for the P values to help readers draw reliable conclusions based on the fragility of the outcomes.

Assessing the robustness of COVID-19 vaccine efficacy trials: systematic review and meta-analysis, January 2023

BackgroundVaccines play a crucial role in the response to COVID-19 and their efficacy is thus of great importance.AimTo assess the robustness of COVID-19 vaccine efficacy (VE) trial results using the fragility index (FI) and fragility quotient (FQ) methodology.MethodsWe conducted a Cochrane and PRISMA-compliant systematic review and meta-analysis of COVID-19 VE trials published worldwide until 22 January 2023. We calculated the FI and FQ for all included studies and assessed their associations with selected trial characteristics using Wilcoxon rank sum tests and Kruskal-Wallis H tests. Spearman correlation coefficients and scatter plots were used to quantify the strength of correlation of FIs and FQs with trial characteristics.ResultsOf 6,032 screened records, we included 40 trials with 54 primary outcomes, comprising 909,404 participants with a median sample size per outcome of 13,993 (interquartile range (IQR): 8,534-25,519). The median FI and FQ was 62 (IQR: 22-123) and 0.50% (IQR: 0.24-0.92), respectively. FIs were positively associated with sample size (p < 0.001), and FQs were positively associated with type of blinding (p = 0.023). The Spearman correlation coefficient for FI with sample size was moderately strong (0.607), and weakly positive for FI and FQ with VE (0.138 and 0.161, respectively).ConclusionsThis was the largest study on trial robustness to date. Robustness of COVID-19 VE trials increased with sample size and varied considerably across several other important trial characteristics. The FI and FQ are valuable complementary parameters for the interpretation of trial results and should be reported alongside established trial outcome measures.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

BACKGROUND

The statistical significance of randomized controlled trials (RCTs) and comparative studies is often conveyed utilizing the P value. However, P values are an imperfect measure and may be vulnerable to a small number of outcome reversals to alter statistical significance. The interpretation of the statistical strength of these studies may be aided by the inclusion of a Fragility Index (FI) and Fragility Quotient (FQ). This study examines the statistical stability of studies comparing operative vs nonoperative management for Achilles tendon rupture.

METHODS

A systematic search was performed of 10 orthopaedic journals between 2000 and 2021 for comparative studies focusing on management of Achilles tendon rupture reporting dichotomous outcome measures. FI for each outcome was determined by the number of event reversals necessary to alter significance (P < .05). FQ was calculated by dividing the FI by the respective sample size. Additional subgroup analyses were performed.

RESULTS

Of 8020 studies screened, 1062 met initial search criteria with 17 comparative studies ultimately included for analysis, 10 of which were RCTs. A total of 40 outcomes were examined. Overall, the median FI was 2.5 (interquartile range [IQR] 2-4), the mean FI was 2.90 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). The FI was less than the number of patients lost to follow-up for 78% of outcomes.

CONCLUSION

Studies examining the efficacy of operative vs nonoperative management of Achilles tendon rupture may not be as statistically stable as previously thought. The average number of outcome reversals needed to alter the significance of a given study was 2.90. Future analyses may benefit from the inclusion of a fragility index and a fragility quotient in their statistical analyses.

Statistical Fragility of Single-Row Versus Double-Row Anchoring for Rotator Cuff Repair: A Systematic Review of Comparative Studies

Background:

Comparative studies and randomized controlled trials (RCTs) often use the P (probability) value to convey the statistical significance of their findings. P values are an imperfect measure, however, and are vulnerable to a small number of outcome reversals to alter statistical significance. The inclusion of a fragility index (FI) and fragility quotient (FQ) may aid in the interpretation of a study’s statistical strength.

Purpose/Hypothesis:

The purpose of this study was to examine the statistical stability of studies comparing single-row to double-row rotator cuff repair. It was hypothesized that the findings of these studies would be vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up.

Study Design:

Systematic review; Level of evidence, 3.

Methods:

We analyzed comparative studies and RCTs on primary single-row versus double-row rotator cuff repair that were published between 2000 and 2021 in 10 leading orthopaedic journals. Statistical significance was defined as a P < .05. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size.

Results:

Of 4896 studies screened, 22 comparative studies, 10 of which were RCTs, were ultimately included for analysis. A total of 74 outcomes were examined. Overall, the median FI was 2 (interquartile range [IQR], 1-3), and the median FQ was 0.035 (IQR, 0.020-0.057). The mean FI was 2.55 ± 1.29, and the mean FQ was 0.043 ± 0.027. In 64% of outcomes, the FI was less than the number of patients lost to follow-up.) Additionally, 81% of significant outcomes needed just a single outcome reversal to lose their significance.

Conclusion:

Over half of the studies currently used to guide clinical practice have a number of patients lost to follow-up greater than their FI. The results of these studies should be interpreted within the context of these limitations. Future analyses may benefit from the inclusion of the FI and the FQ in their statistical analyses.

The Fragility Index of Randomized Controlled Trials for Preterm Neonates

BACKGROUND

As a metric to determine the robustness of trial results, the fragility index (FI) is the number indicating how many patients would be required to reverse the significant results. This study aimed to calculate the FI in randomized controlled trials (RCTs) involving premature.

METHODS

Trials were included if they had a 1:1 study design, reported statistically significant dichotomous outcomes, and had an explicitly stated sample size or power calculation. The FI was calculated for binary outcomes using Fisher's exact test, and the FIs of subgroups were compared. Spearman's correlation was applied to determine correlations between the FI and study characteristics.

RESULTS

Finally, 66 RCTs were included in the analyses. The median FI for these trials was 3.00 (interquartile range [IQR]: 1.00-5.00), with a median fragility quotient of 0.014 (IQR: 0.008-0.028). FI was ≤ 3 in 42 of these 66 RCTs (63.6%), and in 42.4% (28/66) of the studies, the number of patients lost to follow-up was greater than that of the FI. Significant differences were found in the FI among journals (p = 0.011). We observed that FI was associated with the sample size, total number of events, and reported p-values (r _s = 0.437, 0.495, and -0.857, respectively; all p < 0.001).

CONCLUSION

For RCTs in the premature population, a median of only three events was needed to change from a "non-event" to "event" to render a significant result non-significant, indicating that the significance may hinge on a small number of events.

On clinical trial fragility due to patients lost to follow up

BACKGROUND

Clinical trials routinely have patients lost to follow up. We propose a methodology to understand their possible effect on the results of statistical tests by altering the concept of the fragility index to treat the outcomes of observed patients as fixed but incorporate the potential outcomes of patients lost to follow up as random and subject to modification.

METHODS

We reanalyse the statistical results of three clinical trials on coronary artery bypass grafting (CABG) to study the possible effect of patients lost to follow up on the treatment effect statistical significance. To do so, we introduce the LTFU-aware fragility indices as a measure of the robustness of a clinical trial's statistical results with respect to patients lost to follow up.

RESULTS

The analyses illustrate that clinical trials can either be completely robust to the outcomes of patients lost to follow up, extremely sensitive to the outcomes of patients lost to follow up, or in an intermediate state. When a clinical trial is in an intermediate state, the LTFU-aware fragility indices provide an interpretable measure to quantify the degree of fragility or robustness.

CONCLUSIONS

The LTFU-aware fragility indices allow researchers to rigorously explore the outcomes of patients who are lost to follow up, when their data is the appropriate kind. The LTFU-aware fragility indices are sensitivity measures in a way that the original fragility index is not.