The Fragility Index-P Values Reimagined, Flaws and All-Reply

Thresholds for interpreting the fragility index derived from sample of randomised controlled trials in cardiology: a meta-epidemiologic study

The fragility index (FI) was proposed as a simplified way to communicate robustness of statistically significant results and their susceptibility to a change of a handful number of events. While this index is intuitive, it is not anchored by a cut-off or a guide for interpretation. We identified cardiovascular trials published in six high impact journals from 2007 to 2021 (500 or more participants and a dichotomous statistically significant primary outcome). We estimated area under curve (AUC) to determine FI value that best predicts whether the treatment effect was precise, defined as adequately powered for a plausible relative risk reduction (RRR) of 25% or 30% or having a CI that is sufficiently narrow to exclude a risk reduction that is too small (close to the null, <0.05). The median FI of 201 included cardiovascular trials was 13 (range 1-172). FI exceeded the number of patients lost to follow-up in 46/201 (22.89%) trials. FI values of 19 and 22 predicted that trials would be precise (powered for RRR of 30% and 25%; respectively, combined with CI that excluded risk reduction <0.05). AUC for meeting these precision criteria was 0.90 (0.86-0.94). In conclusion, FI values that range 19-22 may meet various definitions of precision and can be used as a rule of thumb to suggest that a treatment effect is likely precise and less susceptible to random error. The number of patients lost to follow-up should be presented alongside FI to better illustrate fragility.

Assessing and visualizing fragility of clinical results with binary outcomes in R using the fragility package

With the growing concerns about research reproducibility and replicability, the assessment of scientific results' fragility (or robustness) has been of increasing interest. The fragility index was proposed to quantify the robustness of statistical significance of clinical studies with binary outcomes. It is defined as the minimal event status modifications that can alter statistical significance. It helps clinicians evaluate the reliability of the conclusions. Many factors may affect the fragility index, including the treatment groups in which event status is modified, the statistical methods used for testing for the association between treatments and outcomes, and the pre-specified significance level. In addition to assessing the fragility of individual studies, the fragility index was recently extended to both conventional pairwise meta-analyses and network meta-analyses of multiple treatment comparisons. It is not straightforward for clinicians to calculate these measures and visualize the results. We have developed an R package called "fragility" to offer user-friendly functions for such purposes. This article provides an overview of methods for assessing and visualizing the fragility of individual studies as well as pairwise and network meta-analyses, introduces the usage of the "fragility" package, and illustrates the implementations with several worked examples.

Systematic Evaluation of the Robustness of the Evidence Supporting Current Guidelines on Myocardial Revascularization Using the Fragility Index

Background:

RCTs (randomized controlled trials) are the preferred source of evidence to support professional societies’ guidelines. The fragility index (FI), defined as the minimum number of patients whose status would need to switch from nonevent to event to render a statistically significant result nonsignificant, quantitatively estimates the robustness of RCT results. We evaluate RCTs supporting current guidelines on myocardial revascularization using the FI and FI minus number of patients lost to follow-up.

Methods and Results:

The FI and FI minus number of patients lost to follow-up of RCTs supporting the 2012 American College of Cardiology/American Heart Association Guideline for the Diagnosis and Management of Patients with Stable Ischemic Heart Disease, the 2014 Focused Update of the American College of Cardiology/American Heart Association Guideline for the Diagnosis and Management of Patients with Stable Ischemic Heart Disease, and the 2018 European Society of Cardiology/European Association for Cardio-Thoracic Surgery Guidelines for Myocardial Revascularization were calculated. Of 414 RCTs identified, 160 were eligible for FI calculation. The median FI was 8.0 (95% CI, 5.0–9.0) and the median FI minus number of patients lost to follow-up was 1.0 (95% CI, 0.0–3.0). FI was ≤3, indicating very limited robustness, in 44 (27.5%) RCTs, and was lower than the number LTF, indicating limited robustness, in 68 (42.5%) RCTs. FI was significantly (all P <0.05) correlated with the sample size, number of events, statistical power, journal impact factor, use of intention-to-treat analysis, and of composite end points and negatively correlated with the use of percutaneous interventions in the treatment arm and the P -value level.

Conclusions:

More than a quarter of RCTs that support current guidelines on myocardial revascularization have a FI of 3 or lower, and over 40% of trials reveal a FI that is lower than the number of patients lost to follow-up. These findings suggest that the robustness of the findings that support current myocardial revascularization guidelines is tenuous and vulnerable to change as new evidence from RCTs appears.