Paediatric surgical trials, their fragility index, and why to avoid using it to evaluate results

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

The Statistical Fragility of Orbital Fractures: A Systematic Review of Randomized Controlled Trials

BACKGROUND

The P value has often been used as a tool to determine the statistical significance and evaluate the statistical robustness of study findings in orthopedic literature. The purpose of this study is to apply both the fragility index (FI) and the fragility quotient (FQ) to evaluate the degree of statistical fragility in orbital fracture literature. We hypothesized that the dichotomous outcomes within the orbital fracture literature will be vulnerable to a small number of outcome event reversals and will be statistically fragile.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), the authors identified all dichotomous data for randomized controlled trials (RCTs) in orbital fracture literature and performed a PubMed search from 2000 to 2022. The FI of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The FQ was calculated by dividing each FI by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 3,329 studies screened, 28 met the criteria with 10 RCTs evaluating orbital fractures included for analysis. A total of 58 outcome events with 22 significant (P < .05) outcomes and 36 nonsignificant (P ≥ .05) outcomes were identified. The overall FI and FQ for all 58 outcomes was 5 (IQR: 4 to 5) and 0.140 (IQR: 0.075 to 0.250), respectively. Fragility analysis of statistical significant outcomes and nonsignificant outcomes had an FI of 3.5 with no IQR and 5 (IQR 4-5), respectively. All of the studies reported a loss to follow-up data, where 20% (2) was greater than the overall FI of 5.

CONCLUSION

The orbital fracture literature provides treatment guidance by relying on statistical significant results from RCTs. However, the RCTs in the orbital fracture peer-reviewed literature may not be statistically stable as previously thought. The sole reliance of the P value may depict misleading results. Thus, we recommend standardizing the reporting of the P value, FI, and FQ in the orbital fracture literature to aid readers in reliably drawing conclusions based on fragility outcome measures impacting clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.