The Statistical Fragility of Hamstring Versus Patellar Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review of Comparative Studies

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

CLINICAL RELEVANCE

Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

Evaluating the Statistical Fragility of Comparative Studies on Autografts for Pediatric ACL Reconstruction

Background

The literature presents conflicting findings regarding outcomes after pediatric anterior cruciate ligament reconstruction (ACLR) with various autograft options, reflecting a lack of consensus on the standard of practice. Fragility analyses may assist in evaluating the statistical robustness of these studies.

Purpose

To evaluate the statistical fragility of comparative studies in pediatric ACLR through the fragility index (FI) and fragility quotient (FQ), as well as qualitative factors such as outcome type, outcome significance, and patients lost to follow-up.

Study Design

Systematic review; Level of evidence, 4.

Methods

A systematic review conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines identified 1139 studies in the PubMed and Embase databases that met the search criteria; ultimately, 6 studies were selected for inclusion. A total of 32 comparative outcomes were assessed for fragility across the 6 studies. Descriptive statistics were employed to summarize the fragility data and generate subgroup comparisons.

Results

The mean FI was 1.5, and the mean reverse FI was 3.19 (P < .01); the mean FQ was 0.0064, and the mean reverse FQ was 0.028 (P≤ .0001). No significant difference was found in the FIs between objective outcomes and patient-reported outcomes (P = .418). These findings suggested that a comparable number of patients would need to transition from a nonevent to an event to alter a statistically significant result to a nonsignificant one. The FI was lower than the estimated number of patients lost to follow-up for 30 of the 32 outcomes (93.7%).

Conclusion

Comparative studies on pediatric ACLR autograft outcomes displayed vulnerability when assessed using fragility metrics, indicating a lack of statistically robust data. The findings revealed that many reported outcomes are fragile and may require further investigation. Future research should incorporate fragility analyses-especially in studies with long-term follow-ups-to enhance the reliability of conclusions regarding optimal graft selection in pediatric ACLR.

The Statistical Fragility of Lateral Extra-articular Tenodesis Research: A Systematic Review

Background

A P value of <.05 is often used to denote statistical significance; however, in many scenarios, this threshold is vulnerable to a small number of outcome reversals. This study joins a body of studies within the orthopaedic literature that evaluate the statistical fragility of existing research via metrics such as fragility index (FI) and fragility quotient (FQ).

Purpose/Hypothesis

The purpose of this study was to investigate the statistical fragility of randomized controlled trials (RCTs) and comparative studies on the topic, given the resurgent interest in lateral extra-articular tenodesis (LET) to augment primary or revision anterior cruciate ligament reconstruction (ACLR). It was hypothesized that the outcomes reported in these studies would be statistically fragile.

Study Design

Systematic review; Level of evidence, 4.

Methods

Comparative studies and RCTs regarding LET as an adjunct procedure to ACLR published between 2000 and 2022 were analyzed. Descriptive characteristics, dichotomous outcomes, and continuous outcomes were extracted. The FI and continuous FI (CFI) were calculated by the number of event reversals to change significance; the FQ and continuous FQ (CFQ) were calculated to normalize the fragility metrics per sample size.

Results

Of 455 studies screened, 29 studies were included (9 RCTs, 20 comparative); 79.3% of included studies were published after 2020. A total of 48 dichotomous and 265 continuous outcomes were analyzed. The median FI was 9.0 (IQR, 7.0-13.3), with FQ of 0.1 (IQR, 0.04-0.17); the median CFI was 7.8 (IQR, 4.2-19.6), with CFQ of 0.12 (IQR, 0.08-0.19). The FQ and CFQ for studies on LET with revision ACLR were larger (0.117 and 0.113, respectively) than those focused on primary ACLR (0.042 and 0.095, respectively).

Conclusion

Studies focused on LET with primary ACLR were more fragile than those on LET with revision, which suggests that further research on the indications for LET with primary ACLR is necessary. Future orthopaedic comparative research should include fragility metrics alongside traditional P values.

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

Interpreting the Current Literature on Outcomes of Robotic-Assisted Versus Conventional Total Knee Arthroplasty Using Fragility Analysis: A Systematic Review and Cross-Sectional Study of Randomized Controlled Trials

BACKGROUND

Fragility analysis is a method of further characterizing outcomes in terms of the stability of statistical findings. This study assesses the statistical fragility of recent randomized controlled trials (RCTs) evaluating robotic-assisted versus conventional total knee arthroplasty (RA-TKA versus C-TKA).

METHODS

We queried PubMed for RCTs comparing alignment, function, and outcomes between RA-TKA and C-TKA. Fragility index (FI) and reverse fragility index (RFI) (collectively, "FI") were calculated for dichotomous outcomes as the number of outcome reversals needed to change statistical significance. Fragility quotient (FQ) was calculated by dividing the FI by the sample size for that outcome event. Median FI and FQ were calculated for all outcomes collectively as well as for each individual outcome. Subanalyses were performed to assess FI and FQ based on outcome event type and statistical significance, as well as study loss to follow-up and year of publication.

RESULTS

The overall median FI was 3.0 (interquartile range, [IQR] 1.0 to 6.3) and the median reverse fragility index was 3.0 (IQR 2.0 to 4.0). The overall median FQ was 0.027 (IQR 0.012 to 0.050). Loss to follow-up was greater than FI for 23 of the 38 outcomes assessed.

CONCLUSIONS

A small number of alternative outcomes is often enough to reverse the statistical significance of findings in RCTs evaluating dichotomous outcomes in RA-TKA versus C-TKA. We recommend reporting FI and FQ alongside P values to improve the interpretability of RCT results.

The Fragility of Statistical Findings in the Femoral Neck Fracture Literature: A Systematic Review of Randomized Controlled Trials

OBJECTIVES

Randomized controlled trials (RCTs) in the femoral neck fracture literature frequently report P -values for outcomes, which have substantial implications in guiding surgical management. This study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to assess the statistical stability of outcomes reported in RCTs evaluating the management and treatment of femoral neck fractures.

METHODS

DATA SOURCES

DESIGN

PubMed, Embase, and MEDLINE were queried for RCTs (January 1, 2010 to February 28, 2023).

SETTING

RCTs that evaluated surgical management or treatment of femoral neck fractures were included.

STUDY SELECTION CRITERIA

RCTs with 2 treatment arms reporting categorical dichotomous outcomes were included. Non-RCT studies, RCTs with greater than 2 treatment arms, and RCTs without a femoral neck fracture cohort were excluded.

DATA EXTRACTION AND SYNTHESIS

OUTCOME MEASURES AND COMPARISONS

The FI and rFI were calculated as the number of outcome event reversals required to alter statistical significance for significant ( P < 0.05) and nonsignificant ( P ≥ 0.05) outcomes, respectively. The FQ was calculated by dividing the FI by the sample size for the study.

RESULTS

Nine hundred eighty-five articles were screened, with 71 studies included for analysis. The median FI across a total of 197 outcomes was 4 [interquartile range (IQR) 2-5] with an associated FQ of 0.033 (IQR 0.017-0.060). Forty-seven outcomes were statistically significant with a median FI of 2 (IQR 1-4) and associated FQ of 0.02 (IQR 0.014-0.043). One hundred fifty outcomes were statistically nonsignificant with a median rFI of 4 (IQR 3-5) and associated FQ of 0.037 (IQR 0.019-0.065).

CONCLUSIONS

Statistical findings in femoral neck fracture RCTs are fragile, with reversal of a median 4 outcomes altering significance of study findings. The authors thus recommend standardized reporting of P -values with FI and FQ metrics to aid in interpreting the robustness of outcomes in femoral neck fracture RCTs.

LEVEL OF EVIDENCE

Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.

The Reverse Fragility Index: Interpreting the Evidence for Arthroscopic Rotator Cuff Repair Healing Associated With Early Versus Delayed Mobilization

Background: The American Academy of Orthopaedic Surgeons (AAOS) clinical practice guidelines (CPGs) note "strong" evidence that early and delayed mobilization protocols after small to medium arthroscopic rotator cuff repairs achieve similar rotator cuff healing rates. Purpose: We utilized the reverse fragility index (RFI) to assess the fragility of randomized controlled trials (RCTs) reporting no statistically significant difference in tendon re-tear rates after rotator cuff repair in those undergoing early versus delayed rehabilitation. Methods: Randomized controlled trials used in the most recent AAOS CPGs on the timing of postoperative mobilization after arthroscopic rotator cuff repairs were analyzed. Only RCTs with a reported P value ≥ .05 were included. The RFI at a threshold of P < .05 was calculated for each study. The reverse fragility quotient (RFQ) was calculated by dividing the RFI by the study sample size. Results: In 6 clinical trials with a total of 542 patients, the number of tendon re-tear events was 48. The median RFI at the P < .05 threshold was 4 (range: 3.25-4.75), and the median RFQ was .05 (range: 0.03-0.08). The median loss to follow-up was 6 patients. Of the 6 studies investigated, 3 reported a loss to follow-up greater than their respective RFI. Conclusion: The equivalence in rotator cuff repair healing rates associated with early and delayed mobilization protocols rests on fragile studies, as their statistical non-significance can be reversed by changing the outcome status of only a handful of patients. Consideration should be given to the routine reporting of RFI in clinical practice guidelines including RCTs with statistically non-significant results.

Statistical Fragility Analysis of Open Reduction Internal Fixation vs Primary Arthrodesis to Treat Lisfranc Injuries: A Systematic Review

BACKGROUND

There is a lack of consensus in the use of open reduction internal fixation (ORIF) vs primary arthrodesis (PA) in the management of Lisfranc injuries. Statistical fragility represents the number of events needed to flip statistical significance and provides context to interpret P values of outcomes from conflicting studies. The current study evaluates the statistical fragility of existing research with an outcome-specific approach to provide statistical clarity to the ORIF vs PA discussion. We hypothesized that statistical fragility analysis would offer clinically relevant insight when interpreting conflicting outcomes regarding ORIF vs PA management of Lisfranc injuries.

METHODS

All comparative studies, RCTs, and case-series investigating ORIF vs PA management of Lisfranc injuries published through October 5, 2023, were identified. Descriptive characteristics, dichotomous outcomes, and continuous outcomes were extracted. Fragility index and continuous fragility index were calculated by the number of event reversals needed to alter significance. Outcomes were categorized by clinical relevance, and median FI and CFI were reported.

RESULTS

A total of 244 studies were screened. Ten studies and 67 outcomes (44 dichotomous, 23 continuous) were included in the fragility analysis. Of the 10 studies, 4 studies claimed PA to correlate with superior outcomes compared to ORIF with regard to functional scores and return to function outcomes. Of these 4 studies, 3 were statistically robust. Six studies claimed PA and ORIF to have no differences in outcomes, in which only 2 studies were statistically robust.

CONCLUSION

The overall research regarding ORIF vs PA is relatively robust compared with other orthopaedic areas of controversy. Although the full statistical context of each article must be considered, studies supporting PA superiority with regard to functional scores and return to function metrics were found to be statistically robust. Outcome-specific analysis revealed moderate fragility in several clinically relevant outcomes such as functional score, return to function, and wound complications.

Reverse Fragility Index Comparing Rates of Rerupture After Open Achilles Tendon Repair Versus Early Functional Rehabilitation: A Systematic Review of Randomized Controlled Trials

BACKGROUND

Despite similar published rates of rerupture among patients treated with early functional rehabilitation and open repair for acute Achilles tendon rupture, uncertainty still exists regarding the optimal treatment modality. The reverse fragility index (RFI) is a statistical tool that provides an objective measure of the study's neutrality by determining the number of events that need to change for a nonsignificant result to be significant.

PURPOSE

The purpose was to utilize the RFI to appraise the strength of neutrality of randomized controlled trials (RCTs) comparing the rerupture rates of acute Achilles tendon ruptures treated with open repair versus early functional rehabilitation.

STUDY DESIGN

Systematic review; Level of evidence, 1.

METHODS

A systematic review was performed including all RCTs comparing the rerupture rates after operative repair and early functional rehabilitation for acute Achilles tendon ruptures. Studies were included that explicitly used early functional rehabilitation, defined as weightbearing and exercise-based interventions initiated within 2 weeks, as compared with open repair and reported a nonsignificant difference in rerupture rates. The RFI, with rerupture as the primary outcome, was calculated for each study (significance threshold, P < .05). The RFI quantifies a study's strength of neutrality and is defined as the minimum number of event reversals necessary to change a nonsignificant result to statistically significant.

RESULTS

Nine RCTs were included, with 713 patients and 46 reruptures. The median (interquartile range) rerupture rate was 7.69% (6.38%-9.64%) overall, 4.00% (2.33%-7.14%) in the operative group, and 10.00% (5.26%-12.20%) in the nonoperative group. The median RFI was 3, indicating that an outcome reversal of 3 patients was necessary to change the results from nonsignificant to statistically significant. The median number of patients lost to follow-up was 6 (3-7). Of 9 studies, 7 (77.8%) had a loss to follow-up greater than or equal to its RFI.

CONCLUSION

The statistical nonsignificance of studies reporting equivalent rerupture rates in the management of acute Achilles tendon ruptures with open repair versus nonoperative management with early functional rehabilitation can be reversed by changing the outcome status of only a few patients.

Fragile Statistical Findings in Randomized Controlled Trials Evaluating Autograft Versus Allograft Use in Anterior Cruciate Ligament Reconstruction: A Systematic Review

PURPOSE

To analyze the statistical stability of randomized controlled trials (RCTs) evaluating the surgical management of autografts versus allografts in the anterior cruciate ligament reconstruction (ACLR) literature and calculate the fragility index (FI) and fragility quotient and explore a subgroup analysis by calculating the proportion of outcome events where the FI was less than the number of patients lost to follow-up.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a systematic search in the PubMed and Cochrane databases to identify RCTs published between 2000 and 2022 that investigated the use of autografts versus allografts in ACLR literature and reported dichotomous data. The fragility index of each dichotomous variable was calculated through the reversal of a single outcome event until significance was reversed. The fragility quotient was calculated by dividing each fragility index by the study sample size. The interquartile range also was calculated.

RESULTS

Of the 4407 articles screened, 23 met the search criteria, with 11 RCTs evaluating ALCR using autografts and allografts included for analysis. Two hundred and 18 outcome events with 32 significant (P < .05) outcomes and 186 nonsignificant (P ≥ .05) outcomes were identified. The overall fragility index and fragility quotient for all 218 outcomes were 6 subjects (interquartile range 5-8) and 0.058 (interquartile range 0.039-0.077). Fragility analysis of statistically significant outcomes and nonsignificant outcomes had a fragility index of 3.5 (interquartile range 1-5.5) and 6 (interquartile range 5-8), respectively. All of the studies reported a loss to follow-up where 45.5% (5) reported a loss to follow-up greater or equal to 6.

CONCLUSIONS

The RCTs in the ACLR peer-reviewed literature evaluating autograft versus allograft use are vulnerable to a small number of outcome event reversals and exemplify significant statistical fragility in statistically significant findings.

LEVEL OF EVIDENCE

Level I, systematic review of Level I studies.

Randomized Controlled Trials Comparing Bone-Patellar Tendon-Bone Versus Hamstring Tendon Autografts in Anterior Cruciate Ligament Reconstruction Surgery Are Statistically Fragile: A Systematic Review

PURPOSE

To assess the statistical fragility of recently published randomized controlled trials (RCTs) comparing the use of hamstring tendon autograft with bone-patellar tendon-bone autograft for anterior cruciate ligament (ACL) reconstruction.

METHODS

The PubMed, Embase, and MEDLINE databases were queried for RCTs published since 2010 comparing autograft type (bone-patellar tendon-bone vs hamstring tendon) in ACL reconstruction surgery. The fragility index (FI) and reverse FI (rFI) were determined for significant and nonsignificant outcomes, respectively, as the number of outcome reversals required to change statistical significance. The fragility quotient (FQ) and reverse FQ, representing fragility as a proportion of the study population, were calculated by dividing the FI and rFI, respectively, by the sample size.

RESULTS

We identified 19 RCTs reporting 55 total dichotomous outcomes. The median FI of the 55 total outcomes was 5 (interquartile range [IQR], 4-7), meaning a median of 5 outcome event reversals would alter the outcomes' significance. Five outcomes were reported as statistically significant with a median FI of 4 (IQR, 2-6), meaning a median of 4 outcome event reversals would change outcomes to be nonsignificant. Fifty outcomes were reported as nonsignificant with a median rFI of 5 (IQR, 4-7), meaning a median of 5 outcome event reversals would change outcomes to be significant. The FQ and reverse FQ for significant and nonsignificant outcomes were 0.025 (IQR, 0.018-0.045) and 0.082 (IQR, 0.041-0.106), respectively. For 61.8% of outcomes, patients lost to follow-up exceeded the corresponding FI or rFI.

CONCLUSIONS

There is substantial statistical fragility in recent RCTs on autograft choice in ACL reconstruction surgery given that altering a few outcome events is sufficient to reverse study findings. For over half of outcomes, maintaining patients lost to follow-up may have been sufficient to reverse study conclusions.

CLINICAL RELEVANCE

We recommend co-reporting FIs and P values to provide a more comprehensive representation of a study's conclusions when conducting an RCT.

Editorial Commentary: The Statistical Fragility Index of Medical Trials Is Low By Design: Critical Evaluation of Confidence Intervals Is Required

The Fragility Index (FI) provides the number of patients whose outcome would need to have changed for the results of a clinical trial to no longer be statistically significant. Although it's a well-intended and easily interpreted metric, its calculation is based on reversing a significant finding and therefore its interpretation is only relevant in the domain of statistical significance. Its interpretation is only relevant in the domain of statistical significance. A well-designed clinical trial includes an a priori sample size calculation that aims to find the bare minimum of patients needed to obtain statistical significance. Such trials are fragile by design! Examining the robustness of clinical trials requires an estimation of uncertainty, rather than a misconstrued, dichotomous focus on statistical significance. Confidence intervals (CIs) provide a range of values that are compatible with a study's data and help determine the precision of results and the compatibility of the data with different hypotheses. The width of the CI speaks to the precision of the results, and the extent to which the values contained within have potential to be clinically important. Finally, one should not assume that a large FI indicates robust findings. Poorly executed trials are prone to bias, leading to large effects, and therefore, small P values, and a large FI. Let's move our future focus from the FI toward the CI.

The Majority of Sports Medicine and Arthroscopy-Related Randomized Controlled Trials Reporting Nonsignificant Results Are Statistically Fragile

PURPOSE

To evaluate the robustness of sports medicine and arthroscopy related randomized controlled trials (RCTs) reporting nonsignificant results by calculating the reverse fragility index (RFI) and reverse fragility quotient (RFQ).

METHODS

All sports medicine and arthroscopic-related RCTs from January 1, 2010, through August 3, 2021, were identified. Randomized-controlled trials comparing dichotomous variables with a reported P value ≥ .05 were included. Study characteristics, such as publication year and sample size, as well as loss to follow-up and number of outcome events were recorded. The RFI at a threshold of P < .05 and respective RFQ were calculated for each study. Coefficients of determination were calculated to determine the relationships between RFI and the number of outcome events, sample size, and number of patients lost to follow-up. The number of RCTs in which the loss to follow-up was greater than the RFI was determined.

RESULTS

Fifty-four studies and 4,638 patients were included in this analysis. The mean sample size and loss to follow-up were 85.9 patients and 12.5 patients, respectively. The mean RFI was 3.7, signifying that a change of 3.7 events in one arm was needed to flip the results of the study from non-significant to significant (P < .05). Of the 54 studies investigated, 33 (61%) had a loss to follow-up greater than their calculated RFI. The mean RFQ was 0.05. A significant correlation between RFI with sample size (R2 = 0.10, P = .02) and the total number of observed events (R2 = 0.13, P < .01) was found. No significant correlation existed between RFI and loss to follow-up in the lesser arm (R2 = 0.01, P = .41).

CONCLUSIONS

The RFI and RFQ are statistical tools that allow the fragility of studies reporting nonsignificant results to be appraised. Using this methodology, we found that the majority of sports medicine and arthroscopy-related RCTs reporting nonsignificant results are fragile.

CLINICAL RELEVANCE

RFI and RFQ serve as tools that can be used to assess the validity of RCT results and provide additional context for appropriate conclusions.

The Reverse Fragility Index: Interpreting the Current Literature on Long-Term Survivorship of Computer-Navigated Versus Conventional TKA: A Systematic Review and Cross-Sectional Study of Randomized Controlled Trials

BACKGROUND

Despite the most recent American Academy of Orthopaedic Surgeons clinical practice guideline making a "strong" recommendation against the use of intraoperative navigation in total knee arthroplasty (TKA), its use is increasing. We utilized the concept of the reverse fragility index (RFI) to assess the strength of neutrality of the randomized controlled trials (RCTs) comparing the long-term survivorship of computer-navigated and conventional TKA.

METHODS

A systematic review was performed including all RCTs through August 3, 2021, comparing the long-term outcomes of computer-navigated and conventional TKA. Randomized trials with mean follow-up of >8 years and survivorship with revision as the end point were included. The RFI quantifies the strength of a study's neutrality by calculating the minimum number of events necessary to flip the result from nonsignificant to significant. The RFI at a threshold of p < 0.05 was calculated for each study reporting nonsignificant results. The reverse fragility quotient (RFQ) was calculated by dividing the RFI by the study sample size.

RESULTS

Ten clinical trials with 2,518 patients and 38 all-cause revisions were analyzed. All 10 studies reported nonsignificant results. The median RFI at the p < 0.05 threshold was 4, meaning that a median of 4 events would be needed to change the results from nonsignificant to significant. The median RFQ was 0.029, indicating that the nonsignificance of the results was contingent on only 2.9 events per 100 participants. The median loss to follow-up was 27 patients. In all studies, the number of patients lost to follow-up was greater than the RFI.

CONCLUSIONS

The equipoise in long-term survivorship between computer-navigated and conventional TKA rests on fragile studies, as their statistical nonsignificance could be reversed by changing the outcome status of only a handful of patients--a number that was always smaller than the number lost to follow-up. Routine reporting of the RFI in trials with nonsignificant findings may provide readers with a measure of confidence in the neutrality of the results.

LEVEL OF EVIDENCE

Prognostic Level II. See Instructions for Authors for a complete description of levels of evidence.

The Statistical Fragility of Foot and Ankle Surgery Randomized Controlled Trials

Fragility index (FI) is a metric used to interpret the results of randomized controlled trials (RCTs), and describes the number of subjects that would need to be switched from event to non-event for a result to no longer be significant. Studies that analyze FI of RCTs in various orthopedic subspecialties have shown the RCTs to be largely underpowered and highly fragile. However, FI has not been assessed in foot and ankle RCTs. The MEDLINE and Embase online databases were searched from 1/1/2011 through 11/19/2021 for RCTs involving foot and ankle conditions. FI, fragility quotient (FQ), and difference between the FI and number of subjects lost to follow-up was calculated. Spearman correlation was performed to determine the relationship between sample size and FI. Overall, 1262 studies were identified of which 18 were included in the final analysis. The median sample size was 65 (interquartile range [IQR] 57-95.5), the median FI was 2 (IQR 1-2.5), and the median FQ was 0.026 (IQR 0.012-0.033). Ten of 15 (67%) studies with non-zero FI values had FI values less than the number of subjects lost to follow-up. There was linear association between FI and sample size (R² = 0.495, p-value: .031). This study demonstrates that RCTs in the field of foot and ankle surgery are highly fragile, similar to other orthopedic subspecialties.

Fragility Part I: a guide to understanding statistical power

The aim of this paper is to close the knowledge-to-practice gap around statistical power. We demonstrate how four factors affect power: p value, effect size, sample size, and variance. This article further delves into the advantages and disadvantages of a priori versus post hoc power analyses, though we believe only understanding of the former is essential to addressing the present-day issue of reproducibility in research. Upon reading this paper, physician-scientists should have expanded their arsenal of statistical tools and have the necessary context to understand statistical fragility.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

BACKGROUND

The statistical significance of randomized controlled trials (RCTs) and comparative studies is often conveyed utilizing the P value. However, P values are an imperfect measure and may be vulnerable to a small number of outcome reversals to alter statistical significance. The interpretation of the statistical strength of these studies may be aided by the inclusion of a Fragility Index (FI) and Fragility Quotient (FQ). This study examines the statistical stability of studies comparing operative vs nonoperative management for Achilles tendon rupture.

METHODS

A systematic search was performed of 10 orthopaedic journals between 2000 and 2021 for comparative studies focusing on management of Achilles tendon rupture reporting dichotomous outcome measures. FI for each outcome was determined by the number of event reversals necessary to alter significance (P < .05). FQ was calculated by dividing the FI by the respective sample size. Additional subgroup analyses were performed.

RESULTS

Of 8020 studies screened, 1062 met initial search criteria with 17 comparative studies ultimately included for analysis, 10 of which were RCTs. A total of 40 outcomes were examined. Overall, the median FI was 2.5 (interquartile range [IQR] 2-4), the mean FI was 2.90 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). The FI was less than the number of patients lost to follow-up for 78% of outcomes.

CONCLUSION

Studies examining the efficacy of operative vs nonoperative management of Achilles tendon rupture may not be as statistically stable as previously thought. The average number of outcome reversals needed to alter the significance of a given study was 2.90. Future analyses may benefit from the inclusion of a fragility index and a fragility quotient in their statistical analyses.

Statistical Fragility of Single-Row Versus Double-Row Anchoring for Rotator Cuff Repair: A Systematic Review of Comparative Studies

Background:

Comparative studies and randomized controlled trials (RCTs) often use the P (probability) value to convey the statistical significance of their findings. P values are an imperfect measure, however, and are vulnerable to a small number of outcome reversals to alter statistical significance. The inclusion of a fragility index (FI) and fragility quotient (FQ) may aid in the interpretation of a study’s statistical strength.

Purpose/Hypothesis:

The purpose of this study was to examine the statistical stability of studies comparing single-row to double-row rotator cuff repair. It was hypothesized that the findings of these studies would be vulnerable to a small number of outcome event reversals, often fewer than the number of patients lost to follow-up.

Study Design:

Systematic review; Level of evidence, 3.

Methods:

We analyzed comparative studies and RCTs on primary single-row versus double-row rotator cuff repair that were published between 2000 and 2021 in 10 leading orthopaedic journals. Statistical significance was defined as a P < .05. The FI for each outcome was determined by the number of event reversals necessary to alter significance. The FQ was calculated by dividing the FI by the respective sample size.

Results:

Of 4896 studies screened, 22 comparative studies, 10 of which were RCTs, were ultimately included for analysis. A total of 74 outcomes were examined. Overall, the median FI was 2 (interquartile range [IQR], 1-3), and the median FQ was 0.035 (IQR, 0.020-0.057). The mean FI was 2.55 ± 1.29, and the mean FQ was 0.043 ± 0.027. In 64% of outcomes, the FI was less than the number of patients lost to follow-up.) Additionally, 81% of significant outcomes needed just a single outcome reversal to lose their significance.

Conclusion:

Over half of the studies currently used to guide clinical practice have a number of patients lost to follow-up greater than their FI. The results of these studies should be interpreted within the context of these limitations. Future analyses may benefit from the inclusion of the FI and the FQ in their statistical analyses.

Static versus dynamic fixation of distal tibiofibular syndesmosis: a systematic review of overlapping meta-analyses

PURPOSE

Multiple Level I meta-analyses were conducted comparing traditional static vs. more recently introduced dynamic strategies of fixation for injuries of the distal tibiofibular syndesmosis (TFS). The aim of this review was to assess their robustness and methodological quality, providing support in the choice of a treatment strategy in case of TFS injury using the highest level of evidence.

METHODS

In this systematic review, conducted in accordance with the PRISMA guidelines, meta-analyses/systematic reviews comparing static and dynamic fixation methods after acute TFS injury were identified. The robustness of studies was evaluated using the fragility index (FI) for meta-analysis and the fragility quotient (FQ). The risk of bias was evaluated using the Assessment of Multiple Systematic Reviews (AMSTAR) instrument. Finally, the Jadad was applied to select the study which provided the highest quality of evidence to develop recommendations for the fixation strategy of these lesions.

RESULTS

Out of 1.302 records, four Level I meta-analyses were included in this study. Analyzing the statistically significant dichotomous outcomes, the median FI was 3.5 (IQR, 2 to 5.5; range, 1 to 9), while the median FQ was 1.9% (IQR, 1 to 3.5; range 0.35 to 4.4). In total, 37% had an FI of 2 or less and 75% of outcomes had a FI of 4 or less. According to the AMSTAR score and Jadad algorithm, the largest meta-analysis was selected as the highest evidence provided so far.

CONCLUSION

The meta-analyses with statistically significant dichotomous outcomes comparing dynamic and static fixation for treating injuries of the distal tibiofibular syndesmosis are fragile, with a change in less than four patients or less than 2% of the study population sufficient to reverse a significant outcome to nonsignificant.

LEVEL OF EVIDENCE

Level I.