The fragility of statistically significant results in gynecologic surgery: A systematic review

Statistical Fragility of Outcomes on Breast Reconstruction with Acellular Dermal Matrix: A Systematic Review of Randomized Controlled Trials

BACKGROUND

Acellular dermal matrix (ADM) is pivotal in breast surgery, yet the statistical robustness of surgical outcomes remains underexplored. This study uses the fragility index (FI), reverse FI, and fragility quotient (FQ) to investigate the statistical fragility of ADM breast reconstruction outcomes.

METHODS

Randomized controlled trials (2013 to present) with dichotomous outcomes were sourced from PubMed, Embase, SCOPUS, Medline, and Cochrane databases. FI and reverse FI (event reversals needed to alter outcome significance) and FQ (standardized fragility across trials) were computed and reported as median (interquartile range [IQR]). Subgroup analysis focused on intervention types.

RESULTS

Of 33 studies screened, 19 RCTs comprising 204 outcomes were included, with a median FI of 4 (IQR, 3 to 5) and FQ of 0.039 (IQR, 0.029 to 0.070). Twenty-six outcomes achieved statistical significance, with a median FI of 3.5 (IQR, 1 to 5) and FQ of 0.033 (IQR, 0.010 to 0.073). The remaining 178 outcomes were not significant, exhibiting a median FI of 4 (IQR, 3 to 5) and FQ of 0.040 (IQR, 0.030 to 0.070). Of the 204 outcomes, 18% had a number of patients lost to follow-up equal to or surpassing the FI. By intervention type, the median FIs were similar in value but remained low.

CONCLUSIONS

ADM-related breast reconstruction outcomes are statistically fragile; thus, reversal of a few outcomes or maintaining follow-up with patients may alter the significance of findings. Future researchers are thus recommended to report FI and FQ metrics with P values to accurately portray reconstructive surgery outcomes.

The fragility of statistical findings in the intertrochanteric fracture fixation literature: a systematic review of randomized controlled trials

INTRODUCTION

Intertrochanteric fractures are common and can lead to significant disability and morality, particularly in the elderly. Utilizing the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ), this study evaluates the statistical fragility of outcomes reported in intertrochanteric fracture fixation randomized controlled trials (RCTs).

MATERIALS AND METHODS

Data sources: Pubmed, Embase, and MEDLINE were queried for RCTs published between 2010-present.

STUDY SELECTION

RCTs reporting 1:1 categorical, dichotomous outcomes were included. Articles were excluded if they were not RCTs, had over two treatment groups, included in vitro/animal/cadaveric data, and did not feature intertrochanteric fractures.

DATA EXTRACTION

Publication and individual outcome data were collected by three independent reviewers.

DATA SYNTHESIS

FI and rFI were calculated as the number of event reversals required to reverse the statistical significance for each outcome. The FQ was calculated by dividing FI by the study sample size. Subgroup analysis was performed based on outcome types.

RESULTS

Two hundred thirty-two articles were screened, and 52 articles with a total of 370 outcomes were included for analysis. The median FI was 5 (IQR 4-6) with a FQ of 0.05 (IQR 0.032-0.078). 57/370 outcomes were statistically significant with a median FI of 3 (IQR 1-8). 313 outcomes were statistically nonsignificant with a median rFI of 5 (IQR 4-6). The number of patients lost to follow-up was greater than or equal to the FI in 127/370 outcomes (34.32%). Outcomes relating to malunion/nonunion were the most fragile, encompassing 11 outcomes with a median FI of 3 (IQR 2.5-5).

CONCLUSION

Outcomes in intertrochanteric fracture fixation RCTs are fragile as reversal of a few outcomes or maintaining follow-up may alter the significance of study findings. Thus, P-values are recommended to be routinely reported with FI and FQ metrics in order to provide a comprehensive understanding of the statistical robustness of outcomes in orthopedic trauma literature.

LEVEL OF EVIDENCE

I.

The fragility index and reverse fragility index of FDA investigational device exemption trials in spinal fusion surgery: a systematic review

PURPOSE

FDA investigational device exemption (IDE) studies are considered a gold standard of assessing safety and efficacy of novel devices through RCTs. The fragility index (FI) has emerged as a means to assess robustness of statistically significant study results and inversely, the reverse fragility index (RFI) for non-significant differences. Previous authors have defined results as fragile if loss to follow up is greater than the FI or RFI. The aim of this study was to assess the FI, RFI, and robustness of data supplied by IDE studies in spinal surgery.

METHODS

This was a systematic review of the literature. Inclusion criteria included randomized controlled trials with dichotomous outcome measures conducted under IDE guidelines between 2000 and 2023. FI and RFI were calculated through successively changing events to non-events until the outcome changed to non-significance or significance, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI and RFI, respectively, by the sample size.

RESULTS

Thirty-two studies met inclusion criteria with a total of 40 unique outcome measures; 240 outcomes were analyzed. Twenty-six studies reported 96 statistically significant results. The median FI was 6 (IQR: 3-9.25), and patients lost to follow up was greater than the FI in 99.0% (95/96) of results. The average FQ was 0.027. Thirty studies reported 144 statistically insignificant results and a median RFI of 6 (IQR: 4-8). The average RFQ extrapolated was 0.021, and loss to follow up was greater than the RFI in 98.6% (142/144) of results.

CONCLUSIONS

IDE studies in spine surgery are surprisingly fragile given their reputations, large sample sizes, and intent to establish safety in investigational devices. This study found a median FI and RFI of 6. The number of patients lost to follow-up was greater than FIand RFI in 98.8% (237/240) of reported outcomes. FQ and RFQ tell us that changes of two to three patients per hundred can flip the significance of reported outcomes. This is an important reminder of the limitations of RCTs. Analysis of fragility in future studies may help clarify the strength of the relationship between reported data and their conclusions.

Fragility Index and Fragility Quotient in Statistically Significant Randomized Controlled Trials in Plastic Breast Surgery

Background

The fragility index (FI) was conceived as an adjunct to the P value, signifying the strength of statistically significant results. The index states the minimal number of patients whose outcome must be changed from "event" to "nonevent" for the results to be statistically nonsignificant. The FI was applied in various medical specialties to assess the robustness of results presented in studies. We aim to assess the robustness of statistically significant results in studies on plastic surgery of the breast and determine factors correlated with studies deemed fragile.

Methods

A systematic literature review of PubMed databases using designated keywords was performed. Background characteristics were extracted from the studies, alongside the significance of outcomes. FI and fragility quotient were calculated for each analyzed outcome and correlated with various baseline characteristics.

Results

FI and fragility quotient were both significantly correlated only with the P value of the analyzed outcomes. However, grouping studies based on the P value into three categories did not demonstrate a difference in FI. Comparisons of fragile and robust studies did not demonstrate a statistically significant change in terms of baseline variables, except for the mean P value of the outcome.

Conclusion

Statistically significant results of randomized controlled trials in plastic surgery of the breast suffer from extensive fragility, and researchers should critically implement their conclusions in their practice.

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

Fragility of Statistically Significant Outcomes in Colonic Diverticular Disease Randomized Trials: A Systematic Review

BACKGROUND

The p value has been criticized as an oversimplified determination of whether a treatment effect exists. One alternative is the fragility index. It is a representation of the minimum number of nonevents that would need to be converted to events to increase the p value above 0.05.

OBJECTIVE

To determine the fragility index of randomized controlled trials assessing the efficacy of interventions for patients with diverticular disease since 2010 to assess the robustness of current evidence.

DESIGN

MEDLINE, Embase, and Cochrane Central Register of Controlled Trials were searched from inception to August 2022.

SETTINGS

Articles were eligible for inclusion if they were randomized trials conducted between 2010 and 2022 with parallel, superiority designs evaluating interventions in patients with diverticular disease. Only randomized trials with dichotomous primary outcomes with an associated p value of <0.05 were considered for inclusion.

PARTICIPANTS

Any surgical or medical intervention for patients with diverticular disease.

MAIN OUTCOME MEASURES

The fragility index was determined by adding events and subtracting nonevents from the groups with the smaller number of events. Events were added until the p value exceeded 0.05. The smallest number of events required was considered the fragility index.

RESULTS

After screening 1271 citations, 15 randomized trials met the inclusion criteria. Nine of the studies evaluated surgical interventions and 6 evaluated medical interventions. The mean number of patients randomly assigned and lost to follow-up per randomized controlled trial was 92 (SD 35.3) and 9 (SD 11.4), respectively. The median fragility index was 1 (range, 0-5). The fragility indices for the included studies did not correlate significantly with any study characteristics.

LIMITATIONS

Small sample, heterogeneity, and lack of inclusion of studies with continuous outcomes.

CONCLUSIONS

The randomized trials evaluating surgical and medical interventions for diverticular disease are not robust. Changing a single-outcome event in most studies was sufficient to make a statistically significant study finding not significant. See Video Abstract .

FRAGILIDAD DE LOS RESULTADOS ESTADSTICAMENTE SIGNIFICATIVOS EN ENSAYOS ALEATORIOS DE ENFERMEDAD DIVERTICULAR DEL COLON UNA REVISIN SISTEMTICA

ANTECEDENTES:El valor p ha sido criticado por una determinación demasiado simplificada de si existe un efecto del tratamiento. Una alternativa es el Índice de Fragilidad. Es una representación del número mínimo de no eventos que deberían convertirse en eventos para aumentar el valor p por encima de 0,05.OBJETIVO:Determinar el IF de ensayos controlados aleatorios que evalúan la eficacia de las intervenciones para pacientes con enfermedad diverticular desde 2010 para evaluar la solidez de la evidencia actual.FUENTES DE DATOS:Se realizaron búsquedas en MEDLINE, Embase y CENTRAL desde el inicio hasta agosto de 2022.SELECCIÓN DE ESTUDIOS:Los artículos eran elegibles para su inclusión si eran ensayos aleatorizados realizados entre 2010 y 2022 con diseños paralelos de superioridad que evaluaran intervenciones en pacientes con enfermedad diverticular. Sólo se consideraron para su inclusión los ensayos aleatorizados con resultados primarios dicotómicos con un valor de p asociado menor que 0,05.INTERVENCIÓNES:Cualquier intervención quirúrgica o médica para pacientes con enfermedad diverticular.PRINCIPALES MEDIDAS DE VALORACIÓN:El índice de fragilidad se determinó sumando eventos y restando no eventos de los grupos con el menor número de eventos. Se agregaron eventos hasta que el valor p superó 0,05. El menor número de eventos requeridos se consideró índice de fragilidad.RESULTADOS:Después de examinar 1271 citas, 15 ensayos aleatorios cumplieron los criterios de inclusión. Nueve de los estudios evaluaron intervenciones quirúrgicas y seis evaluaron intervenciones médicas. El número medio de pacientes aleatorizados y perdidos durante el seguimiento por ECA fue 92 (DE 35,3) y 9 (DE 11,4), respectivamente. La mediana del índice de fragilidad fue 1 (rango: 0-5). Los índices de fragilidad de los estudios incluidos no se correlacionaron significativamente con ninguna característica del estudio.LIMITACIONES:Muestra pequeña, heterogeneidad y falta de inclusión de estudios con resultados continuos.CONCLUSIONES:Los ensayos aleatorios que evalúan las intervenciones quirúrgicas y médicas para la enfermedad diverticular no son sólidos. Cambiar un solo evento de resultado en la mayoría de los estudios fue suficiente para que un hallazgo estadísticamente significativo del estudio no fuera significativo. (Traducción- Dr. Ingrid Melo ).

The Fragility of Landmark Randomized Controlled Trials in the Plastic Surgery Literature

Background

Randomized controlled trials (RCTs) are integral to the progress of evidenced-based medicine and help guide changes in the standards of care. Although results are traditionally evaluated according to their corresponding P value, the universal utility of this statistical metric has been called into question. The fragility index (FI) has been developed as an adjunct method to provide additional statistical perspective. In this study, we aimed to determine the fragility of 25 highly cited RCTs in the plastic surgery literature.

Methods

A PubMed search was used to identify the 25 highest cited RCTs with statistically significant dichotomous outcomes across 24 plastic surgery journals. Article characteristics were extracted, and the FI of each article was calculated. Additionally, Altmetric scores were determined for each study to determine article attention across internet platforms.

Results

The median FI score across included studies was 4 (2-7.5, interquartile range). The two highest FI scores were 208 and 58, respectively. Four studies (16%) had scores of 0 or 1. Three studies (12%) had scores of 2. All other studies (72%) had FI scores of 3 or higher. The median Altmetric score was 0 (0-3).

Conclusion

The FI can provide additional perspective on the robustness of study results, but like the P value, it should be interpreted in the greater context of other study elements.

Statistical Fragility of Randomized Controlled Trials Evaluating Rehabilitation After Arthroscopic Rotator Cuff Repair

Background

Clinical decision-making often relies on evidence-based medicine, derived from objective data with conventional and rigorous statistical tests to evaluate significance. The literature surrounding rehabilitation after rotator cuff repair (RCR) is conflicting, with no defined standard of practice.

Purpose

To determine the fragility index (FI) and the fragility quotient (FQ) of randomized controlled trials (RCTs) evaluating rehabilitation protocols after RCR.

Study Design

Systematic review.

Methods

A systematic review was performed according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines by searching the PubMed, Cochrane Library, and Embase databases for RCTs evaluating rehabilitation protocols after arthroscopic RCRs from 2000 to June 1, 2022. The FI was determined by manipulating the dichotomous outcome events from each article until a reversal of significance with 2 × 2 contingency tables was achieved. The FQ was determined by dividing the FI by the sample size.

Results

Fourteen RCTs with 48 dichotomous outcomes were ultimately included for analysis. The mean FI for the included dichotomous outcomes was 4 (interquartile range, 3-6), suggesting that the reversal of 4 events is required to change study significance. The mean FQ was 0.048. Of the RCTs that reported data regarding loss to follow-up, most studies (58.5%) indicated that >4 patients had been lost to follow-up.

Conclusion

The results of RCT studies of RCR rehabilitation protocols are moderately fragile, something clinicians should be aware of when implementing study results into practice. We recommend the inclusion of FI and FQ in addition to standard P values when reporting statistical results in future RCTs with dichotomous outcome variables on this topic.

Fragility of statistically significant findings from randomized trials in comparing laparoscopic versus robotic abdominopelvic surgeries

BACKGROUND

Utility of robotic over laparoscopic approach has been an area of debate across all surgical specialties over the past decade. The fragility index (FI) is a metric that evaluates the frailty of randomized controlled trials (RCTs) findings by altering the status of patients from an event to non-event until significance is lost. This study aims to evaluate the robustness of RCTs comparing laparoscopic and robotic abdominopelvic surgeries through the FI.

METHODS

A search was conducted in MEDLINE and EMBASE for RCTs with dichotomous outcomes comparing laparoscopic and robot-assisted surgery in general surgery, gynecology, and urology. The FI and reverse fragility Index (RFI) metrics were used to assess the strength of findings reported by RCTs, and bivariate correlation was conducted to analyze relationships between FI and trial characteristics.

RESULTS

A total of 21 RCTs were included, with a median sample size of 89 participants (Interquartile range [IQR] 62-126). The median FI was 2 (IQR 0-15) and median RFI 5.5 (IQR 4-8.5). The median FI was 3 (IQR 1-15) for general surgery (n = 7), 2 (0.5-3.5) for gynecology (n = 4), and 0 (IQR 0-8.5) for urology RCTs (n = 4). Correlation was found between increasing FI and decreasing p-value, but not sample size, number of outcome events, journal impact factor, loss to follow-up, or risk of bias.

CONCLUSION

RCTs comparing laparoscopic and robotic abdominal surgery did not prove to be very robust. While possible advantages of robotic surgery may be emphasized, it remains novel and requires further concrete RCT data.

Statistical Fragility of Venous Thromboembolism Prophylaxis Following Total Joint Arthroplasty

Background

Statistical fragility is a quantitative measure of the robustness of the statistical conclusions drawn in a study. Although statistical fragility has been comprehensively evaluated in the arthroplasty literature, the statistical fragility of large-scale randomized trials evaluating venous thromboembolism (VTE) prophylaxis has not been evaluated. The purpose of this study was to determine the utility of applying the fragility index (FI) and the fragility quotient (FQ) analysis to randomized controlled trials (RCTs) evaluating VTE prophylaxis following total joint arthroplasty.

Methods

A systematic review was performed by searching multiple databases to identify RCTs that evaluated VTE prophylaxis following total joint arthroplasty from 2000 to 2020. The FI was determined by manipulating each reported dichotomous outcome event until a reversal of significance was appreciated with 2 × 2 contingency tables. The associated FQ was determined by dividing the FI by the sample size.

Results

Thirty-two RCTs were ultimately included for analysis. The overall FI incorporating all 32 RCTs was only 7 (interquartile range 3-9), suggesting that the reversal of only 7 events is required to change study significance. The associated FQ was determined to be 0.01. Of the RCTs that reported lost-to-follow-up data, the majority of studies had lost-to-follow-up numbers greater than 7.

Conclusions

Our findings suggest that RCTs evaluating VTE prophylaxis following total hip arthroplasty and total knee arthroplasty may lack statistical stability as few outcome events are required to reverse the significance of outcomes. Future randomized trials should consider reporting FI and FQ along with the P value analysis to provide better context to the integrity of statistical stability.

The Statistical Fragility of Orbital Fractures: A Systematic Review of Randomized Controlled Trials

BACKGROUND

The P value has often been used as a tool to determine the statistical significance and evaluate the statistical robustness of study findings in orthopedic literature. The purpose of this study is to apply both the fragility index (FI) and the fragility quotient (FQ) to evaluate the degree of statistical fragility in orbital fracture literature. We hypothesized that the dichotomous outcomes within the orbital fracture literature will be vulnerable to a small number of outcome event reversals and will be statistically fragile.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), the authors identified all dichotomous data for randomized controlled trials (RCTs) in orbital fracture literature and performed a PubMed search from 2000 to 2022. The FI of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The FQ was calculated by dividing each FI by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 3,329 studies screened, 28 met the criteria with 10 RCTs evaluating orbital fractures included for analysis. A total of 58 outcome events with 22 significant (P < .05) outcomes and 36 nonsignificant (P ≥ .05) outcomes were identified. The overall FI and FQ for all 58 outcomes was 5 (IQR: 4 to 5) and 0.140 (IQR: 0.075 to 0.250), respectively. Fragility analysis of statistical significant outcomes and nonsignificant outcomes had an FI of 3.5 with no IQR and 5 (IQR 4-5), respectively. All of the studies reported a loss to follow-up data, where 20% (2) was greater than the overall FI of 5.

CONCLUSION

The orbital fracture literature provides treatment guidance by relying on statistical significant results from RCTs. However, the RCTs in the orbital fracture peer-reviewed literature may not be statistically stable as previously thought. The sole reliance of the P value may depict misleading results. Thus, we recommend standardizing the reporting of the P value, FI, and FQ in the orbital fracture literature to aid readers in reliably drawing conclusions based on fragility outcome measures impacting clinical decision-making.

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.