Assessing fragility of statistically significant findings from randomized controlled trials assessing pharmacological therapies for opioid use disorders: a systematic review

BACKGROUND

The fragility index is a statistical measure of the robustness or "stability" of a statistically significant result. It has been adapted to assess the robustness of statistically significant outcomes from randomized controlled trials. By hypothetically switching some non-responders to responders, for instance, this metric measures how many individuals would need to have responded for a statistically significant finding to become non-statistically significant. The purpose of this study is to assess the fragility index of randomized controlled trials evaluating opioid substitution and antagonist therapies for opioid use disorder. This will provide an indication as to the robustness of trials in the field and the confidence that should be placed in the trials' outcomes, potentially identifying ways to improve clinical research in the field. This is especially important as opioid use disorder has become a global epidemic, and the incidence of opioid related fatalities have climbed 500% in the past two decades.

METHODS

Six databases were searched from inception to September 25, 2021, for randomized controlled trials evaluating opioid substitution and antagonist therapies for opioid use disorder, and meeting the necessary requirements for fragility index calculation. Specifically, we included all parallel arm or two-by-two factorial design RCTs that assessed the effectiveness of any opioid substitution and antagonist therapies using a binary primary outcome and reported a statistically significant result. The fragility index of each study was calculated using methods described by Walsh and colleagues. The risk of bias of included studies was assessed using the Revised Cochrane Risk of Bias tool for randomized trials.

RESULTS

Ten studies with a median sample size of 82.5 (interquartile range (IQR) 58, 179, range 52-226) were eligible for inclusion. Overall risk of bias was deemed to be low in seven studies, have some concerns in two studies, and be high in one study. The median fragility index was 7.5 (IQR 4, 12, range 1-26).

CONCLUSIONS

Our results suggest that approximately eight participants are needed to overturn the conclusions of the majority of trials in opioid use disorder. Future work should focus on maximizing transparency in reporting of study results, by reporting confidence intervals, fragility indexes, and emphasizing the clinical relevance of findings.

TRIAL REGISTRATION

PROSPERO CRD42013006507. Registered on November 25, 2013.

Statistical fragility of outcomes in acellular dermal matrix literature: A systematic review of randomized controlled trials

BACKGROUND

Acellular dermal matrix (ADM) is commonly used in plastic and reconstructive surgery. With the abundance of randomized controlled trials (RCTs) reporting P-values for ADM outcomes, this study used the fragility index (FI), reverse fragility index (rFI), and fragility quotient (FQ) to evaluate the statistical stability of the outcomes in ADM RCTs.

METHODS

PubMed, Embase, SCOPUS, Medline, and Cochrane databases were reviewed for ADM RCTs (2003-present) reporting a dichotomous, categorical outcome. FI and rFI (event reversals influencing outcome significance) and FQ (standardized fragility) were calculated and reported as median. Subgroup analysis was performed based on intervention types.

RESULTS

Among the 127 studies screened, 56 RCTs with 579 outcomes were included. The median FI stood at 4 (3-5) and FQ was 0.04 (0.03-0.07). Only 101 outcomes were statistically significant with a median FI of 3 (1-6) and FQ of 0.04 (0.02-0.08). The nonsignificant outcomes had a median FI of 4 (3-5) and FQ of 0.04 (0.03-0.07). Notably, 26% of the outcomes had several patients lost to follow up equal to or surpassing the FI. Based on the intervention type, the median FIs showed minor fluctuations but remained low.

CONCLUSIONS

Outcomes from ADM-related RCTs were statistically fragile. Slight outcome reversals or maintenance of patient follow-up can alter the significance of results. Therefore, future researchers are recommended to jointly report FI, FQ, and P-values to offer a comprehensive view of the robustness in ADM literature.

Fragility index analysis for randomized controlled trials of approved biologicals and small molecule drugs in inflammatory bowel diseases

INTRODUCTION

Biologics and small molecules have been increasingly applied in Crohn's disease (CD) and ulcerative colitis (UC). But the robustness of their trials has not been evaluated.

METHODS

We initially collected all the approved biologics or small molecules for CD or UC up to December 1, 2022. Databases were then queried by keywords in chemical name and CD or UC. Randomized controlled trials (RCTs) in the two-arm, 1:1 design were included. Fragility index (FI) and fragility quotient (FQ) were subsequently calculated.

RESULTS

We included twenty-eight RCTs, including nine pivotal trials listed in approval labels, nineteen non-pivotal trials not included in the labels. The median sample size was 99 [IQR, 60-262] and the median number of loss-of-follow-up (LFU) was 14 [IQR, 8-43]. Pivotal trials in the labels had the median FI of 8 [IQR, 4-14, n = 6] that was marginally higher than non-pivotal trials (3 [IQR, 2-4], p = 0.08). The median FQ was 0.0330 [IQR, 0.1220-0.0466] and 0.0310 [IQR, 0.0129-0.0540] for pivotal and non-pivotal trials, respectively (p = 1.0). The sample size and FI were significantly correlated (Spearman correlation coefficient [r] = 0.56, 95 %CI 0.21-0.78, p = 0.003). The number of total events was also significantly correlated with FI (r = 0.53, 95 %CI 0.17-0.77, p = 0.006). Study p-values were significantly associated with FI (p = 0.01): trials with p-values < 0.001 had the highest median FI of 10 [IQR, 6-17]. No factor was found strongly correlated with FQ.

CONCLUSION

Results from trials assessing administration-approved biologics or small molecules for treating CD or UC were vulnerable to small changes by measuring FI or FQ. Pivotal studies contributing to regulatory approvals exhibited a relatively higher degree of resilience compared to non-pivotal trials.

Assessing the robustness of positive vascular surgery randomized controlled trials using their fragility index

OBJECTIVE

The fragility index (FI) measures the robustness of statistically significant findings in randomized controlled trials (RCTs) by quantifying the minimum number of event conversions required to reverse a dichotomous outcome's statistical significance. In vascular surgery, many clinical guidelines and critical decision-making points are informed by a handful of key RCTs, especially regarding open surgical versus endovascular treatment. The objective of this study is to evaluate the FI of RCTs with statistically significant primary outcomes that compared open vs endovascular surgery in vascular surgery.

METHODS

In this meta-epidemiological study and systematic review, MEDLINE, Embase, and CENTRAL were searched for RCTs comparing open versus endovascular treatments for abdominal aortic aneurysms, carotid artery stenosis, and peripheral arterial disease to December 2022. RCTs with statistically significant primary outcomes were included. Data screening and extraction were conducted in duplicate. The FI was calculated by adding an event to the group with the smaller number of events while subtracting a nonevent to the same group until Fisher's exact test produced a nonstatistically significant result. The primary outcome was the FI and proportion of outcomes where the loss to follow-up was greater than the FI. The secondary outcomes assessed the relationship of the FI to disease state, presence of commercial funding, and study design.

RESULTS

Overall, 5133 articles were captured in the initial search with 21 RCTs reporting 23 different primary outcomes being included in the final analysis. The median FI (first quartile, third quartile) was 3 (3, 20) with 16 outcomes (70%) reporting a loss to follow-up greater than its FI. Mann-Whitney U test revealed that commercially funded RCTs and composite outcomes had greater FIs (median, 20.0 [5.5, 24.5] vs median, 3.0 [2.0, 5.5], P = .035; median, 21 [8, 38] vs median, 3.0 [2.0, 8.5], P = .01, respectively). The FI did not vary between disease states (P = .285) or between index and follow-up trials (P = .147). There were significant correlations between the FI and P values (Pearson r = 0.90; 95% confidence interval, 0.77-0.96), and the number of events (r = 0.82; 95% confidence interval, 0.48-0.97).

CONCLUSIONS

A small number of event conversions (median, 3) are needed to alter the statistical significance of primary outcomes in vascular surgery RCTs evaluating open surgical and endovascular treatments. Most studies had loss to follow-up greater than its FI, which can call into question trial results, and commercially funded studies had a greater FI. The FI and these findings should be considered in future trial design in vascular surgery.

The impact of surgical randomised controlled trials on the management of FAI syndrome: a citation analysis

PURPOSE

To identify and assess the clinical impact of randomised controlled trials (RCTs) assessing the surgical management of femoroacetabular impingement syndrome (FAIS) through a citation analysis.

METHODS

MEDLINE, EMBASE and CENTRAL were searched from inception to April 22, 2023 for RCTs assessing the surgical management of FAIS. Study characteristics were directly abstracted from included trials and citation metrics were obtained from the Clarivate Web of Knowledge database on May 19, 2023. The continuous fragility index (CFI) was calculated for eligible outcomes. Univariate regression models were used to explore correlations between total citations per year and various study characteristics.

RESULTS

Ten studies comprising one thousand two hundred ninetypatients were eligible for analysis. Studies were published from 2013 to 2023. Eight countries were represented across various trials with 91% being either North American or European. The mean journal impact factor of published studies was 39.684 (median 2.982; range 1.31-202.73). The mean citation density was 14.17 (range 0.33-48.67). The median CFI was 4.8 (range 1-32.2). Correlation analysis demonstrated strong and statistically significant correlations to study sample size (R = 0.75, p = 0.012), journal impact factor (R = 0.80, p = 0.006) and continuous fragility index (R = 0.95, p = 0.015).

CONCLUSION

Trials assessing the surgical management of FAIS present with a wide range of clinical uptake based on citation density and are published in journals of broadly variable impact factor. Despite promising citation metrics, high-quality evidence on arthroscopy for FAIS is limited to the United States and Europe with an unclear international impact. Future knowledge translation efforts are warranted to maximise the international uptake of evidence regarding arthroscopic management of FAIS.

LEVEL OF EVIDENCE

I.

The statistical fragility of outcomes in calcaneus fractures: A systematic review of randomized controlled trials

INTRODUCTION

The purpose of this study was to utilize the fragility index to assess the robustness of randomized controlled trials (RCTs) evaluating the management of calcaneus fractures. We hypothesize that the dichotomous outcomes in calcaneus fracture literature will be statistically fragile and comparable to other orthopedic specialties.

METHODS

We performed a PubMed search for calcaneus fracture RCTs from 2000 to 2022 using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). The fragility index (FI) of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The fragility quotient (FQ) was calculated by dividing each fragility index by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 3003 studies screened, 97 met the search criteria, with 19 RCTs evaluating calcaneus fractures included in the analysis. Seventy-nine dichotomous outcomes with 30 significant (P < 0.05) outcomes and 49 with nonsignificant (P> 0.05) outcomes were identified. The overall FI and FQ of all outcomes were 6 (IQR 3-8) and 0.067 (IQR 0.032-0.100), respectively.

CONCLUSIONS

The literature surrounding calcaneus fractures may not be as statistically stable as previously thought. The sole reliance on the P value may depict misleading results. We, therefore, recommend reporting the P value in conjunction with the FI and FQ to give a robust contextualization of clinical findings in the calcaneus fracture literature.

The statistical fragility of intramedullary reaming in tibial nail fixation: a systematic review

PURPOSE

To report the statistical stability of prospective clinical trials evaluating the effect of intramedullary reaming on rates of non-union in tibial fractures through calculation of the fragility metrics for non-union rates and all other dichotomous outcomes.

METHODS

Literature search was conducted for prospective clinical trials evaluating the effect of intramedullary reaming on non-union rates in tibial nailing. All dichotomous outcomes were extracted from the manuscripts. The fragility index (FI) and reverse fragility index (RFI) were calculated by determining the number of event reversals required for a statistically significant outcome to lose significance and vice-versa. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI or RFI by the sample size, respectively. Outcomes were defined as "fragile" if the FI or RFI was found to be less than or equal to the number of patients lost to follow-up.

RESULTS

Literature search identified 579 results which produced ten studies meeting the criteria for review. There were 111 outcomes identified for analysis, of which 89 (80%) exhibited statistical fragility. For reported outcomes across the studies the median and mean FI was 2, the median FQ was 0.019, the mean FQ was 0.030, the median RFI was 4, the mean RFI was 3.95, the median RFQ was 0.045, and the mean RFQ was 0.030. Four studies reported outcomes which were found to have an FI of 0.

CONCLUSIONS

The studies evaluating the effect of intramedullary reaming on tibial nail fixation demonstrate considerable fragility. On average, two event reversals for significant findings, and four event reversals for insignificant findings are sufficient to alter statistical significance.

LEVEL OF EVIDENCE

Level II, systematic review of Level I and Level II studies.

The statistical fragility of the management options for reverse shoulder arthroplasty: a systematic review of randomized control trial with fragility analysis

Reverse shoulder arthroplasty (RSA) is used in the treatment of traumatic and arthritic pathologies, with expanding clinical indications and as a result there has been an increase in clinical research on the topic. The purpose of this study was to examine the statistical fragility of randomized control trials (RCTs) reporting outcomes from RSA. A systematic search was undertaken to find RCTs investigating RSA. The Fragility Index (FI) was calculated using Fisher's exact test, by sequentially altering the number of events until there was a reversal of significance. The Fragility Quotient (FQ) was calculated by dividing the FI by the trial population. Each trial was assigned an overall FI and FQ calculated as the median result of its reported findings. Overall, 19 RCTs warranted inclusion in the review, representing 1146 patients, of which 41.2% were male, with a mean age of 74.2 ± 4.3 years and mean follow-up of 22.1 ± 9.9 months. The median RCT population was 59, with a median of 9 patients lost to follow-up. The median FI was 4.5, and median FQ was 0.083, indicating more patients did not complete the trial than the number of outcomes which would have to change to reverse the finding of significance. This review found that the RCT evidence for RSA management may be vulnerable to statistical fragility, with a handful of events required to reverse a finding of significance.

Fragility index of positive phase II and III randomised clinical trials of treatments for hepatocellular carcinoma (2002-2022)

Background & Aims

The fragility index (FI), i.e., theminimum number of best survivors reassigned to the control group required to revert the statistically significant result of a clinical trial to non-significant, is a metric to evaluate the robustness of randomized controlled trials (RCTs). We aimed to assess the FI in the field of HCC.

Methods

This is a retrospective analysis of phase 2 and 3 RCTs for the treatment of HCC published between 2002 and 2022. We included two-arm studies with 1:1 randomization and significant positive results for a primary time-to-event endpoint for the FI calculation, which involves the iterative addition of a best survivor from the experimental group to the control group, until positive significance (p <0,05, Log-rank test) is lost.

Results

We identified 51 phase 2 and 3 positive RCTs, of which 29 (57%) were eligible for fragility index calculation. After reconstruction of the Kaplan-Meier curves, 25/29 studies remained significant, among which the analysis was performed. The median (interquartile range (IQR)) FI was 5 (2-10) and Fragility Quotient (FQ) was 3% (1%-6%). Ten trials (40%) had a FI of 2 or less. FI was positively correlated to the blind assessment of the primary endpoint (median FI 9 with blind assessment versus 2 without, p = 0.01), the number of reported events in the control arm (RS = 0.45, p = 0.02) and to impact factor (RS = 0.58, p = 0.003).

Conclusions

Several phases 2 and 3 RCTs in HCC have a low fragility index, underlying the limited robustness on the conclusion of their superiority over control treatments. The fragility index might provide an additional tool to assess the robustness of clinical trial data in HCC.

Impact and implications

The fragility index is a method to assess robustness of a clinical trial and is defined the minimum number of best survivors reassigned to the control group required to revert the statistically significant result of a clinical trial to non-significant. Among 25 randomised controlled trials in HCC, the median fragility index was 5, and 10 trials among 25 (40%) had a fragility index of 2 or less, indicating an important fragility.

Pulmonary arterial hypertension trials put to the test: Using the fragility index to assess trials robustness

BACKGROUND

Randomized clinical trials (RCTs) are considered the gold standard for evidence-based medicine. The Fragility Index (FI) is a tool to assess the robustness of RCT results. FI was validated for dichotomous outcomes and recent work expanded its use to continuous outcomes. Studying the robustness of RCTs in Pulmonary Arterial Hypertension (PAH) treatments is crucial due to the severity and mortality risks associated with this rare condition.

OBJECTIVES

Analyze FI and Fragility quotient (FQ) of significant primary outcomes in PAH RCTs and study FI correlation with sample size and journal impact factor.

METHODS

FI and FQ calculation followed by Spearman correlation to assess the correlation between FI and sample size, and FI and impact factor.

RESULTS

The median sample size of the 21 trials was 202 patients (IQR 106-267), with 6 trials reporting primary outcomes as dichotomous and 15 reporting continuous primary outcomes. The median FI was 10 (IQR 3-20), and the median FQ was 0.044 (0.026-0.097). A moderate correlation was found between FI and sample size, with r = 0.56; P = 0.008 and FI and journal impact factor (r=0.50; P=0.019). The FI for continuous outcomes was similar to that for dichotomous outcomes.

CONCLUSIONS

This study represents the first analysis of the FI and FQ of PAH treatment RCTs, and expands the use of FI to continuous outcomes in this context. The moderate correlation between FI and sample size suggests that increasing sample size alone is partially correlated to a higher FI. The similarity between FI for continuous and dichotomous outcomes supports the broader use of FI in PAH RCTs.

The reproducibility of interventional radiology randomized controlled trials and external validation of a classification system

PURPOSE

The fragility index (FI) measures the robustness of randomized controlled trials (RCTs). It complements the P value by taking into account the number of outcome events. In this study, the authors measured the FI for major interventional radiology RCTs.

METHODS

Interventional radiology RCTs published between January 2010 and December 2022 relating to trans-jugular intrahepatic portosystemic shunt, trans-arterial chemoembolization, needle biopsy, angiography, angioplasty, thrombolysis, and nephrostomy tube insertion were analyzed to measure the FI and robustness of the studies.

RESULTS

A total of 34 RCTs were included. The median FI of those studies was 4.5 (range 1-68). Seven trials (20.6%) had a number of patients lost to follow-up that was higher than their FI, and 15 (44.1%) had a FI of 1-3.

CONCLUSION

The median FI, and hence the reproducibility of interventional radiology RCTs, is low compared to other medical fields, with some having a FI of 1, which should be interrupted cautiously.

The statistical fragility of studies on rotator cuff repair with graft augmentation

BACKGROUND

Clinical decision-making often relies on evidence-based medicine. Our purpose was to determine the fragility index (FI) and fragility quotient (FQ) for studies evaluating rotator cuff repair (RCR) with graft augmentation. A lost to follow-up (LTF) value greater than the FI indicates statistical instability for the reported outcomes and conclusions.

METHODS

We performed a systematic review using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines by searching PubMed, the Cochrane library, and Embase in June 2022 to identify studies of RCR with graft augmentation. Comparative studies with at least 1 statistically analyzed dichotomous outcome were included. Seventeen studies published in seven peer-reviewed journals from 2003 to 2019 were subsequently evaluated. The FI was determined by changing each reported outcome event within 2 × 2 contingency tables until statistical significance or nonsignificance was reversed. The associated FQ was determined by dividing the FI by the sample size. LTF values were also extracted from each included study.

RESULTS

The included studies had a total of 1098 patients with 36 dichotomous outcomes. The associated median FI was 4 (interquartile range 2-5), indicating that the reversal of 4 patients' outcomes would have reversed the finding of significant difference. The median FQ was 0.08 (interquartile range 0.04-0.15), indicating that in a sample of 100 patients, reversal of 8 patients' outcomes would reverse statistical significance. The median number of patients LTF was 3 (range 0-25), with 56% of reported outcomes having LTF greater than their respective FI.

CONCLUSION

Studies of RCR with graft augmentation lack statistical stability, with few altered outcome events required to reverse statistical significance. Larger comparative studies with better follow-up will strengthen the statistical stability of the evidence for RCR with graft augmentation. For future investigations and reports, we recommend including FI and FQ along with traditional statistical significance analyses to provide better context on the strength of conclusions.

Fragility of statistically significant findings from randomized trials in comparing laparoscopic versus robotic abdominopelvic surgeries

BACKGROUND

Utility of robotic over laparoscopic approach has been an area of debate across all surgical specialties over the past decade. The fragility index (FI) is a metric that evaluates the frailty of randomized controlled trials (RCTs) findings by altering the status of patients from an event to non-event until significance is lost. This study aims to evaluate the robustness of RCTs comparing laparoscopic and robotic abdominopelvic surgeries through the FI.

METHODS

A search was conducted in MEDLINE and EMBASE for RCTs with dichotomous outcomes comparing laparoscopic and robot-assisted surgery in general surgery, gynecology, and urology. The FI and reverse fragility Index (RFI) metrics were used to assess the strength of findings reported by RCTs, and bivariate correlation was conducted to analyze relationships between FI and trial characteristics.

RESULTS

A total of 21 RCTs were included, with a median sample size of 89 participants (Interquartile range [IQR] 62-126). The median FI was 2 (IQR 0-15) and median RFI 5.5 (IQR 4-8.5). The median FI was 3 (IQR 1-15) for general surgery (n = 7), 2 (0.5-3.5) for gynecology (n = 4), and 0 (IQR 0-8.5) for urology RCTs (n = 4). Correlation was found between increasing FI and decreasing p-value, but not sample size, number of outcome events, journal impact factor, loss to follow-up, or risk of bias.

CONCLUSION

RCTs comparing laparoscopic and robotic abdominal surgery did not prove to be very robust. While possible advantages of robotic surgery may be emphasized, it remains novel and requires further concrete RCT data.

The fragility of significant results from randomized controlled trials in esophageal surgeries

While randomized controlled trials (RCTs) are regarded as one of the highest forms of clinical research, the robustness of their P values can be difficult to ascertain. Defined as the minimum number of patients in a study arm that would need to be changed from a non-event to event for the findings to lose significance, the Fragility Index is a method for evaluating results from these trials. This study aims to calculate the Fragility Index for trials evaluating perioperative esophagectomy-related interventions to determine the strength of RCTs in this field. MEDLINE and EMBASE were searched for RCTs related to esophagectomy that reported a significant dichotomous outcome. Two reviewers independently screened articles and performed the data extractions with risk of bias assessment. The Fragility Index was calculated using a two-tailed Fisher's exact test. Bivariate correlation was conducted to evaluate associations between the Fragility Index and study characteristics. 41 RCTs were included, and the median sample size was 80 patients [Interquartile range (IQR) 60-161]. Of the included outcomes, 29 (71%) were primary, and 12 (29%) were secondary. The median Fragility Index was 1 (IQR 1-3), meaning that by changing one patient from a non-event to event, the results would become non-significant. Fragility Index was correlated with P value, number of events, and journal impact factor. The RCTs related to esophagectomy did not prove to be robust, as the significance of their results could be changed by altering the outcome status of a handful of patients in one study arm.

The statistical fragility of the distal fibula fracture literature: A systematic review of randomized controlled trials

BACKGROUND

The purpose of this study was to apply both the fragility index (FI) and fragility quotient (FQ) to evaluate the degree of statistical fragility in the distal fibular fracture (DFF) literature. We hypothesized that the dichotomous outcomes within the DFF literature are statistically fragile.

METHODS

We performed a PubMed search for distal fibular fractures clinical trials from 2000 to 2022 reporting dichotomous outcomes. The FI of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The FQ was calculated by dividing each fragility index by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 1158 articles screened, 23 met the search criteria, with six RCTs included for analysis. Forty-five outcome events with 5 significant (p < 0.05) outcomes and 40 nonsignificant (p ≥ 0.05) outcomes were identified. The overall FI and FQ was 5 (IQR 4-6) and 0.089 (IQR 0.061-0.107), respectively.

CONCLUSIONS

The randomized controlled trials in the peer-reviewed distal fibular fracture literature may not be as robust as previously thought, as incorporating statistical analyses solely on a P value threshold is misleading. Standardized reporting of the P value, FI and FQ can help the clinician reliably draw conclusions based on the fragility of outcome measures.

The Fragility Index in randomised clinical trials supporting clinical practice guidelines for acute coronary syndrome: measuring robustness from a different perspective

BACKGROUND

In randomised clinical trials (RCTs) rejecting the null hypothesis, the fragility index (FI) yields the minimum number of participants who would need to have had a different outcome for the results of the trial to become non-significant. We evaluated the robustness of RCTs supporting ACC/AHA and ESC clinical practice guidelines (CPGs) for ST elevation myocardial infarction (STEMI) and non-ST elevation acute coronary syndrome (NSTE-ACS) using the FI.

METHODS

There were 407 RCTs among the 2128 studies cited in the 2013 and 2014 ACC/AHA and 2017 and 2020 ESC CPGs for STEMI and NSTE-ACS, respectively. The FI could be calculated in 132 RCTs (32.4%) meeting the needed criteria for its estimation (2-arm RCT, 1:1 allocation, binary outcome, p < 0.05).

RESULTS

The median FI was 12 (interquartile range: 4-29). Hence, a change in the outcome status of 12 patients would be needed to reverse the statistical significance of the primary endpoint in 50% of the RCTs. The FI was ≤1% than their sample size in 55.7% RCTs, whereas in 47% of RCTs, the FI was lower than the number of patients lost to follow-up. Some study design features were associated with higher FI (international, multicentre, private funding; all p < 0.05), whilst baseline patient characteristics were not substantially different by FI (e.g., age, female sex, white study participants; all p > 0.05), except for geographic enrolment (p = 0.042).

CONCLUSIONS

FI might be useful to evaluate the robustness of those RCTs with statistically significant findings for the primary endpoint that have an impact on key guideline recommendations.

Robustness of Significant Dichotomous Outcomes in Randomized Controlled Trials in the Treatment of Patients with COVID-19: A Systematic Analysis

Purpose

Significant results of randomized controlled trials (RCTs) should be properly weighed. This study adopted fragility index (FI) to evaluate the robustness of significant dichotomous outcomes from RCTs on coronavirus disease 2019 (COVID-19) treatment.

Materials and methods

ClinicalTrials.gov and PubMed were searched from inception to July 31, 2021. FIs were calculated and their distribution was depicted. FI's categorical influential factors were analyzed. Spearman correlation coefficient (r _s) was reported for the relationship between FI and the continuous characteristics of RCTs.

Results

Fifty RCTs with 120 outcomes in 7869 patients were included. The FI distribution was abnormal with median 3 (interquartile range 1-7, P = 0.0001). The FIs and robustness were affected by the outcomes of interest, various patient populations, and interventions (T = 18.215,16.667, 23.107; P = 0.02,0.0001, 0.001, respectively). A cubic relationship between the FIs and absolute difference of events between groups with R square of 0.848 (T = 215.828, P = 0.0001, R square = 0.865) was observed. A strong negative logarithmic relationship existed between FI and the P value with R square = - 0.834.

Conclusion

The robustness of significant dichotomous outcomes of COVID-19 treatments was fragile and affected by the outcomes of interest, patients, interventions, P value, and absolute difference of events between the groups. FI was an useful quantitative metric for the binary significant outcomes on COVID-19 treatments.

Registration

PROSPERO (CRD42021272455).

Supplementary Information

The online version contains supplementary material available at 10.1007/s44231-022-00027-y.

The fragility of statistical significance in distal femur fractures: systematic review of randomized controlled trials

PURPOSE

The purpose of this study was to apply both the fragility index (FI) and fragility quotient (FQ) to evaluate the degree of statistical fragility in the distal femur fracture (DFF) literature. We hypothesized that the dichotomous outcomes within the DFF literature are statistically fragile.

METHODS

Using preferred reporting items for systematic reviews and meta-analyses, we performed a PubMed search for distal femur fractures clinical trials from 2000 to 2022 reporting dichotomous outcomes. The FI of each outcome was calculated through the reversal of a single outcome event until significance was reversed. The FQ was calculated by dividing each fragility index by study sample size. The interquartile range (IQR) was also calculated for the FI and FQ.

RESULTS

Of the 4258 articles screened, 92 met the search criteria, with eleven RCTs included for analysis. Ninety eight outcome events with 25 significant (P < 0.05) outcomes and 73 nonsignificant (P > 0.05) outcomes were identified. The overall FI and FQ for all 98 outcomes were 5 (IQR 4-6) and 0.130 (IQR 0.087-0.174), respectively. Three studies (33.3%) reported loss to follow (LTF) greater than 5.

CONCLUSIONS

The randomized controlled trials in the peer-reviewed distal femur fracture literature may not be as robust as previously thought, as incorporating statistical analyses solely on a P value threshold is misleading. Standardized reporting of the P value, FI and FQ can help the clinician reliably draw conclusions based on the fragility of outcome measures.

Fragility of statistically significant findings from randomized clinical trials of surgical treatment of humeral shaft fractures: A systematic review

BACKGROUND

Despite recent meta-analyses of randomized controlled trials (RCTs), there remains no consensus regarding the preferred surgical treatment for humeral shaft fractures. The fragility index (FI) is an emerging tool used to evaluate the robustness of RCTs by quantifying the number of participants in a study group that would need to switch outcomes in order to reverse the study conclusions.

AIM

To investigate the fragility index of randomized control trials assessing outcomes of operative fixation in proximal humerus fractures.

METHODS

We completed a systematic review of RCTs evaluating the surgical treatment of humeral shaft fractures. Inclusion criteria included: articles published in English; patients randomized and allotted in 1:1 ratio to 2 parallel arms; and dichotomous outcome variables. The FI was calculated for total complications, each complication individually, and secondary surgeries using the Fisher exact test, as previously published.

RESULTS

Fifteen RCTs were included in the analysis comparing open reduction plate osteosynthesis with dynamic compression plate or locking compression plate, intramedullary nail, and minimally invasive plate osteosynthesis. The median FI was 0 for all parameters analyzed. Regarding individual outcomes, the FI was 0 for 81/91 (89%) of outcomes. The FI exceeded the number lost to follow up in only 2/91 (2%) outcomes.

CONCLUSION

The FI shows that data from RCTs regarding operative treatment of humeral shaft fractures are fragile and does not demonstrate superiority of any particular surgical technique.

Robustness of outcomes in trials evaluating sodium-glucose co-transporter 2 inhibitors for heart failure

AIMS

Recent trials have evaluated sodium-glucose co-transporter 2 inhibitors in patients with heart failure (HF). We sought to assess the robustness of findings from these trials using the fragility index (FI).

METHODS AND RESULTS

Fragility index is defined as the minimum number of patients that must be moved from the 'non-event' to the 'event' group to turn a statistically significant result to non-significant. In addition to FI, fragility quotient [(FQ); FI divided by the sample size] was calculated to assess the proportion of events that must be moved to change the significance. For statistically non-significant outcomes, reverse fragility index (RFI) and reverse fragility quotient (RFQ) were calculated. Robustness of findings after pooling data from all three trials was also assessed. A robust reduction in first HF hospitalization or cardiovascular mortality was seen with dapagliflozin (FI = 62 and FQ = 0.013), empagliflozin (FI = 50 and FQ = 0.013), and sotagliflozin (FI = 60 and FQ = 0.049). Dapagliflozin nominally improved all-cause and cardiovascular mortality, with modest FI (n = 8 and 5) and FQ (0.002 and 0.001). Empagliflozin and sotagliflozin did not demonstrate statistically significant reductions in all-cause mortality, with modest RFI (empagliflozin: RFI = 26 and RFQ = 0.007; sotagliflozin: RFI = 6 and RFQ = 0.005). A similar trend was seen with cardiovascular mortality (empagliflozin: RFI = 24 and RFQ = 0.006; sotagliflozin: RFI = 7 and RFQ = 0.006). Upon meta-analysis, the result for first HF hospitalization or cardiovascular mortality was robust (FI = 95 and FQ = 0.010). The reductions in all-cause (FI = 12 and FQ = 0.001) and cardiovascular mortality (FI = 9 and FQ = 0.001), while statistically significant, were fragile.

CONCLUSION

Improvement in the composite outcome of first HF hospitalization or cardiovascular death was highly concordant and robust across sodium-glucose co-transporter 2 inhibitor trials. In contrast, secondary endpoints of all-cause and cardiovascular mortality were statistically fragile, underscoring the need to power trials for mortality to fully understand the benefit of therapies on fatal events.

On clinical trial fragility due to patients lost to follow up

BACKGROUND

Clinical trials routinely have patients lost to follow up. We propose a methodology to understand their possible effect on the results of statistical tests by altering the concept of the fragility index to treat the outcomes of observed patients as fixed but incorporate the potential outcomes of patients lost to follow up as random and subject to modification.

METHODS

We reanalyse the statistical results of three clinical trials on coronary artery bypass grafting (CABG) to study the possible effect of patients lost to follow up on the treatment effect statistical significance. To do so, we introduce the LTFU-aware fragility indices as a measure of the robustness of a clinical trial's statistical results with respect to patients lost to follow up.

RESULTS

The analyses illustrate that clinical trials can either be completely robust to the outcomes of patients lost to follow up, extremely sensitive to the outcomes of patients lost to follow up, or in an intermediate state. When a clinical trial is in an intermediate state, the LTFU-aware fragility indices provide an interpretable measure to quantify the degree of fragility or robustness.

CONCLUSIONS

The LTFU-aware fragility indices allow researchers to rigorously explore the outcomes of patients who are lost to follow up, when their data is the appropriate kind. The LTFU-aware fragility indices are sensitivity measures in a way that the original fragility index is not.

Fragility Index as a Measure of Randomized Clinical Trial Quality in Adult Reconstruction: A Systematic Review

Background

The Fragility Index (FI) and Reverse Fragility Index are powerful tools to supplement the P value in evaluation of randomized clinical trial (RCT) outcomes. These metrics are defined as the number of patients needed to change the significance level of an outcome. The purpose of this study was to calculate these metrics for published RCTs in total joint arthroplasty (TJA).

Methods

We performed a systematic review of RCTs in TJA over the last decade. For each study, we calculated the FI (for statistically significant outcomes) or Reverse Fragility Index (for nonstatistically significant outcomes) for all dichotomous, categorical outcomes. We also used the Pearson correlation coefficient to evaluate publication-level variables.

Results

We included 104 studies with 473 outcomes; 92 were significant, and 381 were nonstatistically significant. The median FI was 6 overall and 4 and 7 for significant and nonsignificant outcomes, respectively. There was a positive correlation between FI and sample size (R = 0.14, P = .002) and between FI and P values (R = 0.197, P = .000012).

Conclusions

This study is the largest evaluation of FI in orthopedics literature to date. We found a median FI that was comparable to or higher than FIs calculated in other orthopedic subspecialties. Although the mean and median FIs were greater than the 2 recommended by the American Academy of Orthopaedic Surgeons Clinical Practice Guidelines to demonstrate strong evidence, a large percentage of studies have an FI < 2. This suggests that the TJA literature is on par or slightly better than other subspecialties, but improvements must be made.

Level of Evidence

Level I; Systematic Review.

Evaluating the literature on preoperative androgen stimulation for hypospadias repair using the fragility index - can we trust observational studies?

BACKGROUND

Preoperative androgen stimulation (PAS) is typically used in hypospadias repair for patients with a proximal meatus or small glans size. Hypospadias PAS literature suffer from small sample sizes and lack of power to claim robust conclusions. Small changes in the number of events may completely change the statistical significance, making the conclusions drawn unreliable. Fragility index (FI) is the number of additional events needed to occur in either the control or experimental group to turn a statistically significant result to a non-significant result. The objective of the report was to assess the quality of available literature revolving around PAS use in hypospadias repair and its effects on post-operative complication rates using FI.

METHODS

A comprehensive search of MEDLINE, EMBASE, and grey literature (ESPU and SPU abstracts) was conducted to identify RCTs and observational studies investigating the effect of PAS on complications post-hypospadias repair between 1990 and 2020. The FI was calculated for each study. Postoperative complications were defined as: fistula, stricture/stenosis, diverticula, and dehiscence. The odds ratio (OR), 95% confidence intervals (CI), corresponding p-values was calculated for each study. A random effects mixed model was implemented to combine the ORs for each study design.

RESULTS

Fourteen studies qualified for inclusion, of which nine were observational studies and five were RCTs (Figure 1). The median sample size was 110 patients (IQR 69-171). The summary ORs for observational studies was 1.74 (95% CI: 1.10 to 2.74; p = 0.020) and for RCTs was 0.71 (9% CI: 0.34 to 1.47; p = 0.350). The median FI was 0 (IQR 0-2) of the included studies.

DISCUSSION

PAS use does not appear to significantly affect complication rates shown in RCTs, however, observational studies cumulatively suggested significantly greater odds of complications after PAS. The FI is best used for RCTs with 1-to-1 randomization and binary data. Observational studies are rarely balanced for demographics and comorbidities with unequal sample size between comparable groups. The study was limited by substantial variability in how PAS was delivered to patients, leading to restricted comparability.

CONCLUSION

Strong conclusions regarding the influence of PAS on hypospadias repair outcomes cannot be properly drawn based on the current literature due to deficits from either a statistical or methodological standpoint. The current PAS literature has shown inconclusive results, calling for well-designed RCTs, involving standardized surgical techniques and PAS protocols, to evaluate the true effect of PAS on complications post-hypospadias repair.

Fragility of results from randomized controlled trials supporting the guidelines for the treatment of osteoporosis: a retrospective analysis

This is the first report on the fragility of results from randomized controlled trials (RCTs) for the treatment of osteoporosis. The results of aforementioned RCTs appear to depend on a small number of events and are generally statistically fragile.

INTRODUCTION

Osteoporosis remains a health concern worldwide. Evidence-based guideline recommendations that are mainly based on results of clinical trials are important to clinical decision-making. The fragility index (FI) is a novel statistical metric to measure the fragility of results from an RCT. Our study aimed to analyze the fragility of the clinical trials referenced in the guidelines for the treatment of osteoporosis.

METHODS

Trials were included if they investigated primary osteoporosis, randomized patients to treatment or control in a 1:1 design, and reported fracture outcome as the primary endpoint. The FI and fragility quotient (FQ) were calculated for assessing the robustness of results from the eligible RCTs. An FI was defined as the minimum number of events in the intervention group that needs to change from a non-event to an event in order to render a significant result non-significant (or vice versa). The FQ was calculated by dividing the FI by the sample size of the trial.

RESULTS

Of the 372 RCTs identified from the guidelines, 42 were eligible for analyses. Their median FI was 10 (25th-75th percentile [Q1-Q3]: 4-18), with a median FQ of 0.007 (Q1-Q3: 0.0017-0.019). Approximately one third of the RCTs had a FI of less than or equal to 5. There were 17 (40.5%) trials where the number of patients lost to follow-up was greater than the FI. The FI was significantly associated with sample size, journal impact factor, and the percent of patients lost to follow-up.

CONCLUSION

Results from some RCTs supporting guideline recommendations for the treatment of osteoporosis depend on a small number of events. The FI and FQ may provide additional, intuitive metrics to help interpret the robustness of trial results.

The fragility index can be used for sample size calculations in clinical trials

OBJECTIVE

The fragility index is a clinically interpretable metric increasingly used to interpret the robustness of clinical trials results that is generally not incorporated in sample size calculation and applied post-hoc. In this manuscript, we propose to base the sample size calculation on the fragility index in a way that supplements the classical prefixed alpha and power cutoffs and we provide a dedicated R software package for the design and analysis tools.

STUDY DESIGN AND SETTING

This approach follows from a novel hypothesis testing framework that is based on the fragility index and builds on the classical testing approach. As case studies, we re-analyse the design of two important trials in cardiovascular medicine, the FAME and FAMOUS-NSTEMI trials.

RESULTS

The analyses show that approach returns sample sizes which results in a higher power for the P value based test and most importantly a lower and context dependent Type I error rate for the fragility index based test compared to standard tests.

CONCLUSION

Our method allows clinicians to control for the fragility index during clinical trial design.

Randomized controlled trials-a critical re-appraisal

Randomized controlled trials (RCTs) are considered to represent the gold standard of scientific studies and paved the way for evidence-based medicine (EBM). Besides the initial aim to improve the quality of patient care, EBM is used in the meanwhile for political and economic decision-making and legal issues as well. A review of the literature was performed, followed by a search using links and references of the detected articles. Additionally, homepages for German institutions of public health were screened. Substantial limitations of RCTs and EBM health care could be identified. Based on the selected literature, 80% of the medical treatments have low evidence. RCTs are expensive and are mainly performed by the industry nowadays. A publication bias for positive results exists. Some RCTs are of low external validity. Many studies have a low fragility index. Nonetheless, negative RCTs could be of benefit for the patients. The results of RCTs, gained in a distinct patient population, are partially generalized. RCTs should be analyzed critically before adopting the results to daily clinical routine. It is not really justified to use RCTs and EBM for political and economic decision-making and legal issues as seen today.

Data Analysis will not Result in Knowledge Production about Sepsis

How to cite this article: Sampath S. Data Analysis will not Result in Knowledge Production about Sepsis. Indian J Crit Care Med 2021;25(7):750-751.

Fragility index of urological literature regarding medical expulsive treatment

INTRODUCTION

The role of medical expulsive treatment (MET) is controversial. Fragility index is an additional metric to assess randomized controlled trials (RCTs) outcome validity and indicates how many patients would be required to convert a trial from being statistically significant, to not significant. The larger is the FI, the better the trial's data. The aim of this study is to assess FI of RCTs regarding MET for ureteral stones.

MATERIALS AND METHODS

A systematic literature search was performed. RCTs, reporting stone expulsion as a dichotomous outcome, showing statistical significance were eligible. FI (the number of patients needed to change from a non-event to event group, to lose statistical significance) and Fragility quotient (FI divided by total sample size), were calculated while Pearson's correlation and Mann-Whitney U test were used as appropriate.

RESULTS

Thirty-six RCTs were eligible, with median FI = 3.5 and fragility quotient = 0.042, median sample size = 81, median journal impact factor = 1.73 and median reported p value = 0.008. In 33.3% of the studies, number of patients lost during follow-up was larger than FI, while in 13.89% of the studies, FI was 0, indicating use of inappropriate statistical method. Pearson's correlation showed significant positive association between FI and sample size (r = 0.981), number of events (r = 0.982) and impact factor (r = 0.731), while no association was found with p value or publication year.

CONCLUSIONS

In this analysis, a calculated FI of 3.5 indicates that findings from RCTs on MET for ureteral stones are fragile and should be interpreted in combination with clinical thinking and expertise.

Frequent fragility of randomized controlled trials for HCC treatment

Background

The fragility index (FI) of trial results can provide a measure of confidence in the positive effects reported in randomized controlled trials (RCTs). The aim of this study was to calculate the FI of RCTs supporting HCC treatments.

Methods

A methodological systematic review of RCTs in HCC treatments was conducted. Two-arm studies with randomized and positive results for a time-to-event outcome were eligible for the FI calculation.

Results

A total of 6 trails were included in this analysis. The median FI was 0.5 (IQR 0–10). FI was ≤7 in 4 (66.7%) of 6 trials; in those trials the fragility quotient was ≤1%.

Conclusion

Many phase 3 RCTs supporting HCC treatments have a low FI, which challenges the confidence in concluding the superiority of these drugs over control treatments.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-021-08133-8.

ProgPerm: Progressive permutation for a dynamic representation of the robustness of microbiome discoveries

BACKGROUND

Identification of features is a critical task in microbiome studies that is complicated by the fact that microbial data are high dimensional and heterogeneous. Masked by the complexity of the data, the problem of separating signals (differential features between groups) from noise (features that are not differential between groups) becomes challenging and troublesome. For instance, when performing differential abundance tests, multiple testing adjustments tend to be overconservative, as the probability of a type I error (false positive) increases dramatically with the large numbers of hypotheses. Moreover, the grouping effect of interest can be obscured by heterogeneity. These factors can incorrectly lead to the conclusion that there are no differences in the microbiome compositions.

RESULTS

We translate and represent the problem of identifying differential features, which are differential in two-group comparisons (e.g., treatment versus control), as a dynamic layout of separating the signal from its random background. More specifically, we progressively permute the grouping factor labels of the microbiome samples and perform multiple differential abundance tests in each scenario. We then compare the signal strength of the most differential features from the original data with their performance in permutations, and will observe a visually apparent decreasing trend if these features are true positives identified from the data. Simulations and applications on real data show that the proposed method creates a U-curve when plotting the number of significant features versus the proportion of mixing. The shape of the U-Curve can convey the strength of the overall association between the microbiome and the grouping factor. We also define a fragility index to measure the robustness of the discoveries. Finally, we recommend the identified features by comparing p-values in the observed data with p-values in the fully mixed data.

CONCLUSIONS

We have developed this into a user-friendly and efficient R-shiny tool with visualizations. By default, we use the Wilcoxon rank sum test to compute the p-values, since it is a robust nonparametric test. Our proposed method can also utilize p-values obtained from other testing methods, such as DESeq. This demonstrates the potential of the progressive permutation method to be extended to new settings.

A method for calculating the fragility index of continuous outcomes

OBJECTIVE

Clinicians' overdependence on p-values to determine significance in clinical trials is common yet potentially misleading. The Fragility Index (FI) describes how robust a significant result is by determining the number of events the statistical significance hinges on. However, this concept cannot be applied to nondichotomous variables. We describe a method to calculate a Continuous Fragility Index (CFI) for continuous variables. We further provide a method to estimate CFI when original data is not available.

STUDY DESIGN AND SETTING

An iterative substitution algorithm is described to calculate CFI prospectively from data or retrospectively from summary statistics and its response to variations in the data is reported. We then apply this method to a previously published review as a proof-of-concept.

RESULTS

The CFI increases linearly with sample size, logarithmically with mean difference, and decreases exponentially with standard deviation. Forty-eight studies were included of which 30 had significant non-dichotomous outcomes. CFI and FI were uncorrelated and mean CFI was significantly higher than FI (9 vs. 2, P< 0.001).

CONCLUSION

Our algorithm extends fragility to continuous outcomes, expanding the applications of the fragility concept. The fragility of outcomes within a single study may vary based on variable type and should be evaluated independently.

Meta-analyses in paediatric surgery are often fragile: implications and consequences

PURPOSE

Meta-analyses occupy the highest level of evidence and thereby guide clinical decision-making. Recently, randomised-controlled trials were evaluated for the robustness of their findings by calculating the fragility index. The fragility index is the number of events that needs to be added to one treatment arm until the statistical significance collapses. We, therefore, aimed to evaluate the robustness of paediatric surgical meta-analyses.

METHODS

We searched MEDLINE for paediatric surgical meta-analyses in the last decade. All meta-analyses on a paediatric surgical condition were eligible for analysis if they based their conclusion on a statistically significant meta-analysis.

RESULTS

We screened 303 records and conducted a full-text evaluation of 60 manuscripts. Of them, 39 were included in our analysis that conducted 79 individual meta-analyses with significant results. Median fragility index was 5 (Q25-Q75% 2-11). Median fragility in relation to included patients was 0.77% (Q25-Q75% 0.29-1.87%).

CONCLUSION

Paediatric surgical meta-analyses are often fragile. In almost 60% of results, the statistical significance depends on less than 1% of the included population. However, as the fragility index is just a transformation of the P value, it basically conveys the same information in a different format. It therefore should be avoided.

Fragility index of trials supporting approval of anti-cancer drugs in common solid tumours

BACKGROUND

The Fragility Indexquantifies the reliability of positive trials by estimating the number of events, which would change statistically significant results to non-significant results.

METHODS

We identified randomized trials supporting drug approvals by the US FDA between 2009 and 2019 in lung, breast, prostate, and colon cancers and in melanoma. We reconstructed survival tablesand calculated the number of events, which would result in a non-significant result for the primary endpoint. The FI was then compared to the number of patients in each trial who withdrew consent or were lost to follow-up. Regression analyses were used to explore associations between RCT characteristics and FI and trials in which FI was lower or equal to number of participants who withdrew consent or were lost to follow-up.

RESULTS

Among 81 RCTs, the median FI was 28. The median number of patients who withdrew consent or were lost to follow up was 27. FI was equal or lower than the number of patients who withdrew consent or were lost to follow-up in 47 trials (58%). There was a modest increase in FI over time (p = 0.02). Trials with overall survival as the primary endpoint (p = 0.006) and those in the palliative setting (p < 0.001) had lower FI. There was no association with trial sample size or duration of follow-up.

FINDINGS

Statistical significance of RCTs in common solid tumours can be reversed often with a small number of additional events. Post-approval RCTs or real-world data analyses should be performed to ensure results of registration trials are robust.

How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines

Objectives

“Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016” provides guidelines in regard to prompt management and resuscitation of sepsis or septic shock. The study is aimed to assess the robustness of randomized controlled trials (RCTs) that formulate these guidelines in terms of fragility index and reverse fragility index.

Method

RCTs that contributed to these guidelines having parallel two-group design, 1:1 allocation ratio, and at least one dichotomous outcome were included in the study. The median fragility index was calculated for RCTs with significant statistical outcomes, whereas the median reverse fragility index was calculated for RCTs with nonsignificant statistical results.

Results

Hundred RCTs that met the inclusion criteria were analyzed. The median fragility index was 5.5 [95% confidence interval (CI) 1–30] and median reverse fragility index was 13 (95% CI 12.07–16.8) at a p value of 0.05. The median reverse fragility index was 16 (95% CI 10–26) at a p value of 0.01. Most of the RCTs included in this analysis were of good quality, having a median Jadad score of 6.

Conclusion

This analysis found that the surviving sepsis guidelines were based on highly robust RCTs with statistically insignificant results and on some moderately robust RCTs with statistically significant results. RCTs with statistically insignificant results were more robust than RCTs with statistically significant results in regard to these guidelines.

Highlights

The study assessed the robustness of randomized controlled trials (RCTs) that were used to formulate surviving sepsis guidelines. Most RCTs showed statistically nonsignificant results. RCTs with statistically significant results were moderately fragile whereas RCTs with nonsignificant results were more robust.

How to cite this article

Choupoo NS, Das SK, Saikia P, Dey S, Ray S. How Robust are the Evidences that Formulate Surviving Sepsis Guidelines? An Analysis of Fragility and Reverse Fragility of Randomized Controlled Trials that were Referred in these Guidelines. Indian J Crit Care Med 2021;25(7):773–779.

Trends and relevance in the bladder and bowel dysfunction literature: PlumX metrics contrasted with fragility indicators

INTRODUCTION

The concepts of fragility index (FI) and fragility quotient (FQ) have been previously described. PlumX metrics encompass online "footprints" of research in addition to traditional citations. Herein we explore PlumX metrics against the quality of BBD literature.

OBJECTIVE

To explore altmetrics against the quality of bladder and bowel dysfunction (BBD) literature.

STUDY DESIGN

A literature search was conducted using Pubmed, Medline, Embase for BBD and related terms. A total of 54,045 abstracts were screened, followed by 693 full text reviews and data extraction from 126. Studies were included if they reported on 2 groups being compared, had dichotomous outcomes, and had significant results.

RESULTS

The median FI score was 4 (0-500) and there were 20 studies which had a FI of 0. The FQ had a median of 0.04 (0-0.32). PlumX usage was 263 ± 540, captures were 45 ± 60 and social media attention was 2 ± 2. Overall, 42% of papers were clinical trials (RCTs). When compared to other study designs, we noted a significant difference in PlumX captures (57 ± 72 RCT vs. 35 ± 47 other; p = 0.03). RCTs had higher usage, social media engagement and citations however, the differences were not significant. H-Index had a significant correlation with FI (p = 0.036), however correlations for PlumX usage and captures, while modestly positive (0.04-0.10) for the FI and FQ, were not significant. A comparison of FI and FQ by topic can be reviewed in the Summary Table.

DISCUSSION

When considering the FI and FQ robustness indicators of the BBD literature, we found similarities when compared to other studies. It was reported that overall, the hydronephrosis literature was fragile with many studies requiring only a few events to nullify significance, regardless of the study design. Similarly, in a review of pediatric vesicoureteral reflux (VUR) clinical trials, results were also fragile. When comparing fragility measures to altmetric variables we noted that despite the growing popularity of altmetrics, citation counts, and h-indices remain the traditional measures to monitor research consumption. There has been a reported correlation between manuscript citation counts, author h-index, altmetrics measures in several specialties and across many domains of research including medical sciences, arts, and the humanities, however in the present study only weak correlations were noted.

CONCLUSION

The body of BBD comparative studies is fragile in keeping with other pediatric urology literature populations. Despite fragile results, RCTs generate slightly moreattention as measured by select PlumX metrics. These results suggest the need for including fragility measures in our literature, aiming to focus attention towards more robust articles.

Fragility index of network meta-analysis with application to smoking cessation data

BACKGROUND AND OBJECTIVES

The network meta-analysis (NMA) is frequently used to synthesize evidence for multiple treatment comparisons, but its complexity may affect the robustness (or fragility) of the results. The fragility index (FI) is recently proposed to assess the fragility of the results from clinical studies and from pairwise meta-analyses. We extend the FI to NMAs with binary outcomes.

METHODS

We define the FI for each treatment comparison in NMAs. It quantifies the minimal number of events necessary to be modified for altering the comparison's statistical significance. We introduce an algorithm to derive the FI and visualizations of the process. A worked example of smoking cessation data is used to illustrate the proposed methods.

RESULTS

Some treatment comparisons had small FIs; their significance (or nonsignificance) could be altered by modifying a few events' status. They were related to various factors, such as P-values, event counts, and sample sizes, in the original NMA. After modifying event status, treatment ranking measures were also changed to different extents.

CONCLUSION

Many NMAs include insufficiently compared treatments, small event counts, or small sample sizes; their results are potentially fragile. The FI offers a useful tool to evaluate treatment comparisons' robustness and reliability.

The Reproducibility of Cerebrovascular Randomized Controlled Trials

BACKGROUND

Numerous randomized controlled trials (RCTs) relevant to the cerebrovascular field have been performed. The fragility index was recently developed to complement the P value and measure the robustness and reproducibility of clinical findings of RCTs.

OBJECTIVE

In this study, we evaluate the fragility index for key surgical and endovascular cerebrovascular RCTs and propose a novel RCT classification system based on the fragility index.

METHODS

Cerebrovascular RCTs reported between 2000 and 2018 were reviewed. Six key areas were specifically targeted in relation to stroke, carotid stenosis, cerebral aneurysms, and subarachnoid hemorrhage. The correlation between fragility index, number of patients lost to follow-up, and fragility quotient were evaluated to propose a classification system for the robustness of the studies.

RESULTS

A total of 20 RCTs that reported significant differences between both study groups in terms of the primary outcome were included. The median fragility index for the trials was 5.5. An additional 30 randomly selected RCTs were added to propose a classification system with high reliability. The difference between the number of patients lost to follow-up and fragility index inversely correlated with the fragility quotient and was used to divide the robustness of the RCTs into 3 classes reflecting the reproducibility of the trial.

CONCLUSIONS

Neurosurgeons and neurointerventionalists should exercise caution with interpreting the results of cerebrovascular RCTs, especially when the sample size and events numbers are small and there is a high number of patients who were lost to follow-up, as quantitatively identified using the proposed classification system.

The fragility of statistically significant findings from randomized controlled trials in pediatric appendicitis: A systematic review

PURPOSE

Randomized controlled trials (RCT) in pediatric appendicitis remain limited, and the robustness of available evidence is unknown. The aim of this study was to determine the fragility of results in pediatric appendicitis RCTs.

METHODS

A systematic search of Embase and MEDLINE was performed. Eligible studies were two-armed RCTs that included at least one statistically significant dichotomous outcome, had parallel-group allocation, and assessed pediatric patients (0-17) with a primary diagnosis of appendicitis. The Fragility Index (FI) for one statistically significant outcome per trial was calculated using a Fisher's exact test, with statistical significance set at p < 0.05.

RESULTS

Six studies were identified for inclusion. Studies included a median of 103 patients (interquartile range [IQR] 86-127), with a median of 18 (IQR 4.5-41.25) events for analyzed outcomes. The primary outcome variable was included in analysis for 4(67%) studies. The median FI across studies was 3 (IQR 0.75-4.25), with results ranging from 0 to 5. Results indicate that overall, converting 3 patients from non-events to events in a single trial arm would change the significant dichotomous outcome to nonsignificant.

CONCLUSION

The fragility of results in RCTs in pediatric appendicitis should be considered before clinical practice is changed. Investigators should consider reporting the FI alongside study results, as p-values alone may be misleading.

TYPE OF STUDY

Randomized Controlled Trial.

LEVEL OF EVIDENCE

Level I.

Further medical experience will be required to validate these results: How experience -based medicine shapes the validity of medical evidence

This communication contextualizes the importance of clinical experience or experience-based medicine within the current paradigm of evidence-based medicine, complementing our previously published series of five short educational articles on evidence-based medicine. Previously we focused on optimizing medical decisions using publications of well-conducted research. Previous commentaries included a backgrounder, the hierarchy of evidence, data acquisition, tools for critical appraisal, and clinical use of evidence based medicine.

The fragility of findings of randomized controlled trials in shoulder and elbow surgery

BACKGROUND

Considered the gold standard of study designs, randomized controlled trials' (RCTs) results shape clinical practice, effect policy, and influence reimbursement. The fragility index (FI) can be used to quantitate the relative robustness of RCT results, with higher scores indicating more stout results. Unfortunately, most RCTs in surgery have fragile results. The aim of this study was to report on the FI in addition to a qualitative assessment of recent RCTs within the field of shoulder and elbow surgery.

METHODS

A systematic review was performed identifying recently published shoulder/elbow RCTs that included 1:1 allocated parallel study arms, dichotomous primary outcome variables, and statistical significance. The FI was calculated by sequentially modifying outcome groups by exchanging a nonevent in one group to an event until the P value for the outcome comparison, as calculated by the Fisher exact test, was increased above the .05 threshold.

RESULTS

Thirty RCTs were included. The median FI was 4. Sixty percent trials had a FI of 2 or less. Fifty-three percent studies reported that participants were lost to follow-up. In 87.5% of these studies, the losses to follow-up exceeded their respective FIs. Only 53% of studies defined a primary outcome variable and 60% studies performed a prestudy power analysis.

CONCLUSIONS

The median FI reported in the recent shoulder/elbow literature is 4; however, a high proportion of included RCTs display significant methodological concerns. The FI is a useful adjunct to analyze RCT results, but careful analysis of trial methods should be employed in each circumstance before drawing conclusions.

The fragility index applied to liver-related trials

The goal of this manuscript was to apply the fragility index (FI), which is a statistically sound method to evaluate robustness of test results, to liver-related randomized clinical trials. The authors searched the ClinicalTrials.gov database with the following limitations: term "liver," recruitment completed, with results, interventional study type, last updated May 01, 2016, to May 01, 2017. Forty-eight trials were included and four had FI of 0. The median FI for trials moving from significance to non-significance was 6 (IQR 18; 2 to 20), while the median for trials moving from non-significance to significance was 5 (IQR 5; 4 to 9). The median number lost to follow up was 17 (IQR 42; 3 to 45). Of the 21 trials that showed statistical significance, the number lost to follow up was greater than the FI in 13 (61.90%) trials. Investigators of liver-related studies should consider adding the FI to evaluate their work.

The fragility of statistically significant results from clinical nutrition randomized controlled trials

BACKGROUND & AIMS

Recently, a parameter called "Fragility index" (FI) has been proposed, which measures how many events the statistical significance relies on. The lower the FI the more "fragile" the results, and thus more care should be taken when interpreting the results. Our aim in this study was to check FI of nutritional trials.

METHODS

We conducted a systematic review of human clinical nutrition RCTs that report statistically significant dichotomous primary outcomes. We searched the EMBASE, MEDLINE, and Scopus databases. The FI of primary outcomes using the Fisher exact test was calculated and checked the correlations of FI with the number of randomised trials, the p-value of primary outcomes, the publication date, the journal impact factor and the number of patients lost to follow-up.

RESULTS

The initial database search revealed 5790 articles, 37 of which were included in qualitative synthesis. The median (IQR) FI for all studies was 1 (1-3). 28 studies (75.7%) had an FI lower or equal to 2, and in 12 (32.43%) articles, the FI was lower than the number of patients lost to follow-up. No correlations were found between FI and the study characteristics (number of randomized patients, p value of primary outcome, event ratio in experimental group, event ratio in control group, publication date, journal impact factor, lost to follow-up).

CONCLUSION

The results of RCTs in nutritional research often rely on a small number of events or patients. The number of patients lost to follow-up is frequently higher than the FI calculation. Formulating recommendations based on RCTs should be done with caution and FI may be used as auxiliary parameter when assessing the robustness of their findings.

The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses

OBJECTIVES

Meta-analyses inform clinical practice by summarizing treatment effect estimates based on results from several trials. However, the statistical significance of a meta-analysis (i.e., whether the pooled treatment effect is statistically significant or not) may rely on the outcome of only a few patients from specific trials in the meta-analysis. We aimed to evaluate the extent to which the statistical significance of meta-analyses can be changed (from statistically significant to nonsignificant, or vice versa) after modifying the event status of patients in specific arms of specific trials.

METHODS

We conducted a cross-sectional analysis of meta-analyses of trials with a binary outcome from Cochrane Systematic Reviews. We defined the fragility index of meta-analyses as the minimum number of patients from one or more trials included in the meta-analysis for whom an event-status modification (i.e., changing an event to nonevent or a nonevent to event) would change the statistical significance of the pooled treatment effect. For statistically significant and nonsignificant meta-analyses, we evaluated the fragility index, the ratio between the fragility index and the total number of participants included in the trials, and the ratio between the fragility index and the total number of events.

RESULTS

Our sample comprised 906 meta-analyses: 400 and 506 had statistically significant and nonsignificant pooled treatment effects, respectively. For statistically significant meta-analyses, the median fragility index was 12 (Q1-Q3: 4-33); for 29% the fragility index was 5 or less. Overall, 43% and 9% meta-analyses would have become nonsignificant if the event status was modified for less than 1% of the total participants in one or several specific trials, and for less than 1% of the total number of events, respectively. These proportions were similar for statistically nonsignificant meta-analyses. Overall, the statistical significance of 33% of all meta-analyses depended on the event status of five or fewer participants from one or more specific trials.

CONCLUSION

The statistical significance of meta-analyses often depends on the outcome of a few patients. The fragility index of meta-analyses may help in interpreting the conclusions of meta-analyses.

Fragility Index in Randomized Controlled Trials of Ischemic Stroke

OBJECTIVE

The fragility index (FI), a minimum number of events in 1 arm of a clinical trial required to revert the statistically significant result to nonsignificant, has recently been developed as an easy-to-understand novel metric to evaluate the robustness of randomized controlled trials (RCTs). Here, we evaluated the FI of RCTs in the field of neurology, particularly in studies of ischemic stroke.

METHODS

Previous literature published between June 1, 2012 and May 31, 2018 were reviewed from the MEDLINE database by the authors. The original article reporting the significant RCT result, of which a dichotomous outcome was set as its primary outcome measure, was included to evaluate the robustness of the result by calculating the FI. In addition, recent studies examining FI in other clinical fields were reviewed and summarized.

RESULTS

In the 25 eligible RCT studies, the median total number of study participants was 206 (inter quartile range: 144-450) and the median FI was 7 (inter quartile range: 4-15.0). The FI showed a strong negative correlation with the observed P value. There was no significant difference in the FI between RCTs with and without acute settings. Our median FI was higher than the median FI of 2.5 of previous studies examining FI in other clinical fields, as only 20% (5 of 25) of studies included in our study had an FI less than 2.5.

CONCLUSION

Our results suggest that many RCTs in the field of ischemic stroke have a fair robustness, when compared to those in other clinical fields.

Treatments for giant cell arteritis: Meta-analysis and assessment of estimates reliability using the fragility index

BACKGROUND

To better communicate the results of randomized controlled trials (RCTs) of giant cell arteritis (GCA), we propose the use of the fragility index (FI), which is an intuitive measure defined as the minimum number of subjects whose status would have to change (e.g., from having the outcome to not) to render a statistically significant result nonsignificant, or vice-versa.

METHODS

We conducted a systematic review and random-effects meta-analysis of RCTs of glucocorticoid (GC) sparing strategies for relapse-free maintenance in GCA, and used the FI to simplify the presentation of results.

RESULTS

Ten RCTs (nine phase II and one phase III enrolling 645 subjects) were included. Tocilizumab, IV GC and methotrexate significantly improved the likelihood of being relapse free with relative risks and 95% confidence intervals of 3.54 (2.28, 5.51), 5.11 (1.39, 18.81) and 1.54 (1.02, 2.30); respectively. The median FI was 4.5 (range, 1-28), and was generally higher for negative RCTs (n = 6; median FI 4.5) than for positive RCTs (n = 4; median FI 3.5). The range of FI per treatment was (1-8) for methotrexate, (2-6) for anti-TNF agents, 4 for abatacept, 3 for IV GC pulses and (4-28) for tocilizumab.

CONCLUSION

Tocilizumab, IV GC and methotrexate improve the likelihood of being relapse-free in subjects with GCA. Assessment of GC sparing strategies in GCA has long depended on imprecise trials that would change significance if outcomes were reversed for a handful of subjects. FI may be used in rheumatology to simplify communication of statistical significance and overcome limitations of p-value.

Unbreakable? An analysis of the fragility of randomized trials that support diabetes treatment guidelines

AIMS

To analyze the robustness of randomized controlled trials (RCTs) referenced in the American Diabetes Association's Standards of Medical Care in Diabetes-2017 using the Fragility Index (FI) and Fragility Quotient (FQ).

METHODS

We performed a systematic survey of all RCTs referenced in the Standards of Medical Care in Diabetes-2017. One investigator screened for trials and then recorded data from them, including sample size per group, event rate per group, and the dichotomous outcome analyzed by trialists. The FI and FQ for each outcome were calculated. Outcomes were then surveyed to determine whether the outcome used for analysis aligned with the guideline recommendation.

RESULTS

Thirty-five RCTs were included in this study. Thirty-three of 35 (94%) FIs were based on the trial outcomes referenced in the clinical practice guideline. The median sample size was 2548 participants ([IQR], 522-6946). The median total number of events for each outcome was 403 (IQR, 86-969). Nineteen (54%) P Values were below 0.05, 8 (22.4%) were below 0.01. The median FI for all trials was 16 (IQR, 4-29). The median FQ was 0.007 (IQR, 0.003-0.014). FI was not related to risk of bias or Science Citation Index but was significantly correlated with sample size (for significant trials; r=0.77, P < .001, for neutral trials; r=0.76, P < .001).

CONCLUSIONS

We found that the robustness of RCTs varied, but on the whole were not robust in nature. Most trials demonstrated a modest FI and FQ. As a result, conclusions drawn from these trials should take this information into account.