Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

The Number of Patients Lost to Follow-up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

PURPOSE

To 1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF) and 2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature.

METHODS

All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI (RFI) to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements were defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome and were determined by two authors. A theoretical RCT with a sample size of 100, p-value of 0.006, and an FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (the FI) vs. adding patients LTF back to the trial to demonstrate the true sensitivity of RCTs to patients LTF.

RESULTS

Of the 39 FI-based studies, 37 (95%) directly compared the FI to the number of patients lost to follow-up. Of these, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal.

CONCLUSIONS

The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was demonstrated to be greater than the FI, suggesting RCTs with relatively "low" FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.

The Utilization of Preoperative Steroids Safely Decreases the Risk of Postoperative Delirium in Geriatric Patients After Hip Fracture Surgery: A Systematic Review and Meta-analysis of Randomized Controlled Trials

OBJECTIVES

Postoperative delirium is an acute neurocognitive complication that can have adverse effects on outcomes of geriatric patients after undergoing hip fracture surgery. The objective of this study was to examine the efficacy of preoperative steroids in preventing postoperative delirium after hip fracture surgery.

METHODS

DATA SOURCES

A systematic review and meta-analysis was performed using PubMed, SPORTDiscus, CINAHL, MEDLINE, and Web of Science from database inception until September 28, 2023.

STUDY SELECTION

Inclusion criteria were randomized controlled trials of patients who underwent surgical intervention for hip fracture, were examined for postoperative delirium, and used preoperative steroids.

DATA EXTRACTION

Data included the risk of postoperative delirium, postoperative all-cause infection, and postoperative hyperglycemia. Articles were graded via the Cochrane Collaboration's tool.

DATA SYNTHESIS

Statistical analysis included a random-effects binary model with relative risk, 95% confidence intervals along with a defined "number needed to treat" threshold (number needed to treat).

RESULTS

Four randomized controlled trials were included from 128 articles initially retrieved. Patients (n = 416; average age: 82.2 ± 2.2 years) underwent surgical intervention for hip fracture after receiving either preoperative steroids (n = 209) or control interventions (n = 207). There was a statistically significant decrease in the incidence of postoperative delirium among patients who received preoperative steroids (12.9%; 27 cases) as compared with patients who received control interventions (26.7%; 55 cases) after hip fracture surgery ( P < 0.001; RR: 0.84). The absolute risk difference was 13.8%, and the number needed to treat was 7.2 patients. There was no statistically significant difference in the risk of postoperative all-cause infection among patients who received preoperative steroids as compared with patients who received normal saline as placebo after hip fracture surgery ( P = 0.850; RR: 0.96).

CONCLUSIONS

The utilization of preoperative steroids seems to decrease the risk of postoperative delirium after hip fracture surgery in elderly adults. Furthermore, this decreased risk of postoperative delirium was not associated with a significant increase in postoperative infection, indicating possible safety of preoperative steroid administration.

LEVEL OF EVIDENCE

Therapeutic Level II. See Instructions for Authors for a complete description of levels of evidence.

Water birth: a systematic review and meta-analysis of maternal and neonatal outcomes

OBJECTIVE

This systematic review and meta-analysis aimed to conduct a thorough and contemporary assessment of maternal and neonatal outcomes associated with water birth in comparison with land-based birth.

DATA SOURCES

We conducted a comprehensive search of PubMed, EMBASE, CINAHL, and gray literature sources, from inception to February 28, 2023.

STUDY ELIGIBILITY CRITERIA

We included randomized and nonrandomized studies that assessed maternal and neonatal outcomes in patients who delivered either conventionally or while submerged in water.

METHODS

Pooled unadjusted odds ratios with 95% confidence intervals were calculated using a random-effects model (restricted maximum likelihood method). We assessed the 95% prediction intervals to estimate the likely range of future study results. To evaluate the robustness of the results, we calculated fragility indices. Maternal infection was designated as the primary outcome, whereas postpartum hemorrhage, perineal lacerations, obstetrical anal sphincter injury, umbilical cord avulsion, low Apgar scores, neonatal aspiration requiring resuscitation, neonatal infection, neonatal mortality within 30 days of birth, and neonatal intensive care unit admission were considered secondary outcomes.

RESULTS

Of the 20,642 articles identified, 52 were included in the meta-analyses. Based on data from observational studies, water birth was not associated with increased probability of maternal infection compared with land birth (10 articles, 113,395 pregnancies; odds ratio, 0.93; 95% confidence interval, 0.76-1.14). Patients undergoing water birth had decreased odds of postpartum hemorrhage (21 articles, 149,732 pregnancies; odds ratio, 0.80; 95% confidence interval, 0.68-0.94). Neonates delivered while submerged in water had increased odds of cord avulsion (10 articles, 91,504 pregnancies; odds ratio, 1.75; 95% confidence interval, 1.38-2.24) and decreased odds of low Apgar scores (21 articles, 165,917 pregnancies; odds ratio, 0.69; 95% confidence interval, 0.58-0.82), neonatal infection (15 articles, 53,635 pregnancies; odds ratio, 0.64; 95% confidence interval, 0.42-0.97), neonatal aspiration requiring resuscitation (19 articles, 181,001 pregnancies; odds ratio, 0.60; 95% confidence interval, 0.43-0.84), and neonatal intensive care unit admission (30 articles, 287,698 pregnancies; odds ratio, 0.56; 95% confidence interval, 0.45-0.70).

CONCLUSION

When compared with land birth, water birth does not appear to increase the risk of most maternal and neonatal complications. Like any other delivery method, water birth has its unique considerations and potential risks, which health care providers and expectant parents should evaluate thoroughly. However, with proper precautions in place, water birth can be a reasonable choice for mothers and newborns, in facilities equipped to conduct water births safely.

Glucagon-like peptide-1 receptor agonists and risk of thyroid cancer: A systematic review and meta-analysis of randomized controlled trials

AIM

To conduct a meta-analysis of randomized clinical trials (RCTs) to investigate whether there is an association between glucagon-like peptide-1 receptor agonist (GLP-1RA) treatment and thyroid cancer.

MATERIALS AND METHODS

In this meta-analysis of RCTs, we included studies comparing a GLP-1RA with any comparator, lasting at least 52 weeks, and reporting the incidence of adverse events independently of the principal endpoint and population. All cases of thyroid cancer were collected.

RESULTS

We retrieved 64 trials, 26 of which reported at least one incident case of thyroid cancer. GLP-1RA treatment was associated with a significant increase in the risk of overall thyroid cancer (Mantel-Haenzel odds ratio [MH-OR] 1.52 [95% confidence interval {CI} 1.01, 2.29]; P = 0.04, I2  = 0%), with a fragility index of 1, and a 5-year number needed to harm of 1349. The association remained significant when including only trials lasting at least 104 weeks (MH-OR 1.76 [95% CI 1.00, 3.12]; P = 0.05). No significant association was found for papillary thyroid cancer (MH-OR 1.54 [95% CI 0.77, 3.06]; P = 0.22) or medullary thyroid cancer (MH-OR 1.44 [95% CI 0.23, 9.16]; P = 0.55).

CONCLUSIONS

Our meta-analysis showed that GLP-1RA treatment could be associated with a moderate increase in relative risk for thyroid cancer in clinical trials, with a small increase in absolute risk. Studies of longer duration are required to assess the clinical implications of this finding.

The Fragility of Landmark Randomized Controlled Trials in the Plastic Surgery Literature

Background

Randomized controlled trials (RCTs) are integral to the progress of evidenced-based medicine and help guide changes in the standards of care. Although results are traditionally evaluated according to their corresponding P value, the universal utility of this statistical metric has been called into question. The fragility index (FI) has been developed as an adjunct method to provide additional statistical perspective. In this study, we aimed to determine the fragility of 25 highly cited RCTs in the plastic surgery literature.

Methods

A PubMed search was used to identify the 25 highest cited RCTs with statistically significant dichotomous outcomes across 24 plastic surgery journals. Article characteristics were extracted, and the FI of each article was calculated. Additionally, Altmetric scores were determined for each study to determine article attention across internet platforms.

Results

The median FI score across included studies was 4 (2-7.5, interquartile range). The two highest FI scores were 208 and 58, respectively. Four studies (16%) had scores of 0 or 1. Three studies (12%) had scores of 2. All other studies (72%) had FI scores of 3 or higher. The median Altmetric score was 0 (0-3).

Conclusion

The FI can provide additional perspective on the robustness of study results, but like the P value, it should be interpreted in the greater context of other study elements.

Non-Pharmacological Strategies and Interventions for Effective COVID-19 Control: A Narrative Review

The COVID-19 pandemic had a devastating impact on the world, causing widespread illness and death. Focusing on prevention strategies to limit the spread of the disease remains essential. Despite the advent of vaccines, maintaining a vigilant approach to prevention remains paramount. We reviewed effective strategies to prevent COVID-19 transmission, including various prevention measures and interventions and both established practices and unresolved issues that have been addressed in meta-analyses, literature reviews, or in the health care context. Standard precautions are the cornerstone of infection control, with hand hygiene and mask use as key components. The use of surgical masks is recommended to prevent droplet transmission, while eye protection is recommended in combination with masks. In terms of room occupancy, ventilation is critical in reducing the risk of transmission in poorly ventilated environments. Chemical disinfection of indoor air with Triethylene glycol-based products can provide safe additional protection. Since viral RNA detection on surfaces does not necessarily indicate infectivity, the risk of transmission by surface contact remains low if surfaces are properly maintained and hand hygiene is practiced regularly. Thus, prevention of SARS-CoV-2 transmission requires a multifaceted approach, including reducing particle emissions from infected persons by wearing masks, eliminating aerosols by ventilation and air treatment, ensuring physical separation, and protecting exposed persons with masks and eye protection.

Hysteroscopic resection vs ultrasound-guided dilation and evacuation for treatment of cesarean scar ectopic pregnancy: a randomized clinical trial

BACKGROUND

Cesarean scar ectopic pregnancy is a type of ectopic pregnancy in which the fertilized egg is implanted in the muscle or fibrous tissue of the scar after a previous cesarean delivery. The condition can be catastrophic if not managed on time and can lead to significant morbidity and mortality. Several approaches have been studied for the management of cesarean scar ectopic pregnancy in women who opted for termination of pregnancy with no consensus on the best treatment modality reached so far.

OBJECTIVE

This study aimed to compare the success rate of hysteroscopic resection vs ultrasound-guided dilation and evacuation for the treatment of cesarean scar ectopic pregnancy.

STUDY DESIGN

This was a parallel group, nonblinded, randomized clinical trial conducted at a single center in Italy. Women with singleton gestations at <8 weeks and 6 days of gestation were included in the study. Inclusion criteria were women with a cesarean scar ectopic pregnancy with positive embryonic heart activity who opted for termination of pregnancy. Patients were randomized 1:1 to receive either hysteroscopic resection (ie, intervention group) or ultrasound-guided dilation and evacuation (ie, control group). Both groups received 50 mg/m2 of methotrexate intramuscularly at the time of randomization (day 1) and another dose at day 3. A third dose of methotrexate was planned in case of persistence of positive fetal heart activity at day 5. Participants received either ultrasound-guided dilation and evacuation or hysteroscopic resection from 1 to 5 days after the last dose of methotrexate. Hysteroscopic resection was performed under spinal anesthesia using a 15 Fr bipolar mini-resectoscope. Dilation and evacuation were performed by vacuum aspiration with a Karman cannula, followed by sharp curettage, if necessary, under ultrasound guidance. The primary outcome was the success rate of the treatment protocol, defined as no further treatment required until the complete resolution of the cesarean scar ectopic pregnancy. Resolution of the cesarean scar ectopic pregnancy was evaluated based on decline of beta-hCG and the absence of residual gestational material in the endometrial cavity. Treatment failure was defined as the necessity for further treatment required until the complete resolution of the cesarean scar ectopic pregnancy. A sample size calculation indicated that 54 participants were required to test the hypothesis RESULTS: A total of 54 women were enrolled and randomized. Number of previous cesarean deliveries ranged from 1 to 3. Overall, 10 women received a third dose of methotrexate with 7 of 27 (25.9%) participants in the hysteroscopic resection group and 3 of 27 (11.1%) in the dilation and evacuation group. The success rate was 100% (27/27) in the hysteroscopic resection group and 81.5% (22/27) in the dilation and evacuation group (relative risk, 1.22; 95% confidence interval, 1.01-1.48). Additional procedures were required in 5 cases of the control group, namely 3 hysterectomies, 1 laparotomic uterine segmental resection, and 1 hysteroscopic resection. The length of stay in the hospital was 9.0±2.9 days in the intervention group and 10.0±3.5 days in the control group (mean difference, -1.00 days; 95% confidence interval, -2.71 to 0.71). No cases of admission to intensive care unit or maternal death were reported.

CONCLUSION

Hysteroscopic resection was associated with an increased success rate in the treatment of cesarean scar ectopic pregnancy when compared with ultrasound-guided dilation and evacuation.

A fragility index analysis of clinical trials evaluating low-intensity extracorporeal shockwave therapy for erectile dysfunction

Erectile dysfunction is a common sexual dysfunction that affects a significant proportion of men. Low-intensity extracorporeal shockwave therapy has been evaluated in multiple clinical trials as a therapeutic option for men with erectile dysfunction. The robustness of these clinical trials is not well defined, as the trials are hindered by inconsistent treatment protocols, small study arm size and short follow-up intervals. The fragility index is a statistical analysis which is used to evaluate the robustness of clinical trials. It is calculated by evaluating the minimum number of patients in a given trial arm that would be required to have an alternative outcome to alter the statistical significance of the results. The lowest fragility index in statistically significant trials is 1, meaning that if just one participant experienced an alternate outcome, the results would no longer achieve statistical significance. The upper limit is determined by the number of participants in a given arm of the trial. Herein, a scoping review of clinical trials evaluating the efficacy of low-intensity extracorporeal shockwave therapy in erectile dysfunction to determine the fragility index of trials with clinically significant results. We hypothesized that the fragility index would be low, indicating the results are less robust and generalizable.

Empirical assessment of fragility index based on a large database of clinical studies in the Cochrane Library

RATIONALE AIMS AND OBJECTIVES

The fragility index (FI) and fragility quotient (FQ) are increasingly used measures for assessing the robustness of clinical studies with binary outcomes in terms of statistical significance. The FI is the minimum number of event status modifications that can alter a study result's statistical significance (or nonsignificance), and the FQ is calculated as the FI divided by the study's total sample size. The literature has no widely recognized criteria for interpreting the fragility measures' magnitudes. This article aims to provide an empirical assessment for the FI and FQ based on a large database of clinical studies in the Cochrane Library.

METHODS

We explored the overall empirical distributions of the FI and FQ based on five common methods (Fisher's exact test, χ2 test, risk difference, odds ratio, and relative risk) for determining statistical significance of binary outcomes in clinical research. We also considered three different scenarios for the FI calculation and evaluated the relationship between p values and FIs or FQs using Spearman'sρ $\rho $ . Finally, we summarized empirical thresholds based on the overall distributions of the FI and FQ to facilitate their interpretations in future research.

RESULTS

For about 20% of studies with significant results, the statistical significance was changed after modifying the event status of only one participant. Studies with significant results were considered slightly fragile if the significance hinged on the statuses of about five events. Studies were extremely fragile if FI≤ $\le $ 1 or FQ≤ $\le $ 0.01. The FIs were strongly correlated with p values for significant studies, while Spearman'sρ $\rho $ varied according to the total sample sizes of studies.

CONCLUSIONS

The statistical significance of clinical studies could be changed after modifying a few events' statuses. Many studies' findings are fairly fragile. The distributions of the FI and FQ provide insights for appraising the robustness of evidence in clinical decision-making.