1
|
Placer-Galán C, Enriquez-Navascués JM, Lopetegui AE, Ansorena YS. An analysis of randomized controlled trials on anal fistula conducted between 2000 and 2020 based on the Fragility Index and Reverse Fragility Index. Colorectal Dis 2023; 25:1572-1577. [PMID: 37400967 DOI: 10.1111/codi.16645] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Revised: 04/29/2023] [Accepted: 05/30/2023] [Indexed: 07/05/2023]
Abstract
AIM The aim of this work was to evaluate the robustness of randomized controlled trials (RCTs) on anal fistula management using the news tools of Fragility Index (FI), Reverse Fragility Index (RFI) and their corresponding fragility quotients. METHOD A systematic search was conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines which utilized MEDLINE, EMBASE, Cochrane Library and Web of Science databases. Inclusion criteria included RCTs related to the management of anal fistula published from 2000 to 2022 with dichotomous outcomes measures and 1:1 allocation. Calculation of FI and RFI was performed by creating 2 × 2 contingency tables by successively changing one nonevent to an event for each outcome measure until the result was made nonsignificant or significant, respectively. The Fragility Quotients were calculated by dividing the FI or RFI by the total sample size. Fragile results were defined as those with a FI or RFI equal to or less than the number of patients lost to follow-up. Additionally, those with a FI or RFI less than 3 were also considered fragile. Studies were considered extremely fragile if FI was ≤1 or FQ was ≤0.01. RESULTS There were 36 RCTs that met our criteria, with 3223 patients. Among these, 19 (53%) were positive RCTs (p < 0.005) and 17 (47%) were negative RCTs (p > 0.05). The median FI was 2 (0-5). The analysis by categorical subgroup showed a strong correlation between FI and the p-value (p = 0.000) and the number of events (p = 0.011). The median RFI was 5 (3.5-9.5) and the subgroup analysis showed a strong correlation between RFI and the p-value (p = 0.000), sample size (0.021) and number needed to treat/number needed to harm (0.000). We considered 63.2% of positive RCTs to be fragile and 35.3% of negative RCTs. CONCLUSIONS In the present study we demonstrated the lack of robustness of study findings in published RCTs in the field of anal fistula.
Collapse
Affiliation(s)
- Carlos Placer-Galán
- Colorectal Unit, Department of Surgery, Hospital Universitario Donostia, San Sebastian, Spain
| | | | - Ane Etxart Lopetegui
- Colorectal Unit, Department of Surgery, Hospital Universitario Donostia, San Sebastian, Spain
| | | |
Collapse
|
2
|
Usman MS, Khan MS, Fonarow GC, Greene SJ, Friede T, Vaduganathan M, Filippatos G, Coats AJS, Anker SD, Butler J. Robustness of outcomes in trials evaluating sodium-glucose co-transporter 2 inhibitors for heart failure. ESC Heart Fail 2022; 9:885-893. [PMID: 35029056 PMCID: PMC8934993 DOI: 10.1002/ehf2.13785] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2021] [Revised: 10/25/2021] [Accepted: 12/13/2021] [Indexed: 01/10/2023] Open
Abstract
AIMS Recent trials have evaluated sodium-glucose co-transporter 2 inhibitors in patients with heart failure (HF). We sought to assess the robustness of findings from these trials using the fragility index (FI). METHODS AND RESULTS Fragility index is defined as the minimum number of patients that must be moved from the 'non-event' to the 'event' group to turn a statistically significant result to non-significant. In addition to FI, fragility quotient [(FQ); FI divided by the sample size] was calculated to assess the proportion of events that must be moved to change the significance. For statistically non-significant outcomes, reverse fragility index (RFI) and reverse fragility quotient (RFQ) were calculated. Robustness of findings after pooling data from all three trials was also assessed. A robust reduction in first HF hospitalization or cardiovascular mortality was seen with dapagliflozin (FI = 62 and FQ = 0.013), empagliflozin (FI = 50 and FQ = 0.013), and sotagliflozin (FI = 60 and FQ = 0.049). Dapagliflozin nominally improved all-cause and cardiovascular mortality, with modest FI (n = 8 and 5) and FQ (0.002 and 0.001). Empagliflozin and sotagliflozin did not demonstrate statistically significant reductions in all-cause mortality, with modest RFI (empagliflozin: RFI = 26 and RFQ = 0.007; sotagliflozin: RFI = 6 and RFQ = 0.005). A similar trend was seen with cardiovascular mortality (empagliflozin: RFI = 24 and RFQ = 0.006; sotagliflozin: RFI = 7 and RFQ = 0.006). Upon meta-analysis, the result for first HF hospitalization or cardiovascular mortality was robust (FI = 95 and FQ = 0.010). The reductions in all-cause (FI = 12 and FQ = 0.001) and cardiovascular mortality (FI = 9 and FQ = 0.001), while statistically significant, were fragile. CONCLUSION Improvement in the composite outcome of first HF hospitalization or cardiovascular death was highly concordant and robust across sodium-glucose co-transporter 2 inhibitor trials. In contrast, secondary endpoints of all-cause and cardiovascular mortality were statistically fragile, underscoring the need to power trials for mortality to fully understand the benefit of therapies on fatal events.
Collapse
Affiliation(s)
| | | | - Gregg C Fonarow
- Division of Cardiology, Ronald Reagan UCLA Medical Center, Los Angeles, CA, USA
| | - Stephen J Greene
- Division of Cardiology, Duke University Medical Center, Durham, NC, USA
| | - Tim Friede
- Department of Medical Statistics, University Medical Center Göttingen, Göttingen, Germany.,DZHK (German Centre for Cardiovascular Research), partner site Göttingen, Göttingen, Germany
| | | | - Gerasimos Filippatos
- National and Kapodistrian University of Athens School of Medicine, Athens University Hospital Attikon, Athens, Greece
| | - Andrew J Stewart Coats
- Department of Cardiology, IRCCS San Raffaele Pisana, Rome, Italy.,University of Warwick, Coventry, UK
| | - Stefan D Anker
- Department of Cardiology (CVK); and Berlin Institute of Health Center for Regenerative Therapies (BCRT); DZHK (German Centre for Cardiovascular Research), partner site Berlin, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Javed Butler
- Department of Medicine, University of Mississippi, Jackson, MS, USA
| |
Collapse
|
3
|
Pascoal E, Liu M, Lin L, Luketic L. The fragility of statistically significant results in gynecologic surgery: A systematic review. J Obstet Gynaecol Can 2021; 44:508-514. [PMID: 34954411 DOI: 10.1016/j.jogc.2021.11.016] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2021] [Revised: 11/20/2021] [Accepted: 11/22/2021] [Indexed: 11/17/2022]
Abstract
OBJECTIVE To use the fragility index (FI) to evaluate the robustness of gynaecologic surgery trials that report statistically significant results. The FI defines the minimum number of patients who must have an alternative outcome to alter statistical significance. DATA SOURCES We searched MEDLINE, Web of Science, Embase, and ClinicalTrials.gov from 2011 to 2021 to identify gynaecologic surgery randomized controlled trials (RCTs). STUDY SELECTION A total of 4775 trials were screened for eligibility. All included studies evaluated benign gynaecologic surgery interventions or peri-operative medical interventions. Only two-arm RCTs with statistically significant dichotomous primary outcomes were included. Ninety-three trials were ultimately included for analysis. DATA EXTRACTION AND SYNTHESIS Data from the included studies, including sample size, loss to follow-up, and number of events, were recorded. The FI of each study was calculated using a predefined technique. The overall FI and FIs by subgroup (clinical subspecialty, country of origin, and statistical test used) are reported as medians alongside their interquartile ranges (IQRs). The Kruskal-Wallis test was applied to find possible statistically significant relationships between FI and the nominal subgroups. Among this cohort, the median FI was 3 (IQR 1-7). The FI was 0 in 13 trials (14%), and in 39 trials (42%), the number of patients lost to follow-up was greater than the FI. The median FI within clinical subspecialty groups (general gynaecology, anaesthesia, urogynaecology, and fertility) did not differ (P = 0.122). CONCLUSION Statistically significant results of RCTs in gynaecologic surgery are fragile, suggesting that clinicians should interpret results with caution. This is particularly true when the number of patients lost to follow-up is greater than the FI. The FI serves as a quality metric that can be used to evaluate robustness of results when applying the outcomes of RCTs to clinical practice or guideline development.
Collapse
Affiliation(s)
- Erica Pascoal
- Department of Obstetrics and Gynecology, McMaster University, Hamilton, ON.
| | - Marina Liu
- Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON
| | - Lauren Lin
- Michael G. DeGroote School of Medicine, McMaster University, Hamilton, ON
| | - Lea Luketic
- Department of Obstetrics and Gynecology, McMaster University, Hamilton, ON
| |
Collapse
|
4
|
Wayant C, Tritz D, Horn J, Crow M, Vassar M. Evaluation of Risks of Bias in Addiction Medicine Randomized Controlled Trials. Alcohol Alcohol 2021; 56:284-290. [PMID: 32808009 DOI: 10.1093/alcalc/agaa074] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2019] [Revised: 07/06/2020] [Accepted: 07/07/2020] [Indexed: 01/11/2023] Open
Abstract
AIMS Perhaps the most important step when designing and conducting randomized controlled trials (RCTs) in addiction is to put methodological safeguards in place to minimize the likelihood for bias to affect trial outcomes. In this study, we applied the revised Cochrane risk of bias tool (ROB 2) to RCTs of drug, alcohol or tobacco interventions. METHODS We searched for trials published in 15 addiction medicine journals over a 7-year period. Our primary endpoint is the risk of bias of included studies. We conducted a sensitivity analysis of publicly funded trials. RESULTS Overall, included RCTs were most often at high risk of bias per our judgments (244/487, 50.1%). However, significant proportions of included RCTs were at low risk of bias (123/487, 25.3%) or some concerns for bias (120/497, 24.6%). RCTs with behavioral modification interventions (19/44, 43.2%) and alcohol interventions (80/150, 53.3%) had the highest proportion of high-risk judgments. In a sensitivity analysis of publicly funded RCTs), 195/386 (50.5%) were at high risk of bias. CONCLUSIONS Approximately half of included drug, alcohol or tobacco RCTs in our sample were judged to be at high risk of bias with the most common reason being a lack of proper blinding or proper description of blinding. Key action items to reduce bias in future addiction RCTs include adequate randomization, blinding and inclusion of a trial registry number and protocol.
Collapse
Affiliation(s)
- Cole Wayant
- Oklahoma State University Center for Health Sciences, Tulsa, OK 74104, USA
| | - Daniel Tritz
- Oklahoma State University Center for Health Sciences, Tulsa, OK 74104, USA
| | - Jarryd Horn
- Oklahoma State University Center for Health Sciences, Tulsa, OK 74104, USA
| | - Matt Crow
- Oklahoma State University Center for Health Sciences, Tulsa, OK 74104, USA
| | - Matt Vassar
- Oklahoma State University Center for Health Sciences, Tulsa, OK 74104, USA
| |
Collapse
|
5
|
Derici Yıldırım D, Taşdelen B. Tanısal görüntüleme teknikleri kullanılan randomize kontrollü çalışmaların istatistiksel anlamlılığının kırılganlık indeksi ile değerlendirilmesi. Cukurova Medical Journal 2019. [DOI: 10.17826/cumj.512585] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022] Open
|
6
|
Khan MS, Ochani RK, Shaikh A, Usman MS, Yamani N, Khan SU, Murad MH, Mandrola J, Doukky R, Krasuski RA. Fragility Index in Cardiovascular Randomized Controlled Trials. Circ Cardiovasc Qual Outcomes 2019; 12:e005755. [PMID: 31822121 DOI: 10.1161/circoutcomes.119.005755] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
Abstract
BACKGROUND Efficacy of an intervention is commonly evaluated using P values, in addition to effect size measures such as absolute risk reduction, relative risk reduction, and numbers needed to treat. However, these measures are not always intuitive to clinicians. The fragility index (FI) is a more intuitive number that can facilitate interpretation but can only be used with binary outcomes. FI is the minimum number of patients who must be moved from the nonevent group to the event group to turn a significant result nonsignificant. In this retrospective analysis, we assessed the robustness of cardiovascular randomized controlled trials (RCTs), which report a positive (statistically significant) primary outcome by using the FI. METHODS AND RESULTS We searched Medline from 2007 to 2017 to identify cardiovascular RCTs published in 6 high impact journals (The Lancet, New England Journal of Medicine, Journal of the American Medical Association, Circulation, Journal of the American College of Cardiology and European Heart Journal). Only RCTs with sample sizes >500 and a 2-by-2 factorial design or dichotomous primary outcomes were selected. FI was calculated using a defined approach. Among the cohort of 123 RCTs that met inclusion criteria, median FI was 13 (interquartile range, 5-26). In 28 trials (22.8%), FI ranged between 1 and 4. In 37 trials (30.1%), number of patients lost to follow-up was higher than the FI. Pharmaceutical interventions had higher FI compared with other interventions, FI=19 (7-52; P=0.002). Median FI varied according to subspecialty (electrophysiology=2; heart failure=11; interventional cardiology=8; P=0.020) and multiregional RCTs had higher FI=22 (12-53.25; P=0.023). FI did not differ based on risk of bias indicators, funding, or publication year. CONCLUSIONS Considerable variations in FI were observed among cardiovascular trials, suggesting the need for careful interpretation of results, particularly when number of patients lost to follow-up exceeds FI.
Collapse
Affiliation(s)
- Muhammad Shahzeb Khan
- Department of Internal Medicine, John H. Stroger Jr. Hospital of Cook County, Chicago, IL (M.S.K., N.Y.)
| | - Rohan Kumar Ochani
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Asim Shaikh
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Muhammad Shariq Usman
- Department of Internal Medicine, Dow University of Health Sciences, Karachi, Pakistan (R.K.O., A.S., M.S.U.)
| | - Naser Yamani
- Department of Internal Medicine, John H. Stroger Jr. Hospital of Cook County, Chicago, IL (M.S.K., N.Y.)
| | - Safi U Khan
- Department of Internal Medicine, West Virginia University (S.U.K.)
| | - M Hassan Murad
- Evidence-based Practice Center, Mayo Clinic, Rochester, MN (M.H.M.)
| | | | - Rami Doukky
- Division of Cardiology, Rush University Medical Center, Chicago, IL (R.D.)
| | - Richard A Krasuski
- Department of Cardiovascular Medicine, Duke University Health System, Durham, NC (R.A.K.)
| |
Collapse
|
7
|
Rickard M, Lorenzo AJ, Hannick JH, Blais AS, Koyle MA, Bägli DJ. Over-reliance on P Values in Urology: Fragility of Findings in the Hydronephrosis Literature Calls for Systematic Reporting of Robustness Indicators. Urology 2019; 133:204-210. [DOI: 10.1016/j.urology.2019.03.045] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2019] [Revised: 02/28/2019] [Accepted: 03/04/2019] [Indexed: 10/26/2022]
|
8
|
Vassar M, Jellison S, Wendelbo H, Wayant C, Gray H, Bibens M. Using the CONSORT statement to evaluate the completeness of reporting of addiction randomised trials: a cross-sectional review. BMJ Open 2019; 9:e032024. [PMID: 31494625 PMCID: PMC6731848 DOI: 10.1136/bmjopen-2019-032024] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/31/2019] [Revised: 08/07/2019] [Accepted: 08/20/2019] [Indexed: 12/20/2022] Open
Abstract
OBJECTIVES Evaluate the completeness of reporting of addiction randomised controlled trials (RCTs) using the Consolidated Standards of Reporting Trials (CONSORT) statement. SETTING Not applicable. PARTICIPANTS RCTs identified using a PubMed search of 15 addiction journals and a 5-year cross-section. OUTCOME MEASURES Completeness of reporting. RESULTS Our analysis of 394 addiction RCTs found that the mean number of CONSORT items reported was 19.2 (SD 5.2), out of a possible 31. Twelve items were reported in <50% of RCTs; similarly, 12 items were reported in >75% of RCTs. Journal endorsement of CONSORT was found to improve the number of CONSORT items reported. CONCLUSIONS Poor reporting quality may prohibit readers from critically appraising the methodological quality of addiction trials. We recommend journal endorsement of CONSORT since our study and those previous have shown that CONSORT endorsement improves the quality of reporting.
Collapse
Affiliation(s)
- Matthew Vassar
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| | - Sam Jellison
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| | - Hannah Wendelbo
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| | - Cole Wayant
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| | - Harrison Gray
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| | - Michael Bibens
- Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
| |
Collapse
|
9
|
Wayant C, Meyer C, Gupton R, Som M, Baker D, Vassar M. The Fragility Index in a Cohort of HIV/AIDS Randomized Controlled Trials. J Gen Intern Med 2019; 34:1236-1243. [PMID: 31037544 PMCID: PMC6614212 DOI: 10.1007/s11606-019-04928-5] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/15/2018] [Revised: 02/19/2019] [Accepted: 02/20/2019] [Indexed: 12/31/2022]
Abstract
HIV/AIDS is associated with significant morbidity, mortality, and financial burden. For these reasons, robust clinical evidence is critical. We aim to investigate the fragility index, fragility quotient, and risk of bias of clinical trial endpoints in HIV medicine. The fragility index represents the minimum amount of trial endpoint "nonevents" changed to "events" in one trial arm required to nullify statistical significance. The fragility quotient contextualized the fragility index by dividing the index by the total trial sample size. We selected eligible trials from the Department of Health and Human Services guideline for the use of antiretroviral agents in HIV-1-infected adults and adolescents. We calculated the fragility index and fragility quotient for all included trials. The Cochrane "risk of bias" Tool 2.0 was used to evaluate the likelihood and sources of bias in the included trials. Thirty-nine RCTs were included for our analysis of fragility. Thirty-six were included for our analysis of the risk of bias. The median fragility index was 5. Three RCTs were at high risk of bias, all due to the selection of the endpoint or statistical test. Twenty had some concerns for risk of bias. The analyzed HIV medicine RCT endpoints were fragile, overall. This indicates that a median of 5 patients across all included studies would nullify the statistical significance of the endpoints. Furthermore, we found evidence that concerns for bias are present at a high rate.
Collapse
Affiliation(s)
- Cole Wayant
- Department of Biomedical Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA.
| | - Chase Meyer
- Department of Biomedical Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA.,Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
| | - Rebecca Gupton
- Internal Medicine, Oklahoma State University Medical Center, Tulsa, OK, USA
| | - Mousumi Som
- Internal Medicine, Oklahoma State University Medical Center, Tulsa, OK, USA
| | - Damon Baker
- Internal Medicine, Oklahoma State University Medical Center, Tulsa, OK, USA
| | - Matt Vassar
- Department of Biomedical Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA.,Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
| |
Collapse
|