Over-reliance on P Values in Urology: Fragility of Findings in the Hydronephrosis Literature Calls for Systematic Reporting of Robustness Indicators

Fragility of overactive bladder medication clinical trials: A systematic review

PURPOSE

Overactive bladder (OAB) syndrome significantly impairs quality of life, often necessitating pharmacological interventions with associated risks. The fragility of OAB trial outcomes, as measured by the fragility index (FI: smallest number of event changes to reverse statistical significance) and quotient (FQ: FI divided by total sample size expressed as a percentage), is critical yet unstudied.

MATERIALS AND METHODS

We conducted a systematic search for randomized controlled trials on OAB medications published between January 2000 and August 2023. Inclusion criteria were trials with two parallel arms reporting binary outcomes related to OAB medications. We extracted trial details, outcomes, and statistical tests employed. We calculated FI and FQ, analyzing associations with trial characteristics through linear regression.

RESULTS

We included 57 trials with a median sample size of 211 participants and a 12% median lost to follow-up. Most studies investigated anticholinergics (37/57, 65%). The median FI/FQ was 5/3.5%. Larger trials were less fragile (median FI 8; FQ 1.0%) compared to medium (FI: 4; FQ 2.5%) and small trials (FI: 4; FQ 8.3%). Double-blinded studies exhibited higher FQs (median 2.9%) than unblinded trials (6.7%). Primary and secondary outcomes had higher FIs (median 5 and 6, respectively) than adverse events (FI: 4). Each increase in 10 participants was associated with a +0.19 increase in FI (p < 0.001).

CONCLUSIONS

A change in outcome for a median of five participants, or 3.5% of the total sample size, could reverse the direction of statistical significance in OAB trials. Studies with larger sample sizes and efficacy outcomes from blinded trials were less fragile.

The Fragility Index of randomized controlled trials in pediatric anesthesiology

PURPOSE

The P value is a widely used measure of statistical importance but has many drawbacks and limitations, one being that it does not reflect the robustness of the results of a clinical trial. The Fragility Index (FI) was developed as a measure of how many outcome events would need to change to nonevents to render a significant P value nonsignificant (P ≥ 0.05). The FI of trials from other medical specialties is typically < 5. We aimed to determine the FI of pediatric anesthesiology randomized controlled trials (RCT) and to test for association with various characteristics of the included trials.

METHODS

We conducted a comprehensive systematic search of high-impact anesthesia, surgical, and medical journals from the last 25 years for trials comparing an intervention between two groups with a statistically significant P value (< 0.05) for a dichotomous outcome. We also compared FI values for variables that reflect the quality and importance of a trial.

RESULTS

The median [interquartile range] FI was 3 [1-7] and correlated positively with the number of participants (rS = 0.41; P < 0.001) and events (rS = 0.42; P < 0.001), and negatively with the P value (rPB = -0.36; P < 0.001). Other measures of trial quality and impact or importance were not strongly associated with the FI.

CONCLUSIONS

The FI of published trials in pediatric anesthesiology is similarly low as in other medical specialties. Larger trials with more events and P values ≤ 0.01 were associated with a higher FI.

Assessing the robustness of COVID-19 vaccine efficacy trials: systematic review and meta-analysis, January 2023

BackgroundVaccines play a crucial role in the response to COVID-19 and their efficacy is thus of great importance.AimTo assess the robustness of COVID-19 vaccine efficacy (VE) trial results using the fragility index (FI) and fragility quotient (FQ) methodology.MethodsWe conducted a Cochrane and PRISMA-compliant systematic review and meta-analysis of COVID-19 VE trials published worldwide until 22 January 2023. We calculated the FI and FQ for all included studies and assessed their associations with selected trial characteristics using Wilcoxon rank sum tests and Kruskal-Wallis H tests. Spearman correlation coefficients and scatter plots were used to quantify the strength of correlation of FIs and FQs with trial characteristics.ResultsOf 6,032 screened records, we included 40 trials with 54 primary outcomes, comprising 909,404 participants with a median sample size per outcome of 13,993 (interquartile range (IQR): 8,534-25,519). The median FI and FQ was 62 (IQR: 22-123) and 0.50% (IQR: 0.24-0.92), respectively. FIs were positively associated with sample size (p < 0.001), and FQs were positively associated with type of blinding (p = 0.023). The Spearman correlation coefficient for FI with sample size was moderately strong (0.607), and weakly positive for FI and FQ with VE (0.138 and 0.161, respectively).ConclusionsThis was the largest study on trial robustness to date. Robustness of COVID-19 VE trials increased with sample size and varied considerably across several other important trial characteristics. The FI and FQ are valuable complementary parameters for the interpretation of trial results and should be reported alongside established trial outcome measures.

What's in a P-Value? A Fragility Analysis of RCTs in the AUA Guidelines for Benign Prostatic Hyperplasia

OBJECTIVE

To use the fragility index (FI) and fragility quotient (FQ) to assess the strength of statistically significant findings for randomized controlled trials (RCTs) cited in the American Urological Association (AUA) guidelines for benign prostatic hyperplasia.

METHODS

Two investigators independently screened the AUA guidelines for management of benign prostatic hyperplasia for RCTs cited as evidence for recommendations. Investigators extracted data related to event rate per group and loss to follow-up which was compared against the FI. Stata 17.0 was used to calculate the FI and FQ which was then summarized and reported according to primary or secondary endpoints.

RESULTS

Among the 373 citations in the AUA guidelines, 24 RCTs met inclusion criteria with 29 distinct outcomes analyzed. The median fragility index was 12 (IQR = 4-38), indicating that twelve alternative events to either study arm would nullify statistical significance. Six studies had a FI of ≤2, indicating that only 1-2 outcomes would need to be changed in order to render nonsignificance of results. In 10/24 RCTs, the number of patients lost to follow-up was greater than the FI.

CONCLUSION

The AUA Clinical Practice Guidelines for management of benign prostatic hyperplasia cite RCTs with more robust findings when compared to previous studies assessing fragility in the field of Urology. While several included studies had high fragility, the median FI in our analysis was approximately 4-5 times higher than comparable studies of urologic RCTs. However, there are areas where improvement is necessary to support the highest quality of evidence-based medicine.

Prenatal Hydronephrosis: Bridging Pre- and Postnatal Management

Urinary tract dilation (UTD), including hydronephrosis (HN) is the most common prenatally detected anomaly and affects up to 1-2% of pregnancies. Postnatally, it resolves without surgical intervention in approximately 75-80% of children, however this may take several years to occur, necessitating repeated clinic visits and additional invasive testing for many. For the remainder, a surgical intervention will be offered to relieve obstruction or to correct vesicoureteral reflux. During the monitoring period, many of these infants will be offered continuous antibiotic prophylaxis for the prevention of urinary tract infections, however this remains a controversial topic among pediatric urologists and nephrologists. Herein, we present an up-to-date review of the early management of prenatally detected UTD including timing of postnatal imaging, the use of antibiotics, when circumcision may be beneficial and long term outcomes of the most common HN etiologies. We also propose a decision making tool to help guide the care of infants with UTD. This article is protected by copyright. All rights reserved.

Evaluating the literature on preoperative androgen stimulation for hypospadias repair using the fragility index - can we trust observational studies?

BACKGROUND

Preoperative androgen stimulation (PAS) is typically used in hypospadias repair for patients with a proximal meatus or small glans size. Hypospadias PAS literature suffer from small sample sizes and lack of power to claim robust conclusions. Small changes in the number of events may completely change the statistical significance, making the conclusions drawn unreliable. Fragility index (FI) is the number of additional events needed to occur in either the control or experimental group to turn a statistically significant result to a non-significant result. The objective of the report was to assess the quality of available literature revolving around PAS use in hypospadias repair and its effects on post-operative complication rates using FI.

METHODS

A comprehensive search of MEDLINE, EMBASE, and grey literature (ESPU and SPU abstracts) was conducted to identify RCTs and observational studies investigating the effect of PAS on complications post-hypospadias repair between 1990 and 2020. The FI was calculated for each study. Postoperative complications were defined as: fistula, stricture/stenosis, diverticula, and dehiscence. The odds ratio (OR), 95% confidence intervals (CI), corresponding p-values was calculated for each study. A random effects mixed model was implemented to combine the ORs for each study design.

RESULTS

Fourteen studies qualified for inclusion, of which nine were observational studies and five were RCTs (Figure 1). The median sample size was 110 patients (IQR 69-171). The summary ORs for observational studies was 1.74 (95% CI: 1.10 to 2.74; p = 0.020) and for RCTs was 0.71 (9% CI: 0.34 to 1.47; p = 0.350). The median FI was 0 (IQR 0-2) of the included studies.

DISCUSSION

PAS use does not appear to significantly affect complication rates shown in RCTs, however, observational studies cumulatively suggested significantly greater odds of complications after PAS. The FI is best used for RCTs with 1-to-1 randomization and binary data. Observational studies are rarely balanced for demographics and comorbidities with unequal sample size between comparable groups. The study was limited by substantial variability in how PAS was delivered to patients, leading to restricted comparability.

CONCLUSION

Strong conclusions regarding the influence of PAS on hypospadias repair outcomes cannot be properly drawn based on the current literature due to deficits from either a statistical or methodological standpoint. The current PAS literature has shown inconclusive results, calling for well-designed RCTs, involving standardized surgical techniques and PAS protocols, to evaluate the true effect of PAS on complications post-hypospadias repair.

Urologic oncology randomized controlled trials are frequently fragile - A review of the urology literature

BACKGROUND

Randomized controlled trials (RCT) in urologic oncology are the basis of patient management. Considerable debate exists on the limitation of statistical reporting of randomized controlled trials. Fragility index (FI) is a measure of the number of events upon which the trials statistical results depend on. The FI is defined as the minimum number of patients whose status would have to change from a 'non-event' to 'event', in order to turn a statistically significant result to a non-significant result and vice versa. Our aim is to examine the FI of RCT's in urologic oncology published in the urology literature.

MATERIAL AND METHODS

We exhaustively searched MEDLINE and EMBASE from January 1, 2016 to December 31 2019 for RCT's in urology journals. Only studies reporting dichotomous outcomes were included and FI was calculated for each outcome. The distributions of FI across different journals and types of outcome (primary/ secondary, significant/ non-significant) were assessed. We examined the correlation of FI with sample size and P-value.

RESULTS

We identified 216 RCT's, 79 were eligible for analysis. Median FI was 3.0 (2.5, 6.0). One hundred and forty-six (89.6%) outcomes from 72 (94.7%) RCT's had a FI lower than 10 and 6 (3.7%) outcomes from 6 (7.9%) RCT's had a FI = 0. There was no statistically significant difference in FI between different types of outcomes. There was a modest correlation between the FI and the sample size (r_s 0.50, P < 0.01) and a weak correlation between the FI the P value (r_s 0.15 (P < 0.04)).

CONCLUSIONS

Randomized controlled trials in urology journals that study dichotomous outcomes often report fragile results that should be regarded with caution. Reporting the FI alongside P values may enhance the interpretation and implementation of urologic oncology RCT's.

Fragility index of urological literature regarding medical expulsive treatment

INTRODUCTION

The role of medical expulsive treatment (MET) is controversial. Fragility index is an additional metric to assess randomized controlled trials (RCTs) outcome validity and indicates how many patients would be required to convert a trial from being statistically significant, to not significant. The larger is the FI, the better the trial's data. The aim of this study is to assess FI of RCTs regarding MET for ureteral stones.

MATERIALS AND METHODS

A systematic literature search was performed. RCTs, reporting stone expulsion as a dichotomous outcome, showing statistical significance were eligible. FI (the number of patients needed to change from a non-event to event group, to lose statistical significance) and Fragility quotient (FI divided by total sample size), were calculated while Pearson's correlation and Mann-Whitney U test were used as appropriate.

RESULTS

Thirty-six RCTs were eligible, with median FI = 3.5 and fragility quotient = 0.042, median sample size = 81, median journal impact factor = 1.73 and median reported p value = 0.008. In 33.3% of the studies, number of patients lost during follow-up was larger than FI, while in 13.89% of the studies, FI was 0, indicating use of inappropriate statistical method. Pearson's correlation showed significant positive association between FI and sample size (r = 0.981), number of events (r = 0.982) and impact factor (r = 0.731), while no association was found with p value or publication year.

CONCLUSIONS

In this analysis, a calculated FI of 3.5 indicates that findings from RCTs on MET for ureteral stones are fragile and should be interpreted in combination with clinical thinking and expertise.

Trends and relevance in the bladder and bowel dysfunction literature: PlumX metrics contrasted with fragility indicators

INTRODUCTION

The concepts of fragility index (FI) and fragility quotient (FQ) have been previously described. PlumX metrics encompass online "footprints" of research in addition to traditional citations. Herein we explore PlumX metrics against the quality of BBD literature.

OBJECTIVE

To explore altmetrics against the quality of bladder and bowel dysfunction (BBD) literature.

STUDY DESIGN

A literature search was conducted using Pubmed, Medline, Embase for BBD and related terms. A total of 54,045 abstracts were screened, followed by 693 full text reviews and data extraction from 126. Studies were included if they reported on 2 groups being compared, had dichotomous outcomes, and had significant results.

RESULTS

The median FI score was 4 (0-500) and there were 20 studies which had a FI of 0. The FQ had a median of 0.04 (0-0.32). PlumX usage was 263 ± 540, captures were 45 ± 60 and social media attention was 2 ± 2. Overall, 42% of papers were clinical trials (RCTs). When compared to other study designs, we noted a significant difference in PlumX captures (57 ± 72 RCT vs. 35 ± 47 other; p = 0.03). RCTs had higher usage, social media engagement and citations however, the differences were not significant. H-Index had a significant correlation with FI (p = 0.036), however correlations for PlumX usage and captures, while modestly positive (0.04-0.10) for the FI and FQ, were not significant. A comparison of FI and FQ by topic can be reviewed in the Summary Table.

DISCUSSION

When considering the FI and FQ robustness indicators of the BBD literature, we found similarities when compared to other studies. It was reported that overall, the hydronephrosis literature was fragile with many studies requiring only a few events to nullify significance, regardless of the study design. Similarly, in a review of pediatric vesicoureteral reflux (VUR) clinical trials, results were also fragile. When comparing fragility measures to altmetric variables we noted that despite the growing popularity of altmetrics, citation counts, and h-indices remain the traditional measures to monitor research consumption. There has been a reported correlation between manuscript citation counts, author h-index, altmetrics measures in several specialties and across many domains of research including medical sciences, arts, and the humanities, however in the present study only weak correlations were noted.

CONCLUSION

The body of BBD comparative studies is fragile in keeping with other pediatric urology literature populations. Despite fragile results, RCTs generate slightly moreattention as measured by select PlumX metrics. These results suggest the need for including fragility measures in our literature, aiming to focus attention towards more robust articles.