Fragility index of trials supporting approval of anti-cancer drugs in common solid tumours

The fragility index of randomized controlled trials in advanced/metastatic renal cell cancer

PURPOSE

The fragility index (FI) has been applied as a supplement to the noncomprehensive P-values to assess the robustness of randomized controlled trials (RCTs). The objective of this study is to evaluate the statistical robustness of RCTs of advanced/metastatic renal cell cancer (a/mRCC) using the FI.

MATERIALS AND METHODS

RCTs related to a/mRCC published in the 4 highest-impact general medical journals and the 25 highest-impact urological journals between January 1, 2000, and December 31, 2023, were identified from PubMed database. The FI was calculated by using Fisher's exact test. Spearman's correlation analysis was conducted to assess potential correlates regarding FI.

RESULTS

16 eligible RCTs were screened with a median total sample size of 654.5 (IQR, 461-847) and a median patients lost to follow-up of 14 (IQR, 3-23). The median FI was 12.5 (IQR, 8.5-27), suggesting that a switch in outcomes in only 13 patients would have reversed the significance of the trials. The number of patients lost to follow-up exceeded or equaled to the FI in 7 (44%) RCTs. P-values were negatively associated with the FI, while the number of patients lost to follow-up and patients enrolled were not statistically significant.

CONCLUSION

Not all RCTs associated with a/mRCC are as statistically robust as previously considered and should therefore be construed carefully. We suggest that additional reporting of FI in urological RCTs as a supplement to the P-value to assist readers in concluding reliably by considering the fragility of the outcomes.

Clinical benefit and fragility evaluation of systemic therapy trials for advanced soft tissue sarcoma

BACKGROUND

The clinical benefit of systemic anticancer therapies can be unclear despite positive trials, and outcomes may not translate to real-world practice. This study evaluated the benefit of soft tissue sarcoma (STS) treatments using the European Society of Medical Oncology Magnitude of Clinical Benefit Scale (MCBS) v1.1 and measured the robustness of STS trial results using Fragility Index (FI).

METHODS

Database searches for adult phase II or III trials in advanced STS (January 1998-December 2023) were performed. Therapies with trial outcomes that met the criteria for MCBS were scored 1-5 (≥4 represents substantial clinical benefit). For randomized clinical trials with positive time-to-event endpoints, the number of additional events that would render results nonsignificant, FI, was calculated and expressed as a proportion of the experimental arm size (fragility quotient [FQ]). Higher FI/FQ implies more robust results.

RESULTS

Among 194 trials, 19 (9.8%) were phase III. Most phase II trials (146/175; 83.4%) had single-arm or non-comparative design. Trials that were eligible for MCBS scoring (n = 78; 40.2%) evaluated 56 different agents/regimens. Median MCBS score was 2. Only three agents/regimens (all cytotoxic therapies) had an MCBS score ≥4. Among 47 randomized clinical trials, 16 (8 phase II; 8 phase III) trials had positive outcomes. Median FI was 7 (range, 2-52) and 10 trials (62.5%) had an FQ < 10%, with median of 7% (range, 1%-59%).

CONCLUSIONS

Most systemic therapies in STS trials did not confer substantial clinical benefit per European Society of Medical Oncology-MCBS. Additionally, positive randomized trials were often fragile. Novel STS therapy trials should use clinically meaningful endpoints and real-world efficacy confirmation is essential, especially for less robust trials.

Statistical Fragility of Findings From Randomized Phase 3 Trials in Pediatric Oncology

PURPOSE

The fragility index (FI) is an adjunctive metric to facilitate the interpretation of p-values in clinical trials. The FI has not been studied in phase 3 trials in pediatric oncology.

METHODS

PubMed was used to identify phase 3 pediatric oncology trials published between 1980 and 2020. We report trial characteristics and calculate the FI for trials with a binary outcome and survival-inferred fragility index (SIFI) for trials with a time-to-event outcome. FI/SIFI is the number of patients from one arm of a trial who would need to change groups for the statistical conclusion to change. We also report fragility quotients (FQ and SFQ) to normalize FI and SIFI relative to trial size.

RESULTS

One hundred and thirteen trials included sufficient data for analysis. The median FI for trials with a binary outcome (n = 40) was 4.5 (range: 1-33). The median SIFI for trials with a time-to-event outcome (n = 73) was 13 (range: 0-61). The FI or SIFI was less than the number of patients lost to follow-up in 25% of 36 trials. Median FQ and SFQ were 0.026 and 0.03, respectively, and did not significantly vary according to trial characteristics. While sample sizes increased over time, the FQ and SFQ remained stable.

CONCLUSIONS

The statistical conclusions of pediatric oncology phase 3 trials hinge on a relatively small number and proportion of patients. Despite the sample size limitations of low prevalence diseases, pediatric cancer trials are similarly or less fragile than adult oncology trials. Smaller trials do not appear more statistically fragile than larger trials. Statistical fragility appears to have remained constant over the four decades evaluated. We recommend reporting FI or SIFI, in conjunction with p-values, for all phase 3 pediatric oncology trials.

Exploring the fragility of meta-analyses in ophthalmology: a systematic review

OBJECTIVE

The fragility index (FI) of a meta-analysis evaluates the extent that the statistical significance can be changed by modifying the event status of individuals from included trials. Understanding the FI improves the interpretation of the results of meta-analyses and can help to inform changes to clinical practice. This review determined the fragility of ophthalmology-related meta-analyses.

METHODS

Meta-analyses of randomized controlled trials with binary outcomes published in a journal classified as 'Ophthalmology' according to the Journal Citation Report or an Ophthalmology-related Cochrane Review were included. An iterative process determined the FI of each meta-analysis. Multivariable linear regression modeling evaluated the relationship between the FI and potential predictive factors in statistically significant and non-significant meta-analyses.

RESULTS

175 meta-analyses were included. The median FI was 6 (Q1-Q3: 3-12). This meant that moving 6 outcomes from one group to another would reverse the study's findings. The FI was 1 for 18 (10.2%) of the included meta-analyses and was ≤5 for 75 (42.4%) of the included meta-analyses. The number of events (p < 0.001) and the p-value (p < 0.001) were the best predictors of the FI in both significant and non-significant meta-analyses.

CONCLUSION

The statistical significance of meta-analyses in ophthalmology often hinges on the outcome of a few patients. The number of events and the p-value are the most important factors in determining the fragility of the evidence. The FI is an easily interpretable measure that can supplement the reader's understanding of the strength of the evidence being presented.

PROSPERO REGISTRATION

CRD42022377589.

How fragile the positive results of Chinese herbal medicine randomized controlled trials on irritable bowel syndrome are?

OBJECTIVE

The fragility index (FI), which is the minimum number of changes in status from "event" to "non-event" resulting in a loss of statistical significance, serves as a significant supplementary indicator for clinical physicians in interpreting clinical trial results and aids in understanding the outcomes of randomized controlled trials (RCTs). In this systematic literature survey, we evaluated the FI for RCTs evaluating Chinese herbal medicine (CHM) for irritable bowel syndrome (IBS), and explored potential associations between study characteristics and the robustness of RCTs.

METHODS

A comprehensive search was conducted in four databases in Chinese and four databases in English from their inception to January 1, 2023. RCTs encompassed 1:1 ratio into two parallel arms and reported at least one binary outcome that demonstrated statistical significance were included. FI was calculated by the iterative reduction of a target outcome event in the treatment group and concomitant subtraction of a non-target event from that group, until positive significance (defined as P < 0.05 by Fisher's exact test) is lost. The lower the FI (minimum 1) of a trial outcome, the more fragile the positive result of the outcome was. Linear regression models were adopted to explore influence factors of the value of FI.

RESULTS

A total of 30 trials from 2 4118 potentially relevant citations were finally included. The median FI of total trials included was 1.5 (interquartile range [IQR], 1-5), and half of the trials (n = 15) had a FI equal to 1. In 12 trials (40%), the total number of participants lost to follow-up surpassed the respective FI. The study also identified that increased FI was significantly associated with no TCM syndrome differentiation for inclusion criteria of the patients, larger total sample size, low risk of bias, and larger numbers of events.

CONCLUSIONS

The majority of CHM IBS RCTs with positive results were found to be fragile. Ensuring adequate sample size, scientifically rigorous study design, proper control of confounding factors, and a quality control calibration for consistency of TCM diagnostic results among clinicians should be addressed to increase the robustness of the RCTs. We recommend reporting the FI as one of the components of sensitivity analysis in future RCTs to facilitate the assessment of the fragility of trials.

The fragility index and reverse fragility index of FDA investigational device exemption trials in spinal fusion surgery: a systematic review

PURPOSE

FDA investigational device exemption (IDE) studies are considered a gold standard of assessing safety and efficacy of novel devices through RCTs. The fragility index (FI) has emerged as a means to assess robustness of statistically significant study results and inversely, the reverse fragility index (RFI) for non-significant differences. Previous authors have defined results as fragile if loss to follow up is greater than the FI or RFI. The aim of this study was to assess the FI, RFI, and robustness of data supplied by IDE studies in spinal surgery.

METHODS

This was a systematic review of the literature. Inclusion criteria included randomized controlled trials with dichotomous outcome measures conducted under IDE guidelines between 2000 and 2023. FI and RFI were calculated through successively changing events to non-events until the outcome changed to non-significance or significance, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI and RFI, respectively, by the sample size.

RESULTS

Thirty-two studies met inclusion criteria with a total of 40 unique outcome measures; 240 outcomes were analyzed. Twenty-six studies reported 96 statistically significant results. The median FI was 6 (IQR: 3-9.25), and patients lost to follow up was greater than the FI in 99.0% (95/96) of results. The average FQ was 0.027. Thirty studies reported 144 statistically insignificant results and a median RFI of 6 (IQR: 4-8). The average RFQ extrapolated was 0.021, and loss to follow up was greater than the RFI in 98.6% (142/144) of results.

CONCLUSIONS

IDE studies in spine surgery are surprisingly fragile given their reputations, large sample sizes, and intent to establish safety in investigational devices. This study found a median FI and RFI of 6. The number of patients lost to follow-up was greater than FIand RFI in 98.8% (237/240) of reported outcomes. FQ and RFQ tell us that changes of two to three patients per hundred can flip the significance of reported outcomes. This is an important reminder of the limitations of RCTs. Analysis of fragility in future studies may help clarify the strength of the relationship between reported data and their conclusions.

Applying the fragility index to randomized controlled trials evaluating total neoadjuvant therapy for rectal cancer: A methodological survey

BACKGROUND

Recently, there has been significant interest in, and adoption of, total neoadjuvant therapy (TNT) for locally advanced rectal cancer (LARC). We designed the present study to assess the robustness of the randomized controlled trials (RCTs) evaluating contemporary TNTs for LARC using the fragility index (FI).

MATERIALS AND METHODS

Relevant articles were identified through a review article by Johnson et al. in the Canadian Journal of Surgery. Dichotomous outcomes within these RCTs were eligible for inclusion if the reported effect size had a p-value < 0.05. The main outcome was FI for each included outcome. Walsh et al.'s method of calculating FI was utilized. Correlations between FI and research characteristics were assessed using the Spearman's rank correlation coefficients. Risk of bias was assessed using Cochrane recommended tools.

RESULTS

Ten RCTs were identified with 25 outcomes having statistically significant differences between groups. Eleven outcomes were time-to-event outcomes, while the remainder were dichotomous outcomes. Approximately half (n = 13) were oncologic outcomes. The median FI was 2 (interquartile range [IQR] 1-16). The number of patients lost to follow-up exceeded the FI in 17 outcomes (68.0 %) and thus these results were considered "fragile". Lower FI was associated with high risk of bias (rho = -0.5594) and greater loss to follow-up (rho = -0.4394), while higher FI was associated with large study size (rho = 0.5120).

CONCLUSIONS

The robustness of outcomes from trials assessing TNT for LARC was found to be questionable. Most outcomes were fragile, as determined by the FI. This survey is limited by the number of included studies.

Harnessing Real-World Evidence to Advance Cancer Research

Randomized controlled trials (RCTs) form a cornerstone of oncology research by generating evidence about the efficacy of therapies in selected patient populations. However, their implementation is often resource- and cost-intensive, and their generalisability to patients treated in routine practice may be limited. Real-world evidence leverages data collected about patients receiving clinical care in routine practice outside of clinical trial settings and provides opportunities to identify and address gaps in clinical trial evidence. This review outlines the strengths and limitations of real-world and RCT evidence and proposes a framework for the complementary use of the two bodies of evidence to advance cancer research. There are challenges to the implementation of real-world research in oncology, including heterogeneity of data sources, timely access to high-quality data, and concerns about the quality of methods leveraging real-world data, particularly causal inference. Improved understanding of the strengths and limitations of real-world data and ongoing efforts to optimise the conduct of real-world evidence research will improve its reliability, understanding and acceptance, and enable the full potential of real-world evidence to be realised in oncology practice.

The reliability and integrity of overall survival data based on follow-up records only and potential solutions to the challenges

Overall survival (OS) is considered the standard clinical endpoint to support effectiveness claims in new drug applications globally, particularly for lethal conditions such as cancer. However, the source and reliability of OS in the setting of clinical trials have seldom been doubted and discussed. This study first raised the common issue that data integrity and reliability are doubtful when we collect OS information or other time-to-event endpoints based solely on simple follow-up records by investigators without supporting material, especially since the 2019 COVID-19 pandemic. Then, two rounds of discussions with 30 Chinese experts were held and 12 potential source scenarios of three methods for obtaining the time of death of participants, including death certificate, death record and follow-up record, were sorted out and analysed. With a comprehensive assessment of the 12 scenarios by legitimacy, data reliability, data acquisition efficiency, difficulty of data acquisition, and coverage of participants, both short-term and long-term recommended sources, overall strategies and detailed measures for improving the integrity and reliability of death date are presented. In the short term, we suggest integrated sources such as public security systems made available to drug inspection centres appropriately as soon as possible to strengthen supervision. Death certificates provided by participants' family members and detailed standard follow-up records are recommended to investigators as the two channels of mutual compensation, and the acquisition of supporting materials is encouraged as long as it is not prohibited legally. Moreover, we expect that the sharing of electronic medical records and the legal disclosure of death records in established health registries can be realized with the joint efforts of the whole industry in the long-term. The above proposed solutions are mainly based on the context of China and can also provide reference for other countries in the world.

Robustness of Significant Dichotomous Outcomes in Randomized Controlled Trials in the Treatment of Patients with COVID-19: A Systematic Analysis

Purpose

Significant results of randomized controlled trials (RCTs) should be properly weighed. This study adopted fragility index (FI) to evaluate the robustness of significant dichotomous outcomes from RCTs on coronavirus disease 2019 (COVID-19) treatment.

Materials and methods

ClinicalTrials.gov and PubMed were searched from inception to July 31, 2021. FIs were calculated and their distribution was depicted. FI's categorical influential factors were analyzed. Spearman correlation coefficient (r _s) was reported for the relationship between FI and the continuous characteristics of RCTs.

Results

Fifty RCTs with 120 outcomes in 7869 patients were included. The FI distribution was abnormal with median 3 (interquartile range 1-7, P = 0.0001). The FIs and robustness were affected by the outcomes of interest, various patient populations, and interventions (T = 18.215,16.667, 23.107; P = 0.02,0.0001, 0.001, respectively). A cubic relationship between the FIs and absolute difference of events between groups with R square of 0.848 (T = 215.828, P = 0.0001, R square = 0.865) was observed. A strong negative logarithmic relationship existed between FI and the P value with R square = - 0.834.

Conclusion

The robustness of significant dichotomous outcomes of COVID-19 treatments was fragile and affected by the outcomes of interest, patients, interventions, P value, and absolute difference of events between the groups. FI was an useful quantitative metric for the binary significant outcomes on COVID-19 treatments.

Registration

PROSPERO (CRD42021272455).

Supplementary Information

The online version contains supplementary material available at 10.1007/s44231-022-00027-y.

Fragility indices for only sufficiently likely modifications

The fragility index is a clinically meaningful metric based on modifying patient outcomes that is increasingly used to interpret the robustness of clinical trial results. The fragility index relies on a concept that explores alternative realizations of the same clinical trial by modifying patient measurements. In this article, we propose to generalize the fragility index to a family of fragility indices called the incidence fragility indices that permit only outcome modifications that are sufficiently likely and provide an exact algorithm to calculate the incidence fragility indices. Additionally, we introduce a far-reaching generalization of the fragility index to any data type and explain how to permit only sufficiently likely modifications for nondichotomous outcomes. All of the proposed methodologies follow the fragility index concept.

Fragility of randomized trials supporting cancer drug approvals stratified by approval pathway and review designations

BACKGROUND

It has been suggested that the results from fragile trials are less likely to translate into benefit in routine clinical practice.

METHODS

We searched the Food and Drug Administration (FDA) archives to identify drug approvals for solid organ malignancies between 2010 and 2019. We calculated the Fragility Index (FI) supporting each approval, using methods to account for time-to-event. We compared FI and trial and approval characteristics using Mann-Whitney U and Kruskal-Wallis test. Using logistic regression, we examined study characteristics associated with withdrawal of consent or lost to follow-up (WCLFU) exceeding the calculated FI.

RESULTS

The median FI among 125 included studies was 23 (range 1-322). The FI was ≤10 in 35 studies (28%), 11-20 in 21 (17%), and >20 in 69 (55%). The median FI/Nexp was 7.7% (range 0.1-51.7%). The median FI was significantly lower among approvals processed through the accelerated vs regular pathway (5.5 vs 25, p = 0.001), but there was no difference in median FI/Nexp. The WCLFU exceeded FI in 42% of studies. Overall survival endpoints were more likely to have a WCLFU exceeding FI (OR 3.16, p = 0.003). WCLFU exceeding FI was also associated with a lesser magnitude of effect (median HR 0.69 vs 0.55, p < 0.001). In a sensitivity analysis including only studies with 1:1 randomization, 51% of studies had WCLFU >FI.

CONCLUSION

The median FI among all trials was 23, and WCLFU exceeded FI in 42%. Comparative trials in solid tumors supporting approval through the accelerated pathway are more fragile compared to trials approved through the regular pathway, an observation likely explained by a lower sample size in the experimental arm.