1
|
Bi Y, Wei H, Ma Q, Wang R, Jin J, Qu K, Liu Y, Zhai Z, Zhu L, Wang J. The fragility index of randomized controlled trials in advanced/metastatic renal cell cancer. Urol Oncol 2025; 43:333.e9-333.e15. [PMID: 40155257 DOI: 10.1016/j.urolonc.2025.03.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2024] [Revised: 02/06/2025] [Accepted: 03/02/2025] [Indexed: 04/01/2025]
Abstract
PURPOSE The fragility index (FI) has been applied as a supplement to the noncomprehensive P-values to assess the robustness of randomized controlled trials (RCTs). The objective of this study is to evaluate the statistical robustness of RCTs of advanced/metastatic renal cell cancer (a/mRCC) using the FI. MATERIALS AND METHODS RCTs related to a/mRCC published in the 4 highest-impact general medical journals and the 25 highest-impact urological journals between January 1, 2000, and December 31, 2023, were identified from PubMed database. The FI was calculated by using Fisher's exact test. Spearman's correlation analysis was conducted to assess potential correlates regarding FI. RESULTS 16 eligible RCTs were screened with a median total sample size of 654.5 (IQR, 461-847) and a median patients lost to follow-up of 14 (IQR, 3-23). The median FI was 12.5 (IQR, 8.5-27), suggesting that a switch in outcomes in only 13 patients would have reversed the significance of the trials. The number of patients lost to follow-up exceeded or equaled to the FI in 7 (44%) RCTs. P-values were negatively associated with the FI, while the number of patients lost to follow-up and patients enrolled were not statistically significant. CONCLUSION Not all RCTs associated with a/mRCC are as statistically robust as previously considered and should therefore be construed carefully. We suggest that additional reporting of FI in urological RCTs as a supplement to the P-value to assist readers in concluding reliably by considering the fragility of the outcomes.
Collapse
Affiliation(s)
- Yingwei Bi
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China
| | - Haotian Wei
- Department of Urology, The Second Hospital of Tianjin Medical University, Tianjin 300202, China
| | - Qifeng Ma
- College of Basic Medicine, Dalian Medical University, Dalian 116041, China
| | - Rui Wang
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China
| | - Jiacheng Jin
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China
| | - Kexin Qu
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China
| | - Yuxin Liu
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China
| | - Ziwei Zhai
- College of Basic Medicine, Dalian Medical University, Dalian 116041, China
| | - Liang Zhu
- College of Basic Medicine, Dalian Medical University, Dalian 116041, China; College of Basic Medicine, Dalian University of Technology, Dalian 116081, China.
| | - Jianbo Wang
- Department of Urology, The First Affiliated Hospital of Dalian Medical University, Dalian 116011, China.
| |
Collapse
|
2
|
Braik D, Lemieux C, Wilson BE, Salawu A, Abdul Razak AR. Clinical benefit and fragility evaluation of systemic therapy trials for advanced soft tissue sarcoma. Cancer 2025; 131:e35564. [PMID: 39292680 DOI: 10.1002/cncr.35564] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2024] [Revised: 08/14/2024] [Accepted: 08/15/2024] [Indexed: 09/20/2024]
Abstract
BACKGROUND The clinical benefit of systemic anticancer therapies can be unclear despite positive trials, and outcomes may not translate to real-world practice. This study evaluated the benefit of soft tissue sarcoma (STS) treatments using the European Society of Medical Oncology Magnitude of Clinical Benefit Scale (MCBS) v1.1 and measured the robustness of STS trial results using Fragility Index (FI). METHODS Database searches for adult phase II or III trials in advanced STS (January 1998-December 2023) were performed. Therapies with trial outcomes that met the criteria for MCBS were scored 1-5 (≥4 represents substantial clinical benefit). For randomized clinical trials with positive time-to-event endpoints, the number of additional events that would render results nonsignificant, FI, was calculated and expressed as a proportion of the experimental arm size (fragility quotient [FQ]). Higher FI/FQ implies more robust results. RESULTS Among 194 trials, 19 (9.8%) were phase III. Most phase II trials (146/175; 83.4%) had single-arm or non-comparative design. Trials that were eligible for MCBS scoring (n = 78; 40.2%) evaluated 56 different agents/regimens. Median MCBS score was 2. Only three agents/regimens (all cytotoxic therapies) had an MCBS score ≥4. Among 47 randomized clinical trials, 16 (8 phase II; 8 phase III) trials had positive outcomes. Median FI was 7 (range, 2-52) and 10 trials (62.5%) had an FQ < 10%, with median of 7% (range, 1%-59%). CONCLUSIONS Most systemic therapies in STS trials did not confer substantial clinical benefit per European Society of Medical Oncology-MCBS. Additionally, positive randomized trials were often fragile. Novel STS therapy trials should use clinically meaningful endpoints and real-world efficacy confirmation is essential, especially for less robust trials.
Collapse
Affiliation(s)
- Dina Braik
- Princess Margaret Cancer Centre, University Health Network, University of Toronto, Toronto, Ontario, Canada
| | | | - Brooke E Wilson
- Kingston Health Sciences Centre, Queens University, Kingston, Ontario, Canada
| | - Abdulazeez Salawu
- Princess Margaret Cancer Centre, University Health Network, University of Toronto, Toronto, Ontario, Canada
| | - Albiruni R Abdul Razak
- Princess Margaret Cancer Centre, University Health Network, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
3
|
Olsen HE, Kao PC, Richmond C, Shulman DS, London WB, DuBois SG. Statistical Fragility of Findings From Randomized Phase 3 Trials in Pediatric Oncology. Cancer Med 2024; 13:e70356. [PMID: 39676273 DOI: 10.1002/cam4.70356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2024] [Revised: 08/28/2024] [Accepted: 10/11/2024] [Indexed: 12/17/2024] Open
Abstract
PURPOSE The fragility index (FI) is an adjunctive metric to facilitate the interpretation of p-values in clinical trials. The FI has not been studied in phase 3 trials in pediatric oncology. METHODS PubMed was used to identify phase 3 pediatric oncology trials published between 1980 and 2020. We report trial characteristics and calculate the FI for trials with a binary outcome and survival-inferred fragility index (SIFI) for trials with a time-to-event outcome. FI/SIFI is the number of patients from one arm of a trial who would need to change groups for the statistical conclusion to change. We also report fragility quotients (FQ and SFQ) to normalize FI and SIFI relative to trial size. RESULTS One hundred and thirteen trials included sufficient data for analysis. The median FI for trials with a binary outcome (n = 40) was 4.5 (range: 1-33). The median SIFI for trials with a time-to-event outcome (n = 73) was 13 (range: 0-61). The FI or SIFI was less than the number of patients lost to follow-up in 25% of 36 trials. Median FQ and SFQ were 0.026 and 0.03, respectively, and did not significantly vary according to trial characteristics. While sample sizes increased over time, the FQ and SFQ remained stable. CONCLUSIONS The statistical conclusions of pediatric oncology phase 3 trials hinge on a relatively small number and proportion of patients. Despite the sample size limitations of low prevalence diseases, pediatric cancer trials are similarly or less fragile than adult oncology trials. Smaller trials do not appear more statistically fragile than larger trials. Statistical fragility appears to have remained constant over the four decades evaluated. We recommend reporting FI or SIFI, in conjunction with p-values, for all phase 3 pediatric oncology trials.
Collapse
Affiliation(s)
- Hannah E Olsen
- Department of Pediatrics, Boston Children's Hospital and Harvard Medical School, Boston, Massachusetts, USA
| | - Pei-Chi Kao
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Caleb Richmond
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts, USA
| | - David S Shulman
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Wendy B London
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Steven G DuBois
- Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
4
|
Nanji K, Xie J, Hatamnejad A, Pur DR, Phillips M, Zeraatkar D, Wong TY, Guymer RH, Kaiser PK, Sivaprasad S, Bhandari M, Steel DH, Wykoff CC, Chaudhary V. Exploring the fragility of meta-analyses in ophthalmology: a systematic review. Eye (Lond) 2024; 38:3153-3160. [PMID: 39033242 PMCID: PMC11543934 DOI: 10.1038/s41433-024-03255-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2024] [Revised: 06/13/2024] [Accepted: 07/15/2024] [Indexed: 07/23/2024] Open
Abstract
OBJECTIVE The fragility index (FI) of a meta-analysis evaluates the extent that the statistical significance can be changed by modifying the event status of individuals from included trials. Understanding the FI improves the interpretation of the results of meta-analyses and can help to inform changes to clinical practice. This review determined the fragility of ophthalmology-related meta-analyses. METHODS Meta-analyses of randomized controlled trials with binary outcomes published in a journal classified as 'Ophthalmology' according to the Journal Citation Report or an Ophthalmology-related Cochrane Review were included. An iterative process determined the FI of each meta-analysis. Multivariable linear regression modeling evaluated the relationship between the FI and potential predictive factors in statistically significant and non-significant meta-analyses. RESULTS 175 meta-analyses were included. The median FI was 6 (Q1-Q3: 3-12). This meant that moving 6 outcomes from one group to another would reverse the study's findings. The FI was 1 for 18 (10.2%) of the included meta-analyses and was ≤5 for 75 (42.4%) of the included meta-analyses. The number of events (p < 0.001) and the p-value (p < 0.001) were the best predictors of the FI in both significant and non-significant meta-analyses. CONCLUSION The statistical significance of meta-analyses in ophthalmology often hinges on the outcome of a few patients. The number of events and the p-value are the most important factors in determining the fragility of the evidence. The FI is an easily interpretable measure that can supplement the reader's understanding of the strength of the evidence being presented. PROSPERO REGISTRATION CRD42022377589.
Collapse
Affiliation(s)
- Keean Nanji
- Department of Surgery, Division of Ophthalmology, McMaster University, Hamilton, ON, Canada
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada
| | - Jim Xie
- Department of Surgery, Division of Ophthalmology, McMaster University, Hamilton, ON, Canada
| | - Amin Hatamnejad
- Department of Surgery, Division of Ophthalmology, McMaster University, Hamilton, ON, Canada
| | - Daiana R Pur
- Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
| | - Mark Phillips
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada
| | - Dena Zeraatkar
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada
- Department of Anesthesia, McMaster University, Hamilton, ON, Canada
| | - Tien Yin Wong
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
- Tsinghua Medicine, Tsinghua University, Beijing, China
| | - Robyn H Guymer
- Centre for Eye Research Australia, Royal Victorian Eye and Ear Hospital, East Melbourne, Australia
- Department of Surgery (Ophthalmology), The University of Melbourne, Melbourne, Australia
| | - Peter K Kaiser
- Cole Eye Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Sobha Sivaprasad
- NIHR Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK
| | - Mohit Bhandari
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada
- Department of Surgery, Division of Orthopedic Surgery, McMaster University, Hamilton, ON, Canada
| | - David H Steel
- Bioscience Institute, Newcastle University, Newcastle Upon Tyne, UK
- Sunderland Eye Infirmary, Sunderland, UK
| | - Charles C Wykoff
- Retina Consultants of Texas, Houston, TX, USA
- Blanton Eye Institute, Houston Methodist Hospital, Houston, TX, USA
| | - Varun Chaudhary
- Department of Surgery, Division of Ophthalmology, McMaster University, Hamilton, ON, Canada.
| |
Collapse
|
5
|
Luo M, Huang J, Wang Y, Li Y, Liu Z, Liu M, Tao Y, Cao R, Chai Q, Liu J, Fei Y. How fragile the positive results of Chinese herbal medicine randomized controlled trials on irritable bowel syndrome are? BMC Complement Med Ther 2024; 24:300. [PMID: 39143474 PMCID: PMC11323352 DOI: 10.1186/s12906-024-04561-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Accepted: 06/21/2024] [Indexed: 08/16/2024] Open
Abstract
OBJECTIVE The fragility index (FI), which is the minimum number of changes in status from "event" to "non-event" resulting in a loss of statistical significance, serves as a significant supplementary indicator for clinical physicians in interpreting clinical trial results and aids in understanding the outcomes of randomized controlled trials (RCTs). In this systematic literature survey, we evaluated the FI for RCTs evaluating Chinese herbal medicine (CHM) for irritable bowel syndrome (IBS), and explored potential associations between study characteristics and the robustness of RCTs. METHODS A comprehensive search was conducted in four databases in Chinese and four databases in English from their inception to January 1, 2023. RCTs encompassed 1:1 ratio into two parallel arms and reported at least one binary outcome that demonstrated statistical significance were included. FI was calculated by the iterative reduction of a target outcome event in the treatment group and concomitant subtraction of a non-target event from that group, until positive significance (defined as P < 0.05 by Fisher's exact test) is lost. The lower the FI (minimum 1) of a trial outcome, the more fragile the positive result of the outcome was. Linear regression models were adopted to explore influence factors of the value of FI. RESULTS A total of 30 trials from 2 4118 potentially relevant citations were finally included. The median FI of total trials included was 1.5 (interquartile range [IQR], 1-5), and half of the trials (n = 15) had a FI equal to 1. In 12 trials (40%), the total number of participants lost to follow-up surpassed the respective FI. The study also identified that increased FI was significantly associated with no TCM syndrome differentiation for inclusion criteria of the patients, larger total sample size, low risk of bias, and larger numbers of events. CONCLUSIONS The majority of CHM IBS RCTs with positive results were found to be fragile. Ensuring adequate sample size, scientifically rigorous study design, proper control of confounding factors, and a quality control calibration for consistency of TCM diagnostic results among clinicians should be addressed to increase the robustness of the RCTs. We recommend reporting the FI as one of the components of sensitivity analysis in future RCTs to facilitate the assessment of the fragility of trials.
Collapse
Affiliation(s)
- Minjing Luo
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Jinghan Huang
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
| | - Yingqiao Wang
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
| | - Yilin Li
- School of Qi-Huang Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Zhihan Liu
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Meijun Liu
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
| | - Yunci Tao
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Rui Cao
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Qianyun Chai
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Jianping Liu
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China
| | - Yutong Fei
- Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, No.11, Bei San Huan Dong Lu, Chaoyang District, Beijing, 100029, China.
- School of Traditional Chinese Medicine, Beijing University of Chinese Medicine, Beijing, 100029, China.
| |
Collapse
|
6
|
Proal JD, Moon AS, Kwon B. The fragility index and reverse fragility index of FDA investigational device exemption trials in spinal fusion surgery: a systematic review. EUROPEAN SPINE JOURNAL : OFFICIAL PUBLICATION OF THE EUROPEAN SPINE SOCIETY, THE EUROPEAN SPINAL DEFORMITY SOCIETY, AND THE EUROPEAN SECTION OF THE CERVICAL SPINE RESEARCH SOCIETY 2024; 33:2594-2603. [PMID: 38802596 DOI: 10.1007/s00586-024-08317-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Revised: 04/20/2024] [Accepted: 05/16/2024] [Indexed: 05/29/2024]
Abstract
PURPOSE FDA investigational device exemption (IDE) studies are considered a gold standard of assessing safety and efficacy of novel devices through RCTs. The fragility index (FI) has emerged as a means to assess robustness of statistically significant study results and inversely, the reverse fragility index (RFI) for non-significant differences. Previous authors have defined results as fragile if loss to follow up is greater than the FI or RFI. The aim of this study was to assess the FI, RFI, and robustness of data supplied by IDE studies in spinal surgery. METHODS This was a systematic review of the literature. Inclusion criteria included randomized controlled trials with dichotomous outcome measures conducted under IDE guidelines between 2000 and 2023. FI and RFI were calculated through successively changing events to non-events until the outcome changed to non-significance or significance, respectively. The fragility quotient (FQ) and reverse fragility quotient (RFQ) were calculated by dividing the FI and RFI, respectively, by the sample size. RESULTS Thirty-two studies met inclusion criteria with a total of 40 unique outcome measures; 240 outcomes were analyzed. Twenty-six studies reported 96 statistically significant results. The median FI was 6 (IQR: 3-9.25), and patients lost to follow up was greater than the FI in 99.0% (95/96) of results. The average FQ was 0.027. Thirty studies reported 144 statistically insignificant results and a median RFI of 6 (IQR: 4-8). The average RFQ extrapolated was 0.021, and loss to follow up was greater than the RFI in 98.6% (142/144) of results. CONCLUSIONS IDE studies in spine surgery are surprisingly fragile given their reputations, large sample sizes, and intent to establish safety in investigational devices. This study found a median FI and RFI of 6. The number of patients lost to follow-up was greater than FIand RFI in 98.8% (237/240) of reported outcomes. FQ and RFQ tell us that changes of two to three patients per hundred can flip the significance of reported outcomes. This is an important reminder of the limitations of RCTs. Analysis of fragility in future studies may help clarify the strength of the relationship between reported data and their conclusions.
Collapse
Affiliation(s)
- Joshua D Proal
- Tufts University School of Medicine, 145 Harrison Ave, Boston, MA, 02111, USA.
| | - Andrew S Moon
- Department of Orthopedic Surgery, Tufts Medical Center, Tufts University School of Medicine, 800 Washington St, Tufts MC Box #306, Boston, MA, 02111, USA
| | - Brian Kwon
- New England Baptist Hospital, Department of Orthopaedic Surgery, 125 Parker Hill Ave, Boston, MA, 02120, USA
| |
Collapse
|
7
|
McKechnie T, Brennan K, Eskicioglu C, Farooq A, Patel SV. Applying the fragility index to randomized controlled trials evaluating total neoadjuvant therapy for rectal cancer: A methodological survey. Radiother Oncol 2024; 194:110148. [PMID: 38341094 DOI: 10.1016/j.radonc.2024.110148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Revised: 01/08/2024] [Accepted: 02/05/2024] [Indexed: 02/12/2024]
Abstract
BACKGROUND Recently, there has been significant interest in, and adoption of, total neoadjuvant therapy (TNT) for locally advanced rectal cancer (LARC). We designed the present study to assess the robustness of the randomized controlled trials (RCTs) evaluating contemporary TNTs for LARC using the fragility index (FI). MATERIALS AND METHODS Relevant articles were identified through a review article by Johnson et al. in the Canadian Journal of Surgery. Dichotomous outcomes within these RCTs were eligible for inclusion if the reported effect size had a p-value < 0.05. The main outcome was FI for each included outcome. Walsh et al.'s method of calculating FI was utilized. Correlations between FI and research characteristics were assessed using the Spearman's rank correlation coefficients. Risk of bias was assessed using Cochrane recommended tools. RESULTS Ten RCTs were identified with 25 outcomes having statistically significant differences between groups. Eleven outcomes were time-to-event outcomes, while the remainder were dichotomous outcomes. Approximately half (n = 13) were oncologic outcomes. The median FI was 2 (interquartile range [IQR] 1-16). The number of patients lost to follow-up exceeded the FI in 17 outcomes (68.0 %) and thus these results were considered "fragile". Lower FI was associated with high risk of bias (rho = -0.5594) and greater loss to follow-up (rho = -0.4394), while higher FI was associated with large study size (rho = 0.5120). CONCLUSIONS The robustness of outcomes from trials assessing TNT for LARC was found to be questionable. Most outcomes were fragile, as determined by the FI. This survey is limited by the number of included studies.
Collapse
Affiliation(s)
- Tyler McKechnie
- Division of General Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada
| | - Kelly Brennan
- Division of General Surgery, Department of Surgery, Queen's University, Kingston Health Sciences Centre, Kingston, ON, Canada
| | - Cagla Eskicioglu
- Division of General Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada
| | - Ameer Farooq
- Division of General Surgery, Department of Surgery, Queen's University, Kingston Health Sciences Centre, Kingston, ON, Canada
| | - Sunil V Patel
- Division of General Surgery, Department of Surgery, Queen's University, Kingston Health Sciences Centre, Kingston, ON, Canada.
| |
Collapse
|
8
|
Tang M, Pearson SA, Simes RJ, Chua BH. Harnessing Real-World Evidence to Advance Cancer Research. Curr Oncol 2023; 30:1844-1859. [PMID: 36826104 PMCID: PMC9955401 DOI: 10.3390/curroncol30020143] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 01/16/2023] [Accepted: 01/30/2023] [Indexed: 02/05/2023] Open
Abstract
Randomized controlled trials (RCTs) form a cornerstone of oncology research by generating evidence about the efficacy of therapies in selected patient populations. However, their implementation is often resource- and cost-intensive, and their generalisability to patients treated in routine practice may be limited. Real-world evidence leverages data collected about patients receiving clinical care in routine practice outside of clinical trial settings and provides opportunities to identify and address gaps in clinical trial evidence. This review outlines the strengths and limitations of real-world and RCT evidence and proposes a framework for the complementary use of the two bodies of evidence to advance cancer research. There are challenges to the implementation of real-world research in oncology, including heterogeneity of data sources, timely access to high-quality data, and concerns about the quality of methods leveraging real-world data, particularly causal inference. Improved understanding of the strengths and limitations of real-world data and ongoing efforts to optimise the conduct of real-world evidence research will improve its reliability, understanding and acceptance, and enable the full potential of real-world evidence to be realised in oncology practice.
Collapse
Affiliation(s)
- Monica Tang
- Nelune Comprehensive Cancer Centre, Prince of Wales Hospital, Randwick 2031, Australia
- Correspondence:
| | | | - Robert J. Simes
- NHMRC Clinical Trials Centre, University of Sydney, Camperdown 2050, Australia
| | - Boon H. Chua
- Nelune Comprehensive Cancer Centre, Prince of Wales Hospital, Randwick 2031, Australia
- Faculty of Medicine and Health, UNSW Sydney, Sydney 2052, Australia
| |
Collapse
|
9
|
Huang H, Tang Y, Yu Y, Yu A, Wu D, Fang H, Wang S, Sun C, Wang X, Fan Q, Fang Y, Tang Q, Jiang N, Du J, Miao H, Bai Y, Ma P, Xing S, Cui D, Miao S, Jiang Y, Zhu J, Zhu Q, Leng Y, Guo LW, Liao S, Shao Y, Song Y, Liu Z, Hong M, Luo S, Xu B, Lan G, Li N. The reliability and integrity of overall survival data based on follow-up records only and potential solutions to the challenges. THE LANCET REGIONAL HEALTH. WESTERN PACIFIC 2023; 31:100624. [PMID: 36426285 PMCID: PMC9678947 DOI: 10.1016/j.lanwpc.2022.100624] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 09/20/2022] [Accepted: 10/13/2022] [Indexed: 11/19/2022]
Abstract
Overall survival (OS) is considered the standard clinical endpoint to support effectiveness claims in new drug applications globally, particularly for lethal conditions such as cancer. However, the source and reliability of OS in the setting of clinical trials have seldom been doubted and discussed. This study first raised the common issue that data integrity and reliability are doubtful when we collect OS information or other time-to-event endpoints based solely on simple follow-up records by investigators without supporting material, especially since the 2019 COVID-19 pandemic. Then, two rounds of discussions with 30 Chinese experts were held and 12 potential source scenarios of three methods for obtaining the time of death of participants, including death certificate, death record and follow-up record, were sorted out and analysed. With a comprehensive assessment of the 12 scenarios by legitimacy, data reliability, data acquisition efficiency, difficulty of data acquisition, and coverage of participants, both short-term and long-term recommended sources, overall strategies and detailed measures for improving the integrity and reliability of death date are presented. In the short term, we suggest integrated sources such as public security systems made available to drug inspection centres appropriately as soon as possible to strengthen supervision. Death certificates provided by participants' family members and detailed standard follow-up records are recommended to investigators as the two channels of mutual compensation, and the acquisition of supporting materials is encouraged as long as it is not prohibited legally. Moreover, we expect that the sharing of electronic medical records and the legal disclosure of death records in established health registries can be realized with the joint efforts of the whole industry in the long-term. The above proposed solutions are mainly based on the context of China and can also provide reference for other countries in the world.
Collapse
Affiliation(s)
- Huiyao Huang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Yu Tang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Yue Yu
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Anqi Yu
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Dawei Wu
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Hong Fang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Shuhang Wang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Chao Sun
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Xin Wang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Qi Fan
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Yuan Fang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Qiyu Tang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Ning Jiang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Jingting Du
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Huilei Miao
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Ying Bai
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Peiwen Ma
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Shujun Xing
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Dandan Cui
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Hebei Cancer Hospital, Chinese Academy of Medical Sciences, Langfang, 065001, China
| | - Shuangman Miao
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Hebei Cancer Hospital, Chinese Academy of Medical Sciences, Langfang, 065001, China
| | - Yale Jiang
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Jingxiao Zhu
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Hebei Cancer Hospital, Chinese Academy of Medical Sciences, Langfang, 065001, China
| | - Qi Zhu
- School of Basic Medicine and Clinical Pharmacy, China Pharmaceutical University, Nanjing, 211198, China
| | - Ye Leng
- School of Basic Medicine and Clinical Pharmacy, China Pharmaceutical University, Nanjing, 211198, China
| | - Lan Wei Guo
- Department of Cancer Epidemiology and Prevention, Henan International Joint Laboratory of Cancer Prevention, The Affiliated Cancer Hospital of Zhengzhou University, Henan Cancer Hospital, Zhengzhou, 450008, China
| | - Shanmei Liao
- Global Statistics and Data Science Department, BeiGene (Beijing) Co., Ltd, Beijing, 102206, China
| | - Yaguang Shao
- Capital Markets Department, Zhong Lun Law Firm, Shanghai, 200120, China
| | - Yinyin Song
- Department of Clinical Development and Operation, Boehringer Ingelheim (China) Investment Co Ltd, Shanghai, 201203, China
| | - Zeyuan Liu
- Department of Clinical Pharmacology, Affiliated Hospital of Academy of Military Medical Sciences, Beijing, China
| | - Minghuang Hong
- Department of Clinical Research, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer, Guangzhou, 510060, China
| | - Suxia Luo
- Early-phase Clinical Trial Center, Henan International Joint Laboratory of Cancer Prevention, The Affiliated Cancer Hospital of Zhengzhou University, Henan Cancer Hospital, Zhengzhou, 450008, China
| | - Binghe Xu
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| | - Gongtao Lan
- Department of Drug Registration, National Medical Products Administration, Beijing, 100022, China
| | - Ning Li
- Clinical Trials Center, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100021, China
| |
Collapse
|
10
|
Liu Q, Chen H, Gao Y, Zhu C. Robustness of Significant Dichotomous Outcomes in Randomized Controlled Trials in the Treatment of Patients with COVID-19: A Systematic Analysis. INTENSIVE CARE RESEARCH 2023; 3:38-49. [PMID: 36687387 PMCID: PMC9836340 DOI: 10.1007/s44231-022-00027-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Accepted: 12/23/2022] [Indexed: 01/13/2023]
Abstract
Purpose Significant results of randomized controlled trials (RCTs) should be properly weighed. This study adopted fragility index (FI) to evaluate the robustness of significant dichotomous outcomes from RCTs on coronavirus disease 2019 (COVID-19) treatment. Materials and methods ClinicalTrials.gov and PubMed were searched from inception to July 31, 2021. FIs were calculated and their distribution was depicted. FI's categorical influential factors were analyzed. Spearman correlation coefficient (r s) was reported for the relationship between FI and the continuous characteristics of RCTs. Results Fifty RCTs with 120 outcomes in 7869 patients were included. The FI distribution was abnormal with median 3 (interquartile range 1-7, P = 0.0001). The FIs and robustness were affected by the outcomes of interest, various patient populations, and interventions (T = 18.215,16.667, 23.107; P = 0.02,0.0001, 0.001, respectively). A cubic relationship between the FIs and absolute difference of events between groups with R square of 0.848 (T = 215.828, P = 0.0001, R square = 0.865) was observed. A strong negative logarithmic relationship existed between FI and the P value with R square = - 0.834. Conclusion The robustness of significant dichotomous outcomes of COVID-19 treatments was fragile and affected by the outcomes of interest, patients, interventions, P value, and absolute difference of events between the groups. FI was an useful quantitative metric for the binary significant outcomes on COVID-19 treatments. Registration PROSPERO (CRD42021272455). Supplementary Information The online version contains supplementary material available at 10.1007/s44231-022-00027-y.
Collapse
Affiliation(s)
- Qi Liu
- Emergency Department, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, No. 1st, Jianshe Eastern Road, Zhengzhou, Henan Province People's Republic of China.,Department of Translational Medicine Center, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, Zhengzhou, Henan Province People's Republic of China
| | - Hong Chen
- Emergency Department, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, No. 1st, Jianshe Eastern Road, Zhengzhou, Henan Province People's Republic of China.,Department of Translational Medicine Center, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, Zhengzhou, Henan Province People's Republic of China
| | - Yonghua Gao
- Department of Respiratory and Critical Care Medicine, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, People's Republic of China
| | - Changju Zhu
- Emergency Department, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, No. 1st, Jianshe Eastern Road, Zhengzhou, Henan Province People's Republic of China.,Henan Medical Key Laboratory of Emergency and Trauma Research, The First Affiliated Hospital of Zhengzhou University, Zhengzhou University, Zhengzhou, Henan Province People's Republic of China
| |
Collapse
|
11
|
Fragility indices for only sufficiently likely modifications. Proc Natl Acad Sci U S A 2021; 118:2105254118. [PMID: 34848537 DOI: 10.1073/pnas.2105254118] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/13/2021] [Indexed: 11/18/2022] Open
Abstract
The fragility index is a clinically meaningful metric based on modifying patient outcomes that is increasingly used to interpret the robustness of clinical trial results. The fragility index relies on a concept that explores alternative realizations of the same clinical trial by modifying patient measurements. In this article, we propose to generalize the fragility index to a family of fragility indices called the incidence fragility indices that permit only outcome modifications that are sufficiently likely and provide an exact algorithm to calculate the incidence fragility indices. Additionally, we introduce a far-reaching generalization of the fragility index to any data type and explain how to permit only sufficiently likely modifications for nondichotomous outcomes. All of the proposed methodologies follow the fragility index concept.
Collapse
|
12
|
The fragility index in 2010-2021 chronic lymphocytic leukemia randomized controlled trials. Blood Adv 2021; 6:225-227. [PMID: 34753170 PMCID: PMC8753221 DOI: 10.1182/bloodadvances.2021006473] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2021] [Accepted: 11/04/2021] [Indexed: 11/22/2022] Open
|
13
|
Wilson BE, Desnoyers A, Nadler MB, Tibau A, Amir E. Fragility of randomized trials supporting cancer drug approvals stratified by approval pathway and review designations. Cancer Med 2021; 10:5405-5414. [PMID: 34323019 PMCID: PMC8366090 DOI: 10.1002/cam4.4029] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Revised: 04/21/2021] [Accepted: 04/23/2021] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND It has been suggested that the results from fragile trials are less likely to translate into benefit in routine clinical practice. METHODS We searched the Food and Drug Administration (FDA) archives to identify drug approvals for solid organ malignancies between 2010 and 2019. We calculated the Fragility Index (FI) supporting each approval, using methods to account for time-to-event. We compared FI and trial and approval characteristics using Mann-Whitney U and Kruskal-Wallis test. Using logistic regression, we examined study characteristics associated with withdrawal of consent or lost to follow-up (WCLFU) exceeding the calculated FI. RESULTS The median FI among 125 included studies was 23 (range 1-322). The FI was ≤10 in 35 studies (28%), 11-20 in 21 (17%), and >20 in 69 (55%). The median FI/Nexp was 7.7% (range 0.1-51.7%). The median FI was significantly lower among approvals processed through the accelerated vs regular pathway (5.5 vs 25, p = 0.001), but there was no difference in median FI/Nexp. The WCLFU exceeded FI in 42% of studies. Overall survival endpoints were more likely to have a WCLFU exceeding FI (OR 3.16, p = 0.003). WCLFU exceeding FI was also associated with a lesser magnitude of effect (median HR 0.69 vs 0.55, p < 0.001). In a sensitivity analysis including only studies with 1:1 randomization, 51% of studies had WCLFU >FI. CONCLUSION The median FI among all trials was 23, and WCLFU exceeded FI in 42%. Comparative trials in solid tumors supporting approval through the accelerated pathway are more fragile compared to trials approved through the regular pathway, an observation likely explained by a lower sample size in the experimental arm.
Collapse
Affiliation(s)
- Brooke E Wilson
- Princess Margaret Cancer Centre, Department of Medical Oncology, University of Toronto, Toronto, ON, Canada.,University of New South Wales, Kensington, NSW, Australia
| | - Alexandra Desnoyers
- Princess Margaret Cancer Centre, Department of Medical Oncology, University of Toronto, Toronto, ON, Canada
| | - Michelle B Nadler
- Princess Margaret Cancer Centre, Department of Medical Oncology, University of Toronto, Toronto, ON, Canada
| | - Ariadna Tibau
- Oncology Department, Hospital de la Santa Creu i Sant Pau, Institut d'Investigació Biomèdica Sant Pau, and Universitat Autònoma de Barcelona, Barcelona, Catalonia, Spain
| | - Eitan Amir
- Princess Margaret Cancer Centre, Department of Medical Oncology, University of Toronto, Toronto, ON, Canada
| |
Collapse
|