1
|
Lai H, Gao K, Li M, Li T, Zhou X, Zhou X, Guo H, Fu B. Handling missing data and measurement error for early-onset myopia risk prediction models. BMC Med Res Methodol 2024; 24:194. [PMID: 39243025 PMCID: PMC11378546 DOI: 10.1186/s12874-024-02319-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2024] [Accepted: 08/23/2024] [Indexed: 09/09/2024] Open
Abstract
BACKGROUND Early identification of children at high risk of developing myopia is essential to prevent myopia progression by introducing timely interventions. However, missing data and measurement error (ME) are common challenges in risk prediction modelling that can introduce bias in myopia prediction. METHODS We explore four imputation methods to address missing data and ME: single imputation (SI), multiple imputation under missing at random (MI-MAR), multiple imputation with calibration procedure (MI-ME), and multiple imputation under missing not at random (MI-MNAR). We compare four machine-learning models (Decision Tree, Naive Bayes, Random Forest, and Xgboost) and three statistical models (logistic regression, stepwise logistic regression, and least absolute shrinkage and selection operator logistic regression) in myopia risk prediction. We apply these models to the Shanghai Jinshan Myopia Cohort Study and also conduct a simulation study to investigate the impact of missing mechanisms, the degree of ME, and the importance of predictors on model performance. Model performance is evaluated using the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). RESULTS Our findings indicate that in scenarios with missing data and ME, using MI-ME in combination with logistic regression yields the best prediction results. In scenarios without ME, employing MI-MAR to handle missing data outperforms SI regardless of the missing mechanisms. When ME has a greater impact on prediction than missing data, the relative advantage of MI-MAR diminishes, and MI-ME becomes more superior. Furthermore, our results demonstrate that statistical models exhibit better prediction performance than machine-learning models. CONCLUSION MI-ME emerges as a reliable method for handling missing data and ME in important predictors for early-onset myopia risk prediction.
Collapse
Affiliation(s)
- Hongyu Lai
- School of Data Science, Fudan University, Shanghai, China
| | - Kaiye Gao
- School of Economics and Management, Beijing Forestry University, Beijing, China
- Department of Logistics and Maritime Studies, Hong Kong Polytechnic University, Hong Kong, China
- Academy of Mathematics and Systems Sciences, Chinese Academy of Sicences, Beijing, China
| | - Meiyan Li
- Department of Ophthalmology, EYE & ENT Hospital of Fudan University, Shanghai, China
| | - Tao Li
- Department of Ophthalmology, Jinshan Hospital of Fudan University, Shanghai, China
| | - Xiaodong Zhou
- Department of Ophthalmology, Jinshan Hospital of Fudan University, Shanghai, China
| | - Xingtao Zhou
- Department of Ophthalmology, EYE & ENT Hospital of Fudan University, Shanghai, China
| | - Hui Guo
- Centre for Biostatistics, The University of Manchester, Manchester, UK
| | - Bo Fu
- School of Data Science, Fudan University, Shanghai, China.
| |
Collapse
|
2
|
Ahmad D, Sá MP, Brown JA, Yousef S, Wang Y, Thoma F, Chu D, Kaczorowski DJ, West DM, Bonatti J, Yoon PD, Ferdinand FD, Serna-Gallegos D, Phillippi J, Sultan I. External validation of the ARCH score in patients undergoing aortic arch reconstruction under circulatory arrest. J Thorac Cardiovasc Surg 2024:S0022-5223(24)00383-0. [PMID: 38750690 DOI: 10.1016/j.jtcvs.2024.05.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Revised: 05/06/2024] [Accepted: 05/07/2024] [Indexed: 06/07/2024]
Abstract
BACKGROUND Aortic arch surgery with hypothermic circulatory arrest (HCA) carries a higher risk of morbidity and mortality compared to routine cardiac surgical procedures. The newly developed ARCH (arch reconstruction under circulatory arrest with hypothermia) score has not been externally validated. We sought to externally validate this score in our local population. METHODS All consecutive open aortic arch surgeries with HCA performed between 2014 and 2023 were included. Univariable and multivariable analyses were performed. Model discrimination was assessed by the C-statistic with 95% confidence intervals as part of the receiver operating characteristic (ROC) curve analysis. Model performance was visualized by a calibration plot and quantified by the Brier score. RESULTS A total of 760 patients (38.3% females) were included. The mean age was 61 (±13.6) years, with 56.4% of patients' age >60 years. The procedures were carried out mostly emergently or urgently (59.6%). Total arch replacement was performed in 32.5% of the patients, and aortic root procedures were carried out in 74.6%. In-hospital death occurred in 64 patients (8.4%), and stroke occurred in 5.4%. The C-statistic revealed a low discriminatory ability for predicting in-hospital mortality (area under the ROC curve, 0.62; 95% confidence interval, 0.54-0.69; P = .002); however, model calibration was found to be excellent (Brier score of 0.07). CONCLUSIONS The ARCH score for in-hospital mortality showed low discriminatory ability in our local population, although with excellent ability for prediction of mortality.
Collapse
Affiliation(s)
- Danial Ahmad
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Michel Pompeu Sá
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - James A Brown
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Sarah Yousef
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Yisi Wang
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Floyd Thoma
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Danny Chu
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - David J Kaczorowski
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - David M West
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Johannes Bonatti
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Pyongsoo D Yoon
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Francis D Ferdinand
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Derek Serna-Gallegos
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa
| | - Julie Phillippi
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa
| | - Ibrahim Sultan
- Division of Cardiac Surgery, Department of Cardiothoracic Surgery, University of Pittsburgh, Pittsburgh, Pa; UPMC Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pa.
| |
Collapse
|
3
|
Gao M, Ma X. Commentary on the index admission cholecystectomy for acute cholecystitis reduces 30-day readmission rates in pediatric patients. Surg Endosc 2024; 38:2307-2308. [PMID: 38553596 DOI: 10.1007/s00464-024-10825-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 03/22/2024] [Indexed: 08/02/2024]
Affiliation(s)
- Ming Gao
- Department of Gastroenterology (Ward III), Central Hospital of Dalian University of Technology, Dalian, Liaoning, China
| | - Xuefeng Ma
- Department of Gastroenterology (Ward III), Central Hospital of Dalian University of Technology, Dalian, Liaoning, China.
- Department of Gastroenterology (Ward III), Digestive Endoscopy Center, Central Hospital of Dalian University of Technology, 826 Xinan Road, Dalian, China.
| |
Collapse
|
4
|
Zhao J, Bi Y. Comments on ''Lobulated tumor contour as a predictor of preoperative tumor invasion of the lung or pericardium in thymoma patients''. Surg Today 2024:10.1007/s00595-024-02862-6. [PMID: 38683359 DOI: 10.1007/s00595-024-02862-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Accepted: 04/11/2024] [Indexed: 05/01/2024]
Affiliation(s)
- Jing Zhao
- Department of Clinical Laboratory, Ningbo Zhenhai People's Hospital Group, 718 South 2nd West Road, Zhenghai District, Ningbo, Zhejiang, China
| | - Yue Bi
- Department of Clinical Laboratory, Ningbo Zhenhai People's Hospital Group, 718 South 2nd West Road, Zhenghai District, Ningbo, Zhejiang, China.
| |
Collapse
|
5
|
Zhao J, Bi Y. Commentary on ''The postoperative platelet‑to‑lymphocyte ratio predicts the outcome of patients undergoing pancreaticoduodenectomy for pancreatic head cancer''. Surg Today 2024:10.1007/s00595-024-02832-y. [PMID: 38522055 DOI: 10.1007/s00595-024-02832-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Accepted: 02/28/2024] [Indexed: 03/25/2024]
Affiliation(s)
- Jing Zhao
- Department of Clinical Laboratory, Ningbo Zhenhai People's Hospital Group, 718 South 2nd West road, Zhenghai District, Ningbo, 315000, Zhejiang, China
| | - Yue Bi
- Department of Clinical Laboratory, Ningbo Zhenhai People's Hospital Group, 718 South 2nd West road, Zhenghai District, Ningbo, 315000, Zhejiang, China.
| |
Collapse
|
6
|
Tan J, Liu C, Yang M, Xiong Y, Huang S, Qi Y, Chen M, Thabane L, Liu X, He L, Sun X. Investigation of statistical methods used in prognostic prediction models for obstetric care: A 10 year-span cross-sectional study. Acta Obstet Gynecol Scand 2024; 103:611-620. [PMID: 38140844 PMCID: PMC10867372 DOI: 10.1111/aogs.14757] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Revised: 11/06/2023] [Accepted: 12/06/2023] [Indexed: 12/24/2023]
Abstract
INTRODUCTION Obstetric care is a highly active area in the development and application of prognostic prediction models. The development and validation of these models often require the utilization of advanced statistical techniques. However, failure to adhere to rigorous methodological standards could greatly undermine the reliability and trustworthiness of the resultant models. Consequently, the aim of our study was to examine the current statistical practices employed in obstetric care and offer recommendations to enhance the utilization of statistical methods in the development of prognostic prediction models. MATERIAL AND METHODS We conducted a cross-sectional survey using a sample of studies developing or validating prognostic prediction models for obstetric care published in a 10-year span (2011-2020). A structured questionnaire was developed to investigate the statistical issues in five domains, including model derivation (predictor selection and algorithm development), model validation (internal and external), model performance, model presentation, and risk threshold setting. On the ground of survey results and existing guidelines, a list of recommendations for statistical methods in prognostic models was developed. RESULTS A total of 112 eligible studies were included, with 107 reporting model development and five exclusively reporting external validation. During model development, 58.9% of the studies did not include any form of validation. Of these, 46.4% used stepwise regression in a crude manner for predictor selection, while two-thirds made decisions on retaining or dropping candidate predictors solely based on p-values. Additionally, 26.2% transformed continuous predictors into categorical variables, and 80.4% did not consider nonlinear relationships between predictors and outcomes. Surprisingly, 94.4% of the studies did not examine the correlation between predictors. Moreover, 47.1% of the studies did not compare population characteristics between the development and external validation datasets, and only one-fifth evaluated both discrimination and calibration. Furthermore, 53.6% of the studies did not clearly present the model, and less than half established a risk threshold to define risk categories. In light of these findings, 10 recommendations were formulated to promote the appropriate use of statistical methods. CONCLUSIONS The use of statistical methods is not yet optimal. Ten recommendations were offered to assist the statistical methods of prognostic prediction models in obstetric care.
Collapse
Affiliation(s)
- Jing Tan
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
- Department of Health Research Methods, Evidence, and ImpactMcMaster UniversityHamiltonOntarioCanada
- Biostatistics UnitSt Joseph's Healthcare—HamiltonHamiltonOntarioCanada
| | - Chunrong Liu
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
| | - Min Yang
- Department of Epidemiology and Biostatistics, West China School of Public HealthSichuan UniversityChengduChina
- Faculty of Health, Design and ArtSwinburne Technology UniversityMelbourneVictoriaAustralia
| | - Yiquan Xiong
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
| | - Shiyao Huang
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
| | - Yana Qi
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
| | - Meng Chen
- Department of Obstetrics and Gynecology, and Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, West China Second University HospitalSichuan UniversityChengduSichuanChina
| | - Lehana Thabane
- Department of Health Research Methods, Evidence, and ImpactMcMaster UniversityHamiltonOntarioCanada
- Biostatistics UnitSt Joseph's Healthcare—HamiltonHamiltonOntarioCanada
| | - Xinghui Liu
- Department of Obstetrics and Gynecology, and Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, West China Second University HospitalSichuan UniversityChengduSichuanChina
| | - Lin He
- The Intelligence Library Center, Ministry of Science and Technology, Chinese Evidence‐Based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
| | - Xin Sun
- Chinese Evidence‐based Medicine Center, West China HospitalSichuan UniversityChengduSichuanChina
- NMPA Key Laboratory for Real World Data Research and Evaluation in HainanChengduSichuanChina
| |
Collapse
|
7
|
Buick JE, Austin PC, Cheskes S, Ko DT, Atzema CL. Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model. Acad Emerg Med 2023; 30:1150-1160. [PMID: 37266925 DOI: 10.1111/acem.14756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2023] [Revised: 05/24/2023] [Accepted: 05/29/2023] [Indexed: 06/03/2023]
Abstract
Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.
Collapse
Affiliation(s)
- Jason E Buick
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
| | - Peter C Austin
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- ICES, Toronto, Ontario, Canada
- Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
| | - Sheldon Cheskes
- Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
- Division of Emergency Medicine, Department of Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Dennis T Ko
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- ICES, Toronto, Ontario, Canada
- Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
- Department of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Clare L Atzema
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- ICES, Toronto, Ontario, Canada
- Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
- Division of Emergency Medicine, Department of Medicine, University of Toronto, Toronto, Ontario, Canada
| |
Collapse
|
8
|
Jiang T, Gradus JL, Lash TL, Fox MP. Addressing Measurement Error in Random Forests Using Quantitative Bias Analysis. Am J Epidemiol 2021; 190:1830-1840. [PMID: 33517416 DOI: 10.1093/aje/kwab010] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2019] [Revised: 10/27/2020] [Accepted: 11/05/2020] [Indexed: 12/17/2022] Open
Abstract
Although variables are often measured with error, the impact of measurement error on machine-learning predictions is seldom quantified. The purpose of this study was to assess the impact of measurement error on the performance of random-forest models and variable importance. First, we assessed the impact of misclassification (i.e., measurement error of categorical variables) of predictors on random-forest model performance (e.g., accuracy, sensitivity) and variable importance (mean decrease in accuracy) using data from the National Comorbidity Survey Replication (2001-2003). Second, we created simulated data sets in which we knew the true model performance and variable importance measures and could verify that quantitative bias analysis was recovering the truth in misclassified versions of the data sets. Our findings showed that measurement error in the data used to construct random forests can distort model performance and variable importance measures and that bias analysis can recover the correct results. This study highlights the utility of applying quantitative bias analysis in machine learning to quantify the impact of measurement error on study results.
Collapse
|
9
|
Ban JW, Chan MS, Muthee TB, Paez A, Stevens R, Perera R. Design, methods, and reporting of impact studies of cardiovascular clinical prediction rules are suboptimal: a systematic review. J Clin Epidemiol 2021; 133:111-120. [PMID: 33515655 DOI: 10.1016/j.jclinepi.2021.01.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2020] [Revised: 01/08/2021] [Accepted: 01/21/2021] [Indexed: 10/22/2022]
Abstract
OBJECTIVES To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs). STUDY DESIGN AND SETTING We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and nonrandomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we assessed the quality of methods and reporting. We compared the quality of methods and reporting between impact and matched control studies. RESULTS We found 110 impact studies of cardiovascular CPRs. Of these, 65 (59.1%) used inappropriate designs. Of 45 impact studies with appropriate design, 31 (68.9%) had substantial risk of bias. Mean number of reporting domains that impact studies with appropriate study design adhered to was 10.2 of 21 domains (95% confidence interval, 9.3 and 11.1). The quality of methods and reporting was not clearly different between impact and matched control studies. CONCLUSION We found most impact studies either used inappropriate study design, had substantial risk of bias, or poorly complied with reporting guidelines. This appears to be a common feature of complex interventions. Users of CPRs should critically evaluate evidence showing the effectiveness of CPRs.
Collapse
Affiliation(s)
- Jong-Wook Ban
- Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, United Kingdom; Department for Continuing Education, University of Oxford, Rewley House, 1 Wellington Square, Oxford, OX1 2JA, United Kingdom.
| | - Mei Sum Chan
- Nuffield Department of Population Health, University of Oxford, Richard Doll Building, Old Road Campus, Oxford, OX3 7LF, United Kingdom
| | - Tonny Brian Muthee
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, United Kingdom
| | - Arsenio Paez
- Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, United Kingdom; Department for Continuing Education, University of Oxford, Rewley House, 1 Wellington Square, Oxford, OX1 2JA, United Kingdom
| | - Richard Stevens
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, United Kingdom
| | - Rafael Perera
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG, United Kingdom
| |
Collapse
|
10
|
Giardiello D, Hauptmann M, Steyerberg EW, Adank MA, Akdeniz D, Blom JC, Blomqvist C, Bojesen SE, Bolla MK, Brinkhuis M, Chang-Claude J, Czene K, Devilee P, Dunning AM, Easton DF, Eccles DM, Fasching PA, Figueroa J, Flyger H, García-Closas M, Haeberle L, Haiman CA, Hall P, Hamann U, Hopper JL, Jager A, Jakubowska A, Jung A, Keeman R, Koppert LB, Kramer I, Lambrechts D, Le Marchand L, Lindblom A, Lubiński J, Manoochehri M, Mariani L, Nevanlinna H, Oldenburg HSA, Pelders S, Pharoah PDP, Shah M, Siesling S, Smit VTHBM, Southey MC, Tapper WJ, Tollenaar RAEM, van den Broek AJ, van Deurzen CHM, van Leeuwen FE, van Ongeval C, Van't Veer LJ, Wang Q, Wendt C, Westenend PJ, Hooning MJ, Schmidt MK. Prediction of contralateral breast cancer: external validation of risk calculators in 20 international cohorts. Breast Cancer Res Treat 2020; 181:423-434. [PMID: 32279280 PMCID: PMC8380991 DOI: 10.1007/s10549-020-05611-8] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2019] [Accepted: 03/21/2020] [Indexed: 12/18/2022]
Abstract
BACKGROUND Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in patients with invasive breast cancer (BC). METHODS We analyzed data of 132,756 patients (4682 CBC) from 20 international studies with a median follow-up of 8.8 years. Prediction performance included discrimination, quantified as a time-dependent Area-Under-the-Curve (AUC) at 5 and 10 years after diagnosis of primary BC, and calibration, quantified as the expected-observed (E/O) ratio at 5 and 10 years and the calibration slope. RESULTS The AUC at 10 years was: 0.58 (95% confidence intervals [CI] 0.57-0.59) for CBCrisk; 0.60 (95% CI 0.59-0.61) for the Manchester formula; 0.63 (95% CI 0.59-0.66) and 0.59 (95% CI 0.56-0.62) for PredictCBC-1A (for settings where BRCA1/2 mutation status is available) and PredictCBC-1B (for the general population), respectively. The E/O at 10 years: 0.82 (95% CI 0.51-1.32) for CBCrisk; 1.53 (95% CI 0.63-3.73) for the Manchester formula; 1.28 (95% CI 0.63-2.58) for PredictCBC-1A and 1.35 (95% CI 0.65-2.77) for PredictCBC-1B. The calibration slope was 1.26 (95% CI 1.01-1.50) for CBCrisk; 0.90 (95% CI 0.79-1.02) for PredictCBC-1A; 0.81 (95% CI 0.63-0.99) for PredictCBC-1B, and 0.39 (95% CI 0.34-0.43) for the Manchester formula. CONCLUSIONS Current CBC risk prediction tools provide only moderate discrimination and the Manchester formula was poorly calibrated. Better predictors and re-calibration are needed to improve CBC prediction and to identify low- and high-CBC risk patients for clinical decision-making.
Collapse
Affiliation(s)
- Daniele Giardiello
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - Michael Hauptmann
- Brandenburg Medical School, Institute of Biostatistics and Registry Research, Neuruppin, Germany
- Department of Epidemiology and Biostatistics, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
- Department of Public Health, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Muriel A Adank
- Family Cancer Clinic, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Delal Akdeniz
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Jannet C Blom
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Carl Blomqvist
- Department of Oncology, Helsinki University Hospital, University of Helsinki, Helsinki, Finland
- Department of Oncology, Örebro University Hospital, Örebro, Sweden
| | - Stig E Bojesen
- Copenhagen General Population Study, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
- Department of Clinical Biochemistry, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Manjeet K Bolla
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
| | - Mariël Brinkhuis
- Laboratory for Pathology, East-Netherlands, Hengelo, The Netherlands
| | - Jenny Chang-Claude
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
- University Medical Center Hamburg-Eppendorf, Cancer Epidemiology, University Cancer Center Hamburg (UCCH), Hamburg, Germany
| | - Kamila Czene
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Peter Devilee
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
- Department of Human Genetics, Leiden University Medical Center, Leiden, The Netherlands
| | - Alison M Dunning
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Douglas F Easton
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Diana M Eccles
- Cancer Sciences Academic Unit, Faculty of Medicine, University of Southampton, Southampton, UK
| | - Peter A Fasching
- David Geffen School of Medicine, Department of Medicine Division of Hematology and Oncology, University of California At Los Angeles, Los Angeles, CA, USA
- University Hospital Erlangen, Department of Gynecology and Obstetrics, Comprehensive Cancer Center ER-EMN, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany
| | - Jonine Figueroa
- The University of Edinburgh Medical School, Usher Institute of Population Health Sciences and Informatics, Edinburgh, UK
- Cancer Research UK Edinburgh Centre, Edinburgh, UK
- Department of Health and Human Services, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
| | - Henrik Flyger
- Department of Breast Surgery, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
| | - Montserrat García-Closas
- Department of Health and Human Services, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
- Division of Genetics and Epidemiology, Institute of Cancer Research, London, UK
| | - Lothar Haeberle
- University Hospital Erlangen, Department of Gynecology and Obstetrics, Comprehensive Cancer Center ER-EMN, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany
| | - Christopher A Haiman
- Department of Preventive Medicine, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA
| | - Per Hall
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
- Department of Oncology, Södersjukhuset, Stockholm, Sweden
| | - Ute Hamann
- Molecular Genetics of Breast Cancer, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - John L Hopper
- Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, VIC, Australia
| | - Agnes Jager
- Department of Medical Oncology, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Anna Jakubowska
- Department of Genetics and Pathology, Pomeranian Medical University, Szczecin, Poland
- Independent Laboratory of Molecular Biology and Genetic Diagnostics, Pomeranian Medical University, Szczecin, Poland
| | - Audrey Jung
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Renske Keeman
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Linetta B Koppert
- Department of Surgical Oncology, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Iris Kramer
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Diether Lambrechts
- VIB Center for Cancer Biology, Leuven, Belgium
- Laboratory for Translational Genetics, Department of Human Genetics, University of Leuven, Leuven, Belgium
| | - Loic Le Marchand
- Epidemiology Program, University of Hawaii Cancer Center, Honolulu, HI, USA
| | - Annika Lindblom
- Department of Molecular Medicine and Surgery, Karolinska Institutet, Stockholm, Sweden
- Department of Clinical Genetics, Karolinska University Hospital, Stockholm, Sweden
| | - Jan Lubiński
- Department of Genetics and Pathology, Pomeranian Medical University, Szczecin, Poland
| | - Mehdi Manoochehri
- Molecular Genetics of Breast Cancer, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Luigi Mariani
- Unit of Clinical Epidemiology and Trial Organization, Fondazione IRCCS Istituto Nazionale Dei Tumori, Milan, Italy
| | - Heli Nevanlinna
- Department of Obstetrics and Gynecology, Helsinki University Hospital, University of Helsinki, Helsinki, Finland
| | - Hester S A Oldenburg
- Department of Surgical Oncology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Saskia Pelders
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Paul D P Pharoah
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Mitul Shah
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Sabine Siesling
- Department of Research, Netherlands Comprehensive Cancer Organisation, Utrecht, The Netherlands
| | - Vincent T H B M Smit
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Melissa C Southey
- Precision Medicine, School of Clinical Sciences at Monash Health, Monash University, Clayton, VIC, Australia
- Department of Clinical Pathology, The University of Melbourne, Melbourne, VIC, Australia
| | | | - Rob A E M Tollenaar
- Department of Surgery, Leiden University Medical Center, Leiden, The Netherlands
| | - Alexandra J van den Broek
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | | | - Flora E van Leeuwen
- Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Chantal van Ongeval
- Leuven Cancer Institute, Leuven Multidisciplinary Breast Center, Department of Oncology, University Hospitals Leuven, Leuven, Belgium
| | - Laura J Van't Veer
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Qin Wang
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
| | - Camilla Wendt
- Department of Clinical Science and Education, Karolinska Institutet, Södersjukhuset, Stockholm, Sweden
| | | | - Maartje J Hooning
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Marjanka K Schmidt
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
- Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute - Antoni Van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
- Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX, Amsterdam, The Netherlands.
| |
Collapse
|
11
|
The Added Value of Lactate and Lactate Clearance in Prediction of In-Hospital Mortality in Critically Ill Patients With Sepsis. Crit Care Explor 2020; 2:e0087. [PMID: 32259110 PMCID: PMC7098542 DOI: 10.1097/cce.0000000000000087] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022] Open
Abstract
Supplemental Digital Content is available in the text. We investigated the added predictive value of lactate and lactate clearance to the Acute Physiology and Chronic Health Evaluation IV model for predicting in-hospital mortality in critically ill patients with sepsis.
Collapse
|
12
|
Luijken K, Wynants L, van Smeden M, Van Calster B, Steyerberg EW, Groenwold RH, Timmerman D, Bourne T, Ukaegbu C. Changing predictor measurement procedures affected the performance of prediction models in clinical examples. J Clin Epidemiol 2020; 119:7-18. [DOI: 10.1016/j.jclinepi.2019.11.001] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2019] [Revised: 10/30/2019] [Accepted: 11/04/2019] [Indexed: 10/25/2022]
|
13
|
Cowley LE, Farewell DM, Maguire S, Kemp AM. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res 2019; 3:16. [PMID: 31463368 PMCID: PMC6704664 DOI: 10.1186/s41512-019-0060-y] [Citation(s) in RCA: 120] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/13/2018] [Accepted: 05/12/2019] [Indexed: 12/20/2022] Open
Abstract
Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.
Collapse
Affiliation(s)
- Laura E. Cowley
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Daniel M. Farewell
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Sabine Maguire
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Alison M. Kemp
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| |
Collapse
|
14
|
Tan J, Qi Y, Liu C, Xiong Y, He Q, Zhang G, Chen M, He G, Wang W, Liu X, Sun X. The use of rigorous methods was strongly warranted among prognostic prediction models for obstetric care. J Clin Epidemiol 2019; 115:98-105. [PMID: 31326543 DOI: 10.1016/j.jclinepi.2019.07.009] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2019] [Revised: 07/01/2019] [Accepted: 07/15/2019] [Indexed: 02/05/2023]
Abstract
OBJECTIVE The objective of the study was to examine methodological characteristics about the design and conduct in prognostic prediction models used for obstetric care. STUDY DESIGN AND SETTING We searched PubMed for studies on prognostic prediction models for obstetric care, published in top general medicine or major specialty journals between January 2011 and February 2018. Teams of method-trained investigators independently screened titles and abstracts and collected data using a prespecified, pilot-tested, structured questionnaire. RESULTS In total, 91 studies were eligible, of which two were published in top general medicine journals, 20 (22.0%) involved an epidemiologist or statistician, 18 (19.4%) published study protocols, 53 (58.2%) did not include any model validation, 20 (22.0%) did not clearly state the intended timing of use, 23 (25.3%) had no eligibility criteria, 15 (16.5%) did not use clear criteria for ascertaining outcome, and 69 (75.82%) did not apply blinding to outcome assessment. Among those models, 11 (12.1%) included participants fewer than 200 events, 41 (48.8%) had fewer than 100 events, and 19 (24.7%) had fewer than 10 events per variable. CONCLUSION The prognostic prediction models have important limitations in design and conduct. Substantial efforts are needed to strengthen the production of reliable prognostic prediction models for obstetric care.
Collapse
Affiliation(s)
- Jing Tan
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Yana Qi
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Chunrong Liu
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Yiquan Xiong
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Qiao He
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Guiting Zhang
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Meng Chen
- Department of Obstetrics and Gynecology, and Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, West China Second University Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Guolin He
- Department of Obstetrics and Gynecology, and Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, West China Second University Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Wen Wang
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Xinghui Liu
- Department of Obstetrics and Gynecology, and Key Laboratory of Birth Defects and Related Diseases of Women and Children (Sichuan University), Ministry of Education, West China Second University Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Xin Sun
- Chinese Evidence-based Medicine Center and National Clinical Research Center for Geriatrics, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China.
| |
Collapse
|
15
|
Debray TP, de Jong VM, Moons KG, Riley RD. Evidence synthesis in prognosis research. Diagn Progn Res 2019; 3:13. [PMID: 31338426 PMCID: PMC6621956 DOI: 10.1186/s41512-019-0059-4] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Accepted: 04/16/2019] [Indexed: 12/11/2022] Open
Abstract
Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.
Collapse
Affiliation(s)
- Thomas P.A. Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Valentijn M.T. de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Karel G.M. Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Richard D. Riley
- Research Institute for Primary Care & Health Sciences, Keele University, Staffordshire, ST5 5BG UK
| |
Collapse
|
16
|
|
17
|
Winn AN, Neuner JM. Making Sure We Don't Forget the Basics When Using Machine Learning. J Natl Cancer Inst 2019; 111:529-530. [PMID: 30346555 DOI: 10.1093/jnci/djy179] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2018] [Accepted: 09/06/2018] [Indexed: 11/13/2022] Open
Affiliation(s)
- Aaron N Winn
- Department of Clinical Sciences, School of Pharmacy
| | - Joan M Neuner
- Department of Medicine and Center for Advancing Population Science, Medical College of Wisconsin, Milwaukee, WI
| |
Collapse
|
18
|
Luijken K, Groenwold RHH, Van Calster B, Steyerberg EW, van Smeden M. Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective. Stat Med 2019; 38:3444-3459. [PMID: 31148207 PMCID: PMC6619392 DOI: 10.1002/sim.8183] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2018] [Revised: 02/02/2019] [Accepted: 04/08/2019] [Indexed: 12/23/2022]
Abstract
It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.
Collapse
Affiliation(s)
- K Luijken
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - R H H Groenwold
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands.,Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - B Van Calster
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.,Department of Development and Regeneration, University of Leuven, Leuven, Belgium
| | - E W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.,Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - M van Smeden
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| |
Collapse
|