1
|
Keogh RH, Van Geloven N. Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data. Epidemiology 2024; 35:329-339. [PMID: 38630508 PMCID: PMC11332371 DOI: 10.1097/ede.0000000000001713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2023] [Accepted: 01/10/2024] [Indexed: 04/19/2024]
Abstract
Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.
Collapse
Affiliation(s)
- Ruth H. Keogh
- From the Department of Medical Statistics, London School of Hygiene & Tropical Medicine, London, United Kingdom
| | - Nan Van Geloven
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| |
Collapse
|
2
|
Medina-Inojosa JR, Somers VK, Garcia M, Thomas RJ, Allison T, Chaudry R, Wood-Wentz CM, Bailey KR, Mulvagh SL, Lopez-Jimenez F. Performance of the ACC/AHA Pooled Cohort Cardiovascular Risk Equations in Clinical Practice. J Am Coll Cardiol 2023; 82:1499-1508. [PMID: 37793746 DOI: 10.1016/j.jacc.2023.07.018] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2023] [Revised: 05/11/2023] [Accepted: 07/19/2023] [Indexed: 10/06/2023]
Abstract
BACKGROUND The performance of the American College of Cardiology/American Heart Association pooled cohort equation (PCE) for atherosclerotic cardiovascular disease (ASCVD) in real-world clinical practice has not been evaluated extensively. OBJECTIVES The goal of this study was to test the performance of PCE to predict ASCVD risk in the community, and determine if including individuals with values outside the PCE range (ie, age, blood pressure, cholesterol) or statin therapy initiation over follow-up would significantly affect PCE predictive capabilities. METHODS The PCE was validated in a community-based cohort of consecutive patients who sought primary care in Olmsted County, Minnesota, between 1997 and 2000, followed-up through 2016. Inclusion criteria were similar to those of PCE derivation. Patient information was ascertained by using the record linkage system of the Rochester Epidemiology Project. ASCVD events (nonfatal and fatal myocardial infarction and ischemic stroke) were validated in duplicate. Calculated and observed ASCVD risk and c-statistics were compared across predefined groups. RESULTS This study included 30,042 adults, with a mean age of 48.5 ± 12.2 years; 46% were male. Median follow-up was 16.5 years, truncated at 10 years for this analysis. Mean ASCVD risk was 5.6% ± 8.73%. There were 1,555 ASCVD events (5.2%). The PCE revealed good performance overall (c-statistic 0.78) and in sex and race subgroups; it was highest among non-White female subjects (c-statistic 0.81) and lowest in White male subjects (c-statistic 0.77). Out-of-range values and initiation of statin medication did not affect model performance. CONCLUSIONS The PCE performed well in a community cohort representing real-world clinical practice. Values outside PCE ranges and initiation of statin medication did not affect performance. These results have implications for the applicability of current strategies for the prevention of ASCVD.
Collapse
Affiliation(s)
- Jose R Medina-Inojosa
- Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, Minnesota, USA; Division of Epidemiology, Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota, USA
| | - Virend K Somers
- Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Mariana Garcia
- Emory Clinical Cardiovascular Research Institute, Division of Cardiology, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Randal J Thomas
- Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Thomas Allison
- Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Rajeev Chaudry
- Department of Medicine and Division of Preventive Cardiology, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Christina M Wood-Wentz
- Department of Medicine and Division of Preventive Cardiology, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Kent R Bailey
- Department of Medicine and Division of Preventive Cardiology, Mayo Clinic Rochester, Rochester, Minnesota, USA
| | - Sharon L Mulvagh
- Department of Cardiovascular Medicine, Mayo Clinic Rochester, Rochester, Minnesota, USA; Department of Medicine, Division of Cardiology, Dalhousie University, Halifax, Nova Scotia, Canada
| | | |
Collapse
|
3
|
de Jong VMT, Hoogland J, Moons KGM, Riley RD, Nguyen TL, Debray TPA. Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population. Stat Med 2023; 42:3508-3528. [PMID: 37311563 DOI: 10.1002/sim.9817] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2021] [Revised: 02/26/2023] [Accepted: 05/19/2023] [Indexed: 06/15/2023]
Abstract
External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.
Collapse
Affiliation(s)
- Valentijn M T de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Data Analytics and Methods Task Force, European Medicines Agency, Amsterdam, The Netherlands
| | - Jeroen Hoogland
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Tri-Long Nguyen
- Section of Epidemiology, Department of Public Health, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Smart Data Analysis and Statistics, Utrecht, The Netherlands
| |
Collapse
|
4
|
Damen JA, Moons KG, van Smeden M, Hooft L. How to conduct a systematic review and meta-analysis of prognostic model studies. Clin Microbiol Infect 2022; 29:434-440. [PMID: 35934199 PMCID: PMC9351211 DOI: 10.1016/j.cmi.2022.07.019] [Citation(s) in RCA: 25] [Impact Index Per Article: 12.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 07/05/2022] [Accepted: 07/19/2022] [Indexed: 11/03/2022]
Abstract
BACKGROUND Prognostic models are typically developed to estimate the risk that an individual in a particular health state will develop a particular health outcome, to support (shared) decision making. Systematic reviews of prognostic model studies can help identify prognostic models that need to further be validated or are ready to be implemented in healthcare. OBJECTIVES To provide a step-by-step guidance on how to conduct and read a systematic review of prognostic model studies and to provide an overview of methodology and guidance available for every step of the review progress. SOURCES Published, peer-reviewed guidance articles. CONTENT We describe the following steps for conducting a systematic review of prognosis studies: 1) Developing the review question using the Population, Index model, Comparator model, Outcome(s), Timing, Setting format, 2) Searching and selection of articles, 3) Data extraction using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist, 4) Quality and risk of bias assessment using the Prediction model Risk Of Bias ASsessment (PROBAST) tool, 5) Analysing data and undertaking quantitative meta-analysis, and 6) Presenting summary of findings, interpreting results, and drawing conclusions. Guidance for each step is described and illustrated using a case study on prognostic models for patients with COVID-19. IMPLICATIONS Guidance for conducting a systematic review of prognosis studies is available, but the implications of these reviews for clinical practice and further research highly depend on complete reporting of primary studies.
Collapse
|
5
|
van der Endt VHW, Milders J, Penning de Vries BBL, Trines SA, Groenwold RHH, Dekkers OM, Trevisan M, Carrero JJ, van Diepen M, Dekker FW, de Jong Y. Comprehensive comparison of stroke risk score performance: a systematic review and meta-analysis among 6 267 728 patients with atrial fibrillation. Europace 2022; 24:1739-1753. [PMID: 35894866 PMCID: PMC9681133 DOI: 10.1093/europace/euac096] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 05/24/2022] [Indexed: 12/15/2022] Open
Abstract
AIMS Multiple risk scores to predict ischaemic stroke (IS) in patients with atrial fibrillation (AF) have been developed. This study aims to systematically review these scores, their validations and updates, assess their methodological quality, and calculate pooled estimates of the predictive performance. METHODS AND RESULTS We searched PubMed and Web of Science for studies developing, validating, or updating risk scores for IS in AF patients. Methodological quality was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). To assess discrimination, pooled c-statistics were calculated using random-effects meta-analysis. We identified 19 scores, which were validated and updated once or more in 70 and 40 studies, respectively, including 329 validations and 76 updates-nearly all on the CHA2DS2-VASc and CHADS2. Pooled c-statistics were calculated among 6 267 728 patients and 359 373 events of IS. For the CHA2DS2-VASc and CHADS2, pooled c-statistics were 0.644 [95% confidence interval (CI) 0.635-0.653] and 0.658 (0.644-0.672), respectively. Better discriminatory abilities were found in the newer risk scores, with the modified-CHADS2 demonstrating the best discrimination [c-statistic 0.715 (0.674-0.754)]. Updates were found for the CHA2DS2-VASc and CHADS2 only, showing improved discrimination. Calibration was reasonable but available for only 17 studies. The PROBAST indicated a risk of methodological bias in all studies. CONCLUSION Nineteen risk scores and 76 updates are available to predict IS in patients with AF. The guideline-endorsed CHA2DS2-VASc shows inferior discriminative abilities compared with newer scores. Additional external validations and data on calibration are required before considering the newer scores in clinical practice. CLINICAL TRIAL REGISTRATION ID CRD4202161247 (PROSPERO).
Collapse
Affiliation(s)
| | - Jet Milders
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Bas B L Penning de Vries
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Serge A Trines
- Department of Cardiology, Willem Einthoven Center of Arrhythmia Research and Management, Leiden University Medical Center, 2333 ZA Leiden, The Netherlands
| | - Rolf H H Groenwold
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Olaf M Dekkers
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Marco Trevisan
- Department of Medical Epidemiology and Biostatistics (MEB), Karolinska Institutet, 171 77 Stockholm, Sweden
| | - Juan J Carrero
- Department of Medical Epidemiology and Biostatistics (MEB), Karolinska Institutet, 171 77 Stockholm, Sweden
| | - Merel van Diepen
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Friedo W Dekker
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands
| | - Ype de Jong
- Department of Clinical Epidemiology, Leiden University Medical Center, PO Box 9600, 2333 ZA Leiden, The Netherlands,Department of Internal Medicine, Leiden University Medical Center, 2333 ZA Leiden, The Netherlands
| |
Collapse
|
6
|
Angelow A, Klötzer C, Donner-Banzhoff N, Haasenritter J, Oliver Schmidt C, Dörr M, Chenot JF. Validation of Cardiovascular Risk Prediction by the Arriba Instrument. DEUTSCHES ARZTEBLATT INTERNATIONAL 2022; 119:476-482. [PMID: 35635438 PMCID: PMC9664993 DOI: 10.3238/arztebl.m2022.0220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/21/2021] [Revised: 12/21/2021] [Accepted: 05/04/2022] [Indexed: 01/04/2023]
Abstract
BACKGROUND It is recommended in cardiovascular prevention guidelines that treatment should be based on overall cardiovascular risk. The arriba instrument has been widely used for this purpose in Germany. The aim of this study is to validate risk prediction by arriba with the aid of morbidity and mortality data from the population- based Study of Health in Pomerania. METHODS In a longitudinal analysis, the arriba instrument was used to calculate the 10-year overall cardiovascular risk at baseline for subjects who had not sustained any prior cardiovascular event. Cardiovascular event rates were determined from follow-up data, and discrimination and calibration measures for the risk determination algorithm were calculated. RESULTS Data from 1973 subjects (mean age 51 ± 13 years, 48% men) were included in the analysis. After a median follow-up of 10.9 years, cardiovascular events had occurred in 196 subjects, or 9.8%. The ratio of predicted to observed event rate was 0.8 (95% confidence interval: [0.5; 1.1]), 1.3 [1.0; 1.8], and 1.1 [0.8; 1.4] for subjects at low, intermediate, and high cardiovascular risk, respectively. Arriba underestimated cardiovascular event rates in women and overestimated them in persons aged 30-44 and 45-59. The area under curve was 0.84 [95% CI 0.81; 0.86]. CONCLUSION The discrimination scores of the arriba instrument resemble those of SCORE-Germany and PROCAM, but a better adjustment to the target population would be desirable. The results support the recommendation of the German Guideline for Cardiovascular Risk Counseling in General Practice for the use of the arriba instrument. An unresolved problem is the failure to consider intervention effects, resulting in an overall mild overestimation of risk.
Collapse
Affiliation(s)
- Aniela Angelow
- University Hospital of Greifswald, Department of General Practice, Institute for Community Medicine, Greifswald,*Universitätsmedizin Greifswald Abteilung Allgemeinmedizin Institut für Community Medicine Fleischmannstraße 6, 17475 Greifswald
| | - Christine Klötzer
- University Hospital of Greifswald, Department of General Practice, Institute for Community Medicine, Greifswald
| | | | - Jörg Haasenritter
- Philipps-University Marburg, Department of General Practice, Marburg
| | - Carsten Oliver Schmidt
- University Hospital of Greifswald, Department SHIP/KEF, Institute for Community Medicine, Greifswald
| | - Marcus Dörr
- University Hospital of Greifswald, Department of Internal Medicine B, German Centre for Cardiovascular Research, Greifswald,German Centre for Cardiovascular Research e. V. (DZHK), Standort Greifswald
| | - Jean-François Chenot
- University Hospital of Greifswald, Department of General Practice, Institute for Community Medicine, Greifswald
| |
Collapse
|
7
|
Dickerman BA, Dahabreh IJ, Cantos KV, Logan RW, Lodi S, Rentsch CT, Justice AC, Hernán MA. Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV. Eur J Epidemiol 2022; 37:367-376. [PMID: 35190946 PMCID: PMC9189026 DOI: 10.1007/s10654-022-00855-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2021] [Accepted: 02/14/2022] [Indexed: 12/23/2022]
Abstract
The accuracy of a prediction algorithm depends on contextual factors that may vary across deployment settings. To address this inherent limitation of prediction, we propose an approach to counterfactual prediction based on the g-formula to predict risk across populations that differ in their distribution of treatment strategies. We apply this to predict 5-year risk of mortality among persons receiving care for HIV in the U.S. Veterans Health Administration under different hypothetical treatment strategies. First, we implement a conventional approach to develop a prediction algorithm in the observed data and show how the algorithm may fail when transported to new populations with different treatment strategies. Second, we generate counterfactual data under different treatment strategies and use it to assess the robustness of the original algorithm's performance to these differences and to develop counterfactual prediction algorithms. We discuss how estimating counterfactual risks under a particular treatment strategy is more challenging than conventional prediction as it requires the same data, methods, and unverifiable assumptions as causal inference. However, this may be required when the alternative assumption of constant treatment patterns across deployment settings is unlikely to hold and new data is not yet available to retrain the algorithm.
Collapse
Affiliation(s)
- Barbra A Dickerman
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
| | - Issa J Dahabreh
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | | | - Roger W Logan
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Sara Lodi
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Biostatistics, Boston University School of Public Health, Boston, MA, USA
| | - Christopher T Rentsch
- Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK
- Veterans Affairs Connecticut Healthcare System, West Haven, CT, USA
- Department of Internal Medicine, Yale School of Medicine, New Haven, CT, USA
| | - Amy C Justice
- Veterans Affairs Connecticut Healthcare System, West Haven, CT, USA
- Department of Internal Medicine, Yale School of Medicine, New Haven, CT, USA
- Center for Interdisciplinary Research on AIDS, Yale School of Public Health, New Haven, CT, USA
| | - Miguel A Hernán
- CAUSALab, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
- Harvard-MIT Division of Health Sciences and Technology, Boston, MA, USA
| |
Collapse
|
8
|
Housseine N, Rijken MJ, Weller K, Nassor NH, Gbenga K, Dodd C, Debray T, Meguid T, Franx A, Grobbee DE, Browne JL. Development of a clinical prediction model for perinatal deaths in low resource settings. EClinicalMedicine 2022; 44:101288. [PMID: 35252826 PMCID: PMC8888338 DOI: 10.1016/j.eclinm.2022.101288] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/04/2021] [Revised: 12/19/2021] [Accepted: 01/17/2022] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Most pregnancy-related deaths in low and middle income countries occur around the time of birth and are avoidable with timely care. This study aimed to develop a prognostic model to identify women at risk of intrapartum-related perinatal deaths in low-resourced settings, by (1) external validation of an existing prediction model, and subsequently (2) development of a novel model. METHODS A prospective cohort study was conducted among pregnant women who presented consecutively for delivery at the maternity unit of Zanzibar's tertiary hospital, Mnazi Mmoja Hospital, the Republic of Tanzania between October 2017 and May 2018. Candidate predictors of perinatal deaths included maternal and foetal characteristics obtained from routine history and physical examination at the time of admission to the labour ward. The outcomes were intrapartum stillbirths and neonatal death before hospital discharge. An existing stillbirth prediction model with six predictors from Nigeria was applied to the Zanzibar cohort to assess its discrimination and calibration performance. Subsequently, a new prediction model was developed using multivariable logistic regression. Model performance was evaluated through internal validation and corrected for overfitting using bootstrapping methods. FINDINGS 5747 mother-baby pairs were analysed. The existing model showed poor discrimination performance (c-statistic 0·57). The new model included 15 clinical predictors and showed promising discriminative and calibration performance after internal validation (optimism adjusted c-statistic of 0·78, optimism adjusted calibration slope =0·94). INTERPRETATION The new model consisted of predictors easily obtained through history-taking and physical examination at the time of admission to the labour ward. It had good performance in predicting risk of perinatal death in women admitted in labour wards. Therefore, it has the potential to assist skilled birth attendance to triage women for appropriate management during labour. Before routine implementation, external validation and usefulness should be determined in future studies. FUNDING The study received funding from Laerdal Foundation, Otto Kranendonk Fund and UMC Global Health Fellowship. TD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050).
Collapse
Affiliation(s)
- Natasha Housseine
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
- Division of Woman and Baby, University Medical Centre Utrecht, The Netherlands
- Department of Obstetrics and Gynaecology, Mnazi Mmoja Hospital, Zanzibar, Tanzania
- Corresponding author: Natasha Housseine, Julius Center for Health Sciences and Primary Care, UMC Utrecht, Postal address: Huispost nr 1. STR 6.131, P.O. Box 85500, 3508 GA Utrecht, The Netherlands, Telephone number: +255 745 338950.
| | - Marcus J Rijken
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
- Division of Woman and Baby, University Medical Centre Utrecht, The Netherlands
| | - Katinka Weller
- Department of Obstetrics and Gynaecology, Erasmus MC University Medical Centre Rotterdam, The Netherlands
| | | | - Kayode Gbenga
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
| | - Caitlin Dodd
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
| | - Thomas Debray
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
| | - Tarek Meguid
- Department of Obstetrics and Gynaecology, Mnazi Mmoja Hospital, Zanzibar, Tanzania
- School of Health and Medical Sciences, State University of Zanzibar
- Village Health Works, Kigutu, Burundi
| | - Arie Franx
- Department of Obstetrics and Gynaecology, Erasmus MC University Medical Centre Rotterdam, The Netherlands
| | - Diederick E Grobbee
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
| | - Joyce L Browne
- Julius Global Health, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, The Netherlands
| |
Collapse
|
9
|
Sperrin M, Diaz-Ordaz K, Pajouheshnia R. Invited Commentary: Treatment Drop-in-Making the Case for Causal Prediction. Am J Epidemiol 2021; 190:2015-2018. [PMID: 33595073 PMCID: PMC8485150 DOI: 10.1093/aje/kwab030] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Revised: 01/25/2021] [Accepted: 02/02/2021] [Indexed: 11/12/2022] Open
Abstract
Clinical prediction models (CPMs) are often used to guide treatment initiation, with individuals at high risk offered treatment. This implicitly assumes that the probability quoted from a CPM represents the risk to an individual of an adverse outcome in absence of treatment. However, for a CPM to correctly target this estimand requires careful causal thinking. One problem that needs to be overcome is treatment drop-in: where individuals in the development data commence treatment after the time of prediction but before the outcome occurs. In this issue of the Journal, Xu et al. (Am J Epidemiol. 2021;190(10):2000-2014) use causal estimates from external data sources, such as clinical trials, to adjust CPMs for treatment drop-in. This represents a pragmatic and promising approach to address this issue, and it illustrates the value of utilizing causal inference in prediction. Building causality into the prediction pipeline can also bring other benefits. These include the ability to make and compare hypothetical predictions under different interventions, to make CPMs more explainable and transparent, and to improve model generalizability. Enriching CPMs with causal inference therefore has the potential to add considerable value to the role of prediction in healthcare.
Collapse
Affiliation(s)
- Matthew Sperrin
- Correspondence to Matthew Sperrin, Vaughan House, Portsmouth Street, University of Manchester, Manchester M13 9GB, UK (e-mail: )
| | | | | |
Collapse
|
10
|
Xu Z, Arnold M, Stevens D, Kaptoge S, Pennells L, Sweeting MJ, Barrett J, Di Angelantonio E, Wood AM. Prediction of Cardiovascular Disease Risk Accounting for Future Initiation of Statin Treatment. Am J Epidemiol 2021; 190:2000-2014. [PMID: 33595074 PMCID: PMC8485151 DOI: 10.1093/aje/kwab031] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Revised: 12/17/2020] [Accepted: 12/18/2020] [Indexed: 12/15/2022] Open
Abstract
Cardiovascular disease (CVD) risk-prediction models are used to identify high-risk individuals and guide statin initiation. However, these models are usually derived from individuals who might initiate statins during follow-up. We present a simple approach to address statin initiation to predict “statin-naive” CVD risk. We analyzed primary care data (2004–2017) from the UK Clinical Practice Research Datalink for 1,678,727 individuals (aged 40–85 years) without CVD or statin treatment history at study entry. We derived age- and sex-specific prediction models including conventional risk factors and a time-dependent effect of statin initiation constrained to 25% risk reduction (from trial results). We compared predictive performance and measures of public-health impact (e.g., number needed to screen to prevent 1 event) against models ignoring statin initiation. During a median follow-up of 8.9 years, 103,163 individuals developed CVD. In models accounting for (versus ignoring) statin initiation, 10-year CVD risk predictions were slightly higher; predictive performance was moderately improved. However, few individuals were reclassified to a high-risk threshold, resulting in negligible improvements in number needed to screen to prevent 1 event. In conclusion, incorporating statin effects from trial results into risk-prediction models enables statin-naive CVD risk estimation and provides moderate gains in predictive ability but had a limited impact on treatment decision-making under current guidelines in this population.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | - Angela M Wood
- Correspondence to Dr. Angela M. Wood, Department of Public Health and Primary Care, University of Cambridge, Strangeways Research Laboratory, Cambridge CB1 8RN, United Kingdom (e-mail: )
| |
Collapse
|
11
|
Moriarty AS, Paton LW, Snell KIE, Riley RD, Buckman JEJ, Gilbody S, Chew-Graham CA, Ali S, Pilling S, Meader N, Phillips B, Coventry PA, Delgadillo J, Richards DA, Salisbury C, McMillan D. The development and validation of a prognostic model to PREDICT Relapse of depression in adult patients in primary care: protocol for the PREDICTR study. Diagn Progn Res 2021; 5:12. [PMID: 34215317 PMCID: PMC8254312 DOI: 10.1186/s41512-021-00101-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/14/2021] [Accepted: 05/19/2021] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Most patients who present with depression are treated in primary care by general practitioners (GPs). Relapse of depression is common (at least 50% of patients treated for depression will relapse after a single episode) and leads to considerable morbidity and decreased quality of life for patients. The majority of patients will relapse within 6 months, and those with a history of relapse are more likely to relapse in the future than those with no such history. GPs see a largely undifferentiated case-mix of patients, and once patients with depression reach remission, there is limited guidance to help GPs stratify patients according to risk of relapse. We aim to develop a prognostic model to predict an individual's risk of relapse within 6-8 months of entering remission. The long-term objective is to inform the clinical management of depression after the acute phase. METHODS We will develop a prognostic model using secondary analysis of individual participant data drawn from seven RCTs and one longitudinal cohort study in primary or community care settings. We will use logistic regression to predict the outcome of relapse of depression within 6-8 months. We plan to include the following established relapse predictors in the model: residual depressive symptoms, number of previous depressive episodes, co-morbid anxiety and severity of index episode. We will use a "full model" development approach, including all available predictors. Performance statistics (optimism-adjusted C-statistic, calibration-in-the-large, calibration slope) and calibration plots (with smoothed calibration curves) will be calculated. Generalisability of predictive performance will be assessed through internal-external cross-validation. Clinical utility will be explored through net benefit analysis. DISCUSSION We will derive a statistical model to predict relapse of depression in remitted depressed patients in primary care. Assuming the model has sufficient predictive performance, we outline the next steps including independent external validation and further assessment of clinical utility and impact. STUDY REGISTRATION ClinicalTrials.gov ID: NCT04666662.
Collapse
Affiliation(s)
- Andrew S Moriarty
- Department of Health Sciences, University of York, York, England.
- Hull York Medical School, University of York, York, England.
| | - Lewis W Paton
- Department of Health Sciences, University of York, York, England
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, England
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, England
| | - Joshua E J Buckman
- Centre for Outcomes and Research Effectiveness, Research Department of Clinical, Educational and Health Psychology, University College London, London, England
- iCope - Camden and Islington Psychological Therapies Services, Camden & Islington NHS Foundation Trust, London, England
| | - Simon Gilbody
- Department of Health Sciences, University of York, York, England
- Hull York Medical School, University of York, York, England
| | | | - Shehzad Ali
- Department of Health Sciences, University of York, York, England
- Department of Epidemiology and Biostatistics, Schulich School of Medicine & Dentistry, Western University, London, ON, Canada
| | - Stephen Pilling
- Centre for Outcomes and Research Effectiveness, Research Department of Clinical, Educational and Health Psychology, University College London, London, England
- Camden & Islington NHS Foundation Trust, St Pancras Hospital, London, England
| | - Nick Meader
- Centre for Reviews and Dissemination, University of York, York, England
| | - Bob Phillips
- Centre for Reviews and Dissemination, University of York, York, England
| | - Peter A Coventry
- Department of Health Sciences, University of York, York, England
| | - Jaime Delgadillo
- Department of Psychology, University of Sheffield, Sheffield, England
| | - David A Richards
- Institute of Health Research, College of Medicine and Health, University of Exeter, Exeter, England
- Department of Health and Caring Sciences, Western Norway University of Applied Sciences, Inndalsveien 28, 5063 Bergen, Norway, USA
| | - Chris Salisbury
- Centre for Academic Primary Care, University of Bristol, Bristol, England
| | - Dean McMillan
- Department of Health Sciences, University of York, York, England
- Hull York Medical School, University of York, York, England
| |
Collapse
|
12
|
Uffen JW, van Goor H, Reitsma J, Oosterheert JJ, de Regt M, Kaasjager K. Retrospective study on the possible existence of a treatment paradox in sepsis scores in the emergency department. BMJ Open 2021; 11:e046518. [PMID: 33707275 PMCID: PMC7957128 DOI: 10.1136/bmjopen-2020-046518] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVE The quick Sequential Organ Failure Assessment (qSOFA) is developed as a tool to identify patients with infection with increased risk of dying from sepsis in non-intensive care unit settings, like the emergency department (ED). An abnormal score may trigger the initiation of appropriate therapy to reduce that risk. This study assesses the risk of a treatment paradox: the effect of a strong predictor for mortality will be reduced if that predictor also acts as a trigger for initiating treatment to prevent mortality. DESIGN Retrospective analysis on data from a large observational cohort. SETTING ED of a tertiary medical centre in the Netherlands. PARTICIPANTS 3178 consecutive patients with suspected infection. PRIMARY OUTCOME To evaluate the existence of a treatment paradox by determining the influence of baseline qSOFA on treatment decisions within the first 24 hours after admission. RESULTS 226 (7.1%) had a qSOFA ≥2, of which 51 (22.6%) died within 30 days. Area under receiver operating characteristics of qSOFA for 30-day mortality was 0.68 (95% CI 0.61 to 0.75). Patients with a qSOFA ≥2 had higher odds of receiving any form of intensive therapy (OR 11.4 (95% CI 7.5 to 17.1)), such as aggressive fluid resuscitation (OR 8.8 95% CI 6.6 to 11.8), fast antibiotic administration (OR 8.5, 95% CI 5.7 to 12.3) or vasopressic therapy (OR 17.3, 95% CI 11.2 to 26.8), compared with patients with qSOFA <2. CONCLUSION In ED patients with suspected infection, a qSOFA ≥2 was associated with more intensive treatment. This could lead to inadequate prediction of 30-day mortality due to the presence of a treatment paradox. TRIAL REGISTRATION NUMBER 6916.
Collapse
Affiliation(s)
- Jan Willem Uffen
- Department of Internal Medicine and Acute Medicine, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Harriet van Goor
- Department of Internal Medicine and Acute Medicine, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Johannes Reitsma
- Department of Epidemiology, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Jan Jelrik Oosterheert
- Department of Internal Medicine and Infectious Diseases, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Marieke de Regt
- Department of Internal Medicine, Onze Lieve Vrouwe Gasthuis, Amsterdam, Noord-Holland, The Netherlands
| | - Karin Kaasjager
- Department of Internal Medicine and Acute Medicine, University Medical Centre Utrecht, Utrecht, The Netherlands
| |
Collapse
|
13
|
Lin L, Sperrin M, Jenkins DA, Martin GP, Peek N. A scoping review of causal methods enabling predictions under hypothetical interventions. Diagn Progn Res 2021; 5:3. [PMID: 33536082 PMCID: PMC7860039 DOI: 10.1186/s41512-021-00092-9] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/21/2020] [Accepted: 01/02/2021] [Indexed: 02/07/2023] Open
Abstract
BACKGROUND The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions. AIMS We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges. METHODS We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies. RESULTS We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation. CONCLUSIONS There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.
Collapse
Affiliation(s)
- Lijing Lin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - David A Jenkins
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- NIHR Greater Manchester Patient Safety Translational Research Centre, The University of Manchester, Manchester, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Niels Peek
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- NIHR Greater Manchester Patient Safety Translational Research Centre, The University of Manchester, Manchester, UK
- NIHR Manchester Biomedical Research Centre, The University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| |
Collapse
|
14
|
Dickerman BA, Hernán MA. Counterfactual prediction is not only for causal inference. Eur J Epidemiol 2020; 35:615-617. [PMID: 32623620 DOI: 10.1007/s10654-020-00659-8] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Affiliation(s)
- Barbra A Dickerman
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA.
| | - Miguel A Hernán
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA.,Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA.,Harvard-MIT Division of Health Sciences and Technology, Boston, MA, USA
| |
Collapse
|
15
|
van Geloven N, Swanson SA, Ramspek CL, Luijken K, van Diepen M, Morris TP, Groenwold RHH, van Houwelingen HC, Putter H, le Cessie S. Prediction meets causal inference: the role of treatment in clinical prediction models. Eur J Epidemiol 2020; 35:619-630. [PMID: 32445007 PMCID: PMC7387325 DOI: 10.1007/s10654-020-00636-1] [Citation(s) in RCA: 51] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2020] [Accepted: 04/18/2020] [Indexed: 11/29/2022]
Abstract
In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.
Collapse
Affiliation(s)
- Nan van Geloven
- Department of Biomedical Data Sciences, Leiden University Medical Center, Zone S5-P, PO Box 9600, 2300 RC, Leiden, The Netherlands.
| | - Sonja A Swanson
- Department of Epidemiology, Erasmus MC, Rotterdam, The Netherlands
- Department of Epidemiology, Harvard T. H. Chan School of Public Health, Boston, USA
| | - Chava L Ramspek
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Kim Luijken
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Merel van Diepen
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Tim P Morris
- MRC Clinical Trials Unit, UCL London, London, UK
| | - Rolf H H Groenwold
- Department of Biomedical Data Sciences, Leiden University Medical Center, Zone S5-P, PO Box 9600, 2300 RC, Leiden, The Netherlands
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| | - Hans C van Houwelingen
- Department of Biomedical Data Sciences, Leiden University Medical Center, Zone S5-P, PO Box 9600, 2300 RC, Leiden, The Netherlands
| | - Hein Putter
- Department of Biomedical Data Sciences, Leiden University Medical Center, Zone S5-P, PO Box 9600, 2300 RC, Leiden, The Netherlands
| | - Saskia le Cessie
- Department of Biomedical Data Sciences, Leiden University Medical Center, Zone S5-P, PO Box 9600, 2300 RC, Leiden, The Netherlands
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, The Netherlands
| |
Collapse
|
16
|
Geersing GJ, Hendriksen JMT, Zuithoff NPA, Roes KC, Oudega R, Takada T, Schutgens REG, Moons KGM. Effect of tailoring anticoagulant treatment duration by applying a recurrence risk prediction model in patients with venous thromboembolism compared to usual care: A randomized controlled trial. PLoS Med 2020; 17:e1003142. [PMID: 32589630 PMCID: PMC7319277 DOI: 10.1371/journal.pmed.1003142] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/06/2020] [Accepted: 06/03/2020] [Indexed: 11/19/2022] Open
Abstract
BACKGROUND Patients with unprovoked (i.e., without the presence of apparent transient risk factors such as recent surgery) venous thromboembolism (VTE) are at risk of recurrence if anticoagulants are stopped after 3-6 months, yet their risk remains heterogeneous. Thus, prolonging anticoagulant treatment should be considered in high-risk patients, whereas stopping is likely preferred in those with a low predicted risk. The Vienna Prediction Model (VPM) could aid clinicians in estimating this risk, yet its clinical effects and external validity are currently unknown. The aim of this study was to investigate the clinical impact of this model on reducing recurrence risk in patients with unprovoked VTE, compared to usual care. METHODS AND FINDINGS In a randomized controlled trial, the decision to prolong or stop anticoagulant treatment was guided by predicted recurrence risk using the VPM (n = 441), which was compared with usual care (n = 442). Patients with unprovoked VTE were recruited from local thrombosis services in the Netherlands (in Utrecht, Harderwijk, Ede, Amersfoort, Zwolle, Hilversum, Rotterdam, Deventer, and Enschede) between 22 July 2011 and 30 November 2015, with 24-month follow-up complete for all patients by early 2018. The primary outcome was recurrent VTE during 24 months of follow-up. Secondary outcomes included major bleeding and clinically relevant non-major (CRNM) bleeding. In the total study population of 883 patients, mean age was 55 years, and 507 (57.4%) were men. A total of 96 recurrent VTE events (10.9%) were observed, 46 in the intervention arm and 50 in the control arm (risk ratio 0.92, 95% CI 0.63-1.35, p = 0.67). Major bleeding occurred in 4 patients, 2 in each treatment arm, whereas CRNM bleeding occurred in 20 patients (12 in intervention arm versus 8 in control arm). The VPM showed good discriminative power (c-statistic 0.76, 95% CI 0.69-0.83) and moderate to good calibration, notably at the lower spectrum of predicted risk. For instance, in 284 patients with a predicted risk of >2% to 4%, the observed rate of recurrence was 2.5% (95% CI 0.7% to 4.3%). The main limitation of this study is that it did not enroll the preplanned number of 750 patients in each study arm due to declining recruitment rate. CONCLUSIONS Our results show that application of the VPM in all patients with unprovoked VTE is unlikely to reduce overall recurrence risk. Yet, in those with a low predicted risk of recurrence, the observed rate was also low, suggesting that it might be safe to stop anticoagulant treatment in these patients. TRIAL REGISTRATION Netherlands Trial Register NTR2680.
Collapse
Affiliation(s)
- Geert-Jan Geersing
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
- * E-mail:
| | - Janneke M. T. Hendriksen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Nicolaas P. A. Zuithoff
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Kit C. Roes
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
- Biostatistics Research Group, Department of Health Evidence, Radboud University Medical Center, Radboud University, Nijmegen, the Netherlands
| | - Ruud Oudega
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Toshihiko Takada
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Roger E. G. Schutgens
- Van Creveld Clinic, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Karel G. M. Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| |
Collapse
|
17
|
van Bussel EF, Hoevenaar-Blom MP, Poortvliet RKE, Gussekloo J, van Dalen JW, van Gool WA, Richard E, Moll van Charante EP. Predictive value of traditional risk factors for cardiovascular disease in older people: A systematic review. Prev Med 2020; 132:105986. [PMID: 31958478 DOI: 10.1016/j.ypmed.2020.105986] [Citation(s) in RCA: 26] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/13/2019] [Revised: 01/08/2020] [Accepted: 01/12/2020] [Indexed: 01/21/2023]
Abstract
With increasing age, associations between traditional risk factors (TRFs) and cardiovascular disease (CVD) shift. It is unknown which mid-life risk factors remain relevant predictors for CVD in older people. We systematically searched PubMed and EMBASE on August 16th 2019 for studies assessing predictive ability of >1 of fourteen TRFs for fatal and non-fatal CVD, in the general population aged 60+. We included 12 studies, comprising 11 unique cohorts. TRF were evaluated in 2 to 11 cohorts, and retained in 0-70% of the cohorts: age (70%), diabetes (64%), male sex (57%), systolic blood pressure (SBP) (50%), smoking (36%), high-density lipoprotein cholesterol (HDL) (33%), left ventricular hypertrophy (LVH) (33%), total cholesterol (22%), diastolic blood pressure (20%), antihypertensive medication use (AHM) (20%), body mass index (BMI) (0%), hypertension (0%), low-density lipoprotein cholesterol (0%). In studies with low to moderate risk of bias, systolic blood pressure (SBP) (80%), smoking (80%) and HDL cholesterol (60%) were more often retained. Model performance was moderate with C-statistics ranging from 0.61 to 0.77. Compared to middle-aged adults, in people aged 60+ different risk factors predict CVD and current prediction models perform only moderate at best. According to most studies, age, sex and diabetes seem valuable predictors of CVD in old-age. SBP, HDL cholesterol and smoking may also have predictive value. Other blood pressure and cholesterol related variables, BMI, and LVH seem of very limited or no additional value. Without competing risk analysis, predictors are overestimated.
Collapse
Affiliation(s)
- E F van Bussel
- Department of General Practice, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100DD Amsterdam, the Netherlands.
| | - M P Hoevenaar-Blom
- Department of Neurology, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100 DD Amsterdam, the Netherlands; Department of Neurology, Donders Centre for Brain, Behaviour and Cognition, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands.
| | - R K E Poortvliet
- Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, the Netherlands.
| | - J Gussekloo
- Department of Public Health and Primary Care, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, the Netherlands; Department of Gerontology and Geriatrics, Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, Netherlands.
| | - J W van Dalen
- Department of Neurology, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100 DD Amsterdam, the Netherlands; Department of Neurology, Donders Centre for Brain, Behaviour and Cognition, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands.
| | - W A van Gool
- Department of Neurology, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100 DD Amsterdam, the Netherlands.
| | - E Richard
- Department of Neurology, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100 DD Amsterdam, the Netherlands; Department of Neurology, Donders Centre for Brain, Behaviour and Cognition, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands.
| | - E P Moll van Charante
- Department of General Practice, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1100DD Amsterdam, the Netherlands.
| |
Collapse
|
18
|
Groenwold RHH. Informative missingness in electronic health record systems: the curse of knowing. Diagn Progn Res 2020; 4:8. [PMID: 32699824 PMCID: PMC7371469 DOI: 10.1186/s41512-020-00077-0] [Citation(s) in RCA: 46] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/07/2020] [Accepted: 04/22/2020] [Indexed: 12/17/2022] Open
Abstract
Electronic health records provide a potentially valuable data source of information for developing clinical prediction models. However, missing data are common in routinely collected health data and often missingness is informative. Informative missingness can be incorporated in a clinical prediction model, for example by including a separate category of a predictor variable that has missing values. The predictive performance of such a model depends on the transportability of the missing data mechanism, which may be compromised once the model is deployed in practice and the predictive value of certain variables becomes known. Using synthetic data, this phenomenon is explained and illustrated.
Collapse
Affiliation(s)
- Rolf H. H. Groenwold
- grid.10419.3d0000000089452978Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, the Netherlands
- grid.10419.3d0000000089452978Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, the Netherlands
| |
Collapse
|
19
|
Pajouheshnia R, Schuster NA, Groenwold RHH, Rutten FH, Moons KGM, Peelen LM. Accounting for time‐dependent treatment use when developing a prognostic model from observational data: A review of methods. STAT NEERL 2019. [DOI: 10.1111/stan.12193] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Affiliation(s)
- Romin Pajouheshnia
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht Utrecht The Netherlands
- Division of Pharmacoepidemiology and Clinical PharmacologyUtrecht Institute for Pharmaceutical Sciences, Utrecht University Utrecht The Netherlands
| | - Noah A. Schuster
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht Utrecht The Netherlands
| | - Rolf H. H. Groenwold
- Department of Clinical EpidemiologyLeiden University Medical Center Leiden The Netherlands
| | - Frans H. Rutten
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht Utrecht The Netherlands
| | - Karel G. M. Moons
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht Utrecht The Netherlands
| | - Linda M. Peelen
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht Utrecht The Netherlands
| |
Collapse
|
20
|
Multidimensional screening for predicting pain problems in adults: a systematic review of screening tools and validation studies. Pain Rep 2019; 4:e775. [PMID: 31875182 PMCID: PMC6882575 DOI: 10.1097/pr9.0000000000000775] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2019] [Revised: 06/11/2019] [Accepted: 06/26/2019] [Indexed: 12/14/2022] Open
Abstract
Screening tools allowing to predict poor pain outcomes are widely used. Often these screening tools contain psychosocial risk factors. This review (1) identifies multidimensional screening tools that include psychosocial risk factors for the development or maintenance of pain, pain-related distress, and pain-related disability across pain problems in adults, (2) evaluates the quality of the validation studies using Prediction model Risk Of Bias ASsessment Tool (PROBAST), and (3) synthesizes methodological concerns. We identified 32 articles, across 42 study samples, validating 7 screening tools. All tools were developed in the context of musculoskeletal pain, most often back pain, and aimed to predict the maintenance of pain or pain-related disability, not pain-related distress. Although more recent studies design, conduct, analyze, and report according to best practices in prognosis research, risk of bias was most often moderate. Common methodological concerns were identified, related to participant selection (eg, mixed populations), predictors (eg, predictors were administered differently to predictors in the development study), outcomes (eg, overlap between predictors and outcomes), sample size and participant flow (eg, unknown or inappropriate handling of missing data), and analysis (eg, wide variety of performance measures). Recommendations for future research are provided.
Collapse
|
21
|
van Bussel EF, Richard E, Busschers WB, Steyerberg EW, van Gool WA, Moll van Charante EP, Hoevenaar-Blom MP. A cardiovascular risk prediction model for older people: Development and validation in a primary care population. J Clin Hypertens (Greenwich) 2019; 21:1145-1152. [PMID: 31294917 PMCID: PMC6772108 DOI: 10.1111/jch.13617] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2019] [Revised: 04/18/2019] [Accepted: 05/03/2019] [Indexed: 12/01/2022]
Abstract
Cardiovascular risk prediction is mainly based on traditional risk factors that have been validated in middle‐aged populations. However, associations between these risk factors and cardiovascular disease (CVD) attenuate with increasing age. Therefore, for older people the authors developed and internally validated risk prediction models for fatal and non‐fatal CVD, (re)evaluated the predictive value of traditional and new factors, and assessed the impact of competing risks of non‐cardiovascular death. Post hoc analyses of 1811 persons aged 70‐78 year and free from CVD at baseline from the preDIVA study (Prevention of Dementia by Intensive Vascular care, 2006‐2015), a primary care‐based trial that included persons free from dementia and conditions likely to hinder successful long‐term follow‐up, were performed. In 2017‐2018, Cox‐regression analyses were performed for a model including seven traditional risk factors only, and a model to assess incremental predictive ability of the traditional and eleven new factors. Analyses were repeated accounting for competing risk of death, using Fine‐Gray models. During an average of 6.2 years of follow‐up, 277 CVD events occurred. Age, sex, smoking, and type 2 diabetes mellitus were traditional predictors for CVD, whereas total cholesterol, HDL‐cholesterol, and systolic blood pressure (SBP) were not. Of the eleven new factors, polypharmacy and apathy symptoms were predictors. Discrimination was moderate (concordance statistic 0.65). Accounting for competing risks resulted in slightly smaller predicted absolute risks. In conclusion, we found, SBP, HDL, and total cholesterol no longer predict CVD in older adults, whereas polypharmacy and apathy symptoms are two new relevant predictors. Building on the selected risk factors in this study may improve CVD prediction in older adults and facilitate targeting preventive interventions to those at high risk.
Collapse
Affiliation(s)
- Emma F van Bussel
- Department of General Practice, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Edo Richard
- Department of Neurology, Donderds Centre for Brain, Behaviour and Cognition, Radboud University Medical Center, Nijmegen, The Netherlands.,Department of Neurology, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Wim B Busschers
- Department of General Practice, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, LUMC, Leiden, The Netherlands.,Department of Public Health, Erasmus MC, Rotterdam, The Netherlands
| | - Willem A van Gool
- Department of Neurology, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Eric P Moll van Charante
- Department of General Practice, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Marieke P Hoevenaar-Blom
- Department of Neurology, Donderds Centre for Brain, Behaviour and Cognition, Radboud University Medical Center, Nijmegen, The Netherlands.,Department of Neurology, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
22
|
Damen JA, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten RJPM, Hooft L, Debray TPA. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med 2019; 17:109. [PMID: 31189462 PMCID: PMC6563379 DOI: 10.1186/s12916-019-1340-7] [Citation(s) in RCA: 123] [Impact Index Per Article: 24.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Accepted: 05/07/2019] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND The Framingham risk models and pooled cohort equations (PCE) are widely used and advocated in guidelines for predicting 10-year risk of developing coronary heart disease (CHD) and cardiovascular disease (CVD) in the general population. Over the past few decades, these models have been extensively validated within different populations, which provided mounting evidence that local tailoring is often necessary to obtain accurate predictions. The objective is to systematically review and summarize the predictive performance of three widely advocated cardiovascular risk prediction models (Framingham Wilson 1998, Framingham ATP III 2002 and PCE 2013) in men and women separately, to assess the generalizability of performance across different subgroups and geographical regions, and to determine sources of heterogeneity in the findings across studies. METHODS A search was performed in October 2017 to identify studies investigating the predictive performance of the aforementioned models. Studies were included if they externally validated one or more of the original models in the general population for the same outcome as the original model. We assessed risk of bias for each validation and extracted data on population characteristics and model performance. Performance estimates (observed versus expected (OE) ratio and c-statistic) were summarized using a random effects models and sources of heterogeneity were explored with meta-regression. RESULTS The search identified 1585 studies, of which 38 were included, describing a total of 112 external validations. Results indicate that, on average, all models overestimate the 10-year risk of CHD and CVD (pooled OE ratio ranged from 0.58 (95% CI 0.43-0.73; Wilson men) to 0.79 (95% CI 0.60-0.97; ATP III women)). Overestimation was most pronounced for high-risk individuals and European populations. Further, discriminative performance was better in women for all models. There was considerable heterogeneity in the c-statistic between studies, likely due to differences in population characteristics. CONCLUSIONS The Framingham Wilson, ATP III and PCE discriminate comparably well but all overestimate the risk of developing CVD, especially in higher risk populations. Because the extent of miscalibration substantially varied across settings, we highly recommend that researchers further explore reasons for overprediction and that the models be updated for specific populations.
Collapse
Affiliation(s)
- Johanna A Damen
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. .,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands.
| | - Romin Pajouheshnia
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Pauline Heus
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Karel G M Moons
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Johannes B Reitsma
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Rob J P M Scholten
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Lotty Hooft
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Thomas P A Debray
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| |
Collapse
|
23
|
Pajouheshnia R, Groenwold RHH, Peelen LM, Reitsma JB, Moons KGM. When and how to use data from randomised trials to develop or validate prognostic models. BMJ 2019; 365:l2154. [PMID: 31142454 DOI: 10.1136/bmj.l2154] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Affiliation(s)
- Romin Pajouheshnia
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, Netherlands
| | - Rolf H H Groenwold
- Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
| | - Linda M Peelen
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
| | - Johannes B Reitsma
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
- Cochrane Netherlands, Utrecht, Netherlands
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, 3508 GA Utrecht, Netherlands
- Cochrane Netherlands, Utrecht, Netherlands
| |
Collapse
|
24
|
Sperrin M, Martin GP, Pate A, Van Staa T, Peek N, Buchan I. Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models. Stat Med 2018; 37:4142-4154. [PMID: 30073700 PMCID: PMC6282523 DOI: 10.1002/sim.7913] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2017] [Revised: 05/31/2018] [Accepted: 06/25/2018] [Indexed: 01/19/2023]
Abstract
Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are usually derived using data sets where patients received treatment, often initiated postbaseline as "treatment drop-ins." This study proposes the use of marginal structural models (MSMs) to adjust for treatment drop-in. We illustrate the use of MSMs in the CPM framework through simulation studies that represent randomized controlled trials and real-world observational data and the example of statin initiation for cardiovascular disease prevention. The simulations include a binary treatment and a covariate, each recorded at two timepoints and having a prognostic effect on a binary outcome. The bias in predicted risk was examined in a model ignoring treatment, a model fitted on treatment-naïve patients (at baseline), a model including baseline treatment, and the MSM. In all simulation scenarios, all models except the MSM underestimated the risk of outcome given absence of treatment. These results were supported in the statin initiation example, which showed that ignoring statin initiation postbaseline resulted in models that significantly underestimated the risk of a cardiovascular disease event occurring within 10 years. Consequently, CPMs that do not acknowledge treatment drop-in can lead to underallocation of treatment. In conclusion, when developing CPMs to predict treatment-naïve risk, researchers should consider using MSMs to adjust for treatment drop-in, and also seek to exploit the ability of MSMs to allow estimation of individual treatment effects.
Collapse
Affiliation(s)
- Matthew Sperrin
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
| | - Glen P. Martin
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
| | - Alexander Pate
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
| | - Tjeerd Van Staa
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
| | - Niels Peek
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
| | - Iain Buchan
- Farr Institute, Faculty of Biology, Medicine and HealthUniversity of Manchester, Manchester Academic Health Science CentreManchesterUK
- Microsoft ResearchCambridgeUK
| |
Collapse
|
25
|
Meertens LJE, van Montfort P, Scheepers HCJ, van Kuijk SMJ, Aardenburg R, Langenveld J, van Dooren IMA, Zwaan IM, Spaanderman MEA, Smits LJM. Prediction models for the risk of spontaneous preterm birth based on maternal characteristics: a systematic review and independent external validation. Acta Obstet Gynecol Scand 2018; 97:907-920. [PMID: 29663314 PMCID: PMC6099449 DOI: 10.1111/aogs.13358] [Citation(s) in RCA: 44] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Accepted: 04/10/2018] [Indexed: 01/01/2023]
Abstract
Introduction Prediction models may contribute to personalized risk‐based management of women at high risk of spontaneous preterm delivery. Although prediction models are published frequently, often with promising results, external validation generally is lacking. We performed a systematic review of prediction models for the risk of spontaneous preterm birth based on routine clinical parameters. Additionally, we externally validated and evaluated the clinical potential of the models. Material and methods Prediction models based on routinely collected maternal parameters obtainable during first 16 weeks of gestation were eligible for selection. Risk of bias was assessed according to the CHARMS guidelines. We validated the selected models in a Dutch multicenter prospective cohort study comprising 2614 unselected pregnant women. Information on predictors was obtained by a web‐based questionnaire. Predictive performance of the models was quantified by the area under the receiver operating characteristic curve (AUC) and calibration plots for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation. Clinical value was evaluated by means of decision curve analysis and calculating classification accuracy for different risk thresholds. Results Four studies describing five prediction models fulfilled the eligibility criteria. Risk of bias assessment revealed a moderate to high risk of bias in three studies. The AUC of the models ranged from 0.54 to 0.67 and from 0.56 to 0.70 for the outcomes spontaneous preterm birth <37 weeks and <34 weeks of gestation, respectively. A subanalysis showed that the models discriminated poorly (AUC 0.51–0.56) for nulliparous women. Although we recalibrated the models, two models retained evidence of overfitting. The decision curve analysis showed low clinical benefit for the best performing models. Conclusions This review revealed several reporting and methodological shortcomings of published prediction models for spontaneous preterm birth. Our external validation study indicated that none of the models had the ability to predict spontaneous preterm birth adequately in our population. Further improvement of prediction models, using recent knowledge about both model development and potential risk factors, is necessary to provide an added value in personalized risk assessment of spontaneous preterm birth.
Collapse
Affiliation(s)
- Linda J E Meertens
- Department of Epidemiology, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands
| | - Pim van Montfort
- Department of Epidemiology, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands
| | - Hubertina C J Scheepers
- Department of Obstetrics and Gynecology, School for Oncology and Developmental Biology (GROW), Maastricht University Medical Center, Maastricht, The Netherlands
| | - Sander M J van Kuijk
- Department of Clinical Epidemiology and Medical Technology Assessment (KEMTA), Care and Public Health Research Institute (CAPHRI), Maastricht University Medical Center, Maastricht, The Netherlands
| | - Robert Aardenburg
- Department of Obstetrics and Gynecology, Zuyderland Medical Center, Heerlen, The Netherlands
| | - Josje Langenveld
- Department of Obstetrics and Gynecology, Zuyderland Medical Center, Heerlen, The Netherlands
| | - Ivo M A van Dooren
- Department of Obstetrics and Gynecology, Sint Jans Gasthuis Weert, Weert, The Netherlands
| | - Iris M Zwaan
- Department of Obstetrics and Gynecology, Laurentius Hospital, Roermond, The Netherlands
| | - Marc E A Spaanderman
- Department of Obstetrics and Gynecology, School for Oncology and Developmental Biology (GROW), Maastricht University Medical Center, Maastricht, The Netherlands
| | - Luc J M Smits
- Department of Epidemiology, Care and Public Health Research Institute (CAPHRI), Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
26
|
Pajouheshnia R, Damen JAAG, Groenwold RHH, Moons KGM, Peelen LM. Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies. Diagn Progn Res 2017; 1:15. [PMID: 31093544 PMCID: PMC6460846 DOI: 10.1186/s41512-017-0015-0] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/30/2017] [Accepted: 09/14/2017] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Ignoring treatments in prognostic model development or validation can affect the accuracy and transportability of models. We aim to quantify the extent to which the effects of treatment have been addressed in existing prognostic model research and provide recommendations for the handling and reporting of treatment use in future studies. METHODS We first describe how and when the use of treatments by individuals in a prognostic study can influence the development or validation of a prognostic model. We subsequently conducted a systematic review of the handling and reporting of treatment use in prognostic model studies in cardiovascular medicine. Data on treatment use (e.g. medications, surgeries, lifestyle interventions), the timing of their use, and the handling of such treatment use in the analyses were extracted and summarised. RESULTS Three hundred two articles were included in the review. Treatment use was not mentioned in 91 (30%) articles. One hundred forty-six (48%) reported specific information about treatment use in their studies; 78 (26%) provided information about multiple treatments. Three articles (1%) reported changes in medication use ("treatment drop-in") during follow-up. Seventy-nine articles (26%) excluded treated individuals from their analysis, 80 articles (26%) modelled treatment as an outcome, and of the 155 articles that developed a model, 86 (55%) modelled treatment use, almost exclusively at baseline, as a predictor. CONCLUSIONS The use of treatments has been partly considered by the majority of CVD prognostic model studies. Detailed accounts including, for example, information on treatment drop-in were rare. Where relevant, the use of treatments should be considered in the analysis of prognostic model studies, particularly when a prognostic model is designed to guide the use of certain treatments and these treatments have been used by the study participants. Future prognostic model studies should clearly report the use of treatments by study participants and consider the potential impact of treatment use on the study findings.
Collapse
Affiliation(s)
- Romin Pajouheshnia
- 0000000090126352grid.7692.aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
| | - Johanna A. A. G. Damen
- 0000000090126352grid.7692.aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
- 0000000090126352grid.7692.aCochrane Netherlands, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
| | - Rolf H. H. Groenwold
- 0000000090126352grid.7692.aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
| | - Karel G. M. Moons
- 0000000090126352grid.7692.aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
- 0000000090126352grid.7692.aCochrane Netherlands, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
| | - Linda M. Peelen
- 0000000090126352grid.7692.aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Huispost Str.6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands
| |
Collapse
|