1
|
Schut MC, Dongelmans DA, de Lange DW, Brinkman S, de Keizer NF, Abu-Hanna A. Development and evaluation of regression tree models for predicting in-hospital mortality of a national registry of COVID-19 patients over six pandemic surges. BMC Med Inform Decis Mak 2024; 24:7. [PMID: 38166918 PMCID: PMC10762959 DOI: 10.1186/s12911-023-02401-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Accepted: 12/11/2023] [Indexed: 01/05/2024] Open
Abstract
BACKGROUND Objective prognostic information is essential for good clinical decision making. In case of unknown diseases, scarcity of evidence and limited tacit knowledge prevent obtaining this information. Prediction models can be useful, but need to be not only evaluated on how well they predict, but also how stable these models are under fast changing circumstances with respect to development of the disease and the corresponding clinical response. This study aims to provide interpretable and actionable insights, particularly for clinicians. We developed and evaluated two regression tree predictive models for in-hospital mortality of COVID-19 patient at admission and 24 hours (24 h) after admission, using a national registry. We performed a retrospective analysis of observational routinely collected data. METHODS Two regression tree models were developed for admission and 24 h after admission. The complexity of the trees was managed via cross validation to prevent overfitting. The predictive ability of the model was assessed via bootstrapping using the Area under the Receiver-Operating-Characteristic curve, Brier score and calibration curves. The tree models were assessed on the stability of their probabilities and predictive ability, on the selected variables, and compared to a full-fledged logistic regression model that uses variable selection and variable transformations using splines. Participants included COVID-19 patients from all ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry, who were admitted at the ICU between February 27, 2020, and November 23, 2021. From the NICE registry, we included concerned demographic data, minimum and maximum values of physiological data in the first 24 h of ICU admission and diagnoses (reason for admission as well as comorbidities) for model development. The main outcome measure was in-hospital mortality. We additionally analysed the Length-of-Stay (LoS) per patient subgroup per survival status. RESULTS A total of 13,369 confirmed COVID-19 patients from 70 ICUs were included (with mortality rate of 28%). The optimism-corrected AUROC of the admission tree (with seven paths) was 0.72 (95% CI: 0.71-0.74) and of the 24 h tree (with 11 paths) was 0.74 (0.74-0.77). Both regression trees yielded good calibration and variable selection for both trees was stable. Patient subgroups comprising the tree paths had comparable survival probabilities as the full-fledged logistic regression model, survival probabilities were stable over six COVID-19 surges, and subgroups were shown to have added predictive value over the individual patient variables. CONCLUSIONS We developed and evaluated regression trees, which operate at par with a carefully crafted logistic regression model. The trees consist of homogenous subgroups of patients that are described by simple interpretable constraints on patient characteristics thereby facilitating shared decision-making.
Collapse
Affiliation(s)
- M C Schut
- Department of Medical Informatics, Amsterdam Public Health research institute, Amsterdam UMC location University of Amsterdam, Meibergdreef 9, 1105, AZ, Amsterdam, The Netherlands.
- Department of Laboratory Medicine, Amsterdam UMC location Vrije Universiteit, De Boelelaan 1117, 1081, HV, Amsterdam, The Netherlands.
| | - D A Dongelmans
- Department of Intensive Care Medicine, Amsterdam UMC location University of Amsterdam, Meibergdreef 9, 1105, AZ, Amsterdam, The Netherlands
| | - D W de Lange
- Department of Intensive Care Medicine and Dutch Poisons Information Center (DPIC), University Medical Center Utrecht, University Utrecht, Heidelberglaan 100, 3584, CX, Utrecht, The Netherlands
| | - S Brinkman
- Department of Medical Informatics, Amsterdam Public Health research institute, Amsterdam UMC location University of Amsterdam, Meibergdreef 9, 1105, AZ, Amsterdam, The Netherlands
| | - N F de Keizer
- Department of Medical Informatics, Amsterdam Public Health research institute, Amsterdam UMC location University of Amsterdam, Meibergdreef 9, 1105, AZ, Amsterdam, The Netherlands
| | - A Abu-Hanna
- Department of Medical Informatics, Amsterdam Public Health research institute, Amsterdam UMC location University of Amsterdam, Meibergdreef 9, 1105, AZ, Amsterdam, The Netherlands
| |
Collapse
|
2
|
Vagliano I, Dormosh N, Rios M, Luik TT, Buonocore TM, Elbers PWG, Dongelmans DA, Schut MC, Abu-Hanna A. Prognostic models of in-hospital mortality of intensive care patients using neural representation of unstructured text: A systematic review and critical appraisal. J Biomed Inform 2023; 146:104504. [PMID: 37742782 DOI: 10.1016/j.jbi.2023.104504] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 08/29/2023] [Accepted: 09/21/2023] [Indexed: 09/26/2023]
Abstract
OBJECTIVE To review and critically appraise published and preprint reports of prognostic models of in-hospital mortality of patients in the intensive-care unit (ICU) based on neural representations (embeddings) of clinical notes. METHODS PubMed and arXiv were searched up to August 1, 2022. At least two reviewers independently selected the studies that developed a prognostic model of in-hospital mortality of intensive-care patients using free-text represented as embeddings and extracted data using the CHARMS checklist. Risk of bias was assessed using PROBAST. Reporting on the model was assessed with the TRIPOD guideline. To assess the machine learning components that were used in the models, we present a new descriptive framework based on different techniques to represent text and provide predictions from text. The study protocol was registered in the PROSPERO database (CRD42022354602). RESULTS Eighteen studies out of 2,825 were included. All studies used the publicly-available MIMIC dataset. Context-independent word embeddings are widely used. Model discrimination was provided by all studies (AUROC 0.75-0.96), but measures of calibration were scarce. Seven studies used both structural clinical variables and notes. Model discrimination improved when adding clinical notes to variables. None of the models was externally validated and often a simple train/test split was used for internal validation. Our critical appraisal demonstrated a high risk of bias in all studies and concerns regarding their applicability in clinical practice. CONCLUSION All studies used a neural architecture for prediction and were based on one publicly available dataset. Clinical notes were reported to improve predictive performance when used in addition to only clinical variables. Most studies had methodological, reporting, and applicability issues. We recommend reporting both model discrimination and calibration, using additional data sources, and using more robust evaluation strategies, including prospective and external validation. Finally, sharing data and code is encouraged to improve study reproducibility.
Collapse
Affiliation(s)
- I Vagliano
- Dept. of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health (APH), Amsterdam, the Netherlands.
| | - N Dormosh
- Dept. of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health (APH), Amsterdam, the Netherlands
| | - M Rios
- Centre for Translation Studies, University of Vienna, Vienna, Austria. https://twitter.com/zhizhid
| | - T T Luik
- Amsterdam Public Health (APH), Amsterdam, the Netherlands; Dept. of Medical Biology, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - T M Buonocore
- Dept. of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy
| | - P W G Elbers
- Amsterdam Public Health (APH), Amsterdam, the Netherlands; Dept. of Intensive Care Medicine, Center for Critical Care Computational Intelligence (C4I), Amsterdam Medical Data Science (AMDS), Amsterdam Institute for Infection and Immunity (AII), Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands. https://twitter.com/zhizhid
| | - D A Dongelmans
- Amsterdam Public Health (APH), Amsterdam, the Netherlands; National Intensive Care Evaluation (NICE) Foundation, Amsterdam, the Netherlands; Dept. of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
| | - M C Schut
- Dept. of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health (APH), Amsterdam, the Netherlands; Dept. of Clinical Chemistry, Amsterdam UMC, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
| | - A Abu-Hanna
- Dept. of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health (APH), Amsterdam, the Netherlands
| |
Collapse
|
3
|
Vagliano I, Schut MC, Abu-Hanna A, Dongelmans DA, de Lange DW, Gommers D, Cremer OL, Bosman RJ, Rigter S, Wils EJ, Frenzel T, de Jong R, Peters MAA, Kamps MJA, Ramnarain D, Nowitzky R, Nooteboom FGCA, de Ruijter W, Urlings-Strop LC, Smit EGM, Mehagnoul-Schipper DJ, Dormans T, de Jager CPC, Hendriks SHA, Achterberg S, Oostdijk E, Reidinga AC, Festen-Spanjer B, Brunnekreef GB, Cornet AD, van den Tempel W, Boelens AD, Koetsier P, Lens J, Faber HJ, Karakus A, Entjes R, de Jong P, Rettig TCD, Reuland MC, Arbous S, Fleuren LM, Dam TA, Thoral PJ, Lalisang RCA, Tonutti M, de Bruin DP, Elbers PWG, de Keizer NF. Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records. Int J Med Inform 2022; 167:104863. [PMID: 36162166 PMCID: PMC9492397 DOI: 10.1016/j.ijmedinf.2022.104863] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 08/19/2022] [Accepted: 09/03/2022] [Indexed: 11/17/2022]
Abstract
PURPOSE To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. METHODS Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. RESULTS A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. CONCLUSION In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.
Collapse
Affiliation(s)
- Iacopo Vagliano
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands.
| | - Martijn C Schut
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | - Ameen Abu-Hanna
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | - Dave A Dongelmans
- National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, The Netherlands
| | - Dylan W de Lange
- National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands; Department of Intensive Care Medicine, University Medical Center Utrecht, University Utrecht, Utrecht, The Netherlands
| | - Diederik Gommers
- Department of Intensive Care, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Olaf L Cremer
- Intensive Care, UMC Utrecht, Utrecht, The Netherlands
| | | | - Sander Rigter
- Department of Anesthesiology and Intensive Care, St. Antonius Hospital, Nieuwegein, The Netherlands
| | - Evert-Jan Wils
- Department of Intensive Care, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Tim Frenzel
- Department of Intensive Care Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Remko de Jong
- Intensive Care, Bovenij Ziekenhuis, Amsterdam, The Netherlands
| | - Marco A A Peters
- Intensive Care, Canisius Wilhelmina Ziekenhuis, Nijmegen, The Netherlands
| | - Marlijn J A Kamps
- Intensive Care, Catharina Ziekenhuis Eindhoven, Eindhoven, The Netherlands
| | | | - Ralph Nowitzky
- Intensive Care, Haga Ziekenhuis, Den Haag, The Netherlands
| | | | - Wouter de Ruijter
- Department of Intensive Care Medicine, Northwest Clinics, Alkmaar, The Netherlands
| | | | - Ellen G M Smit
- Intensive Care, Spaarne Gasthuis, Haarlem en Hoofddorp, The Netherlands
| | | | - Tom Dormans
- Intensive care, Zuyderland MC, Heerlen, The Netherlands
| | | | | | | | | | - Auke C Reidinga
- ICU, SEH, BWC, Martiniziekenhuis, Groningen, The Netherlands
| | | | - Gert B Brunnekreef
- Department of Intensive Care, Ziekenhuisgroep Twente, Almelo, The Netherlands
| | - Alexander D Cornet
- Department of Intensive Care, Medisch Spectrum Twente, Enschede, The Netherlands
| | - Walter van den Tempel
- Department of Intensive Care, Ikazia Ziekenhuis Rotterdam, Rotterdam, The Netherlands
| | - Age D Boelens
- Anesthesiology, Antonius Ziekenhuis Sneek, Sneek, The Netherlands
| | - Peter Koetsier
- Intensive Care, Medisch Centrum Leeuwarden, Leeuwarden, The Netherlands
| | - Judith Lens
- ICU, IJsselland Ziekenhuis, Capelle aan den IJssel, The Netherlands
| | | | - A Karakus
- Department of Intensive Care, Diakonessenhuis Hospital, Utrecht, The Netherlands
| | - Robert Entjes
- Department of Intensive Care, Adrz, Goes, The Netherlands
| | - Paul de Jong
- Department of Anesthesia and Intensive Care, Slingeland Ziekenhuis, Doetinchem, The Netherlands
| | - Thijs C D Rettig
- Department of Anesthesiology, Intensive Care and Pain Medicine, Amphia Ziekenhuis, Breda, The Netherlands
| | - M C Reuland
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands
| | | | - Lucas M Fleuren
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Tariq A Dam
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Patrick J Thoral
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | | | | | | | - Paul W G Elbers
- Department of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam Medical Data Science, Amsterdam UMC, Vrije Universiteit, Amsterdam, The Netherlands
| | - Nicolette F de Keizer
- Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health Research Institute, Amsterdam, The Netherlands; National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands
| | | |
Collapse
|