1
|
Zahra A, van Smeden M, Abbink EJ, van den Berg JM, Blom MT, van den Dries CJ, Gussekloo J, Wouters F, Joling KJ, Melis R, Mooijaart SP, Peters JB, Polinder-Bos HA, van Raaij BFM, Appelman B, la Roi-Teeuw HM, Moons KGM, Luijken K. External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting. J Clin Epidemiol 2024; 168:111270. [PMID: 38311188 DOI: 10.1016/j.jclinepi.2024.111270] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 01/26/2024] [Accepted: 01/26/2024] [Indexed: 02/10/2024]
Abstract
OBJECTIVES To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes. STUDY DESIGN AND SETTING This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation. MAIN OUTCOME MEASURE In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting. RESULTS All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71. CONCLUSION Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.
Collapse
Affiliation(s)
- Anum Zahra
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Evertine J Abbink
- Department of Internal Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jesse M van den Berg
- Department of General Practice, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands; Amsterdam Public Health, Health Behaviors & Chronic Diseases, Amsterdam, The Netherlands; PHARMO Institute for Drug Outcomes Research, Utrecht, The Netherlands
| | - Marieke T Blom
- Department of General Practice, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, The Netherlands; Amsterdam Public Health, Health Behaviors & Chronic Diseases, Amsterdam, The Netherlands
| | - Carline J van den Dries
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Jacobijn Gussekloo
- Section Gerontology and Geriatrics, LUMC Center for Medicine for Older People & Department of Public Health and Primary Care & Department of Internal Medicine, Leiden University Medical Center, Leiden, The Netherlands
| | - Fenne Wouters
- Department of Medicine for Older People, Amsterdam UMC, Location Vrije Universiteit Amsterdam, De Boelelaan 1117, Amsterdam, The Netherlands; Amsterdam Public Health Research Institute, Aging & Later Life, Amsterdam, The Netherlands
| | - Karlijn J Joling
- Department of Medicine for Older People, Amsterdam UMC, Location Vrije Universiteit Amsterdam, De Boelelaan 1117, Amsterdam, The Netherlands; Amsterdam Public Health Research Institute, Aging & Later Life, Amsterdam, The Netherlands
| | - René Melis
- Department of Geriatric Medicine, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Simon P Mooijaart
- LUMC Center for Medicine for Older People, LUMC, Leiden, The Netherlands
| | - Jeannette B Peters
- Department of Pulmonary Diseases, Radboud University Medical Center, Radboud Institute for Health Sciences, Nijmegen, The Netherlands
| | - Harmke A Polinder-Bos
- Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center Rotterdam, The Netherlands
| | - Bas F M van Raaij
- LUMC Center for Medicine for Older People, LUMC, Leiden, The Netherlands
| | - Brent Appelman
- Amsterdam UMC Location University of Amsterdam, Center for Experimental and Molecular Medicine, Amsterdam, The Netherlands
| | - Hannah M la Roi-Teeuw
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Kim Luijken
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
2
|
Alwazzeh MJ, Subbarayalu AV, Abu Ali BM, Alabdulqader R, Alhajri M, Alwarthan SM, AlShehail BM, Raman V, Almuhanna FA. Performance of CURB-65 and ISARIC 4C mortality scores for hospitalized patients with confirmed COVID-19 infection in Saudi Arabia. INFORMATICS IN MEDICINE UNLOCKED 2023; 39:101269. [PMID: 37193544 PMCID: PMC10167802 DOI: 10.1016/j.imu.2023.101269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Revised: 05/06/2023] [Accepted: 05/07/2023] [Indexed: 05/18/2023] Open
Abstract
Background The COVID-19 pandemic continues with new waves that could persist with the arrival of new SARS-CoV-2 variants. Therefore, the availability of validated and effective triage tools is the cornerstone for proper clinical management. Thus, this study aimed to assess the validity of the ISARIC-4C score as a triage tool for hospitalized COVID-19 patients in Saudi Arabia and to compare its performance with the CURB-65 score. Material and methods This retrospective observational cohort study was conducted between March 2020 and May 2021 at KFHU, Saudi Arabia, using 542 confirmed COVID-19 patient data on the variables relevant to the application of the ISARIC-4C mortality score and the CURB-65 score. Chi-square and t-tests were employed to study the significance of the CURB-65 score and the ISARIC-4C score variables considering the ICU requirements and the mortality of COVID-19 hospitalized patients. In addition, logistic regression was used to predict the variables related to COVID-19 mortality. In addition, the diagnostic accuracy of both scores was validated by calculating sensitivities, specificities, positive predictive value, negative predictive value, and Youden's J indices (YJI). Results ROC analysis showed an AUC value of 0.834 [95% CI; 0.800-0.865]) for the CURB-65 score and 0.809 [95% CI; 0.773-0.841]) for the ISARIC-4C score. The sensitivity for CURB-65 and ISARIC-4C is 75% and 85.71%, respectively, while the specificity was 82.31% and 62.66%, respectively. The difference between AUCs was 0.025 (95% [CI; -0.0203-0.0704], p = 0.2795). Conclusion Study results support external validation of the ISARIC-4C score in predicting the mortality risk of hospitalized COVID-19 patients in Saudi Arabia. In addition, the CURB-65 and ISARIC-4C scores showed comparable performance with good consistent discrimination and are suitable for clinical utility as triage tools for hospitalized COVID-19 patients.
Collapse
Affiliation(s)
- Marwan Jabr Alwazzeh
- Infectious Disease Division, Department of Internal Medicine, Faculty of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
- King Fahad Hospital of the University, Al-Khobar, Saudi Arabia
| | - Arun Vijay Subbarayalu
- Quality Studies and Research Unit, Vice Deanship for Quality, Deanship of Quality and Academic Accreditation, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | | | | | - Mashael Alhajri
- Infectious Disease Division, Department of Internal Medicine, Faculty of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
- King Fahad Hospital of the University, Al-Khobar, Saudi Arabia
| | - Sara M Alwarthan
- Infectious Disease Division, Department of Internal Medicine, Faculty of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
- King Fahad Hospital of the University, Al-Khobar, Saudi Arabia
| | - Bashayer M AlShehail
- Pharmacy Practice Department, College of Clinical Pharmacy, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Vinoth Raman
- Statistics Unit, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
| | - Fahd Abdulaziz Almuhanna
- Nephrology Division, Department of Internal Medicine, Faculty of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia
- King Fahad Hospital of the University, Al-Khobar, Saudi Arabia
| |
Collapse
|
3
|
Buttia C, Llanaj E, Raeisi-Dehkordi H, Kastrati L, Amiri M, Meçani R, Taneri PE, Ochoa SAG, Raguindin PF, Wehrli F, Khatami F, Espínola OP, Rojas LZ, de Mortanges AP, Macharia-Nimietz EF, Alijla F, Minder B, Leichtle AB, Lüthi N, Ehrhard S, Que YA, Fernandes LK, Hautz W, Muka T. Prognostic models in COVID-19 infection that predict severity: a systematic review. Eur J Epidemiol 2023; 38:355-372. [PMID: 36840867 PMCID: PMC9958330 DOI: 10.1007/s10654-023-00973-x] [Citation(s) in RCA: 23] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Accepted: 01/28/2023] [Indexed: 02/26/2023]
Abstract
Current evidence on COVID-19 prognostic models is inconsistent and clinical applicability remains controversial. We performed a systematic review to summarize and critically appraise the available studies that have developed, assessed and/or validated prognostic models of COVID-19 predicting health outcomes. We searched six bibliographic databases to identify published articles that investigated univariable and multivariable prognostic models predicting adverse outcomes in adult COVID-19 patients, including intensive care unit (ICU) admission, intubation, high-flow nasal therapy (HFNT), extracorporeal membrane oxygenation (ECMO) and mortality. We identified and assessed 314 eligible articles from more than 40 countries, with 152 of these studies presenting mortality, 66 progression to severe or critical illness, 35 mortality and ICU admission combined, 17 ICU admission only, while the remaining 44 studies reported prediction models for mechanical ventilation (MV) or a combination of multiple outcomes. The sample size of included studies varied from 11 to 7,704,171 participants, with a mean age ranging from 18 to 93 years. There were 353 prognostic models investigated, with area under the curve (AUC) ranging from 0.44 to 0.99. A great proportion of studies (61.5%, 193 out of 314) performed internal or external validation or replication. In 312 (99.4%) studies, prognostic models were reported to be at high risk of bias due to uncertainties and challenges surrounding methodological rigor, sampling, handling of missing data, failure to deal with overfitting and heterogeneous definitions of COVID-19 and severity outcomes. While several clinical prognostic models for COVID-19 have been described in the literature, they are limited in generalizability and/or applicability due to deficiencies in addressing fundamental statistical and methodological concerns. Future large, multi-centric and well-designed prognostic prospective studies are needed to clarify remaining uncertainties.
Collapse
Affiliation(s)
- Chepkoech Buttia
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Emergency Department, Inselspital, Bern University Hospital, University of Bern, Freiburgstrasse 16C, 3010 Bern, Switzerland
- Epistudia, Bern, Switzerland
| | - Erand Llanaj
- Department of Molecular Epidemiology, German Institute of Human Nutrition Potsdam-Rehbrücke, Nuthetal, Germany
- ELKH-DE Public Health Research Group of the Hungarian Academy of Sciences, Department of Public Health and Epidemiology, Faculty of Medicine, University of Debrecen, Debrecen, Hungary
- Epistudia, Bern, Switzerland
- German Center for Diabetes Research (DZD), München-Neuherberg, Germany
| | - Hamidreza Raeisi-Dehkordi
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Lum Kastrati
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Graduate School for Health Sciences, University of Bern, Bern, Switzerland
- Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Mojgan Amiri
- Department of Epidemiology, Erasmus MC University Medical Center, Rotterdam, The Netherlands
| | - Renald Meçani
- Department of Pediatrics, “Mother Teresa” University Hospital Center, Tirana, University of Medicine, Tirana, Albania
- Division of Endocrinology and Diabetology, Department of Internal Medicine, Medical University of Graz, Graz, Austria
| | - Petek Eylul Taneri
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- HRB-Trials Methodology Research Network College of Medicine, Nursing and Health Sciences University of Galway, Galway, Ireland
| | | | - Peter Francis Raguindin
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Swiss Paraplegic Research, Nottwil, Switzerland
- Faculty of Health Sciences, University of Lucerne, Lucerne, Switzerland
| | - Faina Wehrli
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
| | - Farnaz Khatami
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Graduate School for Health Sciences, University of Bern, Bern, Switzerland
- Department of Community Medicine, Tehran University of Medical Sciences, Tehran, Iran
| | - Octavio Pano Espínola
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Department of Preventive Medicine and Public Health, University of Navarre, Pamplona, Spain
- Navarra Institute for Health Research, IdiSNA, Pamplona, Spain
| | - Lyda Z. Rojas
- Research Group and Development of Nursing Knowledge (GIDCEN-FCV), Research Center, Cardiovascular Foundation of Colombia, Floridablanca, Santander, Colombia
| | | | | | - Fadi Alijla
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Graduate School for Health Sciences, University of Bern, Bern, Switzerland
| | - Beatrice Minder
- Public Health and Primary Care Library, University Library of Bern, University of Bern, Bern, Switzerland
| | - Alexander B. Leichtle
- University Institute of Clinical Chemistry, Inselspital, Bern University Hospital, and Center for Artificial Intelligence in Medicine (CAIM), University of Bern, Bern, Switzerland
| | - Nora Lüthi
- Emergency Department, Inselspital, Bern University Hospital, University of Bern, Freiburgstrasse 16C, 3010 Bern, Switzerland
| | - Simone Ehrhard
- Emergency Department, Inselspital, Bern University Hospital, University of Bern, Freiburgstrasse 16C, 3010 Bern, Switzerland
| | - Yok-Ai Que
- Department of Intensive Care Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Laurenz Kopp Fernandes
- Deutsches Herzzentrum Berlin (DHZB), Berlin, Germany
- Charité Universitätsmedizin Berlin, Berlin, Germany
| | - Wolf Hautz
- Emergency Department, Inselspital, Bern University Hospital, University of Bern, Freiburgstrasse 16C, 3010 Bern, Switzerland
| | - Taulant Muka
- Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland
- Epistudia, Bern, Switzerland
| |
Collapse
|