COVID outcome prediction in the emergency department (COPE): using retrospective Dutch hospital data to develop simple and valid models for predicting mortality and need for intensive care unit admission in patients who present at the emergency department with suspected COVID-19

Development and evaluation of regression tree models for predicting in-hospital mortality of a national registry of COVID-19 patients over six pandemic surges

BACKGROUND

Objective prognostic information is essential for good clinical decision making. In case of unknown diseases, scarcity of evidence and limited tacit knowledge prevent obtaining this information. Prediction models can be useful, but need to be not only evaluated on how well they predict, but also how stable these models are under fast changing circumstances with respect to development of the disease and the corresponding clinical response. This study aims to provide interpretable and actionable insights, particularly for clinicians. We developed and evaluated two regression tree predictive models for in-hospital mortality of COVID-19 patient at admission and 24 hours (24 h) after admission, using a national registry. We performed a retrospective analysis of observational routinely collected data.

METHODS

Two regression tree models were developed for admission and 24 h after admission. The complexity of the trees was managed via cross validation to prevent overfitting. The predictive ability of the model was assessed via bootstrapping using the Area under the Receiver-Operating-Characteristic curve, Brier score and calibration curves. The tree models were assessed on the stability of their probabilities and predictive ability, on the selected variables, and compared to a full-fledged logistic regression model that uses variable selection and variable transformations using splines. Participants included COVID-19 patients from all ICUs participating in the Dutch National Intensive Care Evaluation (NICE) registry, who were admitted at the ICU between February 27, 2020, and November 23, 2021. From the NICE registry, we included concerned demographic data, minimum and maximum values of physiological data in the first 24 h of ICU admission and diagnoses (reason for admission as well as comorbidities) for model development. The main outcome measure was in-hospital mortality. We additionally analysed the Length-of-Stay (LoS) per patient subgroup per survival status.

RESULTS

A total of 13,369 confirmed COVID-19 patients from 70 ICUs were included (with mortality rate of 28%). The optimism-corrected AUROC of the admission tree (with seven paths) was 0.72 (95% CI: 0.71-0.74) and of the 24 h tree (with 11 paths) was 0.74 (0.74-0.77). Both regression trees yielded good calibration and variable selection for both trees was stable. Patient subgroups comprising the tree paths had comparable survival probabilities as the full-fledged logistic regression model, survival probabilities were stable over six COVID-19 surges, and subgroups were shown to have added predictive value over the individual patient variables.

CONCLUSIONS

We developed and evaluated regression trees, which operate at par with a carefully crafted logistic regression model. The trees consist of homogenous subgroups of patients that are described by simple interpretable constraints on patient characteristics thereby facilitating shared decision-making.

Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study

INTRODUCTION

Prediction models for identifying emergency department (ED) patients at high risk of poor outcome are often not externally validated. We aimed to perform a head-to-head comparison of the discriminatory performance of several prediction models in a large cohort of ED patients.

METHODS

In this retrospective study, we selected prediction models that aim to predict poor outcome and we included adult medical ED patients. Primary outcome was 31-day mortality, secondary outcomes were 1-day mortality, 7-day mortality, and a composite endpoint of 31-day mortality and admission to intensive care unit (ICU).The discriminatory performance of the prediction models was assessed using an area under the receiver operating characteristic curve (AUC). Finally, the prediction models with the highest performance to predict 31-day mortality were selected to further examine calibration and appropriate clinical cut-off points.

RESULTS

We included 19 prediction models and applied these to 2185 ED patients. Thirty-one-day mortality was 10.6% (231 patients), 1-day mortality was 1.4%, 7-day mortality was 4.4%, and 331 patients (15.1%) met the composite endpoint. The RISE UP and COPE score showed similar and very good discriminatory performance for 31-day mortality (AUC 0.86), 1-day mortality (AUC 0.87), 7-day mortality (AUC 0.86) and for the composite endpoint (AUC 0.81). Both scores were well calibrated. Almost no patients with RISE UP and COPE scores below 5% had an adverse outcome, while those with scores above 20% were at high risk of adverse outcome. Some of the other prediction models (i.e. APACHE II, NEWS, WPSS, MEWS, EWS and SOFA) showed significantly higher discriminatory performance for 1-day and 7-day mortality than for 31-day mortality.

CONCLUSIONS

Head-to-head validation of 19 prediction models in medical ED patients showed that the RISE UP and COPE score outperformed other models regarding 31-day mortality.

Multimodal fine-tuning of clinical language models for predicting COVID-19 outcomes

Clinical prediction models tend only to incorporate structured healthcare data, ignoring information recorded in other data modalities, including free-text clinical notes. Here, we demonstrate how multimodal models that effectively leverage both structured and unstructured data can be developed for predicting COVID-19 outcomes. The models are trained end-to-end using a technique we refer to as multimodal fine-tuning, whereby a pre-trained language model is updated based on both structured and unstructured data. The multimodal models are trained and evaluated using a multicenter cohort of COVID-19 patients encompassing all encounters at the emergency department of six hospitals. Experimental results show that multimodal models, leveraging the notion of multimodal fine-tuning and trained to predict (i) 30-day mortality, (ii) safe discharge and (iii) readmission, outperform unimodal models trained using only structured or unstructured healthcare data on all three outcomes. Sensitivity analyses are performed to better understand how well the multimodal models perform on different patient groups, while an ablation study is conducted to investigate the impact of different types of clinical notes on model performance. We argue that multimodal models that make effective use of routinely collected healthcare data to predict COVID-19 outcomes may facilitate patient management and contribute to the effective use of limited healthcare resources.

Unveiling the Unexpected: Why Doctors Should Look beyond the Lungs when Predicting COVID-19 Mortality

INTRODUCTION

The main objective of this study was to identify the best combination of admission day parameters for predicting COVID-19 mortality in hospitalized patients. Furthermore, we sought to compare the predictive capacity of pulmonary parameters to that of renal parameters for mortality from COVID-19.

METHODS

In this retrospective study, all patients admitted to a tertiary hospital between September 1st, 2020, and December 31st, 2020, who were clinically symptomatic and tested positive for COVID-19, were included. We gathered extensive data on patient admissions, including laboratory results, comorbidities, chest X-ray (CXR) images, and SpO2 levels, to determine their role in predicting mortality. Experienced radiologists evaluated the CXR images and assigned a score from 0 to 18 based on the severity of COVID-19 pneumonia. Further, we categorized patients into two independent groups based on their renal function using the RIFLE and KDIGO criteria to define the acute kidney injury (AKI) and chronic kidney disease (CKD) groups. The first group ("AKI&CKD") was subdivided into six subgroups: normal renal function (A); CKD grade 2+3a (B); AKI-DROP (C); CKD grade 3b (D); AKI-RISE (E); and grade 4 + 5 CKD (F). The second group was based only on estimated glomerular filtration rate (eGFR) at the admission, and thus it was divided into four grades: grade 1, grade 2+3a, grade 3b, and grade 4 + 5.

RESULTS

The cohort comprised 619 patients. Patients who died during hospitalization had a significantly higher mean radiological score compared to those who survived, with a p value <0.01. Moreover, we observed that the risk for mortality was significantly increased as renal function deteriorated, as evidenced by the AKI&CKD and eGFR groups (p < 0.001 for each group). Regarding mortality prediction, the area under the curve (AUC) for renal parameters (AKI&CKD group, eGFR group, and age) was found to be superior to that of pulmonary parameters (age, radiological score, SpO2, CRP, and D-dimer) with an AUC of 0.8068 versus 0.7667. However, when renal and pulmonary parameters were combined, the AUC increased to 0.8813. Optimal parameter combinations for predicting mortality from COVID-19 were identified for three medical settings: Emergency Medical Service (EMS), the Emergency Department, and the Internal Medicine Floor. The AUC for these settings was 0.7874, 0.8614, and 0.8813, respectively.

CONCLUSIONS

Our study demonstrated that selected renal parameters are superior to pulmonary parameters in predicting COVID-19 mortality for patients requiring hospitalization. When combining both renal and pulmonary factors, the predictive ability of mortality significantly improved. Additionally, we identified the optimal combination of factors for mortality prediction in three distinct settings: EMS, Emergency Department, and Internal Medicine Floor.

Prognostic models in COVID-19 infection that predict severity: a systematic review

Current evidence on COVID-19 prognostic models is inconsistent and clinical applicability remains controversial. We performed a systematic review to summarize and critically appraise the available studies that have developed, assessed and/or validated prognostic models of COVID-19 predicting health outcomes. We searched six bibliographic databases to identify published articles that investigated univariable and multivariable prognostic models predicting adverse outcomes in adult COVID-19 patients, including intensive care unit (ICU) admission, intubation, high-flow nasal therapy (HFNT), extracorporeal membrane oxygenation (ECMO) and mortality. We identified and assessed 314 eligible articles from more than 40 countries, with 152 of these studies presenting mortality, 66 progression to severe or critical illness, 35 mortality and ICU admission combined, 17 ICU admission only, while the remaining 44 studies reported prediction models for mechanical ventilation (MV) or a combination of multiple outcomes. The sample size of included studies varied from 11 to 7,704,171 participants, with a mean age ranging from 18 to 93 years. There were 353 prognostic models investigated, with area under the curve (AUC) ranging from 0.44 to 0.99. A great proportion of studies (61.5%, 193 out of 314) performed internal or external validation or replication. In 312 (99.4%) studies, prognostic models were reported to be at high risk of bias due to uncertainties and challenges surrounding methodological rigor, sampling, handling of missing data, failure to deal with overfitting and heterogeneous definitions of COVID-19 and severity outcomes. While several clinical prognostic models for COVID-19 have been described in the literature, they are limited in generalizability and/or applicability due to deficiencies in addressing fundamental statistical and methodological concerns. Future large, multi-centric and well-designed prognostic prospective studies are needed to clarify remaining uncertainties.

US and Dutch Perspectives on the Use of COVID-19 Clinical Prediction Models: Findings from a Qualitative Analysis

INTRODUCTION

Clinical prediction models (CPMs) for coronavirus disease 2019 (COVID-19) may support clinical decision making, treatment, and communication. However, attitudes about using CPMs for COVID-19 decision making are unknown.

METHODS

Online focus groups and interviews were conducted among health care providers, survivors of COVID-19, and surrogates (i.e., loved ones/surrogate decision makers) in the United States and the Netherlands. Semistructured questions explored experiences about clinical decision making in COVID-19 care and facilitators and barriers for implementing CPMs.

RESULTS

In the United States, we conducted 4 online focus groups with 1) providers and 2) surrogates and survivors of COVID-19 between January 2021 and July 2021. In the Netherlands, we conducted 3 focus groups and 4 individual interviews with 1) providers and 2) surrogates and survivors of COVID-19 between May 2021 and July 2021. Providers expressed concern about CPM validity and the belief that patients may interpret CPM predictions as absolute. They described CPMs as potentially useful for resource allocation, triaging, education, and research. Several surrogates and people who had COVID-19 were not given prognostic estimates but believed this information would have supported and influenced their decision making. A limited number of participants felt the data would not have applied to them and that they or their loved ones may not have survived, as poor prognosis may have suggested withdrawal of treatment.

CONCLUSIONS

Many providers had reservations about using CPMs for people with COVID-19 due to concerns about CPM validity and patient-level interpretation of the outcome predictions. However, several people who survived COVID-19 and their surrogates indicated that they would have found this information useful for decision making. Therefore, information provision may be needed to improve provider-level comfort and patient and surrogate understanding of CPMs.

HIGHLIGHTS

While clinical prediction models (CPMs) may provide an objective means of assessing COVID-19 prognosis, provider concerns about CPM validity and the interpretation of CPM predictions may limit their clinical use.Providers felt that CPMs may be most useful for resource allocation, triage, research, or educational purposes for COVID-19.Several survivors of COVID-19 and their surrogates felt that CPMs would have been informative and may have aided them in making COVID-19 treatment decisions, while others felt the data would not have applied to them.

Starting Home Telemonitoring and Oxygen Therapy Directly after Emergency Department Assessment Appears to Be Safe in COVID-19 Patients

BACKGROUND

Since data on the safety and effectiveness of home telemonitoring and oxygen therapy started directly after Emergency Department (ED) assessment in COVID-19 patients are sparse but could have many advantages, we evaluated these parameters in this study.

METHODS

All COVID-19 patients ≥18 years eligible for receiving home telemonitoring (November 2020-February 2022, Albert Schweitzer hospital, the Netherlands) were included: patients started directly after ED assessment (ED group) or after hospital admission (admission group). Safety (number of ED reassessments and hospital readmissions) and effectiveness (number of phone calls, duration of oxygen usage and home telemonitoring) were described in both groups.

RESULTS

278 patients were included (n = 65 ED group, n = 213 admission group). ED group: 23.8% (n = 15) was reassessed, 15.9% (n = 10) was admitted and 7.7% (n = 5) ICU admitted. Admission group: 15.8% (n = 37) was reassessed, 6.5% (n = 14) was readmitted and 2.4% (n = 5) ICU (re)admitted. Ten patients died, of whom 7 due to COVID-19 (1 in ED group; 6 in the admission group). ED group: median duration of oxygen therapy was 9 (IQR 7-13) days; the total duration of home telemonitoring was 14 (IQR 9-18) days. Admission group: duration of oxygen therapy was 10 (IQR 6-16) days; total duration of home telemonitoring was 14 (IQR 10-20) days.

CONCLUSION

it appears to be safe to start home telemonitoring and oxygen therapy directly after ED assessment.

Prognostic models for COVID-19 needed updating to warrant transportability over time and space

BACKGROUND

Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time.

METHODS

We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit.

RESULTS

Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data.

CONCLUSIONS

NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

Rapid systematic review of respiratory rate as a vital sign of clinical deterioration in COVID-19

OBJECTIVES

This meta-analysis aimed to establish a clinical evidence base for respiratory rate (RR) as a single predictor of early-onset COVID-19. The review also looked to determine the practical implementation of mobile respiratory rate measuring devices where information was available.

METHODS

We focused on domestic settings with older adults. Relevant studies were identified through MEDLINE, Embase, and CENTRAL databases. A snowballing method was also used. Articles published from the beginning of the COVID-19 pandemic (2019) until Feb 2022 were selected. Databases were searched for terms related to COVID-19 and respiratory rate measurements in domestic patients.

RESULTS

A total of 2,889 articles were screened for relevant content, of which 60 full-text publications were included. We compared the Odds Ratios and statistically significant results of both vital signs.

CONCLUSION

Multinational studies across dozens of countries have shown respiratory rate to have predictive accuracy in detecting COVID-19 deterioration. However, considerable variability is present in the data, making it harder to be sure about the effects. There is no meaningful difference in data quality in terms of variability (95% CI intervals) between vital signs as predictors of decline in COVID-19 patients. Contextual and economic factors will likely determine the choice of measurement used.