Development and validation of clinical prediction model to estimate the probability of death in hospitalized patients with COVID-19: Insights from a nationwide database

Prognostic models in COVID-19 infection that predict severity: a systematic review

Current evidence on COVID-19 prognostic models is inconsistent and clinical applicability remains controversial. We performed a systematic review to summarize and critically appraise the available studies that have developed, assessed and/or validated prognostic models of COVID-19 predicting health outcomes. We searched six bibliographic databases to identify published articles that investigated univariable and multivariable prognostic models predicting adverse outcomes in adult COVID-19 patients, including intensive care unit (ICU) admission, intubation, high-flow nasal therapy (HFNT), extracorporeal membrane oxygenation (ECMO) and mortality. We identified and assessed 314 eligible articles from more than 40 countries, with 152 of these studies presenting mortality, 66 progression to severe or critical illness, 35 mortality and ICU admission combined, 17 ICU admission only, while the remaining 44 studies reported prediction models for mechanical ventilation (MV) or a combination of multiple outcomes. The sample size of included studies varied from 11 to 7,704,171 participants, with a mean age ranging from 18 to 93 years. There were 353 prognostic models investigated, with area under the curve (AUC) ranging from 0.44 to 0.99. A great proportion of studies (61.5%, 193 out of 314) performed internal or external validation or replication. In 312 (99.4%) studies, prognostic models were reported to be at high risk of bias due to uncertainties and challenges surrounding methodological rigor, sampling, handling of missing data, failure to deal with overfitting and heterogeneous definitions of COVID-19 and severity outcomes. While several clinical prognostic models for COVID-19 have been described in the literature, they are limited in generalizability and/or applicability due to deficiencies in addressing fundamental statistical and methodological concerns. Future large, multi-centric and well-designed prognostic prospective studies are needed to clarify remaining uncertainties.

Development of a Risk Score for AKI onset in COVID-19 Patients: COV-AKI Score

PURPOSE

Acute Kidney Injury (AKI) in COVID-19 patients is associated with increased morbidity and mortality. In the present study, we aimed to develop a prognostic score to predict AKI development in these patients.

MATERIALS AND METHODS

This was a retrospective observational study of 2334 COVID 19 patients admitted to 23 different hospitals in Brazil, between January 10th and August 30rd, 2020. The primary outcome of AKI was defined as any increase in serum creatinine (SCr) by 0.3 mg/dL within 48 h or a change in SCr by ≥ 1.5 times of baseline within 1 week, based on Kidney Disease Improving Global Outcomes (KDIGO) guidelines. All patients aged ≥ 18 y/o admitted with confirmed SARS-COV-2 infection were included. Discrimination of variables was calculated by the Receiver Operator Characteristic Curve (ROC curve) utilizing area under curve. Some continuous variables were categorized through ROC curve. The cutoff points were calculated using the value with the best sensitivity and specificity.

RESULTS

A total of 1131 patients with COVID-19 admitted to the ICU were included. Patients mean age was 52 ± 15,8 y/o., with a prevalence of males 60% (n = 678). The risk of AKI was 33% (n = 376), 78% (n = 293) of which did not require dialysis. Overall mortality was 11% (n = 127), while for AKI patients, mortality rate was 21% (n = 80). Variables selected for the logistic regression model and inclusion in the final prognostic score were the following: age, diabetes, ACEis, ARBs, chronic kidney disease and hypertension.

CONCLUSION

AKI development in COVID 19 patients is accurately predicted by common clinical variables, allowing early interventions to attenuate the impact of AKI in these patients.

Severity of Illness Scores and Biomarkers for Prognosis of Patients with Coronavirus Disease 2019

The spectrum of disease severity and the insidiousness of clinical presentation make it difficult to recognize patients with coronavirus disease 2019 (COVID-19) at higher risk of worse outcomes or death when they are seen in the early phases of the disease. There are now well-established risk factors for worse outcomes in patients with COVID-19. These should be factored in when assessing the prognosis of these patients. However, a more precise prognostic assessment in an individual patient may warrant the use of predictive tools. In this manuscript, we conduct a literature review on the severity of illness scores and biomarkers for the prognosis of patients with COVID-19. Several COVID-19-specific scores have been developed since the onset of the pandemic. Some of them are promising and can be integrated into the assessment of these patients. We also found that the well-known pneumonia severity index (PSI) and CURB-65 (confusion, uremia, respiratory rate, BP, age ≥ 65 years) are good predictors of mortality in hospitalized patients with COVID-19. While neither the PSI nor the CURB-65 should be used for the triage of outpatient versus inpatient treatment, they can be integrated by a clinician into the assessment of disease severity and can be used in epidemiological studies to determine the severity of illness in patient populations. Biomarkers also provide valuable prognostic information and, importantly, may depict the main physiological derangements in severe disease. We, however, do not advocate the isolated use of severity of illness scores or biomarkers for decision-making in an individual patient. Instead, we suggest the use of these tools on a case-by-case basis with the goal of enhancing clinician judgment.

The Relationship Between In-Hospital Mortality and Frontal QRS-T Angle in Patients With COVID-19

Background

Recent studies have demonstrated that the frontal QRS-T angle, defined as the angle between the mean QRS and T vectors, is a strong independent predictor of mortality in patients with cardiovascular disease and in the normal population. In this study, we aimed to investigate the relationship between frontal QRS-T angle and in-hospital mortality in COVID-19 patients.

Methods

A total of consecutive 532 patients with positive polymerase chain reaction (PCR) tests were enrolled. The patients were divided into two groups as in-hospital mortality and survival groups. Frontal QRS-T angle was automatically calculated from the admission electrocardiography (ECG).

Results

The median age in the study population was 62 (49-72) years and 273 (51.4%) of the patients were male. The median frontal QRS-T angle was 40 degrees (20-67 IQR) in the in-hospital mortality group, while it was 27 (11-48 IQR) in the survival group (p=0.001). In multivariable logistic regression analysis, frontal QRS-T angle was found to be an independent predictor of mortality (Odds ratio (OR):1.01, 95% Confidence interval (CI):1.00-1.02, p=0.036).

Conclusion

Frontal QRS-T angle, which was observed wider in the in-hospital mortality group, was found to be associated with in-hospital mortality in patients hospitalized for COVID-19.

Influence of Intermittent Fasting During Ramadan on Circadian Variation of Symptom-Onset and Prehospital Time Delay in Acute ST-Segment Elevation Myocardial Infarction

Ramadan interferes with circadian rhythms mainly by disturbing the routine patterns of feeding and smoking. The objective of this study was to investigate the circadian pattern of ST elevation acute myocardial infarction (STEMI) during the month of Ramadan. We studied consecutive STEMI patients 1 month before and after Ramadan (non-Ramadan group-NRG) and during Ramadan (Ramadan group-RG). The RG group was also divided into two groups, based on whether they chose to fast: fasting (FG) and non-fasting group (NFG). The time of STEMI onset was compared. A total of 742 consecutive STEMI patients were classified into 4 groups by 6 h intervals according to time-of-day at symptom onset. No consistent circadian variation in the onset of STEMI was observed both between the RG (P = .938) and NRG (P = .766) or between the FG (P = .232) and NFG (P = .523). When analyzed for subgroups of the study sample, neither smoking nor diabetes showed circadian rhythm. There was a trend towards a delay from symptom onset to hospital presentation, particularly at evening hours in the RG compared with the control group. In conclusion, there was no significant difference in STEMI onset time, but the time from symptom onset to hospital admission was significantly delayed during Ramadan.

Development and Internal Validation of a New Prognostic Model Powered to Predict 28-Day All-Cause Mortality in ICU COVID-19 Patients-The COVID-SOFA Score

Background: The sequential organ failure assessment (SOFA) score has poor discriminative ability for death in severely or critically ill patients with Coronavirus disease 2019 (COVID-19) requiring intensive care unit (ICU) admission. Our aim was to create a new score powered to predict 28-day mortality. Methods: Retrospective, observational, bicentric cohort study including 425 patients with COVID-19 pneumonia, acute respiratory failure and SOFA score ≥ 2 requiring ICU admission for ≥72 h. Factors with independent predictive value for 28-day mortality were identified after stepwise Cox proportional hazards (PH) regression. Based on the regression coefficients, an equation was computed representing the COVID-SOFA score. Discriminative ability was tested using receiver operating characteristic (ROC) analysis, concordance statistics and precision-recall curves. This score was internally validated. Results: Median (Q1−Q3) age for the whole sample was 64 [55−72], with 290 (68.2%) of patients being male. The 28-day mortality was 54.58%. After stepwise Cox PH regression, age, neutrophil-to-lymphocyte ratio (NLR) and SOFA score remained in the final model. The following equation was computed: COVID-SOFA score = 10 × [0.037 × Age + 0.347 × ln(NLR) + 0.16 × SOFA]. Harrell’s C-index for the COVID-SOFA score was higher than the SOFA score alone for 28-day mortality (0.697 [95% CI; 0.662−0.731] versus 0.639 [95% CI: 0.605−0.672]). Subsequently, the prediction error rate was improved up to 16.06%. Area under the ROC (AUROC) was significantly higher for the COVID-SOFA score compared with the SOFA score for 28-day mortality: 0.796 [95% CI: 0.755−0.833] versus 0.699 [95% CI: 0.653−0.742, p < 0.001]. Better predictive value was observed with repeated measurement at 48 h after ICU admission. Conclusions: The COVID-SOFA score is better than the SOFA score alone for 28-day mortality prediction. Improvement in predictive value seen with measurements at 48 h after ICU admission suggests that the COVID-SOFA score can be used in a repetitive manner. External validation is required to support these results.

COVIDNearTerm: A simple method to forecast COVID-19 hospitalizations

Introduction

COVID-19 has caused tremendous death and suffering since it first emerged in 2019. Soon after its emergence, models were developed to help predict the course of various disease metrics, and these models have been relied upon to help guide public health policy.

Methods

Here we present a method called COVIDNearTerm to "forecast" hospitalizations in the short term, two to four weeks from the time of prediction. COVIDNearTerm is based on an autoregressive model and utilizes a parametric bootstrap approach to make predictions. It is easy to use as it requires only previous hospitalization data, and there is an open-source R package that implements the algorithm. We evaluated COVIDNearTerm on San Francisco Bay Area hospitalizations and compared it to models from the California COVID Assessment Tool (CalCAT).

Results

We found that COVIDNearTerm predictions were more accurate than the CalCAT ensemble predictions for all comparisons and any CalCAT component for a majority of comparisons. For instance, at the county level our 14-day hospitalization median absolute percentage errors ranged from 16 to 36%. For those same comparisons, the CalCAT ensemble errors were between 30 and 59%.

Conclusion

COVIDNearTerm is a simple and useful tool for predicting near-term COVID-19 hospitalizations.

Prediction models for severe manifestations and mortality due to COVID-19: A systematic review

BACKGROUND

Throughout 2020, the coronavirus disease 2019 (COVID-19) has become a threat to public health on national and global level. There has been an immediate need for research to understand the clinical signs and symptoms of COVID-19 that can help predict deterioration including mechanical ventilation, organ support, and death. Studies thus far have addressed the epidemiology of the disease, common presentations, and susceptibility to acquisition and transmission of the virus; however, an accurate prognostic model for severe manifestations of COVID-19 is still needed because of the limited healthcare resources available.

OBJECTIVE

This systematic review aims to evaluate published reports of prediction models for severe illnesses caused COVID-19.

METHODS

Searches were developed by the primary author and a medical librarian using an iterative process of gathering and evaluating terms. Comprehensive strategies, including both index and keyword methods, were devised for PubMed and EMBASE. The data of confirmed COVID-19 patients from randomized control studies, cohort studies, and case-control studies published between January 2020 and May 2021 were retrieved. Studies were independently assessed for risk of bias and applicability using the Prediction Model Risk Of Bias Assessment Tool (PROBAST). We collected study type, setting, sample size, type of validation, and outcome including intubation, ventilation, any other type of organ support, or death. The combination of the prediction model, scoring system, performance of predictive models, and geographic locations were summarized.

RESULTS

A primary review found 445 articles relevant based on title and abstract. After further review, 366 were excluded based on the defined inclusion and exclusion criteria. Seventy-nine articles were included in the qualitative analysis. Inter observer agreement on inclusion 0.84 (95%CI 0.78-0.89). When the PROBAST tool was applied, 70 of the 79 articles were identified to have high or unclear risk of bias, or high or unclear concern for applicability. Nine studies reported prediction models that were rated as low risk of bias and low concerns for applicability.

CONCLUSION

Several prognostic models for COVID-19 were identified, with varying clinical score performance. Nine studies that had a low risk of bias and low concern for applicability, one from a general public population and hospital setting. The most promising and well-validated scores include Clift et al.,¹⁵ and Knight et al.,¹⁸ which seem to have accurate prediction models that clinicians can use in the public health and emergency department setting.

Persistently elevated complement alternative pathway biomarkers in COVID-19 correlate with hypoxemia and predict in-hospital mortality

Mechanisms underlying the SARS-CoV-2-triggered hyperacute thrombo-inflammatory response that causes multi-organ damage in coronavirus disease 2019 (COVID-19) are poorly understood. Several lines of evidence implicate overactivation of complement. To delineate the involvement of complement in COVID-19, we prospectively studied 25 ICU-hospitalized patients for up to 21 days. Complement biomarkers in patient sera and healthy controls were quantified by enzyme-linked immunosorbent assays. Correlations with respiratory function and mortality were analyzed. Activation of complement via the classical/lectin pathways was variably increased. Strikingly, all patients had increased activation of the alternative pathway (AP) with elevated levels of activation fragments, Ba and Bb. This was associated with a reduction of the AP negative regulator, factor (F) H. Correspondingly, terminal pathway biomarkers of complement activation, C5a and sC5b-9, were significantly elevated in all COVID-19 patient sera. C5a and AP constituents Ba and Bb, were significantly associated with hypoxemia. Ba and FD at the time of ICU admission were strong independent predictors of mortality in the following 30 days. Levels of all complement activation markers were sustained throughout the patients' ICU stays, contrasting with the varying serum levels of IL-6, C-reactive protein, and ferritin. Severely ill COVID-19 patients have increased and persistent activation of complement, mediated strongly via the AP. Complement activation biomarkers may be valuable measures of severity of lung disease and the risk of mortality. Large-scale studies will reveal the relevance of these findings to thrombo-inflammation in acute and post-acute COVID-19.

The effect of resistant hypertension on in-hospital mortality in patients hospitalized with COVID-19

Hypertension is a major concomitant disease in hospitalized patients with COVID-19 (Coronavirus disease 2019) infection. The adverse effect of hypertension on prognosis in COVID-19 is known. Nevertheless, it is not known how COVID-19 progresses in resistant hypertensive patients. In this study, we aimed to examine the effect of resistant hypertension (ResHT) on in-hospital mortality in patients hospitalized with COVID-19. In our single-center retrospective study, included 1897 COVID-19 patients. The patients were divided into three groups according to the non-hypertensive (n = 1211), regulated HT (RegHT) (n = 574), and ResHT (n = 112). These three groups were compared according to demographic features, clinical signs, laboratory findings, and follow-up times. The median age of the study population was 62 (50-72 IQR) and 1000 (52.7%) of patients were male. The total mortality of the study population was 18.7% (n = 356). Mortality rates were similar in the hypertensive patient group (27.5% for the RegHT and 32.1% for ResHT, p = 0.321). In a multivariable analysis, ResHT was independently associated with a significantly increased risk of in-hospital mortality of COVID-19, while no significant increased risk was observed with RegHT [respectively, Odds Ratio (OR) = 2.013, Confidence Interval (CI) 1.085-3.734, p = 0.026 and OR = 1.194, CI 0.795-1.794, p = 0.394]. Also, age, male gender, chronic renal failure, lymphocyte, procalcitonin, creatinine, and admission SpO2 levels were determined as independent predictors of in-hospital mortality. In our study, it was found that ResHT was an independent predictor of mortality in patients hospitalized with COVID-19; however, this situation was not found in RegHT.

Association of entirely claims-based frailty indices with long-term outcomes in patients with acute myocardial infarction, heart failure, or pneumonia: a nationwide cohort study in Turkey

BACKGROUND

Several countries have increasingly focused on improving care for acute myocardial infarction (AMI), heart failure (HF), and pneumonia to reduce their readmissions and mortality rates. Frailty is becoming increasingly important to accurately predict healthcare utilization for the aging population. The preferred method for the measurement of frailty remains unclear, and current risk-adjustment models do not account for frailty. We sought to compare commonly used frailty indices in terms of the ability to predict clinical adverse outcomes in AMI, HF, and pneumonia patients.

METHODS

A nationwide cohort study included AMI, HF, and pneumonia with 65 years and older patients in the Turkey between January 1 and December 31, 2018. The primary predictor of interest was frailty. We used two claims-based frailty indices (Johns Hopkins Claims-Based Frailty Index and Hospital Frailty Risk Score) to assess frailty. The main outcome was all-cause long-term mortality up to 3 years. Time to death was calculated as the time period between the date of first admission and the date of death. Patients were censored as of September 30, 2020, which marked the end of the follow-up period.

FINDINGS

Of the 200,948 patients, 35,096 (17.5%) had AMI, 62,403 (31.1%) had HF, and 103,449 (51.5%) had pneumonia. Johns Hopkins Claims-Based Frailty Index (c-statistics for long-term mortality: 0.68 in AMI, 0.61 in HF, 0.64 in pneumonia) was better compared to Hospital Frailty Risk Score (c-statistics for long-term mortality: AMI=0.62, HF=0.58, pneumonia=0.62) (DeLong p<0.001 in all).

INTERPRETATION

Readmission and mortality rates after AMI, HF, and pneumonia gradually increases with increasing frailty score. While the Hospital Frailty Risk Score had a better discrimination for predicting readmissions, Johns Hopkins Claims-Based Frailty Index had a better discrimination for predicting mortality. These findings should be taken into account for a better evaluation of hospital performance.

FUNDING

This study was supported by funding from The Scientific and Technological Research Council of Turkey (grant 120S422, HK).

External validation of prognostic scores for COVID-19: a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals

Purpose

The Coronavirus disease 2019 (COVID-19) has led to an unparalleled influx of patients. Prognostic scores could help optimizing healthcare delivery, but most of them have not been comprehensively validated. We aim to externally validate existing prognostic scores for COVID-19.

Methods

We used “COVID-19 Evidence Alerts” (McMaster University) to retrieve high-quality prognostic scores predicting death or intensive care unit (ICU) transfer from routinely collected data. We studied their accuracy in a retrospective multicenter cohort of adult patients hospitalized for COVID-19 from January 2020 to April 2021 in the Greater Paris University Hospitals. Areas under the receiver operating characteristic curves (AUC) were computed for the prediction of the original outcome, 30-day in-hospital mortality and the composite of 30-day in-hospital mortality or ICU transfer.

Results

We included 14,343 consecutive patients, 2583 (18%) died and 5067 (35%) died or were transferred to the ICU. We examined 274 studies and found 32 scores meeting the inclusion criteria: 19 had a significantly lower AUC in our cohort than in previously published validation studies for the original outcome; 25 performed better to predict in-hospital mortality than the composite of in-hospital mortality or ICU transfer; 7 had an AUC > 0.75 to predict in-hospital mortality; 2 had an AUC > 0.70 to predict the composite outcome.

Conclusion

Seven prognostic scores were fairly accurate to predict death in hospitalized COVID-19 patients. The 4C Mortality Score and the ABCS stand out because they performed as well in our cohort and their initial validation cohort, during the first epidemic wave and subsequent waves, and in younger and older patients.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00134-021-06524-w.

The prognostic role of cardiac troponin in hospitalized COVID-19 patients

BACKGROUND AND AIMS

Myocardial injury defined by elevation of cardiac troponins (cTn) in the course of coronavirus disease 2019 (COVID-19) pandemic has been reported, though not fully characterized yet. Using the Turkish nationwide centralized COVID-19 database, we sought to determine whether cTn measured within 24 h of admission may help identify 30-day all-cause mortality in hospitalized patients.

METHODS

This retrospective cohort study was conducted at all hospitals in Turkey between March 11, 2020, and June 22, 2020. All hospitalized COVID-19 patients (≥18 years) who had cTn measurements within 24 h of admission were included. The primary outcome was 30-day all-cause mortality.

RESULTS

A total of 14,855 COVID-19 patients (median age 49 years and 54% male) from 81 provinces of Turkey were included. Of these, 2020 patients (13.6%) were transferred to intensive care unit, 1165 patients (7.8%) needed mechanical ventilation, and 882 patients (5.9%) died during hospitalization. The prevalence of cTn positivity was 6.9% (n = 1027) in the hospitalized patients. cTn positivity was 5% in those patients alive at 30-day, and 44% in those who died. In multivariable Cox proportional hazard regression model, age, lactate dehydrogenase, and cTn were the strongest predictors of 30-day mortality, irrespective of cTn definition as a continuous, ordinal variable, or dichotomic variables.

CONCLUSIONS

A single measurement of cTn at admission in patients with COVID-19 is associated with 30-day all-cause mortality and may have an important prognostic role for optimizing risk stratification.

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

OBJECTIVE

To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease.

DESIGN

Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

DATA SOURCES

PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

STUDY SELECTION

Studies that developed or validated a multivariable covid-19 related prediction model.

DATA EXTRACTION

At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

RESULTS

37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models.

CONCLUSION

Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

SYSTEMATIC REVIEW REGISTRATION

Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

READERS' NOTE

This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.