Validation of pneumonia prognostic scores in a statewide cohort of hospitalised patients with COVID-19

Performance of established disease severity scores in predicting severe outcomes among adults hospitalized with influenza-FluSurv-NET, 2017-2018

Background

Influenza is a substantial cause of annual morbidity and mortality; however, correctly identifying those patients at increased risk for severe disease is often challenging. Several severity indices have been developed; however, these scores have not been validated for use in patients with influenza. We evaluated the discrimination of three clinical disease severity scores in predicting severe influenza-associated outcomes.

Methods

We used data from the Influenza Hospitalization Surveillance Network to assess outcomes of patients hospitalized with influenza in the United States during the 2017-2018 influenza season. We computed patient scores at admission for three widely used disease severity scores: CURB-65, Quick Sepsis-Related Organ Failure Assessment (qSOFA), and the Pneumonia Severity Index (PSI). We then grouped patients with severe outcomes into four severity tiers, ranging from ICU admission to death, and calculated receiver operating characteristic (ROC) curves for each severity index in predicting these tiers of severe outcomes.

Results

Among 8252 patients included in this study, we found that all tested severity scores had higher discrimination for more severe outcomes, including death, and poorer discrimination for less severe outcomes, such as ICU admission. We observed the highest discrimination for PSI against in-hospital mortality, at 0.78.

Conclusions

We observed low to moderate discrimination of all three scores in predicting severe outcomes among adults hospitalized with influenza. Given the substantial annual burden of influenza disease in the United States, identifying a prediction index for severe outcomes in adults requiring hospitalization with influenza would be beneficial for patient triage and clinical decision-making.

Haemogram indices are as reliable as CURB-65 to assess 30-day mortality in Covid-19 pneumonia

Background Mortality due to Covid-19 and severe community-acquired pneumonia (CAP) remains high, despite progress in critical care management. We compared the precision of CURB-65 score with monocyte-to-lymphocyte ratio (MLR), neutrophil-to-lymphocyte ratio (NLR), platelet-to-lymphocyte ratio (PLR) in prediction of mortality among patients with Covid-19 and CAP presenting to the emergency department. Methods We retrospectively analysed two cohorts of patients admitted to the emergency department of Canakkale University Hospital, namely (i) Covid-19 patients with severe acute respiratory symptoms presenting between 23 March 2020 and 31 October 2020, and (ii) all patients with CAP either from bacterial or viral infection within the 36 months preceding the Covid-19 pandemic. Mortality was defined as in-hospital death or death occurring within 30 days after discharge. Results The first study group consisted of 324 Covid-19 patients and the second group of 257 CAP patients. The non-survivor Covid-19 group had significantly higher MLR, NLR and PLR values. In univariate analysis, in Covid-19 patients, a 1-unit increase in NLR and PLR was associated with increased mortality, and in multivariate analysis for Covid-19 patients, age and NLR remained significant in the final step of the model. According to this model, we found that in the Covid-19 group an increase in 1-unit in NLR would result in an increase by 5% and 7% in the probability of mortality, respectively. According to pairwise analysis, NLR and PLR are as reliable as CURB-65 in predicting mortality in Covid-19. Conclusions Our study indicates that NLR and PLR may serve as reliable predictive factors as CURB-65 in Covid-19 pneumonia, which could easily be used to triage and manage severe patients in the emergency department.

Leucocyte volume, conductivity, and scatter at presentation in COVID-19 patients

Background

In COVID-19 patients, besides changes in leucocyte count, morphological abnormalities of circulating blood cells have been reported.

Aim

This study aims to investigate the relationship between the morphological and functional properties of leucocytes and the severity of the disease in COVID-19 patients.

Materials and Methods

Blood samples were collected from COVID-19 patients (n = 130) at the time of admission. The patients were stratified according to the comorbidity, age, LDH, lymhocyte count score as mild, moderate, and severe. Complete blood count and the cell population data were analyzed by the Volume, conductivity, scatter (VCS) technology on Beckman Coulter LH-780 hematology analyzer. Kruskal-Wal'lis test was used to assess the differences between the groups with subsequent Bonferroni correction.

Results

Neutrophil count was increased, and lymphocyte count was decreased in severe patients compared to mild patients. The increase in the percent of neutrophils and the neutrophil/lymphocyte ratio in the severe patient group was significant in comparison to both the moderate and the mild group. The dispersion of the neutrophil volume and conductivity showed significant changes depending on the severity of the disease. The lymphocyte volume, lymphocyte-volume-SD and lymphocyte-conductivity as well as the monocyte-volume and monocyte-volume-SD were significantly increased in severe patients in comparison to mild patients. The increase of lymphocyte and monocyte volume in severe patients was also significant in comparison to moderate patients.

Conclusions

COVID-19 infection leads to important changes in cell population data of leucocytes. The volumetric changes in lymphocytes and monocytes are related to the severity of the disease.

Prognostic models in COVID-19 infection that predict severity: a systematic review

Current evidence on COVID-19 prognostic models is inconsistent and clinical applicability remains controversial. We performed a systematic review to summarize and critically appraise the available studies that have developed, assessed and/or validated prognostic models of COVID-19 predicting health outcomes. We searched six bibliographic databases to identify published articles that investigated univariable and multivariable prognostic models predicting adverse outcomes in adult COVID-19 patients, including intensive care unit (ICU) admission, intubation, high-flow nasal therapy (HFNT), extracorporeal membrane oxygenation (ECMO) and mortality. We identified and assessed 314 eligible articles from more than 40 countries, with 152 of these studies presenting mortality, 66 progression to severe or critical illness, 35 mortality and ICU admission combined, 17 ICU admission only, while the remaining 44 studies reported prediction models for mechanical ventilation (MV) or a combination of multiple outcomes. The sample size of included studies varied from 11 to 7,704,171 participants, with a mean age ranging from 18 to 93 years. There were 353 prognostic models investigated, with area under the curve (AUC) ranging from 0.44 to 0.99. A great proportion of studies (61.5%, 193 out of 314) performed internal or external validation or replication. In 312 (99.4%) studies, prognostic models were reported to be at high risk of bias due to uncertainties and challenges surrounding methodological rigor, sampling, handling of missing data, failure to deal with overfitting and heterogeneous definitions of COVID-19 and severity outcomes. While several clinical prognostic models for COVID-19 have been described in the literature, they are limited in generalizability and/or applicability due to deficiencies in addressing fundamental statistical and methodological concerns. Future large, multi-centric and well-designed prognostic prospective studies are needed to clarify remaining uncertainties.

Comparison of Pneumonia Severity Indices, qCSI, 4C-Mortality Score and qSOFA in Predicting Mortality in Hospitalized Patients with COVID-19 Pneumonia

This is a retrospective and observational study on 1511 patients with SARS-CoV-2, who were diagnosed with COVID-19 by real-time PCR testing and hospitalized due to COVID-19 pneumonia. 1511 patients, 879 male (58.17%) and 632 female (41.83%) with a mean age of 60.1 ± 14.7 were included in the study. Survivors and non-survivors groups were statistically compared with respect to survival, discharge, ICU admission and in-hospital death. Although gender was not statistically significant different between two groups, 80 (60.15%) of the patients who died were male. Mean age was 72.8 ± 11.8 in non-survivors vs. 59.9 ± 14.7 in survivors (p < 0.001). Overall in-hospital mortality was found to be 8.8% (133/1511 cases), and overall ICU admission was 10.85% (164/1511 cases). The PSI/PORT score of the non-survivors group was higher than that of the survivors group (144.38 ± 28.64 versus 67.17 ± 25.63, p < 0.001). The PSI/PORT yielding the highest performance was the best predictor for in-hospital mortality, since it incorporates the factors as advanced age and comorbidity (AUROC 0.971; % 95 CI 0.961−0.981). The use of A-DROP may also be preferred as an easier alternative to PSI/PORT, which is a time-consuming evaluation although it is more comprehensive.

Evaluation and Comparison of the Predictive Value of 4C Mortality Score, NEWS, and CURB-65 in Poor Outcomes in COVID-19 Patients: A Retrospective Study from a Single Center in Romania

To date, the COVID-19 pandemic has caused millions of deaths across the world. Prognostic scores can improve the clinical management of COVID-19 diagnosis and treatment. The objective of this study was to assess the predictive role of 4C Mortality, CURB-65, and NEWS in COVID-19 mortality among the Romanian population. A single-center, retrospective, observational study was conducted on patients with reverse transcriptase-polymerase chain reaction (RT-PCR)-proven COVID-19 admitted to the Municipal Emergency Clinical Hospital of Timisoara, Romania, between 1 October 2020 and 15 March 2021. Receiver operating characteristic (ROC) and area under the curve (AUC) analyses were performed to determine the discrimination accuracy of the three scores. The mean values of the risk scores were higher in the non-survivors group (survivors group vs. non-survivors group: 8 vs. 15 (4C Mortality Score); 3 vs. 8.5 (NEWS); 1 vs. 3 (CURB-65)). In terms of mortality risk prediction, the NEWS performed best, with an AUC of 0.86, and the CURB-65 score performed poorly, with an AUC of 0.80. CURB-65, NEWS, and 4C Mortality scores were significant mortality predictors in the analysis, with acceptable calibration. Among the scores assessed in our study, NEWS had the highest performance in predicting in-hospital mortality in COVID-19 patients. Thus, the findings from this study suggest that the use of NEWS may be beneficial to the early identification of high-risk COVID-19 patients and the provision of more aggressive care to reduce mortality associated with COVID-19.

Clinical Management of Adult Patients with COVID-19 Outside Intensive Care Units: Guidelines from the Italian Society of Anti-Infective Therapy (SITA) and the Italian Society of Pulmonology (SIP)

INTRODUCTION

The Italian Society of Anti-Infective Therapy (SITA) and the Italian Society of Pulmonology (SIP) constituted an expert panel for developing evidence-based guidance for the clinical management of adult patients with coronavirus disease 2019 (COVID-19) outside intensive care units.

METHODS

Ten systematic literature searches were performed to answer ten different key questions. The retrieved evidence was graded according to the Grading of Recommendations Assessment, Development, and Evaluation methodology (GRADE).

RESULTS AND CONCLUSION

The literature searches mostly assessed the available evidence on the management of COVID-19 patients in terms of antiviral, anticoagulant, anti-inflammatory, immunomodulatory, and continuous positive airway pressure (CPAP)/non-invasive ventilation (NIV) treatment. Most evidence was deemed as of low certainty, and in some cases, recommendations could not be developed according to the GRADE system (best practice recommendations were provided in similar situations). The use of neutralizing monoclonal antibodies may be considered for outpatients at risk of disease progression. For inpatients, favorable recommendations were provided for anticoagulant prophylaxis and systemic steroids administration, although with low certainty of evidence. Favorable recommendations, with very low/low certainty of evidence, were also provided for, in specific situations, remdesivir, alone or in combination with baricitinib, and tocilizumab. The presence of many best practice recommendations testified to the need for further investigations by means of randomized controlled trials, whenever possible, with some possible future research directions stemming from the results of the ten systematic reviews.

Lung Ultrasound, Clinical and Analytic Scoring Systems as Prognostic Tools in SARS-CoV-2 Pneumonia: A Validating Cohort

At the moment, several COVID-19 scoring systems have been developed. It is necessary to determine which one better predicts a poor outcome of the disease. We conducted a single-center prospective cohort study to validate four COVID-19 prognosis scores in adult patients with confirmed infection at ward. These are National Early Warning Score (NEWS) 2, Lung Ultrasound Score (LUS), COVID-19 Worsening Score (COWS), and Spanish Society of Infectious Diseases and Clinical Microbiology score (SEIMC Score). Our outcomes were the combined variable “poor outcome” (non-invasive mechanical ventilation, intubation, intensive care unit admission, and death at 28 days) and death at 28 days. Scores were analysed using univariate logistic regression models, receiver operating characteristic curves, and areas under the curve. Eighty-one patients were included, from which 21 had a poor outcome, and 9 died. We found a statistically significant correlation between poor outcome and NEWS2, LUS > 15, and COWS. Death at 28 days was statistically correlated with NEWS2 and SEIMC Score although COWS also performs well. NEWS2, LUS, and COWS accurately predict poor outcome; and NEWS2, SEIMC Score, and COWS are useful for anticipating death at 28 days. Lung ultrasound is a diagnostic tool that should be included in COVID-19 patients evaluation.

Evaluating risk stratification scoring systems to predict mortality in patients with COVID-19

Background

The COVID-19 pandemic has necessitated efficient and accurate triaging of patients for more effective allocation of resources and treatment.

Objectives

The objectives are to investigate parameters and risk stratification tools that can be applied to predict mortality within 90 days of hospital admission in patients with COVID-19.

Methods

A literature search of original studies assessing systems and parameters predicting mortality of patients with COVID-19 was conducted using MEDLINE and EMBASE.

Results

589 titles were screened, and 76 studies were found investigating the prognostic ability of 16 existing scoring systems (area under the receiving operator curve (AUROC) range: 0.550–0.966), 38 newly developed COVID-19-specific prognostic systems (AUROC range: 0.6400–0.9940), 15 artificial intelligence (AI) models (AUROC range: 0.840–0.955) and 16 studies on novel blood parameters and imaging.

Discussion

Current scoring systems generally underestimate mortality, with the highest AUROC values found for APACHE II and the lowest for SMART-COP. Systems featuring heavier weighting on respiratory parameters were more predictive than those assessing other systems. Cardiac biomarkers and CT chest scans were the most commonly studied novel parameters and were independently associated with mortality, suggesting potential for implementation into model development. All types of AI modelling systems showed high abilities to predict mortality, although none had notably higher AUROC values than COVID-19-specific prediction models. All models were found to have bias, including lack of prospective studies, small sample sizes, single-centre data collection and lack of external validation.

Conclusion

The single parameters established within this review would be useful to look at in future prognostic models in terms of the predictive capacity their combined effect may harness.

Predictors of severity and mortality among patients hospitalized with COVID-19 in Rhode Island

BACKGROUND

In order for healthcare systems to prepare for future waves of COVID-19, an in-depth understanding of clinical predictors is essential for efficient triage of hospitalized patients.

METHODS

We performed a retrospective cohort study of 259 patients admitted to our hospitals in Rhode Island to examine differences in baseline characteristics (demographics and comorbidities) as well as presenting symptoms, signs, labs, and imaging findings that predicted disease progression and in-hospital mortality.

RESULTS

Patients with severe COVID-19 were more likely to be older (p = 0.02), Black (47.2% vs. 32.0%, p = 0.04), admitted from a nursing facility (33.0% vs. 17.9%, p = 0.006), have diabetes (53.9% vs. 30.4%, p<0.001), or have COPD (15.4% vs. 6.6%, p = 0.02). In multivariate regression, Black race (adjusted odds ratio [aOR] 2.0, 95% confidence interval [CI]: 1.1-3.9) and diabetes (aOR 2.2, 95%CI: 1.3-3.9) were independent predictors of severe disease, while older age (aOR 1.04, 95% CI: 1.01-1.07), admission from a nursing facility (aOR 2.7, 95% CI 1.1-6.7), and hematological co-morbidities predicted mortality (aOR 3.4, 95% CI 1.1-10.0). In the first 24 hours, respiratory symptoms (aOR 7.0, 95% CI: 1.4-34.1), hypoxia (aOR 19.9, 95% CI: 2.6-152.5), and hypotension (aOR 2.7, 95% CI) predicted progression to severe disease, while tachypnea (aOR 8.7, 95% CI: 1.1-71.7) and hypotension (aOR 9.0, 95% CI: 3.1-26.1) were associated with increased in-hospital mortality.

CONCLUSIONS

Certain patient characteristics and clinical features can help clinicians with early identification and triage of high-risk patients during subsequent waves of COVID-19.

Real-Time Electronic Health Record Mortality Prediction During the COVID-19 Pandemic: A Prospective Cohort Study

Objective

To rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon sequential organ failure assessment (SOFA) for decision support for a Crisis Standards of Care team.

Materials and Methods

We developed, verified, and deployed a stacked generalization model to predict mortality using data available in the electronic health record (EHR) by combining 5 previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We verified the model with prospectively collected data from 12 hospitals in Colorado between March 2020 and July 2020. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index.

Results

The prospective cohort included 27 296 encounters, of which 1358 (5.0%) were positive for SARS-CoV-2, 4494 (16.5%) required intensive care unit care, 1480 (5.4%) required mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94. In the subset of patients with COVID-19, the stacked model predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85.

Discussion

Stacked regression allows a flexible, updatable, live-implementable, ethically defensible predictive analytics tool for decision support that begins with validated models and includes only novel information that improves prediction.

Conclusion

We developed and validated an accurate in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model that improved upon SOFA.

Real-Time Electronic Health Record Mortality Prediction During the COVID-19 Pandemic: A Prospective Cohort Study

Background

The SARS-CoV-2 virus has infected millions of people, overwhelming critical care resources in some regions. Many plans for rationing critical care resources during crises are based on the Sequential Organ Failure Assessment (SOFA) score. The COVID-19 pandemic created an emergent need to develop and validate a novel electronic health record (EHR)-computable tool to predict mortality.

Research Questions

To rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon SOFA.

Study Design and Methods

We conducted a prospective cohort study of a regional health system with 12 hospitals in Colorado between March 2020 and July 2020. All patients >14 years old hospitalized during the study period without a do not resuscitate order were included. Patients were stratified by the diagnosis of COVID-19. From this cohort, we developed and validated a model using stacked generalization to predict mortality using data widely available in the EHR by combining five previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index.

Results

We prospectively analyzed 27,296 encounters, of which 1,358 (5.0%) were positive for SARS-CoV-2, 4,494 (16.5%) included intensive care unit (ICU)-level care, 1,480 (5.4%) included invasive mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted overall mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted overall mortality with AUROC 0.94. In the subset of patients with COVID-19, we predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85.

Interpretation

We developed and validated an accurate, in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model, that improved upon SOFA.

Take Home Points

Study Question: Can we improve upon the SOFA score for real-time mortality prediction during the COVID-19 pandemic by leveraging electronic health record (EHR) data?Results: We rapidly developed and implemented a novel yet SOFA-anchored mortality model across 12 hospitals and conducted a prospective cohort study of 27,296 adult hospitalizations, 1,358 (5.0%) of which were positive for SARS-CoV-2. The Charlson Comorbidity Index and SOFA scores predicted all-cause mortality with AUROCs of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94.Interpretation: A novel EHR-based mortality score can be rapidly implemented to better predict patient outcomes during an evolving pandemic.

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

OBJECTIVE

To review and appraise the validity and usefulness of published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital with the disease.

DESIGN

Living systematic review and critical appraisal by the COVID-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

DATA SOURCES

PubMed and Embase through Ovid, up to 1 July 2020, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

STUDY SELECTION

Studies that developed or validated a multivariable covid-19 related prediction model.

DATA EXTRACTION

At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

RESULTS

37 421 titles were screened, and 169 studies describing 232 prediction models were included. The review identified seven models for identifying people at risk in the general population; 118 diagnostic models for detecting covid-19 (75 were based on medical imaging, 10 to diagnose disease severity); and 107 prognostic models for predicting mortality risk, progression to severe disease, intensive care unit admission, ventilation, intubation, or length of hospital stay. The most frequent types of predictors included in the covid-19 prediction models are vital signs, age, comorbidities, and image features. Flu-like symptoms are frequently predictive in diagnostic models, while sex, C reactive protein, and lymphocyte counts are frequent prognostic factors. Reported C index estimates from the strongest form of validation available per model ranged from 0.71 to 0.99 in prediction models for the general population, from 0.65 to more than 0.99 in diagnostic models, and from 0.54 to 0.99 in prognostic models. All models were rated at high or unclear risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, high risk of model overfitting, and unclear reporting. Many models did not include a description of the target population (n=27, 12%) or care setting (n=75, 32%), and only 11 (5%) were externally validated by a calibration plot. The Jehi diagnostic model and the 4C mortality score were identified as promising models.

CONCLUSION

Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that almost all pubished prediction models are poorly reported, and at high risk of bias such that their reported predictive performance is probably optimistic. However, we have identified two (one diagnostic and one prognostic) promising models that should soon be validated in multiple cohorts, preferably through collaborative efforts and data sharing to also allow an investigation of the stability and heterogeneity in their performance across populations and settings. Details on all reviewed models are publicly available at https://www.covprecise.org/. Methodological guidance as provided in this paper should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction model authors should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

SYSTEMATIC REVIEW REGISTRATION

Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

READERS' NOTE

This article is a living systematic review that will be updated to reflect emerging evidence. Updates may occur for up to two years from the date of original publication. This version is update 3 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.