Derivation With Internal Validation of a Multivariable Predictive Model to Predict COVID-19 Test Results in Emergency Department Patients

Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19

BACKGROUND

COVID-19 illness is highly variable, ranging from infection with no symptoms through to pneumonia and life-threatening consequences. Symptoms such as fever, cough, or loss of sense of smell (anosmia) or taste (ageusia), can help flag early on if the disease is present. Such information could be used either to rule out COVID-19 disease, or to identify people who need to go for COVID-19 diagnostic tests. This is the second update of this review, which was first published in 2020.

OBJECTIVES

To assess the diagnostic accuracy of signs and symptoms to determine if a person presenting in primary care or to hospital outpatient settings, such as the emergency department or dedicated COVID-19 clinics, has COVID-19.

SEARCH METHODS

We undertook electronic searches up to 10 June 2021 in the University of Bern living search database. In addition, we checked repositories of COVID-19 publications. We used artificial intelligence text analysis to conduct an initial classification of documents. We did not apply any language restrictions.

SELECTION CRITERIA

Studies were eligible if they included people with clinically suspected COVID-19, or recruited known cases with COVID-19 and also controls without COVID-19 from a single-gate cohort. Studies were eligible when they recruited people presenting to primary care or hospital outpatient settings. Studies that included people who contracted SARS-CoV-2 infection while admitted to hospital were not eligible. The minimum eligible sample size of studies was 10 participants. All signs and symptoms were eligible for this review, including individual signs and symptoms or combinations. We accepted a range of reference standards.

DATA COLLECTION AND ANALYSIS

Pairs of review authors independently selected all studies, at both title and abstract, and full-text stage. They resolved any disagreements by discussion with a third review author. Two review authors independently extracted data and assessed risk of bias using the QUADAS-2 checklist, and resolved disagreements by discussion with a third review author. Analyses were restricted to prospective studies only. We presented sensitivity and specificity in paired forest plots, in receiver operating characteristic (ROC) space and in dumbbell plots. We estimated summary parameters using a bivariate random-effects meta-analysis whenever five or more primary prospective studies were available, and whenever heterogeneity across studies was deemed acceptable.

MAIN RESULTS

We identified 90 studies; for this update we focused on the results of 42 prospective studies with 52,608 participants. Prevalence of COVID-19 disease varied from 3.7% to 60.6% with a median of 27.4%. Thirty-five studies were set in emergency departments or outpatient test centres (46,878 participants), three in primary care settings (1230 participants), two in a mixed population of in- and outpatients in a paediatric hospital setting (493 participants), and two overlapping studies in nursing homes (4007 participants). The studies did not clearly distinguish mild COVID-19 disease from COVID-19 pneumonia, so we present the results for both conditions together. Twelve studies had a high risk of bias for selection of participants because they used a high level of preselection to decide whether reverse transcription polymerase chain reaction (RT-PCR) testing was needed, or because they enrolled a non-consecutive sample, or because they excluded individuals while they were part of the study base. We rated 36 of the 42 studies as high risk of bias for the index tests because there was little or no detail on how, by whom and when, the symptoms were measured. For most studies, eligibility for testing was dependent on the local case definition and testing criteria that were in effect at the time of the study, meaning most people who were included in studies had already been referred to health services based on the symptoms that we are evaluating in this review. The applicability of the results of this review iteration improved in comparison with the previous reviews. This version has more studies of people presenting to ambulatory settings, which is where the majority of assessments for COVID-19 take place. Only three studies presented any data on children separately, and only one focused specifically on older adults. We found data on 96 symptoms or combinations of signs and symptoms. Evidence on individual signs as diagnostic tests was rarely reported, so this review reports mainly on the diagnostic value of symptoms. Results were highly variable across studies. Most had very low sensitivity and high specificity. RT-PCR was the most often used reference standard (40/42 studies). Only cough (11 studies) had a summary sensitivity above 50% (62.4%, 95% CI 50.6% to 72.9%)); its specificity was low (45.4%, 95% CI 33.5% to 57.9%)). Presence of fever had a sensitivity of 37.6% (95% CI 23.4% to 54.3%) and a specificity of 75.2% (95% CI 56.3% to 87.8%). The summary positive likelihood ratio of cough was 1.14 (95% CI 1.04 to 1.25) and that of fever 1.52 (95% CI 1.10 to 2.10). Sore throat had a summary positive likelihood ratio of 0.814 (95% CI 0.714 to 0.929), which means that its presence increases the probability of having an infectious disease other than COVID-19. Dyspnoea (12 studies) and fatigue (8 studies) had a sensitivity of 23.3% (95% CI 16.4% to 31.9%) and 40.2% (95% CI 19.4% to 65.1%) respectively. Their specificity was 75.7% (95% CI 65.2% to 83.9%) and 73.6% (95% CI 48.4% to 89.3%). The summary positive likelihood ratio of dyspnoea was 0.96 (95% CI 0.83 to 1.11) and that of fatigue 1.52 (95% CI 1.21 to 1.91), which means that the presence of fatigue slightly increases the probability of having COVID-19. Anosmia alone (7 studies), ageusia alone (5 studies), and anosmia or ageusia (6 studies) had summary sensitivities below 50% but summary specificities over 90%. Anosmia had a summary sensitivity of 26.4% (95% CI 13.8% to 44.6%) and a specificity of 94.2% (95% CI 90.6% to 96.5%). Ageusia had a summary sensitivity of 23.2% (95% CI 10.6% to 43.3%) and a specificity of 92.6% (95% CI 83.1% to 97.0%). Anosmia or ageusia had a summary sensitivity of 39.2% (95% CI 26.5% to 53.6%) and a specificity of 92.1% (95% CI 84.5% to 96.2%). The summary positive likelihood ratios of anosmia alone and anosmia or ageusia were 4.55 (95% CI 3.46 to 5.97) and 4.99 (95% CI 3.22 to 7.75) respectively, which is just below our arbitrary definition of a 'red flag', that is, a positive likelihood ratio of at least 5. The summary positive likelihood ratio of ageusia alone was 3.14 (95% CI 1.79 to 5.51). Twenty-four studies assessed combinations of different signs and symptoms, mostly combining olfactory symptoms. By combining symptoms with other information such as contact or travel history, age, gender, and a local recent case detection rate, some multivariable prediction scores reached a sensitivity as high as 90%.

AUTHORS' CONCLUSIONS

Most individual symptoms included in this review have poor diagnostic accuracy. Neither absence nor presence of symptoms are accurate enough to rule in or rule out the disease. The presence of anosmia or ageusia may be useful as a red flag for the presence of COVID-19. The presence of cough also supports further testing. There is currently no evidence to support further testing with PCR in any individuals presenting only with upper respiratory symptoms such as sore throat, coryza or rhinorrhoea. Combinations of symptoms with other readily available information such as contact or travel history, or the local recent case detection rate may prove more useful and should be further investigated in an unselected population presenting to primary care or hospital outpatient settings. The diagnostic accuracy of symptoms for COVID-19 is moderate to low and any testing strategy using symptoms as selection mechanism will result in both large numbers of missed cases and large numbers of people requiring testing. Which one of these is minimised, is determined by the goal of COVID-19 testing strategies, that is, controlling the epidemic by isolating every possible case versus identifying those with clinically important disease so that they can be monitored or treated to optimise their prognosis. The former will require a testing strategy that uses very few symptoms as entry criterion for testing, the latter could focus on more specific symptoms such as fever and anosmia.

Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study

BACKGROUND

Since the beginning of the pandemic, hospitals have been constantly overcrowded, with several observed waves of infected cases and hospitalisations. To avoid as much as possible this situation, efficient tools to facilitate the diagnosis of COVID-19 are needed.

OBJECTIVE

To evaluate and compare prediction models to diagnose COVID-19 identified in a systematic review published recently using performance indicators such as discrimination and calibration measures.

METHODS

A total of 1618 adult patients present at two Emergency Department triage centers and for whom qRT-PCR tests had been performed were included in this study. Six previously published models were reconstructed and assessed using diagnostic tests as sensitivity (Se) and negative predictive value (NPV), discrimination (Area Under the Roc Curve (AUROC)) and calibration measures. Agreement was also measured between them using Kappa's coefficient and IntraClass Correlation Coefficient (ICC). A sensitivity analysis has been conducted by waves of patients.

RESULTS

Among the 6 selected models, those based only on symptoms and/or risk exposure were found to be less efficient than those based on biological parameters and/or radiological examination with smallest AUROC values (< 0.80). However, all models showed good calibration and values above > 0.75 for Se and NPV but poor agreement (Kappa and ICC < 0.5) between them. The results of the first wave were similar to those of the second wave.

CONCLUSION

Although quite acceptable and similar results were found between all models, the importance of radiological examination was also emphasized, making it difficult to find an appropriate triage system to classify patients at risk for COVID-19.

XGboost Prediction Model Based on 3.0T Diffusion Kurtosis Imaging Improves the Diagnostic Accuracy of MRI BiRADS 4 Masses

BACKGROUND

The malignant probability of MRI BiRADS 4 breast lesions ranges from 2% to 95%, leading to unnecessary biopsies. The purpose of this study was to construct an optimal XGboost prediction model through a combination of DKI independently or jointly with other MR imaging features and clinical characterization, which was expected to reduce false positive rate of MRI BiRADS 4 masses and improve the diagnosis efficiency of breast cancer.

METHODS

120 patients with 158 breast lesions were enrolled. DKI, Diffusion-weighted Imaging (DWI), Proton Magnetic Resonance Spectroscopy (¹H-MRS) and Dynamic Contrast-Enhanced MRI (DCE-MRI) were performed on a 3.0-T scanner. Wilcoxon signed-rank test and χ2 test were used to compare patient's clinical characteristics, mean kurtosis (MK), mean diffusivity (MD), apparent diffusion coefficient (ADC), total choline (tCho) peak, extravascular extracellular volume fraction (V_e), flux rate constant (K_ep) and volume transfer constant (K^trans). ROC curve analysis was used to analyze the diagnostic performances of the imaging parameters. Spearman correlation analysis was performed to evaluate the associations of imaging parameters with prognostic factors and breast cancer molecular subtypes. The Least Absolute Shrinkage and Selectionator operator (lasso) and the area under the curve (AUC) of imaging parameters were used to select discriminative features for differentiating the breast benign lesions from malignant ones. Finally, an XGboost prediction model was constructed based on the discriminative features and its diagnostic efficiency was verified in BiRADS 4 masses.

RESULTS

MK derived from DKI performed better for differentiating between malignant and benign lesions than ADC, MD, tCho, K_ep and K^trans (p < 0.05). Also, MK was shown to be more strongly correlated with histological grade, Ki-67 expression and lymph node status. MD, MK, age, shape and menstrual status were selected to be the optimized feature subsets to construct an XGboost model, which exhibited superior diagnostic ability for breast cancer characterization and an improved evaluation of suspicious breast tumors in MRI BiRADS 4.

CONCLUSIONS

DKI is promising for breast cancer diagnosis and prognostic factor assessment. An optimized XGboost model that included DKI, age, shape and menstrual status is effective in improving the diagnostic accuracy of BiRADS 4 masses.

CCEDRRN COVID-19 Infection Score (CCIS): development and validation in a Canadian cohort of a clinical risk score to predict SARS-CoV-2 infection in patients presenting to the emergency department with suspected COVID-19

OBJECTIVES

To develop and validate a clinical risk score that can accurately quantify the probability of SARS-CoV-2 infection in patients presenting to an emergency department without the need for laboratory testing.

DESIGN

Cohort study of participants in the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) registry. Regression models were fitted to predict a positive SARS-CoV-2 test result using clinical and demographic predictors, as well as an indicator of local SARS-CoV-2 incidence.

SETTING

32 emergency departments in eight Canadian provinces.

PARTICIPANTS

27 665 consecutively enrolled patients who were tested for SARS-CoV-2 in participating emergency departments between 1 March and 30 October 2020.

MAIN OUTCOME MEASURES

Positive SARS-CoV-2 nucleic acid test result within 14 days of an index emergency department encounter for suspected COVID-19 disease.

RESULTS

We derived a 10-item CCEDRRN COVID-19 Infection Score using data from 21 743 patients. This score included variables from history and physical examination and an indicator of local disease incidence. The score had a c-statistic of 0.838 with excellent calibration. We externally validated the rule in 5295 patients. The score maintained excellent discrimination and calibration and had superior performance compared with another previously published risk score. Score cut-offs were identified that can rule-in or rule-out SARS-CoV-2 infection without the need for nucleic acid testing with 97.4% sensitivity (95% CI 96.4 to 98.3) and 95.9% specificity (95% CI 95.5 to 96.0).

CONCLUSIONS

The CCEDRRN COVID-19 Infection Score uses clinical characteristics and publicly available indicators of disease incidence to quantify a patient's probability of SARS-CoV-2 infection. The score can identify patients at sufficiently high risk of SARS-CoV-2 infection to warrant isolation and empirical therapy prior to test confirmation while also identifying patients at sufficiently low risk of infection that they may not need testing.

TRIAL REGISTRATION NUMBER

NCT04702945.

Process Improvement Approaches for Increasing the Response of Emergency Departments against the COVID-19 Pandemic: A Systematic Review

The COVID-19 pandemic has strongly affected the dynamics of Emergency Departments (EDs) worldwide and has accentuated the need for tackling different operational inefficiencies that decrease the quality of care provided to infected patients. The EDs continue to struggle against this outbreak by implementing strategies maximizing their performance within an uncertain healthcare environment. The efforts, however, have remained insufficient in view of the growing number of admissions and increased severity of the coronavirus disease. Therefore, the primary aim of this paper is to review the literature on process improvement interventions focused on increasing the ED response to the current COVID-19 outbreak to delineate future research lines based on the gaps detected in the practical scenario. Therefore, we applied the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to perform a review containing the research papers published between December 2019 and April 2021 using ISI Web of Science, Scopus, PubMed, IEEE, Google Scholar, and Science Direct databases. The articles were further classified taking into account the research domain, primary aim, journal, and publication year. A total of 65 papers disseminated in 51 journals were concluded to satisfy the inclusion criteria. Our review found that most applications have been directed towards predicting the health outcomes in COVID-19 patients through machine learning and data analytics techniques. In the overarching pandemic, healthcare decision makers are strongly recommended to integrate artificial intelligence techniques with approaches from the operations research (OR) and quality management domains to upgrade the ED performance under social-economic restrictions.

Managing Pandemics with Health Informatics: Successes and Challenges

INTRODUCTION

The novel COVID-19 pandemic struck the world unprepared. This keynote outlines challenges and successes using data to inform providers, government officials, hospitals, and patients in a pandemic.

METHODS

The authors outline the data required to manage a novel pandemic including their potential uses by governments, public health organizations, and individuals.

RESULTS

An extensive discussion on data quality and on obstacles to collecting data is followed by examples of successes in clinical care, contact tracing, and forecasting. Generic local forecast model development is reviewed followed by ethical consideration around pandemic data. We leave the reader with thoughts on the next inevitable outbreak and lessons learned from the COVID-19 pandemic.

CONCLUSION

COVID-19 must be a lesson for the future to direct us to better planning and preparing to manage the next pandemic with health informatics.

Inter-rater reliability and prospective validation of a clinical prediction rule for SARS-CoV-2 infection

Objectives

Accurate estimation of the risk of SARS‐CoV‐2 infection based on bedside data alone has importance to emergency department (ED) operations and throughput. The 13‐item CORC (COVID [or coronavirus] Rule‐out Criteria) rule had good overall diagnostic accuracy in retrospective derivation and validation. The objective of this study was to prospectively test the inter‐rater reliability and diagnostic accuracy of the CORC score and rule (score ≤ 0 negative, > 0 positive) and compare the CORC rule performance with physician gestalt.

Methods

This noninterventional study was conducted at an urban academic ED from February 2021 to March 2021. Two practitioners were approached by research coordinators and asked to independently complete a form capturing the CORC criteria for their shared patient and their gestalt binary prediction of the SARS‐CoV‐2 test result and confidence (0%–100%). The criterion standard for SARS‐CoV‐2 was from reverse transcriptase polymerase chain reaction performed on a nasopharyngeal swab. The primary analysis was from weighted Cohen's kappa and likelihood ratios (LRs).

Results

For 928 patients, agreement between observers was good for the total CORC score, κ = 0.613 (95% confidence interval [CI] = 0.579–0.646), and for the CORC rule, κ = 0.644 (95% CI = 0.591–0.697). The agreement for clinician gestalt binary determination of SARs‐CoV‐2 status was κ = 0.534 (95% CI = 0.437–0.632) with median confidence of 76% (first–third quartile = 66–88.5). For 425 patients who had the criterion standard, a negative CORC rule (both observers scored CORC < 0), the sensitivity was 88%, and specificity was 51%, with a negative LR (LR−) of 0.24 (95% CI = 0.10–0.50). Among patients with a mean CORC score of >4, the prevalence of a positive SARS‐CoV‐2 test was 58% (95% CI = 28%–85%) and positive LR was 13.1 (95% CI = 4.5–37.2). Clinician gestalt demonstrated a sensitivity of 51% and specificity of 86% with a LR− of 0.57 (95% CI = 0.39–0.74).

Conclusion

In this prospective study, the CORC score and rule demonstrated good inter‐rater reliability and reproducible diagnostic accuracy for estimating the pretest probability of SARs‐CoV‐2 infection.

A systematic review of prediction models to diagnose COVID-19 in adults admitted to healthcare centers

BACKGROUND

The COVID-19 pandemic is putting significant pressure on the hospital system. To help clinicians in the rapid triage of patients at high risk of COVID-19 while waiting for RT-PCR results, different diagnostic prediction models have been developed. Our objective is to identify, compare, and evaluate performances of prediction models for the diagnosis of COVID-19 in adult patients in a health care setting.

METHODS

A search for relevant references has been conducted on the MEDLINE and Scopus databases. Rigorous eligibility criteria have been established (e.g., adult participants, suspicion of COVID-19, medical setting) and applied by two independent investigators to identify suitable studies at 2 different stages: (1) titles and abstracts screening and (2) full-texts screening. Risk of bias (RoB) has been assessed using the Prediction model study Risk of Bias Assessment Tool (PROBAST). Data synthesis has been presented according to a narrative report of findings.

RESULTS

Out of the 2334 references identified by the literature search, 13 articles have been included in our systematic review. The studies, carried out all over the world, were performed in 2020. The included articles proposed a model developed using different methods, namely, logistic regression, score, machine learning, XGBoost. All the included models performed well to discriminate adults at high risks of presenting COVID-19 (all area under the ROC curve (AUROC) > 0.500). The best AUROC was observed for the model of Kurstjens et al (AUROC = 0.940 (0.910-0.960), which was also the model that achieved the highest sensitivity (98%). RoB was evaluated as low in general.

CONCLUSION

Thirteen models have been developed since the start of the pandemic in order to diagnose COVID-19 in suspected patients from health care centers. All these models are effective, to varying degrees, in identifying whether patients were at high risk of having COVID-19.

Clinical prediction rule for SARS-CoV-2 infection from 116 U.S. emergency departments 2-22-2021

Objectives

Accurate and reliable criteria to rapidly estimate the probability of infection with the novel coronavirus-2 that causes the severe acute respiratory syndrome (SARS-CoV-2) and associated disease (COVID-19) remain an urgent unmet need, especially in emergency care. The objective was to derive and validate a clinical prediction score for SARS-CoV-2 infection that uses simple criteria widely available at the point of care.

Methods

Data came from the registry data from the national REgistry of suspected COVID-19 in EmeRgency care (RECOVER network) comprising 116 hospitals from 25 states in the US. Clinical variables and 30-day outcomes were abstracted from medical records of 19,850 emergency department (ED) patients tested for SARS-CoV-2. The criterion standard for diagnosis of SARS-CoV-2 required a positive molecular test from a swabbed sample or positive antibody testing within 30 days. The prediction score was derived from a 50% random sample (n = 9,925) using unadjusted analysis of 107 candidate variables as a screening step, followed by stepwise forward logistic regression on 72 variables.

Results

Multivariable regression yielded a 13-variable score, which was simplified to a 13-point score: +1 point each for age>50 years, measured temperature>37.5°C, oxygen saturation<95%, Black race, Hispanic or Latino ethnicity, household contact with known or suspected COVID-19, patient reported history of dry cough, anosmia/dysgeusia, myalgias or fever; and -1 point each for White race, no direct contact with infected person, or smoking. In the validation sample (n = 9,975), the probability from logistic regression score produced an area under the receiver operating characteristic curve of 0.80 (95% CI: 0.79–0.81), and this level of accuracy was retained across patients enrolled from the early spring to summer of 2020. In the simplified score, a score of zero produced a sensitivity of 95.6% (94.8–96.3%), specificity of 20.0% (19.0–21.0%), negative likelihood ratio of 0.22 (0.19–0.26). Increasing points on the simplified score predicted higher probability of infection (e.g., >75% probability with +5 or more points).

Conclusion

Criteria that are available at the point of care can accurately predict the probability of SARS-CoV-2 infection. These criteria could assist with decisions about isolation and testing at high throughput checkpoints.

Establishing Classifiers With Clinical Laboratory Indicators to Distinguish COVID-19 From Community-Acquired Pneumonia: Retrospective Cohort Study

Background

The initial symptoms of patients with COVID-19 are very much like those of patients with community-acquired pneumonia (CAP); it is difficult to distinguish COVID-19 from CAP with clinical symptoms and imaging examination.

Objective

The objective of our study was to construct an effective model for the early identification of COVID-19 that would also distinguish it from CAP.

Methods

The clinical laboratory indicators (CLIs) of 61 COVID-19 patients and 60 CAP patients were analyzed retrospectively. Random combinations of various CLIs (ie, CLI combinations) were utilized to establish COVID-19 versus CAP classifiers with machine learning algorithms, including random forest classifier (RFC), logistic regression classifier, and gradient boosting classifier (GBC). The performance of the classifiers was assessed by calculating the area under the receiver operating characteristic curve (AUROC) and recall rate in COVID-19 prediction using the test data set.

Results

The classifiers that were constructed with three algorithms from 43 CLI combinations showed high performance (recall rate >0.9 and AUROC >0.85) in COVID-19 prediction for the test data set. Among the high-performance classifiers, several CLIs showed a high usage rate; these included procalcitonin (PCT), mean corpuscular hemoglobin concentration (MCHC), uric acid, albumin, albumin to globulin ratio (AGR), neutrophil count, red blood cell (RBC) count, monocyte count, basophil count, and white blood cell (WBC) count. They also had high feature importance except for basophil count. The feature combination (FC) of PCT, AGR, uric acid, WBC count, neutrophil count, basophil count, RBC count, and MCHC was the representative one among the nine FCs used to construct the classifiers with an AUROC equal to 1.0 when using the RFC or GBC algorithms. Replacing any CLI in these FCs would lead to a significant reduction in the performance of the classifiers that were built with them.

Conclusions

The classifiers constructed with only a few specific CLIs could efficiently distinguish COVID-19 from CAP, which could help clinicians perform early isolation and centralized management of COVID-19 patients.