Predicting in-hospital mortality and unanticipated admissions to the intensive care unit using routinely collected blood tests and vital signs: Development and validation of a multivariable model

Development of an enhanced scoring system to predict ICU readmission or in-hospital death within 24 hours using routine patient data from two NHS Foundation Trusts

RATIONALE

Intensive care units (ICUs) admit the most severely ill patients. Once these patients are discharged from the ICU to a step-down ward, they continue to have their vital signs monitored by nursing staff, with Early Warning Score (EWS) systems being used to identify those at risk of deterioration.

OBJECTIVES

We report the development and validation of an enhanced continuous scoring system for predicting adverse events, which combines vital signs measured routinely on acute care wards (as used by most EWS systems) with a risk score of a future adverse event calculated on discharge from the ICU.

DESIGN

A modified Delphi process identified candidate variables commonly available in electronic records as the basis for a 'static' score of the patient's condition immediately after discharge from the ICU. L1-regularised logistic regression was used to estimate the in-hospital risk of future adverse event. We then constructed a model of physiological normality using vital sign data from the day of hospital discharge. This is combined with the static score and used continuously to quantify and update the patient's risk of deterioration throughout their hospital stay.

SETTING

Data from two National Health Service Foundation Trusts (UK) were used to develop and (externally) validate the model.

PARTICIPANTS

A total of 12 394 vital sign measurements were acquired from 273 patients after ICU discharge for the development set, and 4831 from 136 patients in the validation cohort.

RESULTS

Outcome validation of our model yielded an area under the receiver operating characteristic curve of 0.724 for predicting ICU readmission or in-hospital death within 24 hours. It showed an improved performance with respect to other competitive risk scoring systems, including the National EWS (0.653).

CONCLUSIONS

We showed that a scoring system incorporating data from a patient's stay in the ICU has better performance than commonly used EWS systems based on vital signs alone.

TRIAL REGISTRATION NUMBER

ISRCTN32008295.

Safer and more efficient vital signs monitoring protocols to identify the deteriorating patients in the general hospital ward: an observational study

Background

The frequency at which patients should have their vital signs (e.g. blood pressure, pulse, oxygen saturation) measured on hospital wards is currently unknown. Current National Health Service monitoring protocols are based on expert opinion but supported by little empirical evidence. The challenge is finding the balance between insufficient monitoring (risking missing early signs of deterioration and delays in treatment) and over-observation of stable patients (wasting resources needed in other aspects of care).

Objective

Provide an evidence-based approach to creating monitoring protocols based on a patient's risk of deterioration and link these to nursing workload and economic impact.

Design

Our study consisted of two parts: (1) an observational study of nursing staff to ascertain the time to perform vital sign observations; and (2) a retrospective study of historic data on patient admissions exploring the relationships between National Early Warning Score and risk of outcome over time. These were underpinned by opinions and experiences from stakeholders.

Setting and participants

Observational study: observed nursing staff on 16 randomly selected adult general wards at four acute National Health Service hospitals. Retrospective study: extracted, linked and analysed routinely collected data from two large National Health Service acute trusts; data from over 400,000 patient admissions and 9,000,000 vital sign observations.

Results

Observational study found a variety of practices, with two hospitals having registered nurses take the majority of vital sign observations and two favouring healthcare assistants or student nurses. However, whoever took the observations spent roughly the same length of time. The average was 5:01 minutes per observation over a 'round', including time to locate and prepare the equipment and travel to the patient area. Retrospective study created survival models predicting the risk of outcomes over time since the patient was last observed. For low-risk patients, there was little difference in risk between 4 hours and 24 hours post observation.

Conclusions

We explored several different scenarios with our stakeholders (clinicians and patients), based on how 'risk' could be managed in different ways. Vital sign observations are often done more frequently than necessary from a bald assessment of the patient's risk, and we show that a maximum threshold of risk could theoretically be achieved with less resource. Existing resources could therefore be redeployed within a changed protocol to achieve better outcomes for some patients without compromising the safety of the rest. Our work supports the approach of the current monitoring protocol, whereby patients' National Early Warning Score 2 guides observation frequency. Existing practice is to observe higher-risk patients more frequently and our findings have shown that this is objectively justified. It is worth noting that important nurse-patient interactions take place during vital sign monitoring and should not be eliminated under new monitoring processes. Our study contributes to the existing evidence on how vital sign observations should be scheduled. However, ultimately, it is for the relevant professionals to decide how our work should be used.

Study registration

This study is registered as ISRCTN10863045.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: 17/05/03) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 6. See the NIHR Funding and Awards website for further award information.

An Evaluation of the National Early Warning Score 2 and the Laboratory Data Decision Tree Early Warning Score in Predicting Mortality in Geriatric Patients

BACKGROUND

Due to the high rate of geriatric patient visits, scoring systems are needed to predict increasing mortality rates.

OBJECTIVE

In this study, we aimed to investigate the in-hospital mortality prediction power of the National Early Warning Score 2 (NEWS2) and the Laboratory Data Decision Tree Early Warning Score (LDT-EWS), which consists of frequently performed laboratory parameters.

METHODS

We retrospectively analyzed 651 geriatric patients who visited the emergency department (ED), were not discharged on the same day from ED, and were hospitalized. The patients were categorized according to their in-hospital mortality status. The NEWS2 and LDT-EWS values of these patients were calculated and compared on the basis of deceased and living patients.

RESULTS

Median (interquartile range [IQR]) NEWS2 and LDT-EWS values of the 127 patients who died were found to be statistically significantly higher than those of the patients who survived (NEWS2: 5 [3-8] vs. 3 [1-5]; p < 0.001; LDT-EWS: 8 [7-10] vs. 6 [5-8]; p < 0.001). In the receiver operating characteristic curve analysis, the NEWS2, LDT-EWS, and NEWS2+LDT-EWS-formed by the sum of the two scoring systems-resulted in 0.717, 0.705, and 0.775 area under curve values, respectively.

CONCLUSIONS

The NEWS2 and LDT-EWS were found to be valuable for predicting in-hospital mortality in geriatric patients. The power of the NEWS2 to predict in-hospital mortality increased when used with the LDT-EWS.

Advancing prognostic precision in pulmonary embolism: A clinical and laboratory-based artificial intelligence approach for enhanced early mortality risk stratification

BACKGROUND

Acute pulmonary embolism (PE) is a critical medical emergency that necessitates prompt identification and intervention. Accurate prognostication of early mortality is vital for recognizing patients at elevated risk for unfavourable outcomes and administering suitable therapy. Machine learning (ML) algorithms hold promise for enhancing the precision of early mortality prediction in PE patients.

OBJECTIVE

To devise an ML algorithm for early mortality prediction in PE patients by employing clinical and laboratory variables.

METHODS

This study utilized diverse oversampling techniques to improve the performance of various machine learning models including ANN, SVM, DT, RF, and AdaBoost for early mortality prediction. Appropriate oversampling methods were chosen for each model based on algorithm characteristics and dataset properties. Predictor variables included four lab tests, eight physiological time series indicators, and two general descriptors. Evaluation used metrics like accuracy, F1_score, precision, recall, Area Under the Curve (AUC) and Receiver Operating Characteristic (ROC) curves, providing a comprehensive view of models' predictive abilities.

RESULTS

The findings indicated that the RF model with random oversampling exhibited superior performance among the five models assessed, achieving elevated accuracy and precision alongside high recall for predicting the death class. The oversampling approaches effectively equalized the sample distribution among the classes and enhanced the models' performance.

CONCLUSIONS

The suggested ML technique can efficiently prognosticate mortality in patients afflicted with acute PE. The RF model with random oversampling can aid healthcare professionals in making well-informed decisions regarding the treatment of patients with acute PE. The study underscores the significance of oversampling methods in managing imbalanced data and emphasizes the potential of ML algorithms in refining early mortality prediction for PE patients.

Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges

The rapid adoption of electronic health record (EHR) systems in US hospitals from 2008 to 2014 produced novel data elements for analysis. Concurrent innovations in computing architecture and machine learning (ML) algorithms have made rapid consumption of health data feasible and a powerful engine for clinical innovation. In critical care research, the net convergence of these trends has resulted in an exponential increase in outcome prediction research. In the following article, we explore the history of outcome prediction in the intensive care unit (ICU), the growing use of EHR data, and the rise of artificial intelligence and ML (AI) in critical care.

Prediction of postoperative patient deterioration and unanticipated intensive care unit admission using perioperative factors

BACKGROUND AND OBJECTIVES

Currently, no evidence-based criteria exist for decision making in the post anesthesia care unit (PACU). This could be valuable for the allocation of postoperative patients to the appropriate level of care and beneficial for patient outcomes such as unanticipated intensive care unit (ICU) admissions. The aim is to assess whether the inclusion of intra- and postoperative factors improves the prediction of postoperative patient deterioration and unanticipated ICU admissions.

METHODS

A retrospective observational cohort study was performed between January 2013 and December 2017 in a tertiary Dutch hospital. All patients undergoing surgery in the study period were selected. Cardiothoracic surgeries, obstetric surgeries, catheterization lab procedures, electroconvulsive therapy, day care procedures, intravenous line interventions and patients under the age of 18 years were excluded. The primary outcome was unanticipated ICU admission.

RESULTS

An unanticipated ICU admission complicated the recovery of 223 (0.9%) patients. These patients had higher hospital mortality rates (13.9% versus 0.2%, p<0.001). Multivariable analysis resulted in predictors of unanticipated ICU admissions consisting of age, body mass index, general anesthesia in combination with epidural anesthesia, preoperative score, diabetes, administration of vasopressors, erythrocytes, duration of surgery and post anesthesia care unit stay, and vital parameters such as heart rate and oxygen saturation. The receiver operating characteristic curve of this model resulted in an area under the curve of 0.86 (95% CI 0.83-0.88).

CONCLUSIONS

The prediction of unanticipated ICU admissions from electronic medical record data improved when the intra- and early postoperative factors were combined with preoperative patient factors. This emphasizes the need for clinical decision support tools in post anesthesia care units with regard to postoperative patient allocation.

Novel predictors of mortality in emergency bowel surgery: a single-centre cohort study

Pre-operative risk stratification is a key part of the care pathway for emergency bowel surgery, as it facilitates the identification of high-risk patients. Several novel risk scores have recently been published that are designed to identify patients who are frail or significantly unwell. They can also be calculated pre-operatively from routinely collected clinical data. This study aimed to investigate the ability of these scores to predict 30-day mortality after emergency bowel surgery. A single centre cohort study was performed using our local data from the National Emergency Laparotomy Audit database. Further data were extracted from electronic hospital records (n = 1508). The National Early Warning Score, Laboratory Decision Tree Early Warning Score and Hospital Frailty Risk Score were then calculated. The most abnormal National or Laboratory Decision Tree Early Warning Score in the 24 or 72 h before surgery was used in analysis. Individual scores were reasonable predictors of mortality (c-statistic 0.699-0.740) but all were poorly calibrated. A National Early Warning Score ≥ 4 was associated with a high overall mortality rate (> 10%). A logistic regression model was developed using age, National Early Warning Score, Laboratory Decision Tree Early Warning Score and Hospital Frailty Risk Score as predictor variables, and its performance compared with other established risk models. The model demonstrated good discrimination and calibration (c-statistic 0.827) but was marginally outperformed by the National Emergency Laparotomy Audit score (c-statistic 0.861). All other models compared performed less well (c-statistics 0.734-0.808). Pre-operative patient vital signs, blood tests and markers of frailty can be used to accurately predict the risk of 30-day mortality after emergency bowel surgery.

Development and validation of nomogram for unplanned ICU admission in patients with dilated cardiomyopathy

Objective

Unplanned admission to the intensive care unit (ICU) is the major in-hospital adverse event for patients with dilated cardiomyopathy (DCM). We aimed to establish a nomogram of individualized risk prediction for unplanned ICU admission in DCM patients.

Methods

A total of 2,214 patients diagnosed with DCM from the First Affiliated Hospital of Xinjiang Medical University from January 01, 2010, to December 31, 2020, were retrospectively analyzed. Patients were randomly divided into training and validation groups at a 7:3 ratio. The least absolute shrinkage and selection operator and multivariable logistic regression analysis were used for nomogram model development. The area under the receiver operating characteristic curve, calibration curves, and decision curve analysis (DCA) were used to evaluate the model. The primary outcome was defined as unplanned ICU admission.

Results

A total of 209 (9.44%) patients experienced unplanned ICU admission. The variables in our final nomogram included emergency admission, previous stroke, New York Heart Association Class, heart rate, neutrophil count, and levels of N-terminal pro b-type natriuretic peptide. In the training group, the nomogram showed good calibration (Hosmer-Lemeshow χ 2 = 14.40, P = 0.07) and good discrimination, with an optimal-corrected C-index of 0.76 (95% confidence interval: 0.72-0.80). DCA confirmed the clinical net benefit of the nomogram model, and the nomogram maintained excellent performances in the validation group.

Conclusion

This is the first risk prediction model for predicting unplanned ICU admission in patients with DCM by simply collecting clinical information. This model may assist physicians in identifying individuals at a high risk of unplanned ICU admission for DCM inpatients.

Multi-center validation of machine learning model for preoperative prediction of postoperative mortality

Accurate prediction of postoperative mortality is important for not only successful postoperative patient care but also for information-based shared decision-making with patients and efficient allocation of medical resources. This study aimed to create a machine-learning prediction model for 30-day mortality after a non-cardiac surgery that adapts to the manageable amount of clinical information as input features and is validated against multi-centered rather than single-centered data. Data were collected from 454,404 patients over 18 years of age who underwent non-cardiac surgeries from four independent institutions. We performed a retrospective analysis of the retrieved data. Only 12–18 clinical variables were used for model training. Logistic regression, random forest classifier, extreme gradient boosting (XGBoost), and deep neural network methods were applied to compare the prediction performances. To reduce overfitting and create a robust model, bootstrapping and grid search with tenfold cross-validation were performed. The XGBoost method in Seoul National University Hospital (SNUH) data delivers the best performance in terms of the area under receiver operating characteristic curve (AUROC) (0.9376) and the area under the precision-recall curve (0.1593). The predictive performance was the best when the SNUH model was validated with Ewha Womans University Medical Center data (AUROC, 0.941). Preoperative albumin, prothrombin time, and age were the most important features in the model for each hospital. It is possible to create a robust artificial intelligence prediction model applicable to multiple institutions through a light predictive model using only minimal preoperative information that can be automatically extracted from each hospital.

Which model is superior in predicting ICU survival: artificial intelligence versus conventional approaches

Background

A disease severity classification system is widely used to predict the survival of patients admitted to the intensive care unit with different diagnoses. In the present study, conventional severity classification systems were compared with artificial intelligence predictive models (Artificial Neural Network and Decision Tree) in terms of the prediction of the survival rate of the patients admitted to the intensive care unit.

Methods

This retrospective cohort study was performed on the data of the patients admitted to the ICU of Ghaemshahr’s Razi Teaching Care Center from March 20th, 2017, to September 22nd, 2019. The required data for calculating conventional severity classification models (SOFA, SAPS II, APACHE II, and APACHE IV) were collected from the patients’ medical records. Subsequently, the score of each model was calculated. Artificial intelligence predictive models (Artificial Neural Network and Decision Tree) were developed in the next step. Lastly, the performance of each model in predicting the survival of the patients admitted to the intensive care unit was evaluated using the criteria of sensitivity, specificity, accuracy, F-measure, and area under the ROC curve. Also, each model was validated externally. The R program, version 4.1, was used to create the artificial intelligence models, and SPSS Statistics Software, version 21, was utilized to perform statistical analysis.

Results

The area under the ROC curve of SOFA, SAPS II, APACHE II, APACHE IV, multilayer perceptron artificial neural network, and CART decision tree were 76.0, 77.1, 80.3, 78.5, 84.1, and 80.0, respectively.

Conclusion

The results showed that although the APACHE II model had better results than other conventional models in predicting the survival rate of the patients admitted to the intensive care unit, the other conventional models provided acceptable results too. Moreover, the findings showed that the artificial neural network model had the best performance among all the studied models, indicating the discrimination power of this model in predicting patient survival compared to the other models.

Artificial intelligence and clinical deterioration

PURPOSE OF REVIEW

To provide an overview of the systems being used to identify and predict clinical deterioration in hospitalised patients, with focus on the current and future role of artificial intelligence (AI).

RECENT FINDINGS

There are five leading AI driven systems in this field: the Advanced Alert Monitor (AAM), the electronic Cardiac Arrest Risk Triage (eCART) score, Hospital wide Alert Via Electronic Noticeboard, the Mayo Clinic Early Warning Score, and the Rothman Index (RI). Each uses Electronic Patient Record (EPR) data and machine learning to predict adverse events. Less mature but relevant evolutions are occurring in the fields of Natural Language Processing, Time and Motion Studies, AI Sepsis and COVID-19 algorithms.

SUMMARY

Research-based AI-driven systems to predict clinical deterioration are increasingly being developed, but few are being implemented into clinical workflows. Escobar et al. (AAM) provide the current gold standard for robust model development and implementation methodology. Multiple technologies show promise, however, the pathway to meaningfully affect patient outcomes remains challenging.

Age-adjusted interpretation of biomarkers of renal function and homeostasis, inflammation, and circulation in Emergency Department patients

Appropriate interpretation of blood tests is important for risk stratification and guidelines used in the Emergency Department (ED) (such as SIRS or CURB-65). The impact of abnormal blood test values on mortality may change with increasing age due to (patho)-physiologic changes. The aim of this study was therefore to assess the effect of age on the case-mix adjusted association between biomarkers of renal function and homeostasis, inflammation and circulation and in-hospital mortality. This observational multi-center cohort study has used the Netherlands Emergency department Evaluation Database (NEED), including all consecutive ED patients ≥ 18 years of three hospitals. A generalized additive logistic regression model was used to visualize the association between in-hospital mortality, age and five blood tests (creatinine, sodium, leukocytes, C-reactive Protein, and hemoglobin). Multivariable logistic regression analyses were used to assess the association between the number of abnormal blood test values and mortality per age category (18-50; 51-65; 66-80; > 80 years). Of the 94,974 included patients, 2550 (2.7%) patients died in-hospital. Mortality increased gradually for C-reactive Protein (CRP), and had a U-shaped association for creatinine, sodium, leukocytes, and hemoglobin. Age significantly affected the associations of all studied blood tests except in leukocytes. In addition, with increasing age categories, case-mix adjusted mortality increased with the number of abnormal blood tests. In summary, the association between blood tests and (adjusted) mortality depends on age. Mortality increases gradually or in a U-shaped manner with increasing blood test values. Age-adjusted numerical scores may improve risk stratification. Our results have implications for interpretation of blood tests and their use in risk stratification tools and acute care guidelines.Trial registration number Netherlands Trial Register (NTR) NL8422, 03/2020.

Vital Signs Prediction for COVID-19 Patients in ICU

This study introduces machine learning predictive models to predict the future values of the monitored vital signs of COVID-19 ICU patients. The main vital sign predictors include heart rate, respiration rate, and oxygen saturation. We investigated the performances of the developed predictive models by considering different approaches. The first predictive model was developed by considering the following vital signs: heart rate, blood pressure (systolic, diastolic and mean arterial, pulse pressure), respiration rate, and oxygen saturation. Similar to the first approach, the second model was developed using the same vital signs, but it was trained and tested based on a leave-one-subject-out approach. The third predictive model was developed by considering three vital signs: heart rate (HR), respiration rate (RR), and oxygen saturation (SpO2). The fourth model was a leave-one-subject-out model for the three vital signs. Finally, the fifth predictive model was developed based on the same three vital signs, but with a five-minute observation rate, in contrast with the aforementioned four models, where the observation rate was hourly to bi-hourly. For the five models, the predicted measurements were those of the three upcoming observations (on average, three hours ahead). Based on the obtained results, we observed that by limiting the number of vital sign predictors (i.e., three vital signs), the prediction performance was still acceptable, with the average mean absolute percentage error (MAPE) being 12%,5%, and 21.4% for heart rate, oxygen saturation, and respiration rate, respectively. Moreover, increasing the observation rate could enhance the prediction performance to be, on average, 8%,4.8%, and 17.8% for heart rate, oxygen saturation, and respiration rate, respectively. It is envisioned that such models could be integrated with monitoring systems that could, using a limited number of vital signs, predict the health conditions of COVID-19 ICU patients in real-time.

The Impact of Age on Predictive Performance of National Early Warning Score at Arrival to Emergency Departments: Development and External Validation

STUDY OBJECTIVE

To investigate how age affects the predictive performance of the National Early Warning Score (NEWS) at arrival to the emergency department (ED) regarding inhospital mortality and intensive care admission.

METHODS

International multicenter retrospective cohorts from 2 Danish and 3 Dutch ED. Development cohort: 14,809 Danish patients aged ≥18 years with at least systolic blood pressure or pulse measured from the Danish Multicenter Cohort. External validation cohort: 50,448 Dutch patients aged ≥18 years with all vital signs measured from the Netherlands Emergency Department Evaluation Database (NEED). Multivariable logistic regression was used for model building. Performance was evaluated overall and within age categories: 18 to 64 years, 65 to 80 years, and more than 80 years.

RESULTS

In the Danish Multicenter Cohort, a total of 2.5% died inhospital, and 2.8% were admitted to the ICU, compared with 2.8% and 1.6%, respectively, in the NEED. Age did not add information for the prediction of intensive care admission but was the strongest predictor for inhospital mortality. For NEWS alone, severe underestimation of risk was observed for persons above 80 while overall Area Under Receiver Operating Characteristic (AUROC) was 0.82 (confidence interval [CI] 0.80 to 0.84) in the Danish Multicenter Cohort versus 0.75 (CI 0.75 to 0.77) in the NEED. When combining NEWS with age, underestimation of risks was eliminated for persons above 80, and overall AUROC increased significantly to 0.86 (CI 0.85 to 0.88) in the Danish Multicenter Cohort versus 0.82 (CI 0.81 to 0.83) in the NEED.

CONCLUSION

Combining NEWS with age improved the prediction performance regarding inhospital mortality, mostly for persons aged above 80, and can potentially improve decision policies at arrival to EDs.

Predicting Patient Deterioration: A Review of Tools in the Digital Hospital Setting

BACKGROUND

Early warning tools identify patients at risk of deterioration in hospitals. Electronic medical records in hospitals offer real-time data and the opportunity to automate early warning tools and provide real-time, dynamic risk estimates.

OBJECTIVE

This review describes published studies on the development, validation, and implementation of tools for predicting patient deterioration in general wards in hospitals.

METHODS

An electronic database search of peer reviewed journal papers from 2008-2020 identified studies reporting the use of tools and algorithms for predicting patient deterioration, defined by unplanned transfer to the intensive care unit, cardiac arrest, or death. Studies conducted solely in intensive care units, emergency departments, or single diagnosis patient groups were excluded.

RESULTS

A total of 46 publications were eligible for inclusion. These publications were heterogeneous in design, setting, and outcome measures. Most studies were retrospective studies using cohort data to develop, validate, or statistically evaluate prediction tools. The tools consisted of early warning, screening, or scoring systems based on physiologic data, as well as more complex algorithms developed to better represent real-time data, deal with complexities of longitudinal data, and warn of deterioration risk earlier. Only a few studies detailed the results of the implementation of deterioration warning tools.

CONCLUSIONS

Despite relative progress in the development of algorithms to predict patient deterioration, the literature has not shown that the deployment or implementation of such algorithms is reproducibly associated with improvements in patient outcomes. Further work is needed to realize the potential of automated predictions and update dynamic risk estimates as part of an operational early warning system for inpatient deterioration.

Machine learning based early mortality prediction in the emergency department

BACKGROUND

It is a great challenge for emergency physicians to early detect the patient's deterioration and prevent unexpected death through a large amount of clinical data, which requires sufficient experience and keen insight.

OBJECTIVE

To evaluate the performance of machine learning models in quantifying the severity of emergency department (ED) patients and identifying high-risk patients.

METHODS

Using routinely-available demographics, vital signs and laboratory tests extracted from electronic health records (EHRs), a framework based on machine learning and feature engineering was proposed for mortality prediction. Patients who had one complete record of vital signs and laboratory tests in ED were included. The following patients were excluded: pediatric patients aged < 18 years, pregnant woman, and patients died or were discharged or hospitalized within 12 h after admission. Based on 76 original features extracted, 9 machine learning models were adopted to validate our proposed framework. Their optimal hyper-parameters were fine-tuned using the grid search method. The prediction results were evaluated on performance metrics (i.e., accuracy, area under the curve (AUC), recall and precision) with repeated 5-fold cross-validation (CV). The time window from patient admission to the prediction was analyzed at 12 h, 24 h, 48 h, and entire stay.

RESULTS

We studied a total of 1114 ED patients with 71.54% (797/1114) survival and 28.46% (317/1114) death in the hospital. The results revealed a more complete time window leads to better prediction performance. Using the entire stay records, the LightGBM model with refined feature engineering demonstrated high discrimination and achieved 93.6% (±0.008) accuracy, 97.6% (±0.003) AUC, 97.1% (±0.008) recall, and 94.2% (±0.006) precision, even if no diagnostic information was utilized.

CONCLUSIONS

This study quantifies the criticality of ED patients and appears to have significant potential as a clinical decision support tool in assisting physicians in their clinical routine. While the model requires validation before use elsewhere, the same methodology could be used to create a strong model for the new hospital.

Detecting Deteriorating Patients in the Hospital: Development and Validation of a Novel Scoring System

Rationale: Late recognition of patient deterioration in hospital is associated with worse outcomes, including higher mortality. Despite the widespread introduction of early warning score (EWS) systems and electronic health records, deterioration still goes unrecognized. Objectives: To develop and externally validate a Hospital- wide Alerting via Electronic Noticeboard (HAVEN) system to identify hospitalized patients at risk of reversible deterioration. Methods: This was a retrospective cohort study of patients 16 years of age or above admitted to four UK hospitals. The primary outcome was cardiac arrest or unplanned admission to the ICU. We used patient data (vital signs, laboratory tests, comorbidities, and frailty) from one hospital to train a machine-learning model (gradient boosting trees). We internally and externally validated the model and compared its performance with existing scoring systems (including the National EWS, laboratory-based acute physiology score, and electronic cardiac arrest risk triage score). Measurements and Main Results: We developed the HAVEN model using 230,415 patient admissions to a single hospital. We validated HAVEN on 266,295 admissions to four hospitals. HAVEN showed substantially higher discrimination (c-statistic, 0.901 [95% confidence interval, 0.898-0.903]) for the primary outcome within 24 hours of each measurement than other published scoring systems (which range from 0.700 [0.696-0.704] to 0.863 [0.860-0.865]). With a precision of 10%, HAVEN was able to identify 42% of cardiac arrests or unplanned ICU admissions with a lead time of up to 48 hours in advance, compared with 22% by the next best system. Conclusions: The HAVEN machine-learning algorithm for early identification of in-hospital deterioration significantly outperforms other published scores such as the National EWS.

Development and validation of early warning score systems for COVID-19 patients

COVID‐19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of November 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. The ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high‐flow nasal oxygen, continuous positive airways pressure, non‐invasive ventilation, intubation) within a prediction window of 24 h is evaluated. It is shown that these scores perform sub‐optimally at this specific task. Therefore, an alternative EWS based on the Gradient Boosting Trees (GBT) algorithm is developed that is able to predict deterioration within the next 24 h with high AUROC 94% and an accuracy, sensitivity, and specificity of 70%, 96%, 70%, respectively. The GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.

Disposition Decision Support by Laboratory Based Outcome Prediction

Disposition is one of the main tasks in the emergency department. However, there is a lack of objective and reliable disposition criteria, and diagnosis-based risk prediction is not feasible at early time points. The aim was to derive a risk score (TRIAL) based on routinely collected baseline (TRIage level and Age) and Laboratory data—supporting disposition decisions by risk stratification based on mortality. We prospectively included consecutive patients presenting to the emergency department over 18 weeks. Data sets of routinely collected baseline (triage level and age) and laboratory data were used for multivariable logistic regression to develop the TRIAL risk score predicting mortality. Routine laboratory variables and disposition cut-offs were chosen beforehand by expert consensus. Risk stratification was based on low risk (<1%), intermediate risk (1–10%), and high risk (>10%) of in-hospital mortality. In total, 8687 data sets were analyzed. Variables identified to develop the TRIAL risk score were triage level (Emergency Severity Index), age, lactate dehydrogenase, creatinine, albumin, bilirubin, and leukocyte count. The area under the ROC curve for in-hospital mortality was 0.93. Stratification according to the TRIAL score showed that 67.5% of all patients were in the low-risk category. Mortality was 0.1% in low-risk, 3.5% in intermediate-risk, and 26.2% in high-risk patients. The TRIAL risk score based on routinely available baseline and laboratory data provides prognostic information for disposition decisions. TRIAL could be used to minimize admission in low-risk and to maximize observation in high-risk patients.

Machine learning-based prediction of in-hospital mortality using admission laboratory data: A retrospective, single-site study using electronic health record data

Risk assessment of in-hospital mortality of patients at the time of hospitalization is necessary for determining the scale of required medical resources for the patient depending on the patient's severity. Because recent machine learning application in the clinical area has been shown to enhance prediction ability, applying this technique to this issue can lead to an accurate prediction model for in-hospital mortality prediction. In this study, we aimed to generate an accurate prediction model of in-hospital mortality using machine learning techniques. Patients 18 years of age or older admitted to the University of Tokyo Hospital between January 1, 2009 and December 26, 2017 were used in this study. The data were divided into a training/validation data set (n = 119,160) and a test data set (n = 33,970) according to the time of admission. The prediction target of the model was the in-hospital mortality within 14 days. To generate the prediction model, 25 variables (age, sex, 21 laboratory test items, length of stay, and mortality) were used to predict in-hospital mortality. Logistic regression, random forests, multilayer perceptron, and gradient boost decision trees were performed to generate the prediction models. To evaluate the prediction capability of the model, the model was tested using a test data set. Mean probabilities obtained from trained models with five-fold cross-validation were used to calculate the area under the receiver operating characteristic (AUROC) curve. In a test stage using the test data set, prediction models of in-hospital mortality within 14 days showed AUROC values of 0.936, 0.942, 0.942, and 0.938 for logistic regression, random forests, multilayer perceptron, and gradient boosting decision trees, respectively. Machine learning-based prediction of short-term in-hospital mortality using admission laboratory data showed outstanding prediction capability and, therefore, has the potential to be useful for the risk assessment of patients at the time of hospitalization.

Consensus on predictors of clinical deterioration in emergency departments: A Delphi process study

AIM

The study aim was to determine relevance and applicability of generic predictors of clinical deterioration in emergency departments based on consensus among clinicians.

METHODS

Thirty-three predictors of clinical deterioration identified from literature were assessed in a modified two-stage Delphi-process. Sixty-eight clinicians (physicians and nurses) participated in the first round and 48 in the second round; all treating hospitalized patients in Danish emergency departments, some with pre-hospital experience. The panel rated the predictors for relevance (relevant marker of clinical deterioration) and applicability (change in clinical presentation over time, generic in nature and possible to detect bedside). They rated their level of agreement on a 9-point Likert scale and were also invited to propose additional generic predictors between the rounds. New predictors suggested by more than one clinician were included in the second round along with non-consensus predictors from the first round. Final decisions of non-consensus predictors after second round were made by a research group and an impartial physician.

RESULTS

The Delphi-process resulted in 19 clinically relevant and applicable predictors based on vital signs and parameters (respiratory rate, saturation, dyspnoea, systolic blood pressure, pulse rate, abnormal electrocardiogram, altered mental state and temperature), biochemical tests (serum c-reactive protein, serum bicarbonate, serum lactate, serum pH, serum potassium, glucose, leucocyte counts and serum haemoglobin), objective clinical observations (skin conditions) and subjective clinical observations (pain reported as new or escalating, and relatives' concerns).

CONCLUSION

The Delphi-process led to consensus of 19 potential predictors of clinical deterioration widely accepted as relevant and applicable in emergency departments.

A deep learning backcasting approach to the electrolyte, metabolite, and acid-base parameters that predict risk in ICU patients

Background

A powerful risk model allows clinicians, at the bedside, to ensure the early identification of and decision-making for patients showing signs of developing physiological instability during treatment. The aim of this study was to enhance the identification of patients at risk for deterioration through an accurate model using electrolyte, metabolite, and acid-base parameters near the end of patients’ intensive care unit (ICU) stays.

Methods

This retrospective study included 5157 adult patients during the last 72 hours of their ICU stays. The patients from the MIMIC-III database who had serum lactate, pH, bicarbonate, potassium, calcium, glucose, chloride, and sodium values available, along with the times at which those data were recorded, were selected. Survivor data from the last 24 hours before discharge and four sets of nonsurvivor data from 48–72, 24–48, 8–24, and 0–8 hours before death were analyzed. Deep learning (DL), random forest (RF) and generalized linear model (GLM) analyses were applied for model construction and compared in terms of performance according to the area under the receiver operating characteristic curve (AUC). A DL backcasting approach was used to assess predictors of death vs. discharge up to 72 hours in advance.

Results

The DL, RF and GLM models achieved the highest performance for nonsurvivors 0–8 hours before death versus survivors compared with nonsurvivors 8–24, 24–48 and 48–72 hours before death versus survivors. The DL assessment outperformed the RF and GLM assessments and achieved discrimination, with an AUC of 0.982, specificity of 0.947, and sensitivity of 0.935. The DL backcasting approach achieved discrimination with an AUC of 0.898 compared with the DL native model of nonsurvivors from 8–24 hours before death versus survivors with an AUC of 0.894. The DL backcasting approach achieved discrimination with an AUC of 0.871 compared with the DL native model of nonsurvivors from 48–72 hours before death versus survivors with an AUC of 0.846.

Conclusions

The DL backcasting approach could be used to simultaneously monitor changes in the electrolyte, metabolite, and acid-base parameters of patients who develop physiological instability during ICU treatment and predict the risk of death over a period of hours to days.

Let Sleeping Patients Lie, avoiding unnecessary overnight vitals monitoring using a clinically based deep-learning model

Impaired sleep for hospital patients is an all too common reality. Sleep disruptions due to unnecessary overnight vital sign monitoring are associated with delirium, cognitive impairment, weakened immunity, hypertension, increased stress, and mortality. It is also one of the most common complaints of hospital patients while imposing additional burdens on healthcare providers. Previous efforts to forgo overnight vital sign measurements and improve patient sleep used providers’ subjective stability assessment or utilized an expanded, thus harder to retrieve, set of vitals and laboratory results to predict overnight clinical risk. Here, we present a model that incorporates past values of a small set of vital signs and predicts overnight stability for any given patient-night. Using data obtained from a multi-hospital health system between 2012 and 2019, a recurrent deep neural network was trained and evaluated using ~2.3 million admissions and 26 million vital sign assessments. The algorithm is agnostic to patient location, condition, and demographics, and relies only on sequences of five vital sign measurements, a calculated Modified Early Warning Score, and patient age. We achieved an area under the receiver operating characteristic curve of 0.966 (95% confidence interval [CI] 0.956–0.967) on the retrospective testing set, and 0.971 (95% CI 0.965–0.974) on the prospective set to predict overnight patient stability. The model enables safe avoidance of overnight monitoring for ~50% of patient-nights, while only misclassifying 2 out of 10,000 patient-nights as stable. Our approach is straightforward to deploy, only requires regularly obtained vital signs, and delivers easily actionable clinical predictions for a peaceful sleep in hospitals.

[Health care safety: The discrepancies between experience and degree of satisfaction of hospitalized patients observed in interviews performed by user representatives]

INTRODUCTION

The purpose of this article is to present the results of a qualitative survey conducted by user representatives (URs) focusing on the health care safety experience of hospitalized patients. The authors wished to identify factors associated with safety of care and, more specifically, with the possibly ominous medical events reported by patients.

METHODS

After being trained with these objectives in mind, eight URs conducted semi-directive interviews with fourteen patients hospitalized in eleven separate hospital units in nine different hospitals.

RESULTS

Eight types of factors consisting in 30 contributing factors liable to be reported by patients were identified: 1) factors related to patients' basic needs; 2) personalization of care; 3) professional factors; 4) organizational factors; 5) communication factors; 6) caregiver responsiveness; 7) infectious risks; 8) continuity of care. Patients' overall feelings about their hospitalization remained excellent notwithstanding more tempered, even negative experiences.

CONCLUSION

This paradoxical result shows that the patients' actual experience is far more instructive than their degree of satisfaction. In light of this study, the acceptability of this type of research (i.e. research conducted by URs) is excellent and it also appears highly feasible, whatever the limitations imposed by organizational considerations.

AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records

BACKGROUND

Risk scores can be useful in clinical risk stratification and accurate allocations of medical resources, helping health providers improve patient care. Point-based scores are more understandable and explainable than other complex models and are now widely used in clinical decision making. However, the development of the risk scoring model is nontrivial and has not yet been systematically presented, with few studies investigating methods of clinical score generation using electronic health records.

OBJECTIVE

This study aims to propose AutoScore, a machine learning-based automatic clinical score generator consisting of 6 modules for developing interpretable point-based scores. Future users can employ the AutoScore framework to create clinical scores effortlessly in various clinical applications.

METHODS

We proposed the AutoScore framework comprising 6 modules that included variable ranking, variable transformation, score derivation, model selection, score fine-tuning, and model evaluation. To demonstrate the performance of AutoScore, we used data from the Beth Israel Deaconess Medical Center to build a scoring model for mortality prediction and then compared the data with other baseline models using the receiver operating characteristic analysis. A software package in R 3.5.3 (R Foundation) was also developed to demonstrate the implementation of AutoScore.

RESULTS

Implemented on the data set with 44,918 individual admission episodes of intensive care, the AutoScore-created scoring models performed comparably well as other standard methods (ie, logistic regression, stepwise regression, least absolute shrinkage and selection operator, and random forest) in terms of predictive accuracy and model calibration but required fewer predictors and presented high interpretability and accessibility. The nine-variable, AutoScore-created, point-based scoring model achieved an area under the curve (AUC) of 0.780 (95% CI 0.764-0.798), whereas the model of logistic regression with 24 variables had an AUC of 0.778 (95% CI 0.760-0.795). Moreover, the AutoScore framework also drives the clinical research continuum and automation with its integration of all necessary modules.

CONCLUSIONS

We developed an easy-to-use, machine learning-based automatic clinical score generator, AutoScore; systematically presented its structure; and demonstrated its superiority (predictive performance and interpretability) over other conventional methods using a benchmark database. AutoScore will emerge as a potential scoring tool in various medical applications.

Early warning score validation methodologies and performance metrics: a systematic review

Background

Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. In the past few years, there has been a proliferation of studies that describe the development and validation of novel machine learning-based EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason is the heterogeneity in validation methods applied. In this review, we aim to examine the methodologies and metrics used in studies which perform EWS validation.

Methods

A systematic review of all eligible studies from the MEDLINE database and other sources, was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and inpatient mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity.

Results

The key differences in validation methodologies identified were (1) validation dataset used, (2) outcomes of interest, (3) case definition, time of EWS use and aggregation methods, and (4) handling of missing values. In terms of case definition, among the 48 eligible studies, 34 used the patient episode case definition while 12 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 18 studies validated the EWS at a single point of time, mostly using the first recorded observation. The review also found more than 10 different performance metrics reported among the studies.

Conclusions

Methodologies and performance metrics used in studies performing validation on EWS were heterogeneous hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology

OBJECTIVE

To provide an overview and critical appraisal of early warning scores for adult hospital patients.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, PsycInfo, and Embase until June 2019.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

Studies describing the development or external validation of an early warning score for adult hospital inpatients.

RESULTS

13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias.

CONCLUSIONS

Early warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017053324.

Early Detection of In-Patient Deterioration: One Prediction Model Does Not Fit All

Supplemental Digital Content is available in the text.

Objectives:

Early detection of subacute potentially catastrophic illnesses using available data is a clinical imperative, and scores that report risk of imminent events in real time abound. Patients deteriorate for a variety of reasons, and it is unlikely that a single predictor such as an abnormal National Early Warning Score will detect all of them equally well. The objective of this study was to test the idea that the diversity of reasons for clinical deterioration leading to ICU transfer mandates multiple targeted predictive models.

Design:

Individual chart review to determine the clinical reason for ICU transfer; determination of relative risks of individual vital signs, laboratory tests and cardiorespiratory monitoring measures for prediction of each clinical reason for ICU transfer; and logistic regression modeling for the outcome of ICU transfer for a specific clinical reason.

Setting:

Cardiac medical-surgical ward; tertiary care academic hospital.

Patients:

Eight-thousand one-hundred eleven adult patients, 457 of whom were transferred to an ICU for clinical deterioration.

Interventions:

None.

Measurements and Main Results:

We calculated the contributing relative risks of individual vital signs, laboratory tests and cardiorespiratory monitoring measures for prediction of each clinical reason for ICU transfer, and used logistic regression modeling to calculate receiver operating characteristic areas and relative risks for the outcome of ICU transfer for a specific clinical reason. The reasons for clinical deterioration leading to ICU transfer were varied, as were their predictors. For example, the three most common reasons—respiratory instability, infection and suspected sepsis, and heart failure requiring escalated therapy—had distinct signatures of illness. Statistical models trained to target-specific reasons for ICU transfer performed better than one model targeting combined events.

Conclusions:

A single predictive model for clinical deterioration does not perform as well as having multiple models trained for the individual specific clinical events leading to ICU transfer.

Development and validation of early warning score system: A systematic literature review

OBJECTIVES

This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies.

METHODOLOGY

A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy.

RESULTS

A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database.

CONCLUSION

This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.

Predicting Adverse Outcomes for Febrile Patients in the Emergency Department Using Sparse Laboratory Data: Development of a Time Adaptive Model

BACKGROUND

A timely decision in the initial stages for patients with an acute illness is important. However, only a few studies have determined the prognosis of patients based on insufficient laboratory data during the initial stages of treatment.

OBJECTIVE

This study aimed to develop and validate time adaptive prediction models to predict the severity of illness in the emergency department (ED) using highly sparse laboratory test data (test order status and test results) and a machine learning approach.

METHODS

This retrospective study used ED data from a tertiary academic hospital in Seoul, Korea. Two different models were developed based on laboratory test data: order status only (OSO) and order status and results (OSR) models. A binary composite adverse outcome was used, including mortality or hospitalization in the intensive care unit. Both models were evaluated using various performance criteria, including the area under the receiver operating characteristic curve (AUC) and balanced accuracy (BA). Clinical usefulness was examined by determining the positive likelihood ratio (PLR) and negative likelihood ratio (NLR).

RESULTS

Of 9491 eligible patients in the ED (mean age, 55.2 years, SD 17.7 years; 4839/9491, 51.0% women), the model development cohort and validation cohort included 6645 and 2846 patients, respectively. The OSR model generally exhibited better performance (AUC=0.88, BA=0.81) than the OSO model (AUC=0.80, BA=0.74). The OSR model was more informative than the OSO model to predict patients at low or high risk of adverse outcomes (P<.001 for differences in both PLR and NLR).

CONCLUSIONS

Early-stage adverse outcomes for febrile patients could be predicted using machine learning models of highly sparse data including test order status and laboratory test results. This prediction tool could help medical professionals who are simultaneously treating the same patient share information, lead dynamic communication, and consequently prevent medical errors.

Testing a digital system that ranks the risk of unplanned intensive care unit admission in all ward patients: protocol for a prospective observational cohort study

INTRODUCTION

Traditional early warning scores (EWSs) use vital sign derangements to detect clinical deterioration in patients treated on hospital wards. Combining vital signs with demographics and laboratory results improves EWS performance. We have developed the Hospital Alerting Via Electronic Noticeboard (HAVEN) system. HAVEN uses vital signs, as well as demographic, comorbidity and laboratory data from the electronic patient record, to quantify and rank the risk of unplanned admission to an intensive care unit (ICU) within 24 hours for all ward patients. The primary aim of this study is to find additional variables, potentially missed during development, which may improve HAVEN performance. These variables will be sought in the medical record of patients misclassified by the HAVEN risk score during testing.

METHODS

This will be a prospective, observational, cohort study conducted at the John Radcliffe Hospital, part of the Oxford University Hospitals NHS Foundation Trust in the UK. Each day during the study periods, we will document all highly ranked patients (ie, those with the highest risk for unplanned ICU admission) identified by the HAVEN system. After 48 hours, we will review the progress of the identified patients. Patients who were subsequently admitted to the ICU will be removed from the study (as they will have been correctly classified by HAVEN). Highly ranked patients not admitted to ICU will undergo a structured medical notes review. Additionally, at the end of the study periods, all patients who had an unplanned ICU admission but whom HAVEN failed to rank highly will have a structured medical notes review. The review will identify candidate variables, likely associated with unplanned ICU admission, not included in the HAVEN risk score.

ETHICS AND DISSEMINATION

Approval has been granted for gathering the data used in this study from the South Central Oxford C Research Ethics Committee (16/SC/0264, 13 June 2016) and the Confidentiality Advisory Group (16/CAG/0066).

DISCUSSION

Our study will use a clinical expert conducting a structured medical notes review to identify variables, associated with unplanned ICU admission, not included in the development of the HAVEN risk score. These variables will then be added to the risk score and evaluated for potential performance gain. To the best of our knowledge, this is the first study of this type. We anticipate that documenting the HAVEN development methods will assist other research groups developing similar technology.

TRIAL REGISTRATION NUMBER

ISRCTN12518261.

Is it possible to quickly identify acutely unwell patients who can be safely managed as outpatients? The need for a "Universal Safe to Discharge Score"

If scores or algorithms were developed that quickly identified patients who are bound to have 100% survival, if even only for a few days, more patients could be safely discharged from emergency department, this eliminating the risks of hospitalization for many patients. This hypothesis proposes that it is possible to develop a "Universal Safe to Discharge Score", and suggests how it might be developed and validated.