Mortality prediction models in the adult critically ill: A scoping review

Real-time machine learning model to predict short-term mortality in critically ill patients: development and international validation

BACKGROUND

A real-time model for predicting short-term mortality in critically ill patients is needed to identify patients at imminent risk. However, the performance of the model needs to be validated in various clinical settings and ethnicities before its clinical application. In this study, we aim to develop an ensemble machine learning model using routinely measured clinical variables at a single academic institution in South Korea.

METHODS

We developed an ensemble model using deep learning and light gradient boosting machine models. Internal validation was performed using the last two years of the internal cohort dataset, collected from a single academic hospital in South Korea between 2007 and 2021. External validation was performed using the full Medical Information Mart for Intensive Care (MIMIC), eICU-Collaborative Research Database (eICU-CRD), and Amsterdam University Medical Center database (AmsterdamUMCdb) data. The area under the receiver operating characteristic curve (AUROC) was calculated and compared to that for the National Early Warning Score (NEWS).

RESULTS

The developed model (iMORS) demonstrated high predictive performance with an internal AUROC of 0.964 (95% confidence interval [CI] 0.963-0.965) and external AUROCs of 0.890 (95% CI 0.889-0.891) for MIMIC, 0.886 (95% CI 0.885-0.887) for eICU-CRD, and 0.870 (95% CI 0.868-0.873) for AmsterdamUMCdb. The model outperformed the NEWS with higher AUROCs in the internal and external validation (0.866 for the internal, 0.746 for MIMIC, 0.798 for eICU-CRD, and 0.819 for AmsterdamUMCdb; p < 0.001).

CONCLUSIONS

Our real-time machine learning model to predict short-term mortality in critically ill patients showed excellent performance in both internal and external validations. This model could be a useful decision-support tool in the intensive care units to assist clinicians.

Modest performance of text mining to extract health outcomes may be almost sufficient for high-quality prognostic model development

BACKGROUND

Across medicine, prognostic models are used to estimate patient risk of certain future health outcomes (e.g., cardiovascular or mortality risk). To develop (or train) prognostic models, historic patient-level training data is needed containing both the predictive factors (i.e., features) and the relevant health outcomes (i.e., labels). Sometimes, when the health outcomes are not recorded in structured data, these are first extracted from textual notes using text mining techniques. Because there exist many studies utilizing text mining to obtain outcome data for prognostic model development, our aim is to study the impact of the text mining quality on downstream prognostic model performance.

METHODS

We conducted a simulation study charting the relationship between text mining quality and prognostic model performance using an illustrative case study about in-hospital mortality prediction in intensive care unit patients. We repeatedly developed and evaluated a prognostic model for in-hospital mortality, using outcome data extracted by multiple text mining models of varying quality.

RESULTS

Interestingly, we found in our case study that a relatively low-quality text mining model (F1 score ≈ 0.50) could already be used to train a prognostic model with quite good discrimination (area under the receiver operating characteristic curve of around 0.80). The calibration of the risks estimated by the prognostic model seemed unreliable across the majority of settings, even when text mining models were of relatively high quality (F1 ≈ 0.80).

DISCUSSION

Developing prognostic models on text-extracted outcomes using imperfect text mining models seems promising. However, it is likely that prognostic models developed using this approach may not produce well-calibrated risk estimates, and require recalibration in (possibly a smaller amount of) manually extracted outcome data.

A novel score for predicting 1-year mortality of intensive care patients

BACKGROUND

We aimed to develop a simple scoring table for predicting probability of death within 1-year after admission to an intensive care unit. We analysed data on emergency admissions from the nationwide Finnish intensive care quality registry.

METHODS

We included first admissions of adult patients with data available on 1-year vital status (dead or alive) and all five variables included in a premorbid functional status score, which is the number of activities the person can manage independently of the following five: get out of bed, move indoors, dress, climb stairs and walk 400 m. We analysed data on patient characteristics and admission-associated factors from 2012 to 2014 to find predictors of 1-year mortality and to develop a score for predicting probability of death. We tested the performance of this score in data from 2015. We assessed the 1-year functional status score of survivors with data available.

RESULTS

Out of 25,261 patients, 20,628 (81.7%) patients were able to perform all five functional activities independently prior to the intensive care unit admission. At 1-year post admission, 19,625 (77.7%) patients were alive. 1-year functional status score was known for 11,011 patients and 8970 (81.5%) patients achieved functional status score 5, managing all five activities independently. The score based on age, sex, preceding functional status, type of intensive care unit admission, severity of acute illness and the most significant diagnoses predicted 1-year mortality with an area under the receiver operating characteristic curve 0.78 (95% CI, 0.76-0.79). The calibration of our prediction model was good, with calibration intercept -0.01 (-0.07 to 0.05) and calibration slope 0.96 (0.90 to 1.02).

CONCLUSION

Our score based on data available at intensive care unit admission predicted 1-year mortality with fairly good discrimination. Most survivors achieved good functional recovery.

Heart rate complexity helps mortality prediction in the intensive care unit: A pilot study using artificial intelligence

BACKGROUND

In intensive care units (ICUs), accurate mortality prediction is crucial for effective patient management and resource allocation. The Simplified Acute Physiology Score II (SAPS-2), though commonly used, relies heavily on comprehensive clinical data and blood samples. This study sought to develop an artificial intelligence (AI) model utilizing key hemodynamic parameters to predict ICU mortality within the first 24 h and assess its performance relative to SAPS-2.

METHODS

We conducted an analysis of select hemodynamic parameters and the structure of heart rate curves to identify potential predictors of ICU mortality. A machine-learning model was subsequently trained and validated on distinct patient cohorts. The AI algorithm's performance was then compared to the SAPS-2, focusing on classification accuracy, calibration, and generalizability.

MEASUREMENTS AND MAIN RESULTS

The study included 1298 ICU admissions from March 27th, 2015, to March 27th, 2017. An additional cohort from 2022 to 2023 comprised 590 patients, resulting in a total dataset of 1888 patients. The observed mortality rate stood at 24.0%. Key determinants of mortality were the Glasgow Coma Scale score, heart rate complexity, patient age, duration of diastolic blood pressure below 50 mmHg, heart rate variability, and specific mean and systolic blood pressure thresholds. The AI model, informed by these determinants, exhibited a performance profile in predicting mortality that was comparable, if not superior, to the SAPS-2.

CONCLUSIONS

The AI model, which integrates heart rate and blood pressure curve analyses with basic clinical parameters, provides a methodological approach to predict in-hospital mortality in ICU patients. This model offers an alternative to existing tools that depend on extensive clinical data and laboratory inputs. Its potential integration into ICU monitoring systems may facilitate more streamlined mortality prediction processes.

Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study

INTRODUCTION

Prediction models for identifying emergency department (ED) patients at high risk of poor outcome are often not externally validated. We aimed to perform a head-to-head comparison of the discriminatory performance of several prediction models in a large cohort of ED patients.

METHODS

In this retrospective study, we selected prediction models that aim to predict poor outcome and we included adult medical ED patients. Primary outcome was 31-day mortality, secondary outcomes were 1-day mortality, 7-day mortality, and a composite endpoint of 31-day mortality and admission to intensive care unit (ICU).The discriminatory performance of the prediction models was assessed using an area under the receiver operating characteristic curve (AUC). Finally, the prediction models with the highest performance to predict 31-day mortality were selected to further examine calibration and appropriate clinical cut-off points.

RESULTS

We included 19 prediction models and applied these to 2185 ED patients. Thirty-one-day mortality was 10.6% (231 patients), 1-day mortality was 1.4%, 7-day mortality was 4.4%, and 331 patients (15.1%) met the composite endpoint. The RISE UP and COPE score showed similar and very good discriminatory performance for 31-day mortality (AUC 0.86), 1-day mortality (AUC 0.87), 7-day mortality (AUC 0.86) and for the composite endpoint (AUC 0.81). Both scores were well calibrated. Almost no patients with RISE UP and COPE scores below 5% had an adverse outcome, while those with scores above 20% were at high risk of adverse outcome. Some of the other prediction models (i.e. APACHE II, NEWS, WPSS, MEWS, EWS and SOFA) showed significantly higher discriminatory performance for 1-day and 7-day mortality than for 31-day mortality.

CONCLUSIONS

Head-to-head validation of 19 prediction models in medical ED patients showed that the RISE UP and COPE score outperformed other models regarding 31-day mortality.

Machine Learning for Benchmarking Critical Care Outcomes

OBJECTIVES

Enhancing critical care efficacy involves evaluating and improving system functioning. Benchmarking, a retrospective comparison of results against standards, aids risk-adjusted assessment and helps healthcare providers identify areas for improvement based on observed and predicted outcomes. The last two decades have seen the development of several models using machine learning (ML) for clinical outcome prediction. ML is a field of artificial intelligence focused on creating algorithms that enable computers to learn from and make predictions or decisions based on data. This narrative review centers on key discoveries and outcomes to aid clinicians and researchers in selecting the optimal methodology for critical care benchmarking using ML.

METHODS

We used PubMed to search the literature from 2003 to 2023 regarding predictive models utilizing ML for mortality (592 articles), length of stay (143 articles), or mechanical ventilation (195 articles). We supplemented the PubMed search with Google Scholar, making sure relevant articles were included. Given the narrative style, papers in the cohort were manually curated for a comprehensive reader perspective.

RESULTS

Our report presents comparative results for benchmarked outcomes and emphasizes advancements in feature types, preprocessing, model selection, and validation. It showcases instances where ML effectively tackled critical care outcome-prediction challenges, including nonlinear relationships, class imbalances, missing data, and documentation variability, leading to enhanced results.

CONCLUSIONS

Although ML has provided novel tools to improve the benchmarking of critical care outcomes, areas that require further research include class imbalance, fairness, improved calibration, generalizability, and long-term validation of published models.

Urine output as one of the most important features in differentiating in-hospital death among patients receiving extracorporeal membrane oxygenation: a random forest approach

BACKGROUND

It is common to support cardiovascular function in critically ill patients with extracorporeal membrane oxygenation (ECMO). The purpose of this study was to identify patients receiving ECMO with a considerable risk of dying in hospital using machine learning algorithms.

METHODS

A total of 1342 adult patients on ECMO support were randomly assigned to the training and test groups. The discriminatory power (DP) for predicting in-hospital mortality was tested using both random forest (RF) and logistic regression (LR) algorithms.

RESULTS

Urine output on the first day of ECMO implantation was found to be one of the most predictive features that were related to in-hospital death in both RF and LR models. For those with oliguria, the hazard ratio for 1 year mortality was 1.445 (p < 0.001, 95% CI 1.265-1.650).

CONCLUSIONS

Oliguria within the first 24 h was deemed especially significant in differentiating in-hospital death and 1 year mortality.

Updating mortality risk estimation in intensive care units from high-dimensional electronic health records with incomplete data

BACKGROUND

The risk of mortality in intensive care units (ICUs) is currently addressed by the implementation of scores using admission data. Their performances are satisfactory when complications occur early after admission; however, they may become irrelevant in the case of long hospital stays. In this study, we developed predictive models of short-term mortality in the ICU from longitudinal data.

METHODS

Using data collected throughout patients' stays of at least 48 h from the MIMIC-III database, several statistical learning approaches were compared, including deep neural networks and penalized regression. Missing data were handled using complete-case analysis or multiple imputation.

RESULTS

Complete-case analyses from 19 predictors showed good discrimination (AUC > 0.77 for several approaches) to predict death between 12 and 24 h onward, yet excluded 75% of patients from the initial target cohort, as data was missing for some of the predictors. Multiple imputation allowed us to include 70 predictors and keep 95% of patients, with similar performances.

CONCLUSION

This proof-of-concept study supports that automated analysis of electronic health records can be of great interest throughout patients' stays as a surveillance tool. Although this framework relies on a large set of predictors, it is robust to data imputation and may be effective early after admission, when data are still scarce.

Predictive models in extracorporeal membrane oxygenation (ECMO): a systematic review

PURPOSE

Extracorporeal membrane oxygenation (ECMO) has been increasingly used in the last years to provide hemodynamic and respiratory support in critically ill patients. In this scenario, prognostic scores remain essential to choose which patients should initiate ECMO. This systematic review aims to assess the current landscape and inform subsequent efforts in the development of risk prediction tools for ECMO.

METHODS

PubMed, CINAHL, Embase, MEDLINE and Scopus were consulted. Articles between Jan 2011 and Feb 2022, including adults undergoing ECMO reporting a newly developed and validated predictive model for mortality, were included. Studies based on animal models, systematic reviews, case reports and conference abstracts were excluded. Data extraction aimed to capture study characteristics, risk model characteristics and model performance. The risk of bias was evaluated through the prediction model risk-of-bias assessment tool (PROBAST). The protocol has been registered in Open Science Framework ( https://osf.io/fevw5 ).

RESULTS

Twenty-six prognostic scores for in-hospital mortality were identified, with a study size ranging from 60 to 4557 patients. The most common candidate variables were age, lactate concentration, creatinine concentration, bilirubin concentration and days in mechanical ventilation prior to ECMO. Five out of 16 venous-arterial (VA)-ECMO scores and 3 out of 9 veno-venous (VV)-ECMO scores had been validated externally. Additionally, one score was developed for both VA and VV populations. No score was judged at low risk of bias.

CONCLUSION

Most models have not been validated externally and apply after ECMO initiation; thus, some uncertainty whether ECMO should be initiated still remains. It has yet to be determined whether and to what extent a new methodological perspective may enhance the performance of predictive models for ECMO, with the ultimate goal to implement a model that positively influences patient outcomes.

Intercontinental validation of a clinical prediction model for predicting 90-day and 2-year mortality in an Israeli cohort of 2033 patients with a femoral neck fracture aged 65 or above

PURPOSE

Mortality prediction in elderly femoral neck fracture patients is valuable in treatment decision-making. A previously developed and internally validated clinical prediction model shows promise in identifying patients at risk of 90-day and 2-year mortality. Validation in an independent cohort is required to assess the generalizability; especially in geographically distinct regions. Therefore we questioned, is the SORG Orthopaedic Research Group (SORG) femoral neck fracture mortality algorithm externally valid in an Israeli cohort to predict 90-day and 2-year mortality?

METHODS

We previously developed a prediction model in 2022 for estimating the risk of mortality in femoral neck fracture patients using a multicenter institutional cohort of 2,478 patients from the USA. The model included the following input variables that are available on clinical admission: age, male gender, creatinine level, absolute neutrophil, hemoglobin level, international normalized ratio (INR), congestive heart failure (CHF), displaced fracture, hemiplegia, chronic obstructive pulmonary disease (COPD), history of cerebrovascular accident (CVA) and beta-blocker use. To assess the generalizability, we used an intercontinental institutional cohort from the Sheba Medical Center in Israel (level I trauma center), queried between June 2008 and February 2022. Generalizability of the model was assessed using discrimination, calibration, Brier score, and decision curve analysis.

RESULTS

The validation cohort included 2,033 patients, aged 65 years or above, that underwent femoral neck fracture surgery. Most patients were female 64.8% (n = 1317), the median age was 81 years (interquartile range = 75-86), and 80.4% (n = 1635) patients sustained a displaced fracture (Garden III/IV). The 90-day mortality was 9.4% (n = 190) and 2-year mortality was 30.0% (n = 610). Despite numerous baseline differences, the model performed acceptably to the validation cohort on discrimination (c-statistic 0.67 for 90-day, 0.67 for 2-year), calibration, Brier score, and decision curve analysis.

CONCLUSIONS

The previously developed SORG femoral neck fracture mortality algorithm demonstrated good performance in an independent intercontinental population. Current iteration should not be relied on for patient care, though suggesting potential utility in assessing patients at low risk for 90-day or 2-year mortality. Further studies should evaluate this tool in a prospective setting and evaluate its feasibility and efficacy in clinical practice. The algorithm can be freely accessed: https://sorg-apps.shinyapps.io/hipfracturemortality/ .

LEVEL OF EVIDENCE

Level III, Prognostic study.

External Validation of Mortality Prediction Models for Critical Illness Reveals Preserved Discrimination but Poor Calibration

OBJECTIVES

In a recent scoping review, we identified 43 mortality prediction models for critically ill patients. We aimed to assess the performances of these models through external validation.

DESIGN

Multicenter study.

SETTING

External validation of models was performed in the Simple Intensive Care Studies-I (SICS-I) and the Finnish Acute Kidney Injury (FINNAKI) study.

PATIENTS

The SICS-I study consisted of 1,075 patients, and the FINNAKI study consisted of 2,901 critically ill patients.

MEASUREMENTS AND MAIN RESULTS

For each model, we assessed: 1) the original publications for the data needed for model reconstruction, 2) availability of the variables, 3) model performance in two independent cohorts, and 4) the effects of recalibration on model performance. The models were recalibrated using data of the SICS-I and subsequently validated using data of the FINNAKI study. We evaluated overall model performance using various indexes, including the (scaled) Brier score, discrimination (area under the curve of the receiver operating characteristics), calibration (intercepts and slopes), and decision curves. Eleven models (26%) could be externally validated. The Acute Physiology And Chronic Health Evaluation (APACHE) II, APACHE IV, Simplified Acute Physiology Score (SAPS)-Reduced (SAPS-R)' and Simplified Mortality Score for the ICU models showed the best scaled Brier scores of 0.11' 0.10' 0.10' and 0.06' respectively. SAPS II, APACHE II, and APACHE IV discriminated best; overall discrimination of models ranged from area under the curve of the receiver operating characteristics of 0.63 (0.61-0.66) to 0.83 (0.81-0.85). We observed poor calibration in most models, which improved to at least moderate after recalibration of intercepts and slopes. The decision curve showed a positive net benefit in the 0-60% threshold probability range for APACHE IV and SAPS-R.

CONCLUSIONS

In only 11 out of 43 available mortality prediction models, the performance could be studied using two cohorts of critically ill patients. External validation showed that the discriminative ability of APACHE II, APACHE IV, and SAPS II was acceptable to excellent, whereas calibration was poor.

Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

OBJECTIVES

Many prediction models for coronavirus disease 2019 (COVID-19) have been developed. External validation is mandatory before implementation in the intensive care unit (ICU). We selected and validated prognostic models in the Euregio Intensive Care COVID (EICC) cohort.

STUDY DESIGN AND SETTING

In this multinational cohort study, routine data from COVID-19 patients admitted to ICUs within the Euregio Meuse-Rhine were collected from March to August 2020. COVID-19 models were selected based on model type, predictors, outcomes, and reporting. Furthermore, general ICU scores were assessed. Discrimination was assessed by area under the receiver operating characteristic curves (AUCs) and calibration by calibration-in-the-large and calibration plots. A random-effects meta-analysis was used to pool results.

RESULTS

551 patients were admitted. Mean age was 65.4 ± 11.2 years, 29% were female, and ICU mortality was 36%. Nine out of 238 published models were externally validated. Pooled AUCs were between 0.53 and 0.70 and calibration-in-the-large between -9% and 6%. Calibration plots showed generally poor but, for the 4C Mortality score and Spanish Society of Infectious Diseases and Clinical Microbiology (SEIMC) score, moderate calibration.

CONCLUSION

Of the nine prognostic models that were externally validated in the EICC cohort, only two showed reasonable discrimination and moderate calibration. For future pandemics, better models based on routine data are needed to support admission decision-making.

Using Second Measurement of De Ritis Ratio to Improve Mortality Prediction in Adult Trauma Patients in Intensive Care Unit

The De Ritis ratio (DRR), the ratio of serum levels of aspartate aminotransferase/alanine aminotransferase, has been reported to be a valuable biomarker in risk stratification for many liver and non-liver diseases. This study aimed to explore whether the inclusion of DRR at the date of intensive care unit (ICU) admission or days after ICU admission improves the predictive performance of various prognosis prediction models. This study reviewed 888 adult trauma patients (74 deaths and 814 survivors) in the trauma registered database between 1 January 2009, and 31 December 2020. Medical information with AST and ALT levels and derived DRR at the date of ICU admission (1st DRR) and 3-7 day after ICU admission (2nd DRR) was retrieved. Logistic regression was used to build new probability models for mortality prediction using additional DRR variables in various mortality prediction models. There was no significant difference in the 1st DRR between the death and survival patients; however, there was a significantly higher 2nd DRR in the death patients than the survival patients. This study showed that the inclusion of the additional DRR variable, measured 3-7 days after ICU admission, significantly increased the prediction performance in all studied prognosis prediction models.

Using long-term predicted Quality of Life in ICU clinical practice to prepare patients for life post-ICU: A feasibility study

PURPOSE

To examine the feasibility of using the PREdicting PAtients' long-term outcome for Recovery (PREPARE) prediction model for Quality of Life (QoL) 1 year after ICU admission in ICU practice to prepare expected ICU survivors and their relatives for life post-ICU.

MATERIALS AND METHODS

Between June 2020 and February 2021, the predicted change in QoL after 1 year was discussed in 25 family conferences in the ICU. 13 physicians, 10 nurses and 19 patients and/or family members were interviewed to evaluate intervention feasibility in ICU practice. Interviews were analysed qualitatively using thematic coding.

RESULTS

Patients' median age was 68.0 years, five patients (20.0%) were female and seven patients (28.0%) died during ICU stay. Generally, study participants thought the intervention, which clarified the concept of QoL through visualization and served as a reminder to discuss QoL and expectations for life post-ICU, had merit. However, some participants, especially physicians, thought the prediction model needed more data on more severely ill ICU patients to curb uncertainty.

CONCLUSIONS

Using predicted QoL scores in ICU practice to prepare patients and family members for life after ICU discharge is feasible. After optimising the model and implementation strategy, its effectiveness can be evaluated in a larger trial.

Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable

How to cite this article: Kumar AAK. Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable. Indian J Crit Care Med 2022;26(1):13-14.

Hourly Kinetics of Critical Organ Dysfunction in Extremely Preterm Infants

Rationale: Use of severity of illness scores to classify patients for clinical care and research is common outside of the neonatal ICU. Extremely premature (<29 weeks' gestation) infants with extremely low birth weight (<1,000 g) experience significant mortality and develop severe pathology during the protracted birth hospitalization. Objectives: To measure at high resolution the changes in organ dysfunction that occur from birth to death or discharge home by gestational age and time, and among extremely preterm infants with and without clinically meaningful outcomes using the neonatal sequential organ failure assessment score. Methods: A single-center, retrospective, observational cohort study of inborn, extremely preterm infants with extremely low birth weight admitted between January 2012 and January 2020. Neonatal sequential organ failure assessment scores were calculated every hour for every patient from admission until death or discharge. Measurements and Main Results: Longitudinal, granular scores from 436 infants demonstrated early and sustained discrimination of those who died versus those who survived to discharge. The discrimination for mortality by the maximum score was excellent (area under curve, 0.91; 95% confidence intervals, 0.88-0.94). Among survivors with and without adverse outcomes, most score variation occurred at the patient level. The weekly average score over the first 28 days was associated with the sum of adverse outcomes at discharge. Conclusions: The neonatal sequential organ failure assessment score discriminates between survival and nonsurvival on the first day of life. The major contributor to score variation occurred at the patient level. There was a direct association between scores and major adverse outcomes, including death.

Elevated Serum Tenascin-C Predicts Mortality in Critically Ill Patients With Multiple Organ Dysfunction

Background: Multiple organ dysfunction is a complex and lethal clinical feature with heterogeneous causes and is usually characterized by tissue injury of multiple organs. Tenascin-C (TNC) is a matricellular protein that is rarely expressed in most of the adult tissues, but re-induced following injury. This study aimed to evaluate serum TNC in predicting mortality in critically ill patients with multiple organ dysfunction. Methods: Adult critically ill patients with at least two organs dysfunction and an increase of Sequential Organ Failure Assess (SOFA) score ≥ 2 points within 7 days were prospectively enrolled into two independent cohorts. The emergency (derivation) cohort was a consecutive series and the patients were from Emergency Department. The inpatient (validation) cohort was a convenience series and the patients were from medical wards. Their serum samples at the first 24 h after enrollment were collected and subjected to TNC measurement using ELISA. The association between serum TNC level and 28-day all-cause mortality was investigated, and then the predictive value of serum TNC was analyzed. Results: A total of 110 patients with a median age of 64 years (53, 73) were enrolled in the emergency cohort. Compared to the survivors, serum TNC in the non-survivors was significantly higher (467.7 vs. 197.5 ng/ml, p < 0.001). Multivariate logistic regression analysis revealed that the association between serum TNC and 28-day mortality was independent of sepsis or critical illness scores such as SOFA, Acute Physiology and Chronic Health Evaluation (APACHE II), and Simplified Acute Physiology Score (SAPS II), respectively (p < 0.001 for each). The area under receiver operating characteristic curve of serum TNC for predicting mortality was 0.803 (0.717-0.888) (p < 0.001), similar with SOFA 0.808 (0.725-0.891), APACHE II 0.762 (0.667-0.857), and SAPS II 0.779 (0.685-0.872). The optimal cut-off value of serum TNC was 298.2 ng/ml. Kaplan-Meier analysis showed that the survival of patients with serum TNC ≥ 300 ng/ml was significantly worse than that of patients with serum TNC < 300 ng/ml. This result was validated in the inpatient cohort. The sensitivity and specificity of serum TNC ≥ 300 ng/ml for predicting mortality were 74.3 and 74.7% in the emergency cohort, and 63.0 and 70.1% in the inpatient cohort, respectively. Conclusion: Serum TNC was associated with mortality in critically ill patients with multiple organ dysfunction, and would be used as a prognostic tool for predicting mortality in this population.

Comparison of Machine Learning Techniques for Mortality Prediction in a Prospective Cohort of Older Adults

As global demographics change, ageing is a global phenomenon which is increasingly of interest in our modern and rapidly changing society. Thus, the application of proper prognostic indices in clinical decisions regarding mortality prediction has assumed a significant importance for personalized risk management (i.e., identifying patients who are at high or low risk of death) and to help ensure effective healthcare services to patients. Consequently, prognostic modelling expressed as all-cause mortality prediction is an important step for effective patient management. Machine learning has the potential to transform prognostic modelling. In this paper, results on the development of machine learning models for all-cause mortality prediction in a cohort of healthy older adults are reported. The models are based on features covering anthropometric variables, physical and lab examinations, questionnaires, and lifestyles, as well as wearable data collected in free-living settings, obtained for the "Healthy Ageing Initiative" study conducted on 2291 recruited participants. Several machine learning techniques including feature engineering, feature selection, data augmentation and resampling were investigated for this purpose. A detailed empirical comparison of the impact of the different techniques is presented and discussed. The achieved performances were also compared with a standard epidemiological model. This investigation showed that, for the dataset under consideration, the best results were achieved with Random UnderSampling in conjunction with Random Forest (either with or without probability calibration). However, while including probability calibration slightly reduced the average performance, it increased the model robustness, as indicated by the lower 95% confidence intervals. The analysis showed that machine learning models could provide comparable results to standard epidemiological models while being completely data-driven and disease-agnostic, thus demonstrating the opportunity for building machine learning models on health records data for research and clinical practice. However, further testing is required to significantly improve the model performance and its robustness.

Performance of prediction models for short-term outcome in COVID-19 patients in the emergency department: a retrospective study

INTRODUCTION

Coronavirus disease 2019 (COVID-19) has a high burden on the healthcare system. Prediction models may assist in triaging patients. We aimed to assess the value of several prediction models in COVID-19 patients in the emergency department (ED).

METHODS

In this retrospective study, ED patients with COVID-19 were included. Prediction models were selected based on their feasibility. Primary outcome was 30-day mortality, secondary outcomes were 14-day mortality and a composite outcome of 30-day mortality and admission to medium care unit (MCU) or intensive care unit (ICU). The discriminatory performance of the prediction models was assessed using an area under the receiver operating characteristic curve (AUC).

RESULTS

We included 403 patients. Thirty-day mortality was 23.6%, 14-day mortality was 19.1%, 66 patients (16.4%) were admitted to ICU, 48 patients (11.9%) to MCU, and 152 patients (37.7%) met the composite endpoint. Eleven prediction models were included. The RISE UP score and 4 C mortality scores showed very good discriminatory performance for 30-day mortality (AUC 0.83 and 0.84, 95% CI 0.79-0.88 for both), significantly higher than that of the other models.

CONCLUSION

The RISE UP score and 4 C mortality score can be used to recognise patients at high risk for poor outcome and may assist in guiding decision-making and allocating resources.

Rule-Based Models for Risk Estimation and Analysis of In-hospital Mortality in Emergency and Critical Care

We propose a novel method that uses associative classification and odds ratios to predict in-hospital mortality in emergency and critical care. Manual mortality risk scores have previously been used to assess the care needed for each patient and their need for palliative measures. Automated approaches allow providers to get a quick and objective estimation based on electronic health records. We use association rule mining to find relevant patterns in the dataset. The odds ratio is used instead of classical association rule mining metrics as a quality measure to analyze association instead of frequency. The resulting measures are used to estimate the in-hospital mortality risk. We compare two prediction models: one minimal model with socio-demographic factors that are available at the time of admission and can be provided by the patients themselves, namely gender, ethnicity, type of insurance, language, and marital status, and a full model that additionally includes clinical information like diagnoses, medication, and procedures. The method was tested and validated on MIMIC-IV, a publicly available clinical dataset. The minimal prediction model achieved an area under the receiver operating characteristic curve value of 0.69, while the full prediction model achieved a value of 0.98. The models serve different purposes. The minimal model can be used as a first risk assessment based on patient-reported information. The full model expands on this and provides an updated risk assessment each time a new variable occurs in the clinical case. In addition, the rules in the models allow us to analyze the dataset based on data-backed rules. We provide several examples of interesting rules, including rules that hint at errors in the underlying data, rules that correspond to existing epidemiological research, and rules that were previously unknown and can serve as starting points for future studies.

The simple observational critical care studies: estimations by students, nurses, and physicians of in-hospital and 6-month mortality

Background

Prognostic assessments of the mortality of critically ill patients are frequently performed in daily clinical practice and provide prognostic guidance in treatment decisions. In contrast to several sophisticated tools, prognostic estimations made by healthcare providers are always available and accessible, are performed daily, and might have an additive value to guide clinical decision-making. The aim of this study was to evaluate the accuracy of students’, nurses’, and physicians’ estimations and the association of their combined estimations with in-hospital mortality and 6-month follow-up.

Methods

The Simple Observational Critical Care Studies is a prospective observational single-center study in a tertiary teaching hospital in the Netherlands. All patients acutely admitted to the intensive care unit were included. Within 3 h of admission to the intensive care unit, a medical or nursing student, a nurse, and a physician independently predicted in-hospital and 6-month mortality. Logistic regression was used to assess the associations between predictions and the actual outcome; the area under the receiver operating characteristics (AUROC) was calculated to estimate the discriminative accuracy of the students, nurses, and physicians.

Results

In 827 out of 1,010 patients, in-hospital mortality rates were predicted to be 11%, 15%, and 17% by medical students, nurses, and physicians, respectively. The estimations of students, nurses, and physicians were all associated with in-hospital mortality (OR 5.8, 95% CI [3.7, 9.2], OR 4.7, 95% CI [3.0, 7.3], and OR 7.7 95% CI [4.7, 12.8], respectively). Discriminative accuracy was moderate for all students, nurses, and physicians (between 0.58 and 0.68). When more estimations were of non-survival, the odds of non-survival increased (OR 2.4 95% CI [1.9, 3.1]) per additional estimate, AUROC 0.70 (0.65, 0.76). For 6-month mortality predictions, similar results were observed.

Conclusions

Based on the initial examination, students, nurses, and physicians can only moderately predict in-hospital and 6-month mortality in critically ill patients. Combined estimations led to more accurate predictions and may serve as an example of the benefit of multidisciplinary clinical care and future research efforts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-021-03809-w.

Workforce management and patient outcomes in the intensive care unit during the COVID-19 pandemic and beyond: a discursive paper

AIMS

To highlight the need for the development of effective and realistic workforce strategies for critical care nurses, in both a steady state and pandemic.

BACKGROUND

In acute care settings, there is an inverse relationship between nurse staffing and iatrogenesis, including mortality. Despite this, there remains a lack of consensus on how to determine safe staffing levels. Intensive care units (ICU) provide highly specialised complex healthcare treatments. In developed countries, mortality rates in the ICU setting are high and significantly varied after adjustment for diagnosis. The variability has been attributed to systems, patient and provider issues including the workload of critical care nurses.

DESIGN

Discursive paper.

FINDINGS

Nursing workforce is the single most influential mediating variable on ICU patient outcomes. Numerous systematic reviews have been undertaken in an effort to quantify the effect of critical care nurses on mortality and morbidity, invariably leading to the conclusion that the association is similar to that reported in acute care studies. This is a consequence of methodological limitations, inconsistent operational definitions and variability in endpoint measures. We evaluated the impact inadequate measurement has had on capturing relevant critical care data, and we argue for the need to develop effective and realistic ICU workforce measures.

CONCLUSION

COVID-19 has placed an unprecedented demand on providing health care in the ICU. Mortality associated with ICU admission has been startling during the pandemic. While ICU systems have largely remained static, the context in which care is provided is profoundly dynamic and the role and impact of the critical care nurse needs to be measured accordingly. Often, nurses are passive recipients of unplanned and under-resourced changes to workload, and this has been brought into stark visibility with the current COVID-19 situation. Unless critical care nurses are engaged in systems management, achieving consistently optimal ICU patient outcomes will remain elusive.

RELEVANCE TO CLINICAL PRACTICE

Objective measures commonly fail to capture the complexity of the critical care nurses' role despite evidence to indicate that as workload increases so does risk of patient mortality, job stress and attrition. Critical care nurses must lead system change to develop and evaluate valid and reliable workforce measures.

Risk factors for adverse outcomes during mechanical ventilation of 1152 COVID-19 patients: a multicenter machine learning study with highly granular data from the Dutch Data Warehouse

Background

The identification of risk factors for adverse outcomes and prolonged intensive care unit (ICU) stay in COVID-19 patients is essential for prognostication, determining treatment intensity, and resource allocation. Previous studies have determined risk factors on admission only, and included a limited number of predictors. Therefore, using data from the highly granular and multicenter Dutch Data Warehouse, we developed machine learning models to identify risk factors for ICU mortality, ventilator-free days and ICU-free days during the course of invasive mechanical ventilation (IMV) in COVID-19 patients.

Methods

The DDW is a growing electronic health record database of critically ill COVID-19 patients in the Netherlands. All adult ICU patients on IMV were eligible for inclusion. Transfers, patients admitted for less than 24 h, and patients still admitted at time of data extraction were excluded. Predictors were selected based on the literature, and included medication dosage and fluid balance. Multiple algorithms were trained and validated on up to three sets of observations per patient on day 1, 7, and 14 using fivefold nested cross-validation, keeping observations from an individual patient in the same split.

Results

A total of 1152 patients were included in the model. XGBoost models performed best for all outcomes and were used to calculate predictor importance. Using Shapley additive explanations (SHAP), age was the most important demographic risk factor for the outcomes upon start of IMV and throughout its course. The relative probability of death across age values is visualized in Partial Dependence Plots (PDPs), with an increase starting at 54 years. Besides age, acidaemia, low P/F-ratios and high driving pressures demonstrated a higher probability of death. The PDP for driving pressure showed a relative probability increase starting at 12 cmH₂O.

Conclusion

Age is the most important demographic risk factor of ICU mortality, ICU-free days and ventilator-free days throughout the course of invasive mechanical ventilation in critically ill COVID-19 patients. pH, P/F ratio, and driving pressure should be monitored closely over the course of mechanical ventilation as risk factors predictive of these outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40635-021-00397-5.

Development and validation of high definition phenotype-based mortality prediction in critical care units

Objectives

The objectives of this study are to construct the high definition phenotype (HDP), a novel time-series data structure composed of both primary and derived parameters, using heterogeneous clinical sources and to determine whether different predictive models can utilize the HDP in the neonatal intensive care unit (NICU) to improve neonatal mortality prediction in clinical settings.

Materials and Methods

A total of 49 primary data parameters were collected from July 2018 to May 2020 from eight level-III NICUs. From a total of 1546 patients, 757 patients were found to contain sufficient fixed, intermittent, and continuous data to create HDPs. Two different predictive models utilizing the HDP, one a logistic regression model (LRM) and the other a deep learning long–short-term memory (LSTM) model, were constructed to predict neonatal mortality at multiple time points during the patient hospitalization. The results were compared with previous illness severity scores, including SNAPPE, SNAPPE-II, CRIB, and CRIB-II.

Results

A HDP matrix, including 12 221 536 minutes of patient stay in NICU, was constructed. The LRM model and the LSTM model performed better than existing neonatal illness severity scores in predicting mortality using the area under the receiver operating characteristic curve (AUC) metric. An ablation study showed that utilizing continuous parameters alone results in an AUC score of >80% for both LRM and LSTM, but combining fixed, intermittent, and continuous parameters in the HDP results in scores >85%. The probability of mortality predictive score has recall and precision of 0.88 and 0.77 for the LRM and 0.97 and 0.85 for the LSTM.

Conclusions and Relevance

The HDP data structure supports multiple analytic techniques, including the statistical LRM approach and the machine learning LSTM approach used in this study. LRM and LSTM predictive models of neonatal mortality utilizing the HDP performed better than existing neonatal illness severity scores. Further research is necessary to create HDP–based clinical decision tools to detect the early onset of neonatal morbidities.

Machine learning predicts mortality in septic patients using only routinely available ABG variables: a multi-centre evaluation

PURPOSE

To evaluate the application of machine learning methods, specifically Deep Neural Networks (DNN) models for intensive care (ICU) mortality prediction. The aim was to predict mortality within 96 hours after admission to mirror the clinical situation of patient evaluation after an ICU trial, which consists of 24-48 hours of ICU treatment and then "re-triage". The input variables were deliberately restricted to ABG values to maximise real-world practicability.

METHODS

We retrospectively evaluated septic patients in the multi-centre eICU dataset as well as single centre MIMIC-III dataset. Included were all patients alive after 48 hours with available data on ABG (n = 3979 and n = 9655 ICU stays for the multi-centre and single centre respectively). The primary endpoint was 96 -h-mortality.

RESULTS

The model was developed using long short-term memory (LSTM), a type of DNN designed to learn temporal dependencies between variables. Input variables were all ABG values within the first 48 hours. The SOFA score (AUC of 0.72) was moderately predictive. Logistic regression showed good performance (AUC of 0.82). The best performance was achieved by the LSTM-based model with AUC of 0.88 in the multi-centre study and AUC of 0.85 in the single centre study.

CONCLUSIONS

An LSTM-based model could help physicians with the "re-triage" and the decision to restrict treatment in patients with a poor prognosis.

Performance of Prognostic Scoring Systems in Trauma Patients in the Intensive Care Unit of a Trauma Center

BACKGROUND

Prediction of mortality outcomes in trauma patients in the intensive care unit (ICU) is important for patient care and quality improvement. We aimed to measure the performance of 11 prognostic scoring systems for predicting mortality outcomes in trauma patients in the ICU.

METHODS

Prospectively registered data in the Trauma Registry System from 1 January 2016 to 31 December 2018 were used to extract scores from prognostic scoring systems for 1554 trauma patients in the ICU. The following systems were used: the Trauma and Injury Severity Score (TRISS); the Acute Physiology and Chronic Health Evaluation (APACHE II); the Simplified Acute Physiology Score (SAPS II); mortality prediction models (MPM II) at admission, 24, 48, and 72 h; the Multiple Organ Dysfunction Score (MODS); the Sequential Organ Failure Assessment (SOFA); the Logistic Organ Dysfunction Score (LODS); and the Three Days Recalibrated ICU Outcome Score (TRIOS). Predictive performance was determined according to the area under the receiver operator characteristic curve (AUC).

RESULTS

MPM II at 24 h had the highest AUC (0.9213), followed by MPM II at 48 h (AUC: 0.9105). MPM II at 24, 48, and 72 h (0.8956) had a significantly higher AUC than the TRISS (AUC: 0.8814), APACHE II (AUC: 0.8923), SAPS II (AUC: 0.9044), MPM II at admission (AUC: 0.9063), MODS (AUC: 0.8179), SOFA (AUC: 0.7073), LODS (AUC: 0.9013), and TRIOS (AUC: 0.8701). There was no significant difference in the predictive performance of MPM II at 24 and 48 h (p = 0.37) or at 72 h (p = 0.10).

CONCLUSIONS

We compared 11 prognostic scoring systems and demonstrated that MPM II at 24 h had the best predictive performance for 1554 trauma patients in the ICU.

A simplified critical illness severity scoring system (CISSS): Development and internal validation

PURPOSE

To create a simplified critical illness severity scoring system with high prediction accuracy for 30-day mortality using only commonly available variables.

MATERIALS AND METHODS

This is a retrospective cohort study of ICU admissions 2010-2015 in 306 ICUs in 117 Veterans Affairs (VA) hospitals. We randomly divided our cohort into a training dataset (75%) and a validation dataset (25%). We created a critical illness severity scoring system (CISSS) using age, comorbidities, heart rate, mean arterial blood pressure, temperature, respiratory rate, hematocrit, white blood cell count, creatinine, sodium, glucose, albumin, bilirubin, bicarbonate, use of invasive mechanical ventilation, and whether the admission was surgical or not. We validated the performance of CISSS to predict 30-day mortality internally.

RESULTS

After excluding 31,743 re-admissions, we divided our sample (n = 534,001) into a training (n = 400,613) and a validation dataset (n = 133,388). In the training dataset, the area under the curve (AUC) of CISSS was 0.847(95%CI = 0.845-0.850). In the validation dataset, the AUC was 0.848 (95%CI = 0.844-0.852), the standardized mortality ratio (SMR) was 1.00 (95%CI = 0.98-1.02), and Brier's score for 30-day mortality was 0.058 (95%CI = 0.057-0.059). CISSS calibration was acceptable.

CONCLUSIONS

CISSS has very good performance and requires only commonly used variables that can be easily extracted by electronic health records.