A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis

PREDICTING IN-HOSPITAL MORTALITY IN CRITICAL ORTHOPEDIC TRAUMA PATIENTS WITH SEPSIS USING MACHINE LEARNING MODELS

ABSTRACT

Purpose: This study aims to establish and validate machine learning-based models to predict death in hospital among critical orthopedic trauma patients with sepsis or respiratory failure. Methods: This study collected 523 patients from the Medical Information Mart for Intensive Care database. All patients were randomly classified into a training cohort and a validation cohort. Six algorithms, including logistic regression (LR), extreme gradient boosting machine (eXGBM), support vector machine (SVM), random forest (RF), neural network (NN), and decision tree (DT), were used to develop and optimize models in the training cohort, and internal validation of these models were conducted in the validation cohort. Based on a comprehensive scoring system, which incorporated 10 evaluation metrics, the optimal model was obtained with the highest scores. An artificial intelligence (AI) application was deployed based on the optimal model in the study. Results: The in-hospital mortality was 19.69%. Among all developed models, the eXGBM had the highest area under the curve (AUC) value (0.951, 95% CI: 0.934-0.967), and it also showed the highest accuracy (0.902), precise (0.893), recall (0.915), and F1 score (0.904). Based on the scoring system, the eXGBM had the highest score of 53, followed by the RF model (43) and the NN model (39). The scores for the LR, SVM, and DT were 22, 36, and 17, respectively. The decision curve analysis confirmed that both the eXGBM and RF models provided substantial clinical net benefits. However, the eXGBM model consistently outperformed the RF model across multiple evaluation metrics, establishing itself as the superior option for predictive modeling in this scenario, with the RF model as a strong secondary choice. The Shapley Additive Explanation analysis revealed that Simplified Acute Physiology Score II, age, respiratory rate, Oxford Acute Severity of Illness Score, and temperature were the most important five features contributing to the outcome. Conclusions: This study develops an artificial intelligence application to predict in-hospital mortality among critical orthopedic trauma patients with sepsis or respiratory failure.

Integrating Multimodal EHR Data for Mortality Prediction in ICU Sepsis Patients

Rapid and accurate prediction of mortality risk among intensive care unit (ICU) sepsis patients is crucial for timely intervention and improving patient outcomes. However, due to the multimodal and dynamic time-series nature of patient visit information and the limited data samples, it is challenging to obtain discriminative patient representations, leading to suboptimal mortality prediction results. To address this issue, we design a time-aware graph embedding attention model (TGAM) to integrate multimodal data and predict mortality in ICU sepsis patients. Our approach involves modeling and generating patient representations that encompass not only demographic information but also dynamic time-series data reflecting patient health status. Additionally, the graph convolutional network is used to obtain informative concept embeddings from medical ontologies, and an improved transformer is used to capture the temporal information of the patient's health status and handle missing values, overcoming the limitations of small samples. The experimental results on the MIMIC-III and MIMIC-IV datasets demonstrate that TGAM significantly improves prediction accuracy, with AUROC scores of 87.65% and 87.00% on the MIMIC-III and MIMIC-IV datasets, respectively, outperforming baseline models by over 5 percentage points. TGAM also achieves higher sensitivity, specificity, and AUPRC metrics, and lower Brier Score compared with baseline models, highlighting its effectiveness in identifying high-risk patients. These findings suggest that TGAM has the potential to become a valuable tool for identifying high-risk sepsis patients, enabling clinicians to make more informed and timely intervention decisions.

The role of artificial intelligence in sepsis in the Emergency Department: a narrative review

Background and Objective

Early recognition and treatment of sepsis in the emergency department (ED) is important. Traditional predictive analytics and clinical decision rules lack accuracy in identifying patients with sepsis. Artificial intelligence (AI) is increasingly prevalent in healthcare and offers application potential in the care of patients with sepsis. This review examines the evidence of AI in diagnosing, managing and prognosticating sepsis in the ED.

Methods

We performed literature search in PubMed, Embase, Google Scholar and Scopus databases for studies published between 1 January 2010 and 30 June 2024 that evaluated the use of AI in adult patients with sepsis in ED, using the following search terms: ("artificial intelligence" OR "machine learning" OR "neural networks, computer" OR "deep learning" OR "natural language processing"), AND ("sepsis" OR "septic shock", AND "emergency services" OR "emergency department"). Independent searches were conducted in duplicate with discrepancies adjudicated by a third member.

Key Content and Findings

Incorporating multiple variables such as vital signs, free text input, laboratory tests and electrocardiogram was possible with AI compared to traditional models leading to improvement in diagnostic performance. Machine learning (ML) models outperformed traditional scoring tools in both diagnosis and prognosis of sepsis. ML models were able to analyze trends over time and showed utility in predicting mortality, severe sepsis and septic shock. Additionally, real-time ML-assisted alert systems are effective in improving time-to-antibiotic administration and ML algorithms can differentiate sepsis patients into distinct phenotypes to tailor management (especially fluid therapy and critical care interventions), potentially improving outcomes. Existing AI tools for sepsis currently lack generalizability and user acceptance. This is risk of automation bias with loss of clinicians' skills if over-reliance develops.

Conclusions

Overall, AI holds great promise in revolutionizing management of patients with sepsis in the ED as a clinical support tool. However, its application is currently still constrained by inherent limitations. Balanced integration of AI technology with clinician input is essential to harness its full potential and ensure optimal patient outcomes.

Machine Learning Models in Sepsis Outcome Prediction for ICU Patients: Integrating Routine Laboratory Tests-A Systematic Review

Background. Sepsis presents significant diagnostic and prognostic challenges, and traditional scoring systems, such as SOFA and APACHE, show limitations in predictive accuracy. Machine learning (ML)-based predictive survival models can support risk assessment and treatment decision-making in the intensive care unit (ICU) by accounting for the numerous and complex factors that influence the outcome in the septic patient. Methods. A systematic literature review of studies published from 2014 to 2024 was conducted using the PubMed database. Eligible studies investigated the development of ML models incorporating commonly available laboratory and clinical data for predicting survival outcomes in adult ICU patients with sepsis. Study selection followed the PRISMA guidelines and relied on predefined inclusion criteria. All records were independently assessed by two reviewers, with conflicts resolved by a third senior reviewer. Data related to study design, methodology, results, and interpretation of the results were extracted in a predefined grid. Results. Overall, 19 studies were identified, encompassing primarily logistic regression, random forests, and neural networks. Most used datasets were US-based (MIMIC-III, MIMIC-IV, and eICU-CRD). The most common variables used in model development were age, albumin levels, lactate levels, and ventilator. ML models demonstrated superior performance metrics compared to conventional methods and traditional scoring systems. The best-performing model was a gradient boosting decision tree, with an area under curve of 0.992, an accuracy of 0.954, and a sensitivity of 0.917. However, several critical limitations should be carefully considered when interpreting the results, such as population selection bias (i.e., single center studies), small sample sizes, limited external validation, and model interpretability. Conclusions. Through real-time integration of routine laboratory and clinical data, ML-based tools can assist clinical decision-making and enhance the consistency and quality of sepsis management across various healthcare contexts, including ICUs with limited resources.

Prediction of mortality in sepsis patients using stacked ensemble machine learning algorithm

INTRODUCTION

Machine learning (ML) has been tried in predicting outcomes following sepsis. This study aims to identify the utility of stacked ensemble algorithm in predicting mortality.

METHODS

The study was a cohort of adults admitted to a medical unit of a tertiary care hospital with sepsis. The data were divided into a training data set (70%) and a test data set (30%). Boruta algorithm was used to identify important features. In the first phase of stacked ensemble model, weak learners such as random forest (RF), support vector machine (SVM), elastic net, and gradient boosting machine were trained. The SVM was used in phase 2 as meta learner to combine the results of all weak learners. All models were validated using test data.

RESULTS

In our cohort of 1,453 patients, the mortality rate was 27% (95% confidence interval [CI]: 25, 29). The Boruta algorithm identified inotrope use and assisted ventilation as the most important variables, which could predict mortality. The random forest outperforms (area under the curve [AUC]: 97.91%) the other algorithms. The AUCs for the other models are SVM (95.21%), GBM (93.67%), and GLM net (91.42%). However, the stacking of all the above models had an AUC of 92.14%. In the test data set, the accuracy of all methods including the RF method accuracy decreased (92.6 to 85.5%).

CONCLUSIONS

The random forest showed high accuracy in train and moderate accuracy in the test data. We suggest more regional open-access intensive care databases that can aid making machine learning a bigger support for healthcare personnel.

Investigating computational models for diagnosis and prognosis of sepsis based on clinical parameters: Opportunities, challenges, and future research directions

This study investigates the use of computational frameworks for sepsis. We consider two dimensions for investigation - early diagnosis of sepsis (EDS) and mortality prediction rate for sepsis patients (MPS). We concentrate on the clinical parameters on which sepsis diagnosis and prognosis are currently done, including customized treatment plans based on historical data of the patient. We identify the most notable literature that uses computational models to address EDS and MPS based on those clinical parameters. In addition to the review of the computational models built upon the clinical parameters, we also provide details regarding the popular publicly available data sources. We provide brief reviews for each model in terms of prior art and present an analysis of their results, as claimed by the respective authors. With respect to the use of machine learning models, we have provided avenues for model analysis in terms of model selection, model validation, model interpretation, and model comparison. We further present the challenges and limitations of the use of computational models, providing future research directions. This study intends to serve as a benchmark for first-hand impressions on the use of computational models for EDS and MPS of sepsis, along with the details regarding which model has been the most promising to date. We have provided details regarding all the ML models that have been used to date for EDS and MPS of sepsis.

Machine learning-based prognostic model for 30-day mortality prediction in Sepsis-3

BACKGROUND

Sepsis poses a critical threat to hospitalized patients, particularly those in the Intensive Care Unit (ICU). Rapid identification of Sepsis is crucial for improving survival rates. Machine learning techniques offer advantages over traditional methods for predicting outcomes. This study aimed to develop a prognostic model using a Stacking-based Meta-Classifier to predict 30-day mortality risks in Sepsis-3 patients from the MIMIC-III database.

METHODS

A cohort of 4,240 Sepsis-3 patients was analyzed, with 783 experiencing 30-day mortality and 3,457 surviving. Fifteen biomarkers were selected using feature ranking methods, including Extreme Gradient Boosting (XGBoost), Random Forest, and Extra Tree, and the Logistic Regression (LR) model was used to assess their individual predictability with a fivefold cross-validation approach for the validation of the prediction. The dataset was balanced using the SMOTE-TOMEK LINK technique, and a stacking-based meta-classifier was used for 30-day mortality prediction. The SHapley Additive explanations analysis was performed to explain the model's prediction.

RESULTS

Using the LR classifier, the model achieved an area under the curve or AUC score of 0.99. A nomogram provided clinical insights into the biomarkers' significance. The stacked meta-learner, LR classifier exhibited the best performance with 95.52% accuracy, 95.79% precision, 95.52% recall, 93.65% specificity, and a 95.60% F1-score.

CONCLUSIONS

In conjunction with the nomogram, the proposed stacking classifier model effectively predicted 30-day mortality in Sepsis patients. This approach holds promise for early intervention and improved outcomes in treating Sepsis cases.

Development and validation of an interpretable machine learning for mortality prediction in patients with sepsis

Introduction

Sepsis is a leading cause of death. However, there is a lack of useful model to predict outcome in sepsis. Herein, the aim of this study was to develop an explainable machine learning (ML) model for predicting 28-day mortality in patients with sepsis based on Sepsis 3.0 criteria.

Methods

We obtained the data from the Medical Information Mart for Intensive Care (MIMIC)-III database (version 1.4). The overall data was randomly assigned to the training and testing sets at a ratio of 3:1. Following the application of LASSO regression analysis to identify the modeling variables, we proceeded to develop models using Extreme Gradient Boost (XGBoost), Logistic Regression (LR), Support Vector Machine (SVM), and Random Forest (RF) techniques with 5-fold cross-validation. The optimal model was selected based on its area under the curve (AUC). Finally, the Shapley additive explanations (SHAP) method was used to interpret the optimal model.

Results

A total of 5,834 septic adults were enrolled, the median age was 66 years (IQR, 54-78 years) and 2,342 (40.1%) were women. After feature selection, 14 variables were included for developing model in the training set. The XGBoost model (AUC: 0.806) showed superior performance with AUC, compared with RF (AUC: 0.794), LR (AUC: 0.782) and SVM model (AUC: 0.687). SHAP summary analysis for XGBoost model showed that urine output on day 1, age, blood urea nitrogen and body mass index were the top four contributors. SHAP dependence analysis demonstrated insightful nonlinear interactive associations between factors and outcome. SHAP force analysis provided three samples for model prediction.

Conclusion

In conclusion, our study successfully demonstrated the efficacy of ML models in predicting 28-day mortality in sepsis patients, while highlighting the potential of the SHAP method to enhance model transparency and aid in clinical decision-making.

ChatGPT Predicts In-Hospital All-Cause Mortality for Sepsis: In-Context Learning with the Korean Sepsis Alliance Database

OBJECTIVES

Sepsis is a leading global cause of mortality, and predicting its outcomes is vital for improving patient care. This study explored the capabilities of ChatGPT, a state-of-the-art natural language processing model, in predicting in-hospital mortality for sepsis patients.

METHODS

This study utilized data from the Korean Sepsis Alliance (KSA) database, collected between 2019 and 2021, focusing on adult intensive care unit (ICU) patients and aiming to determine whether ChatGPT could predict all-cause mortality after ICU admission at 7 and 30 days. Structured prompts enabled ChatGPT to engage in in-context learning, with the number of patient examples varying from zero to six. The predictive capabilities of ChatGPT-3.5-turbo and ChatGPT-4 were then compared against a gradient boosting model (GBM) using various performance metrics.

RESULTS

From the KSA database, 4,786 patients formed the 7-day mortality prediction dataset, of whom 718 died, and 4,025 patients formed the 30-day dataset, with 1,368 deaths. Age and clinical markers (e.g., Sequential Organ Failure Assessment score and lactic acid levels) showed significant differences between survivors and non-survivors in both datasets. For 7-day mortality predictions, the area under the receiver operating characteristic curve (AUROC) was 0.70-0.83 for GPT-4, 0.51-0.70 for GPT-3.5, and 0.79 for GBM. The AUROC for 30-day mortality was 0.51-0.59 for GPT-4, 0.47-0.57 for GPT-3.5, and 0.76 for GBM. Zero-shot predictions using GPT-4 for mortality from ICU admission to day 30 showed AUROCs from the mid-0.60s to 0.75 for GPT-4 and mainly from 0.47 to 0.63 for GPT-3.5.

CONCLUSIONS

GPT-4 demonstrated potential in predicting short-term in-hospital mortality, although its performance varied across different evaluation metrics.

Predicting Chronic Pain and Treatment Outcomes Using Machine Learning Models Based on High-dimensional Clinical Data From a Large Retrospective Cohort

PURPOSE

To identify factors and indicators that affect chronic pain and pain relief, and to develop predictive models using machine learning.

METHODS

We analyzed the data of 67,028 outpatient cases and 11,310 valid samples with pain from a large retrospective cohort. We used decision tree, random forest, AdaBoost, neural network, and logistic regression to discover significant indicators and to predict pain and treatment relief.

FINDINGS

The random forest model had the highest accuracy, F1 value, precision, and recall rates for predicting pain relief. The main factors affecting pain and treatment relief included body mass index, blood pressure, age, body temperature, heart rate, pulse, and neutrophil/lymphocyte × platelet ratio. The logistic regression model had high sensitivity and specificity for predicting pain occurrence.

IMPLICATIONS

Machine learning models can be used to analyze the risk factors and predictors of chronic pain and pain relief, and to provide personalized and evidence-based pain management.

Modeling mortality prediction in older adults with dementia receiving COVID-19 vaccination

OBJECTIVE

This study compared COVID-19 outcomes between vaccinated and unvaccinated older adults with and without cognitive impairment.

METHOD

Electronic health records from Israel from March 2020-February 2022 were analyzed for a large cohort (N = 85,288) aged 65 + . Machine learning constructed models to predict mortality risk from patient factors. Outcomes examined were COVID-19 mortality and hospitalization post-vaccination.

RESULTS

Our study highlights the significant reduction in mortality risk among older adults with cognitive disorders following COVID-19 vaccination, showcasing a survival rate improvement to 93%. Utilizing machine learning for mortality prediction, we found the XGBoost model, enhanced with inverse probability of treatment weighting, to be the most effective, achieving an AUC-PR value of 0.89. This underscores the importance of predictive analytics in identifying high-risk individuals, emphasizing the critical role of vaccination in mitigating mortality and supporting targeted healthcare interventions.

CONCLUSIONS

COVID-19 vaccination strongly reduced poor outcomes in older adults with cognitive impairment. Predictive analytics can help identify highest-risk cases requiring targeted interventions.

Development and external validation of deep learning clinical prediction models using variable-length time series data

OBJECTIVES

To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection).

MATERIALS AND METHODS

This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC).

RESULTS

The study comprised 373 825 admissions for training and 256 128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks.

DISCUSSION

When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC.

CONCLUSION

The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

Predicting Outcomes Following Lower Extremity Endovascular Revascularization Using Machine Learning

BACKGROUND

Lower extremity endovascular revascularization for peripheral artery disease carries nonnegligible perioperative risks; however, outcome prediction tools remain limited. Using machine learning, we developed automated algorithms that predict 30-day outcomes following lower extremity endovascular revascularization.

METHODS AND RESULTS

The National Surgical Quality Improvement Program targeted vascular database was used to identify patients who underwent lower extremity endovascular revascularization (angioplasty, stent, or atherectomy) for peripheral artery disease between 2011 and 2021. Input features included 38 preoperative demographic/clinical variables. The primary outcome was 30-day postprocedural major adverse limb event (composite of major reintervention, untreated loss of patency, or major amputation) or death. Data were split into training (70%) and test (30%) sets. Using 10-fold cross-validation, 6 machine learning models were trained using preoperative features. The primary model evaluation metric was area under the receiver operating characteristic curve. Overall, 21 886 patients were included, and 30-day major adverse limb event/death occurred in 1964 (9.0%) individuals. The best performing model for predicting 30-day major adverse limb event/death was extreme gradient boosting, achieving an area under the receiver operating characteristic curve of 0.93 (95% CI, 0.92-0.94). In comparison, logistic regression had an area under the receiver operating characteristic curve of 0.72 (95% CI, 0.70-0.74). The calibration plot showed good agreement between predicted and observed event probabilities with a Brier score of 0.09. The top 3 predictive features in our algorithm were (1) chronic limb-threatening ischemia, (2) tibial intervention, and (3) congestive heart failure.

CONCLUSIONS

Our machine learning models accurately predict 30-day outcomes following lower extremity endovascular revascularization using preoperative data with good discrimination and calibration. Prospective validation is warranted to assess for generalizability and external validity.

Comparative Analysis of Machine Learning Models for Prediction of Acute Liver Injury in Sepsis Patients

Introduction

Acute liver injury (ALI) is a common complication of sepsis and is associated with adverse clinical outcomes. We aimed to develop a model to predict the risk of ALI in patients with sepsis after hospitalization.

Methods

Medical records of 3196 septic patients treated at the Lishui Central Hospital in Zhejiang Province from January 2015 to May 2023 were selected. Cohort 1 was divided into ALI and non-ALI groups for model training and internal validation. The initial laboratory test results of the study subjects were used as features for machine learning (ML), and models built using nine different ML algorithms were compared to select the best algorithm and model. The predictive performance of model stacking methods was then explored. The best model was externally validated in Cohort 2.

Results

In Cohort 1, LightGBM demonstrated good stability and predictive performance with an area under the curve (AUC) of 0.841. The top five most important variables in the model were diabetes, congestive heart failure, prothrombin time, heart rate, and platelet count. The LightGBM model showed stable and good ALI risk prediction ability in the external validation of Cohort 2 with an AUC of 0.815. Furthermore, an online prediction website was developed to assist healthcare professionals in applying this model more effectively.

Conclusions

The Light GBM model can predict the risk of ALI in patients with sepsis after hospitalization.

Prognostic prediction of sepsis patient using transformer with skip connected token for tabular data

Sepsis is known as a common syndrome in intensive care units (ICU), and severe sepsis and septic shock are among the leading causes of death worldwide. The purpose of this study is to develop a deep learning model that supports clinicians in efficiently managing sepsis patients in the ICU by predicting mortality, ICU length of stay (>14 days), and hospital length of stay (>30 days). The proposed model was developed using 591 retrospective data with 16 tabular data related to a sequential organ failure assessment (SOFA) score. To analyze tabular data, we designed the modified architecture of the transformer that has achieved extraordinary success in the field of languages and computer vision tasks in recent years. The main idea of the proposed model is to use a skip-connected token, which combines both local (feature-wise token) and global (classification token) information as the output of a transformer encoder. The proposed model was compared with four machine learning models (ElasticNet, Extreme Gradient Boosting [XGBoost]), and Random Forest) and three deep learning models (Multi-Layer Perceptron [MLP], transformer, and Feature-Tokenizer transformer [FT-Transformer]) and achieved the best performance (mortality, area under the receiver operating characteristic (AUROC) 0.8047; ICU length of stay, AUROC 0.8314; hospital length of stay, AUROC 0.7342). We anticipate that the proposed model architecture will provide a promising approach to predict the various clinical endpoints using tabular data such as electronic health and medical records.

Explainable Machine Learning Models for Rapid Risk Stratification in the Emergency Department: A Multicenter Study

BACKGROUND

Risk stratification of patients presenting to the emergency department (ED) is important for appropriate triage. Diagnostic laboratory tests are an essential part of the workup and risk stratification of these patients. Using machine learning, the prognostic power and clinical value of these tests can be amplified greatly. In this study, we applied machine learning to develop an accurate and explainable clinical decision support tool model that predicts the likelihood of 31-day mortality in ED patients (the RISKINDEX). This tool was developed and evaluated in four Dutch hospitals.

METHODS

Machine learning models included patient characteristics and available laboratory data collected within the first 2 h after ED presentation, and were trained using 5 years of data from consecutive ED patients from the Maastricht University Medical Center (Maastricht), Meander Medical Center (Amersfoort), and Zuyderland Medical Center (Sittard and Heerlen). A sixth year of data was used to evaluate the models using area under the receiver-operating-characteristic curve (AUROC) and calibration curves. The Shapley additive explanations (SHAP) algorithm was used to obtain explainable machine learning models.

RESULTS

The present study included 266 327 patients with 7.1 million laboratory results available. Models show high diagnostic performance with AUROCs of 0.94, 0.98, 0.88, and 0.90 for Maastricht, Amersfoort, Sittard and Heerlen, respectively. The SHAP algorithm was utilized to visualize patient characteristics and laboratory data patterns that underlie individual RISKINDEX predictions.

CONCLUSIONS

Our clinical decision support tool has excellent diagnostic performance in predicting 31-day mortality in ED patients. Follow-up studies will assess whether implementation of these algorithms can improve clinically relevant end points.

Early Prediction of Mortality for Septic Patients Visiting Emergency Room Based on Explainable Machine Learning: A Real-World Multicenter Study

BACKGROUND

Worldwide, sepsis is the leading cause of death in hospitals. If mortality rates in patients with sepsis can be predicted early, medical resources can be allocated efficiently. We constructed machine learning (ML) models to predict the mortality of patients with sepsis in a hospital emergency department.

METHODS

This study prospectively collected nationwide data from an ongoing multicenter cohort of patients with sepsis identified in the emergency department. Patients were enrolled from 19 hospitals between September 2019 and December 2020. For acquired data from 3,657 survivors and 1,455 deaths, six ML models (logistic regression, support vector machine, random forest, extreme gradient boosting [XGBoost], light gradient boosting machine, and categorical boosting [CatBoost]) were constructed using fivefold cross-validation to predict mortality. Through these models, 44 clinical variables measured on the day of admission were compared with six sequential organ failure assessment (SOFA) components (PaO2/FIO2 [PF], platelets (PLT), bilirubin, cardiovascular, Glasgow Coma Scale score, and creatinine). The confidence interval (CI) was obtained by performing 10,000 repeated measurements via random sampling of the test dataset. All results were explained and interpreted using Shapley's additive explanations (SHAP).

RESULTS

Of the 5,112 participants, CatBoost exhibited the highest area under the curve (AUC) of 0.800 (95% CI, 0.756-0.840) using clinical variables. Using the SOFA components for the same patient, XGBoost exhibited the highest AUC of 0.678 (95% CI, 0.626-0.730). As interpreted by SHAP, albumin, lactate, blood urea nitrogen, and international normalization ratio were determined to significantly affect the results. Additionally, PF and PLTs in the SOFA component significantly influenced the prediction results.

CONCLUSION

Newly established ML-based models achieved good prediction of mortality in patients with sepsis. Using several clinical variables acquired at the baseline can provide more accurate results for early predictions than using SOFA components. Additionally, the impact of each variable was identified.

Machine learning for risk stratification in the emergency department (MARS-ED) study protocol for a randomized controlled pilot trial on the implementation of a prediction model based on machine learning technology predicting 31-day mortality in the emergency department

BACKGROUND

Many prediction models have been developed to help identify emergency department (ED) patients at high risk of poor outcome. However, these models often underperform in clinical practice and their actual clinical impact has hardly ever been evaluated. We aim to perform a clinical trial to investigate the clinical impact of a prediction model based on machine learning (ML) technology.

METHODS

The study is a prospective, randomized, open-label, non-inferiority pilot clinical trial. We will investigate the clinical impact of a prediction model based on ML technology, the RISKINDEX, which has been developed to predict the risk of 31-day mortality based on the results of laboratory tests and demographic characteristics. In previous studies, the RISKINDEX was shown to outperform internal medicine specialists and to have high discriminatory performance. Adults patients (18 years or older) will be recruited in the ED. All participants will be randomly assigned to the control group or the intervention group in a 1:1 ratio. Participants in the control group will receive care as usual in which the study team asks the attending physicians questions about their clinical intuition. Participants in the intervention group will also receive care as usual, but in addition to asking the clinical impression questions, the study team presents the RISKINDEX to the attending physician in order to assess the extent to which clinical treatment is influenced by the results.

DISCUSSION

This pilot clinical trial investigates the clinical impact and implementation of an ML based prediction model in the ED. By assessing the clinical impact and prognostic accuracy of the RISKINDEX, this study aims to contribute valuable insights to optimize patient care and inform future research in the field of ML based clinical prediction models.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05497830. Machine Learning for Risk Stratification in the Emergency Department (MARS-ED). Registered on August 11, 2022. URL: https://clinicaltrials.gov/study/NCT05497830 .

Machine learning algorithms in sepsis

Sepsis remains a significant global health challenge due to its high mortality and morbidity, compounded by the difficulty of early detection given its variable clinical manifestations. The integration of machine learning (ML) into laboratory medicine for timely sepsis identification and outcome forecasting is an emerging field of interest. This comprehensive review assesses the current body of research on ML applications for sepsis within the realm of laboratory diagnostics, detailing both their strengths and shortcomings. An extensive literature search was performed by two independent investigators across PubMed and Scopus databases, employing the keywords "Sepsis," "Machine Learning," and "Laboratory" without publication date limitations, culminating in January 2023. Each selected study was meticulously evaluated for various aspects, including its design, intent (diagnostic or prognostic), clinical environment, demographics, sepsis criteria, data gathering period, and the scope and nature of features, in addition to the ML methodologies and their validation procedures. Out of 135 articles reviewed, 39 fulfilled the criteria for inclusion. Among these, the majority (30 studies) were focused on devising ML algorithms for diagnosis, fewer (8 studies) on prognosis, and one study addressed both aspects. The dissemination of these studies across an array of journals reflects the interdisciplinary engagement in the development of ML algorithms for sepsis. This analysis highlights the promising role of ML in the early diagnosis of sepsis while drawing attention to the need for uniformity in validating models and defining features, crucial steps for ensuring the reliability and practicality of ML in clinical setting.

Deep learning-based prediction of in-hospital mortality for sepsis

As a serious blood infection disease, sepsis is characterized by a high mortality risk and many complications. Accurate assessment of mortality risk of patients with sepsis can help physicians in Intensive Care Unit make optimal clinical decisions, which in turn can effectively save patients' lives. However, most of the current clinical models used for assessing mortality risk in sepsis patients are based on conventional indicators. Unfortunately, some of the conventional indicators have been shown to be inapplicable in the accurate clinical diagnosis nowadays. Meanwhile, traditional evaluation models only focus on a small amount of personal data, causing misdiagnosis of sepsis patients. We refine the core indicators for mortality risk assessment of sepsis from massive clinical electronic medical records with machine learning, and propose a new mortality risk assessment model, DGFSD, for sepsis patients based on deep learning. The DGFSD model can not only learn individual clinical information about unassessed patients, but also obtain information about the structure of the similarity graph between diagnosed patients and patients to be assessed. Numerous experiments have shown that the accuracy of the DGFSD model is superior to baseline methods, and can significantly improve the efficiency of clinical auxiliary diagnosis.

Machine learning for the prediction of sepsis-related death: a systematic review and meta-analysis

BACKGROUND AND OBJECTIVES

Sepsis is accompanied by a considerably high risk of mortality in the short term, despite the availability of recommended mortality risk assessment tools. However, these risk assessment tools seem to have limited predictive value. With the gradual integration of machine learning into clinical practice, some researchers have attempted to employ machine learning for early mortality risk prediction in sepsis patients. Nevertheless, there is a lack of comprehensive understanding regarding the construction of predictive variables using machine learning and the value of various machine learning methods. Thus, we carried out this systematic review and meta-analysis to explore the predictive value of machine learning for sepsis-related death at different time points.

METHODS

PubMed, Embase, Cochrane, and Web of Science databases were searched until August 9th, 2022. The risk of bias in predictive models was assessed using the Prediction model Risk of Bias Assessment Tool (PROBAST). We also performed subgroup analysis according to time of death and type of model and summarized current predictive variables used to construct models for sepsis death prediction.

RESULTS

Fifty original studies were included, covering 104 models. The combined Concordance index (C-index), sensitivity, and specificity of machine learning models were 0.799, 0.81, and 0.80 in the training set, and 0.774, 0.71, and 0.68 in the validation set, respectively. Machine learning outperformed conventional clinical scoring tools and showed excellent C-index, sensitivity, and specificity in different subgroups. Random Forest (RF) and eXtreme Gradient Boosting (XGBoost) are the preferred machine learning models because they showed more favorable accuracy with similar modeling variables. This study found that lactate was the most frequent predictor but was seriously ignored by current clinical scoring tools.

CONCLUSION

Machine learning methods demonstrate relatively favorable accuracy in predicting the mortality risk in sepsis patients. Given the limitations in accuracy and applicability of existing prediction scoring systems, there is an opportunity to explore updates based on existing machine learning approaches. Specifically, it is essential to develop or update more suitable mortality risk assessment tools based on the specific contexts of use, such as emergency departments, general wards, and intensive care units.

Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study

INTRODUCTION

Prediction models for identifying emergency department (ED) patients at high risk of poor outcome are often not externally validated. We aimed to perform a head-to-head comparison of the discriminatory performance of several prediction models in a large cohort of ED patients.

METHODS

In this retrospective study, we selected prediction models that aim to predict poor outcome and we included adult medical ED patients. Primary outcome was 31-day mortality, secondary outcomes were 1-day mortality, 7-day mortality, and a composite endpoint of 31-day mortality and admission to intensive care unit (ICU).The discriminatory performance of the prediction models was assessed using an area under the receiver operating characteristic curve (AUC). Finally, the prediction models with the highest performance to predict 31-day mortality were selected to further examine calibration and appropriate clinical cut-off points.

RESULTS

We included 19 prediction models and applied these to 2185 ED patients. Thirty-one-day mortality was 10.6% (231 patients), 1-day mortality was 1.4%, 7-day mortality was 4.4%, and 331 patients (15.1%) met the composite endpoint. The RISE UP and COPE score showed similar and very good discriminatory performance for 31-day mortality (AUC 0.86), 1-day mortality (AUC 0.87), 7-day mortality (AUC 0.86) and for the composite endpoint (AUC 0.81). Both scores were well calibrated. Almost no patients with RISE UP and COPE scores below 5% had an adverse outcome, while those with scores above 20% were at high risk of adverse outcome. Some of the other prediction models (i.e. APACHE II, NEWS, WPSS, MEWS, EWS and SOFA) showed significantly higher discriminatory performance for 1-day and 7-day mortality than for 31-day mortality.

CONCLUSIONS

Head-to-head validation of 19 prediction models in medical ED patients showed that the RISE UP and COPE score outperformed other models regarding 31-day mortality.

Clinical judgment shows similar and sometimes superior discrimination compared to prognostic clinical prediction models. A systematic review

OBJECTIVES

To systematically review the comparative statistical performance (discrimination and /or calibration) of prognostic clinical prediction models (CPMs) and clinician judgment (CJ).

STUDY DESIGN AND SETTING

We conducted a systematic review of observational studies in PubMed, Medline, Embase, and CINAHL. Eligible studies reported direct statistical comparison between prognostic CPMs and CJ. Risk of bias was assessed using the PROBAST tool.

RESULTS

We identified 41 studies, most with high risk of bias (39 studies). Of these, 41 studies, 39 examined discrimination and 12 studies assessed calibration. Prognostic CPMs had a median AUC of 0.73 (IQR, 0.62 - 0.81), while CJ had a median AUC of 0.71 (IQR, 0.62 - 0.81). 29 studies provided 124 discrimination metrics useful for comparative analysis. Among these, 58 (46.7%) found no significant difference between prognostic CPMs and CJ (p > 0.05); 31 (25%) favored prognostic CPMs, and 35 (28.2%) favored CJ. Four studies compared calibration, showing better performance on prognostic CPMs.

CONCLUSIONS

In many instances CJ frequently demonstrates comparable or superior discrimination compared to prognostic CPMs, although models outperform CJ on calibration. Studies comparing performance of prognostic CPMs and CJ require large improvements in reporting.

Clinician Trust in Artificial Intelligence: What is Known and How Trust Can Be Facilitated

Predictive analytics based on artificial intelligence (AI) offer clinicians the opportunity to leverage big data available in electronic health records (EHR) to improve clinical decision-making, and thus patient outcomes. Despite this, many barriers exist to facilitating trust between clinicians and AI-based tools, limiting its current impact. Potential solutions are available at both the local and national level. It will take a broad and diverse coalition of stakeholders, from health-care systems, EHR vendors, and clinical educators to regulators, researchers and the patient community, to help facilitate this trust so that the promise of AI in health care can be realized.

Using machine learning for process improvement in sepsis management

INTRODUCTION

In the U.S., sepsis afflicts 1.7 million adults, causing 270,000 deaths each year. Early detection of sepsis could decrease the number of deaths by 92,000 annually and decrease hospital expenditures by 1.5 billion USD. Few prior studies and reviews have presented a holistic understanding of the relationship between machine learning and existing process improvement measures. This study, in addition to discussing machine learning and existing process improvements measures, elaborates on the disadvantages and the barriers to integrating machine learning into the clinic. This article synthesizes previous studies to educate healthcare professionals on effectively managing sepsis by leveraging the benefits of machine learning.

METHODS

This study used the PubMed database. Search terms include sepsis antibiotics, sepsis process improvement, sepsis machine learning. Our search criteria included previous studies published between January 1, 2017, and February 1, 2022.

RESULTS/DISCUSSION

Although machine learning algorithms have better predictive capabilities, their effectiveness in the clinical setting is limited as studies show mixed results because the medical staff often fails to intervene. To overcome poor interventional response, clinicians need to work with the facility's IT department to ensure integration into clinical workflow and minimize alert-fatigue. Algorithms should enhance the productivity of clinical teams, not attempt to replace them entirely.

CONCLUSION

Hospitals can employ process improvement measures that effectively utilize machine learning algorithms to ensure integration into clinical workflows. Healthcare professionals can utilize workflow tools in addition to the predictive capabilities of machine learning to enhance clinical decisions in sepsis.

Comparing In-Hospital Mortality Prediction by Senior Emergency Resident's Judgment and Prognostic Models in the Emergency Department

Background

A comparison of emergency residents' judgments and two derivatives of the Sequential Organ Failure Assessment (SOFA), namely, the mSOFA and the qSOFA, was conducted to determine the accuracy of predicting in-hospital mortality among critically ill patients in the emergency department (ED).

Methods

A prospective cohort research was performed on patients over 18 years of age presented to the ED. We used logistic regression to develop a model for predicting in-hospital mortality by using qSOFA, mSOFA, and residents' judgment scores. We compared the accuracy of prognostic models and residents' judgment in terms of the overall accuracy of the predicted probabilities (Brier score), discrimination (area under the ROC curve), and calibration (calibration graph). Analyses were carried out using R software version R-4.2.0.

Results

In the study, 2,205 patients with median age of 64 (IQR: 50-77) years were included. There were no significant differences between the qSOFA (AUC 0.70; 95% CI: 0.67-0.73) and physician's judgment (AUC 0.68; 0.65-0.71). Despite this, the discrimination of mSOFA (AUC 0.74; 0.71-0.77) was significantly higher than that of the qSOFA and residents' judgments. Additionally, the AUC-PR of mSOFA, qSOFA, and emergency resident's judgments was 0.45 (0.43-0.47), 0.38 (0.36-0.40), and 0.35 (0.33-0.37), respectively. The mSOFA appears stronger in terms of overall performance: 0.13 vs. 0.14 and 0.15. All three models showed good calibration.

Conclusion

The performance of emergency residents' judgment and the qSOFA was the same in predicting in-hospital mortality. However, the mSOFA predicted better-calibrated mortality risk. Large-scale studies should be conducted to determine the utility of these models.

Developing and Validating a Prediction Model For Death or Critical Illness in Hospitalized Adults, an Opportunity for Human-Computer Collaboration

Hospital early warning systems that use machine learning (ML) to predict clinical deterioration are increasingly being used to aid clinical decision-making. However, it is not known how ML predictions complement physician and nurse judgment. Our objective was to train and validate a ML model to predict patient deterioration and compare model predictions with real-world physician and nurse predictions.

DESIGN

Retrospective and prospective cohort study.

SETTING

Academic tertiary care hospital.

PATIENTS

Adult general internal medicine hospitalizations.

MEASUREMENTS AND MAIN RESULTS

We developed and validated a neural network model to predict in-hospital death and ICU admission in 23,528 hospitalizations between April 2011 and April 2019. We then compared model predictions with 3,374 prospectively collected predictions from nurses, residents, and attending physicians about their own patients in 960 hospitalizations between April 30, and August 28, 2019. ML model predictions achieved clinician-level accuracy for predicting ICU admission or death (ML median F1 score 0.32 [interquartile range (IQR) 0.30-0.34], AUC 0.77 [IQ 0.76-0.78]; clinicians median F1-score 0.33 [IQR 0.30-0.35], AUC 0.64 [IQR 0.63-0.66]). ML predictions were more accurate than clinicians for ICU admission. Of all ICU admissions and deaths, 36% occurred in hospitalizations where the model and clinicians disagreed. Combining human and model predictions detected 49% of clinical deterioration events, improving sensitivity by 16% compared with clinicians alone and 24% compared with the model alone while maintaining a positive predictive value of 33%, thus keeping false alarms at a clinically acceptable level.

CONCLUSIONS

ML models can complement clinician judgment to predict clinical deterioration in hospital. These findings demonstrate important opportunities for human-computer collaboration to improve prognostication and personalized medicine in hospital.

Sepsis biomarkers and diagnostic tools with a focus on machine learning

Over the last years, there have been advances in the use of data-driven techniques to improve the definition, early recognition, subtypes characterisation, prognostication and treatment personalisation of sepsis. Some of those involve the discovery or evaluation of biomarkers or digital signatures of sepsis or sepsis sub-phenotypes. It is hoped that their identification may improve timeliness and accuracy of diagnosis, suggest physiological pathways and therapeutic targets, inform targeted recruitment into clinical trials, and optimise clinical management. Given the complexities of the sepsis response, panels of biomarkers or models combining biomarkers and clinical data are necessary, as well as specific data analysis methods, which broadly fall under the scope of machine learning. This narrative review gives a brief overview of the main machine learning techniques (mainly in the realms of supervised and unsupervised methods) and published applications that have been used to create sepsis diagnostic tools and identify biomarkers.

Identifying infected patients using semi-supervised and transfer learning

Objectives

Early identification of infection improves outcomes, but developing models for early identification requires determining infection status with manual chart review, limiting sample size. Therefore, we aimed to compare semi-supervised and transfer learning algorithms with algorithms based solely on manual chart review for identifying infection in hospitalized patients.

Materials and Methods

This multicenter retrospective study of admissions to 6 hospitals included “gold-standard” labels of infection from manual chart review and “silver-standard” labels from nonchart-reviewed patients using the Sepsis-3 infection criteria based on antibiotic and culture orders. “Gold-standard” labeled admissions were randomly allocated to training (70%) and testing (30%) datasets. Using patient characteristics, vital signs, and laboratory data from the first 24 hours of admission, we derived deep learning and non-deep learning models using transfer learning and semi-supervised methods. Performance was compared in the gold-standard test set using discrimination and calibration metrics.

Results

The study comprised 432 965 admissions, of which 2724 underwent chart review. In the test set, deep learning and non-deep learning approaches had similar discrimination (area under the receiver operating characteristic curve of 0.82). Semi-supervised and transfer learning approaches did not improve discrimination over models fit using only silver- or gold-standard data. Transfer learning had the best calibration (unreliability index P value: .997, Brier score: 0.173), followed by self-learning gradient boosted machine (P value: .67, Brier score: 0.170).

Discussion

Deep learning and non-deep learning models performed similarly for identifying infection, as did models developed using Sepsis-3 and manual chart review labels.

Conclusion

In a multicenter study of almost 3000 chart-reviewed patients, semi-supervised and transfer learning models showed similar performance for model discrimination as baseline XGBoost, while transfer learning improved calibration.

Explainable Machine-Learning Model for Prediction of In-Hospital Mortality in Septic Patients Requiring Intensive Care Unit Readmission

INTRODUCTION

Septic patients requiring intensive care unit (ICU) readmission are at high risk of mortality, but research focusing on the association of ICU readmission due to sepsis and mortality is limited. The aim of this study was to develop and validate a machine learning (ML) model for predicting in-hospital mortality in septic patients readmitted to the ICU using routinely available clinical data.

METHODS

The data used in this study were obtained from the Medical Information Mart for Intensive Care (MIMIC-IV, v1.0) database, between 2008 and 2019. The study cohort included patients with sepsis requiring ICU readmission. The data were randomly split into a training (75%) data set and a validation (25%) data set. Nine popular ML models were developed to predict mortality in septic patients readmitted to the ICU. The model with the best accuracy and area under the curve (A.C.) in the validation cohort was defined as the optimal model and was selected for further prediction studies. The SHAPELY Additive explanations (SHAP) values and Local Interpretable Model-Agnostic Explanation (LIME) methods were used to improve the interpretability of the optimal model.

RESULTS

A total of 1117 septic patients who had required ICU readmission during the study period were enrolled in the study. Of these participants, 434 (38.9%) were female, and the median (interquartile range [IQR]) age was 68.6 (58.4-79.2) years. The median (IQR) ICU interval duration was 2.60 (0.64-5.78) days. After feature selection, 31 of 47 clinical factors were ultimately chosen for use in model construction. Of the nine ML models tested, the best performance was achieved with the random forest (RF) model, with an A.C. of 0.81, an accuracy of 85% and a precision of 62% in the validation cohort. The SHAP summary analysis revealed that Glasgow Coma Scale score, urine output, blood urea nitrogen, lactate, platelet count and systolic blood pressure were the top six most important factors contributing to the RF model. Additionally, the LIME method demonstrated how the RF model works in terms of explaining risk of death prediction in septic patients requiring ICU readmission.

CONCLUSION

The ML models reported here showed a good prognostic prediction ability in septic patients requiring ICU readmission. Of the features selected, the parameters related to organ perfusion made the largest contribution to outcome prediction during ICU readmission in septic patients.

In-Hospital Mortality of Sepsis Differs Depending on the Origin of Infection: An Investigation of Predisposing Factors

Sepsis is a heterogeneous syndrome characterized by a variety of clinical features. Analysis of large clinical datasets may serve to define groups of sepsis with different risks of adverse outcomes. Clinical experience supports the concept that prognosis, treatment, severity, and time course of sepsis vary depending on the source of infection. We analyzed a large publicly available database to test this hypothesis. In addition, we developed prognostic models for the three main types of sepsis: pulmonary, urinary, and abdominal sepsis. We used logistic regression using routinely available clinical data for mortality prediction in each of these groups. The data was extracted from the eICU collaborative research database, a multi-center intensive care unit with over 200,000 admissions. Sepsis cohorts were defined using admission diagnosis codes. We used univariate and multivariate analyses to establish factors relevant for outcome prediction in all three cohorts of sepsis (pulmonary, urinary and abdominal). For logistic regression, input variables were automatically selected using a sequential forward search algorithm over 10 dataset instances. Receiver operator characteristics were generated for each model and compared with established prognostication tools (APACHE IV and SOFA). A total of 3,958 sepsis admissions were included in the analysis. Sepsis in-hospital mortality differed depending on the cause of infection: abdominal 18.93%, pulmonary 19.27%, and renal 12.81%. Higher average heart rate was associated with increased mortality risk. Increased average Mean Arterial Pressure (MAP) showed a reduced mortality risk across all sepsis groups. Results from the LR models found significant factors that were relevant for specific sepsis groups. Our models outperformed APACHE IV and SOFA scores with AUC between 0.63 and 0.74. Predictive power decreased over time, with the best results achieved for data extracted for the first 24 h of admission. Mortality varied significantly between the three sepsis groups. We also demonstrate that factors of importance show considerable heterogeneity depending on the source of infection. The factors influencing in-hospital mortality vary depending on the source of sepsis which may explain why most sepsis trials have failed to identify an effective treatment. The source of infection should be considered when considering mortality risk. Planning of sepsis treatment trials may benefit from risk stratification based on the source of infection.

A Pipeline for the Implementation and Visualization of Explainable Machine Learning for Medical Imaging Using Radiomics Features

Machine learning (ML) models have been shown to predict the presence of clinical factors from medical imaging with remarkable accuracy. However, these complex models can be difficult to interpret and are often criticized as “black boxes”. Prediction models that provide no insight into how their predictions are obtained are difficult to trust for making important clinical decisions, such as medical diagnoses or treatment. Explainable machine learning (XML) methods, such as Shapley values, have made it possible to explain the behavior of ML algorithms and to identify which predictors contribute most to a prediction. Incorporating XML methods into medical software tools has the potential to increase trust in ML-powered predictions and aid physicians in making medical decisions. Specifically, in the field of medical imaging analysis the most used methods for explaining deep learning-based model predictions are saliency maps that highlight important areas of an image. However, they do not provide a straightforward interpretation of which qualities of an image area are important. Here, we describe a novel pipeline for XML imaging that uses radiomics data and Shapley values as tools to explain outcome predictions from complex prediction models built with medical imaging with well-defined predictors. We present a visualization of XML imaging results in a clinician-focused dashboard that can be generalized to various settings. We demonstrate the use of this workflow for developing and explaining a prediction model using MRI data from glioma patients to predict a genetic mutation.

Sepsis Mortality Prediction Using Wearable Monitoring in Low-Middle Income Countries

Sepsis is associated with high mortality-particularly in low-middle income countries (LMICs). Critical care management of sepsis is challenging in LMICs due to the lack of care providers and the high cost of bedside monitors. Recent advances in wearable sensor technology and machine learning (ML) models in healthcare promise to deliver new ways of digital monitoring integrated with automated decision systems to reduce the mortality risk in sepsis. In this study, firstly, we aim to assess the feasibility of using wearable sensors instead of traditional bedside monitors in the sepsis care management of hospital admitted patients, and secondly, to introduce automated prediction models for the mortality prediction of sepsis patients. To this end, we continuously monitored 50 sepsis patients for nearly 24 h after their admission to the Hospital for Tropical Diseases in Vietnam. We then compared the performance and interpretability of state-of-the-art ML models for the task of mortality prediction of sepsis using the heart rate variability (HRV) signal from wearable sensors and vital signs from bedside monitors. Our results show that all ML models trained on wearable data outperformed ML models trained on data gathered from the bedside monitors for the task of mortality prediction with the highest performance (area under the precision recall curve = 0.83) achieved using time-varying features of HRV and recurrent neural networks. Our results demonstrate that the integration of automated ML prediction models with wearable technology is well suited for helping clinicians who manage sepsis patients in LMICs to reduce the mortality risk of sepsis.

Development and Structure of an Accurate Machine Learning Algorithm to Predict Inpatient Mortality and Hospice Outcomes in the Coronavirus Disease 2019 Era

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic has challenged the accuracy and racial biases present in traditional mortality scores. An accurate prognostic model that can be applied to hospitalized patients irrespective of race or COVID-19 status may benefit patient care.

RESEARCH DESIGN

This cohort study utilized historical and ongoing electronic health record features to develop and validate a deep-learning model applied on the second day of admission predicting a composite outcome of in-hospital mortality, discharge to hospice, or death within 30 days of admission. Model features included patient demographics, diagnoses, procedures, inpatient medications, laboratory values, vital signs, and substance use history. Conventional performance metrics were assessed, and subgroup analysis was performed based on race, COVID-19 status, and intensive care unit admission.

SUBJECTS

A total of 35,521 patients hospitalized between April 2020 and October 2020 at a single health care system including a tertiary academic referral center and 9 community hospitals.

RESULTS

Of 35,521 patients, including 9831 non-White patients and 2020 COVID-19 patients, 2838 (8.0%) met the composite outcome. Patients who experienced the composite outcome were older (73 vs. 61 y old) with similar sex and race distributions between groups. The model achieved an area under the receiver operating characteristic curve of 0.89 (95% confidence interval: 0.88, 0.91) and an average positive predictive value of 0.46 (0.40, 0.52). Model performance did not differ significantly in White (0.89) and non-White (0.90) subgroups or when grouping by COVID-19 status and intensive care unit admission.

CONCLUSION

A deep-learning model using large-volume, structured electronic health record data can effectively predict short-term mortality or hospice outcomes on the second day of admission in the general inpatient population without significant racial bias.

Interpretable Machine Learning for Early Prediction of Prognosis in Sepsis: A Discovery and Validation Study

Introduction

This study aimed to develop and validate an interpretable machine-learning model based on clinical features for early predicting in-hospital mortality in critically ill patients with sepsis.

Methods

We enrolled all patients with sepsis in the Medical Information Mart for Intensive Care IV (MIMIC-IV, v.1.0) database from 2008 to 2019. Lasso regression was used for feature selection. Seven machine-learning methods were applied to develop the models. The best model was selected based on its accuracy and area under curve (AUC) in the validation cohort. Furthermore, we employed the SHapley Additive exPlanations (SHAP) method to illustrate the effects of the features attributed to the model, and to analyze how the individual features affect the output of the model, and to visualize the Shapley value for a single individual.

Results

In total, 8,817 patients with sepsis were eligible for participation, the median age was 66.8 years (IQR, 55.9–77.1 years), and 3361 of 8817 participants (38.1%) were women. After selection, 25 of a total 57 clinical parameters collected on day 1 after ICU admission remained associated with prognosis and were used for developing the machine-learning models. Among seven constructed models, the eXtreme Gradient Boosting (XGBoost) model achieved the best performance with an AUC of 0.884 and an accuracy of 89.5% in the validation cohort. Feature importance analysis showed that Glasgow Coma Scale (GCS) score, blood urea nitrogen, respiratory rate, urine output, and age were the top 5 features of the XGBoost model with the greatest impact. Furthermore, SHAP force analysis illustrated how the constructed model visualized the individualized prediction of death.

Conclusions

We have demonstrated the potential of machine-learning approaches for predicting outcome early in patients with sepsis. The SHAP method could improve the interpretability of machine-learning models and help clinicians better understand the reasoning behind the outcome.

Graphical abstract

Supplementary Information

The online version contains supplementary material available at 10.1007/s40121-022-00628-6.

Artificial Intelligence in Infection Management in the ICU

This article is one of ten reviews selected from the Annual Update in Intensive Care and Emergency Medicine 2022. Other selected articles can be found online at https://www.biomedcentral.com/collections/annualupdate2022. Further information about the Annual Update in Intensive Care and Emergency Medicine is available from https://link.springer.com/bookseries/8901.

Understanding Artificial Intelligence and Predictive Analytics: A Clinically Focused Review of Machine Learning Techniques

»

Machine learning and artificial intelligence have seen tremendous growth in recent years and have been applied in numerous studies in the field of orthopaedics.

»

Machine learning will soon become critical in the day-to-day operations of orthopaedic practice; therefore, it is imperative that providers become accustomed to and familiar with not only the terminology but also the fundamental techniques behind the technology.

»

A foundation of knowledge regarding machine learning is critical for physicians so they can begin to understand the details in the algorithms that are being developed, which provide improved accuracy compared with clinicians, decreased time required, and a heightened ability to triage patients.

Evaluating machine learning models for sepsis prediction: A systematic review of methodologies

Studies for sepsis prediction using machine learning are developing rapidly in medical science recently. In this review, we propose a set of new evaluation criteria and reporting standards to assess 21 qualified machine learning models for quality analysis based on PRISMA. Our assessment shows that (1.) the definition of sepsis is not consistent among the studies; (2.) data sources and data preprocessing methods, machine learning models, feature engineering, and inclusion types vary widely among the studies; (3.) the closer to the onset of sepsis, the higher the value of AUROC is; (4.) the improvement in AUROC is primarily due to using machine learning as a feature engineering tool; (5.) deep neural networks coupled with Sepsis-3 diagnostic criteria tend to yield better results on the time series data collected from patients with sepsis. The new evaluation criteria and reporting standards will facilitate the development of improved machine learning models for clinical applications.

•

New evaluation/reporting standard for sepsis prediction machine learning models

•

Major limitations in the current models for sepsis prediction have been identified

•

We strongly suggest using machine learning as a feature engineering tool

•

Recommending multilayer neural networks and Sepsis 3.0 for yield better result

Machine Learning for Cardiovascular Outcomes From Wearable Data: Systematic Review From a Technology Readiness Level Point of View

BACKGROUND

Wearable technology has the potential to improve cardiovascular health monitoring by using machine learning. Such technology enables remote health monitoring and allows for the diagnosis and prevention of cardiovascular diseases. In addition to the detection of cardiovascular disease, it can exclude this diagnosis in symptomatic patients, thereby preventing unnecessary hospital visits. In addition, early warning systems can aid cardiologists in timely treatment and prevention.

OBJECTIVE

This study aims to systematically assess the literature on detecting and predicting outcomes of patients with cardiovascular diseases by using machine learning with data obtained from wearables to gain insights into the current state, challenges, and limitations of this technology.

METHODS

We searched PubMed, Scopus, and IEEE Xplore on September 26, 2020, with no restrictions on the publication date and by using keywords such as "wearables," "machine learning," and "cardiovascular disease." Methodologies were categorized and analyzed according to machine learning-based technology readiness levels (TRLs), which score studies on their potential to be deployed in an operational setting from 1 to 9 (most ready).

RESULTS

After the removal of duplicates, application of exclusion criteria, and full-text screening, 55 eligible studies were included in the analysis, covering a variety of cardiovascular diseases. We assessed the quality of the included studies and found that none of the studies were integrated into a health care system (TRL<6), prospective phase 2 and phase 3 trials were absent (TRL<7 and 8), and group cross-validation was rarely used. These issues limited these studies' ability to demonstrate the effectiveness of their methodologies. Furthermore, there seemed to be no agreement on the sample size needed to train these studies' models, the size of the observation window used to make predictions, how long participants should be observed, and the type of machine learning model that is suitable for predicting cardiovascular outcomes.

CONCLUSIONS

Although current studies show the potential of wearables to monitor cardiovascular events, their deployment as a diagnostic or prognostic cardiovascular clinical tool is hampered by the lack of a realistic data set and proper systematic and prospective evaluation.

Machine learning techniques for mortality prediction in emergency departments: a systematic review

OBJECTIVES

This systematic review aimed to assess the performance and clinical feasibility of machine learning (ML) algorithms in prediction of in-hospital mortality for medical patients using vital signs at emergency departments (EDs).

DESIGN

A systematic review was performed.

SETTING

The databases including Medline (PubMed), Scopus and Embase (Ovid) were searched between 2010 and 2021, to extract published articles in English, describing ML-based models utilising vital sign variables to predict in-hospital mortality for patients admitted at EDs. Critical appraisal and data extraction for systematic reviews of prediction modelling studies checklist was used for study planning and data extraction. The risk of bias for included papers was assessed using the prediction risk of bias assessment tool.

PARTICIPANTS

Admitted patients to the ED.

MAIN OUTCOME MEASURE

In-hospital mortality.

RESULTS

Fifteen articles were included in the final review. We found that eight models including logistic regression, decision tree, K-nearest neighbours, support vector machine, gradient boosting, random forest, artificial neural networks and deep neural networks have been applied in this domain. Most studies failed to report essential main analysis steps such as data preprocessing and handling missing values. Fourteen included studies had a high risk of bias in the statistical analysis part, which could lead to poor performance in practice. Although the main aim of all studies was developing a predictive model for mortality, nine articles did not provide a time horizon for the prediction.

CONCLUSION

This review provided an updated overview of the state-of-the-art and revealed research gaps; based on these, we provide eight recommendations for future studies to make the use of ML more feasible in practice. By following these recommendations, we expect to see more robust ML models applied in the future to help clinicians identify patient deterioration earlier.

Prognostic tools for elderly patients with sepsis: in search of new predictive models

As a tool to support clinical decision-making, Mortality Prediction Models (MPM) can help clinicians stratify and predict patient risk. There are numerous scoring systems for patients with sepsis that predict sepsis-related mortality and the severity of sepsis. But there are currently no MPMs for adults with sepsis who meet the criteria of "good." Clinicians are unlikely to use complex MPMs that require extensive or expensive data collection to impede workflow. Machine learning applied to minimal medical records of patients diagnosed with sepsis can be a useful tool. Progress is needed in the development and validation of clinical decision support tools that can assist in patient risk stratification, prognosis, discussion of patient outcomes, and shared decision making.

Between competence and warmth: the remaining place of the physician in the era of artificial intelligence

Competence and warmth are two essential dimensions of patient care. During the twentieth century, the industrial revolution in data collection, with the increasing use of machines and the division of labor that led to the development of many subspecialities, increased the overall competence of physicians at the expense of the warmth dimension. The spread of patient-centered care principles aimed to rebalance the two dimensions. In the twenty-first century, the industrial revolution in data processing with the emergence of algorithmic decision-making systems based on artificial intelligence is likely to disrupt further this balance. Competence will no longer be the prerogative of physicians, but a dimension to be shared between physicians and autonomous algorithmic decision-making systems, by contrast to warmth which should remain a human attribute. In this comment, we discuss the extent to which competence and warmth can remain the core dimensions of physician care in the era of artificial intelligence.

Machine Learning and Syncope Management in the ED: The Future Is Coming

In recent years, machine learning (ML) has been promisingly applied in many fields of clinical medicine, both for diagnosis and prognosis prediction. Aims of this narrative review were to summarize the basic concepts of ML applied to clinical medicine and explore its main applications in the emergency department (ED) setting, with a particular focus on syncope management. Through an extensive literature search in PubMed and Embase, we found increasing evidence suggesting that the use of ML algorithms can improve ED triage, diagnosis, and risk stratification of many diseases. However, the lacks of external validation and reliable diagnostic standards currently limit their implementation in clinical practice. Syncope represents a challenging problem for the emergency physician both because its diagnosis is not supported by specific tests and the available prognostic tools proved to be inefficient. ML algorithms have the potential to overcome these limitations and, in the future, they could support the clinician in managing syncope patients more efficiently. However, at present only few studies have addressed this issue, albeit with encouraging results.