Benchmarking emergency department prediction models with machine learning and public electronic health records

An explainable machine learning approach for hospital emergency department visits forecasting using continuous training and multi-model regression

BACKGROUND AND OBJECTIVE

In the last years, the Emergency Department (ED) has become an important source of admissions for hospitals. Since late 90s, the number of ED visits has been steadily increasing, and since Covid19 pandemic this trend has been much stronger. Accurate prediction of ED visits, even for moderate forecasting time-horizons, can definitively improve operational efficiency, quality of care, and patient outcomes in hospitals.

METHODS

In this paper we propose two different interpretable approaches, based on Machine Learning algorithms, to accurately forecast hospital emergency visits. The proposed approaches involve a first step of data segmentation based on two different criteria, depending on the approach considered: first, a threshold-based strategy is adopted, where data is divided depending on the value of specific predictor variables. In a second approach, a cluster-based ensemble learning is proposed, in such a way that a clustering algorithm is applied to the training dataset, and ML models are then trained for each cluster.

RESULTS

The two proposed methodologies have been evaluated in real data from two hospital ED visits datasets in Spain. We have shown that the proposed approaches are able to obtain accurate ED visits forecasting, in short-term and also long-term prediction time-horizons up to one week, improving the efficiency of alternative prediction methods for this problem.

CONCLUSIONS

The proposed forecasting approaches have a strong emphasis on providing explainability to the problem. An analysis on which variables govern the problem and are pivotal for obtaining accurate predictions is finally carried out and included in the discussion of the paper.

Machine learning models for predicting unscheduled return visits to an emergency department: a scoping review

BACKGROUND

Unscheduled return visits (URVs) to emergency departments (EDs) are used to assess the quality of care in EDs. Machine learning (ML) models can incorporate a wide range of complex predictors to identify high-risk patients and reduce errors to save time and cost. However, the accuracy and practicality of such models are questionable. This review compares the predictive power of multiple ML models and examines the effects of multiple research factors on these models' performance in predicting URVs to EDs.

METHODS

We conducted the present scoping review by searching eight databases for data from 2010 to 2023. The criteria focused on eligible articles that used ML to predict ED return visits. The primary outcome was the predictive performances of the ML models, and results were analyzed on the basis of intervals of return visits, patient population, and research scale.

RESULTS

A total of 582 articles were identified through the database search, with 14 articles selected for detailed analysis. Logistic regression was the most widely used method; however, eXtreme Gradient Boosting generally exhibited superior performance. Variations in visit interval, target group, and research scale did not significantly affect the predictive power of the models.

CONCLUSION

This is the first study to summarize the use of ML for predicting URVs in ED patients. The development of practical ML prediction models for ED URVs is feasible, but improving the accuracy of predicting ED URVs to beyond 0.75 remains a challenge. Including multiple data sources and dimensions is key for enabling ML models to achieve high accuracy; however, such inclusion could be challenging within a limited timeframe. The application of ML models for predicting ED URVs may improve patient safety and reduce medical costs by decreasing the frequency of URVs. Further research is necessary to explore the real-world efficacy of ML models.

A QUEST for Model Assessment: Identifying Difficult Subgroups via Epistemic Uncertainty Quantification

Uncertainty quantification in machine learning can provide powerful insight into a model's capabilities and enhance human trust in opaque models. Well-calibrated uncertainty quantification reveals a connection between high uncertainty and an increased likelihood of an incorrect classification. We hypothesize that if we are able to explain the model's uncertainty by generating rules that define subgroups of data with high and low levels of classification uncertainty, then those same rules will identify subgroups of data on which the model performs well and subgroups on which the model does not perform well. If true, then the utility of uncertainty quantification is not limited to understanding the certainty of individual predictions; it can also be used to provide a more global understanding of the model's understanding of patient subpopulations. We evaluate our proposed technique and hypotheses on deep neural networks and tree-based gradient boosting ensemble across benchmark and real-world medical datasets.

Impact of SARS-CoV-2 Positivity on Delivery Outcomes for Pregnant Women between 2020 and 2021: A Single-Center Population-Based Analysis

Despite the existing body of evidence, there is still limited knowledge about the impact of SARS-CoV-2 positivity on delivery outcomes. We aimed to assess the impact of SARS-CoV-2 infection in women who gave birth at the University Hospital "Federico II" of Naples, Italy, between 2020 and 2021. We conducted a retrospective single-center population-based observational study to assess the differences in the caesarean section and preterm labor rates and the length of stay between women who tested positive for SARS-CoV-2 and those who tested negative at the time of labor. We further stratified the analyses considering the time period, dividing them into three-month intervals, and changes in SARS-CoV-2 as the most prevalent variant. The study included 5236 women with 353 positive cases. After vaccination availability, only 4% had undergone a complete vaccination cycle. The Obstetric Comorbidity Index was higher than 0 in 41% of the sample. When compared with negative women, positive ones had 80% increased odds of caesarean section, and it was confirmed by adjusting for the SARS-CoV-2 variant. No significant differences were found in preterm birth risks. The length of stay was 11% higher in positive cases but was not significant after adjusting for the SARS-CoV-2 variant. When considering only positive women in the seventh study period (July-September 2021), they had a 61% decrease in the odds of receiving a caesarean section compared to the fourth (October-December 2020). Guidelines should be implemented to improve the safety and efficiency of the delivery process, considering the transition of SARS-CoV-2 from pandemic to endemic. Furthermore, these guidelines should aim to improve the management of airborne infections in pregnant women.

Prediction of clinical outcomes after percutaneous coronary intervention: Machine-learning analysis of the National Inpatient Sample

BACKGROUND

This study aimed to develop a multiclass machine-learning (ML) model to predict all-cause mortality, ischemic and hemorrhagic events in unselected hospitalized patients undergoing percutaneous coronary intervention (PCI).

METHODS

This retrospective study included 1,815,595 unselected weighted hospitalizations undergoing PCI from the National Inpatient Sample (2016-2019). Five most common ML algorithms (logistic regression, support vector machine (SVM), naive Bayes, random forest (RF), and extreme gradient boosting (XGBoost)) were trained and tested with 101 input features. The study endpoints were different combinations of all-cause mortality, ischemic cerebrovascular events (CVE) and major bleeding. An area under the curve (AUC) with 95% confidence interval (95% CI) was selected as a performance metric.

RESULTS

The study population was split to a training cohort of 1,186,880 PCI discharges, validation cohort (for calibration) of 296,725 hospitalizations and a test cohort of 331,990 PCI discharges. A total of 98,180 (5.4%) hospital entries included study outcomes. Logistic regression, SVM, naive Bayes, and RF model demonstrated AUCs of 0.83 (95% CI 0.82-0.84), 0.84 (95% CI 0.83-0.86), 0.81 (95% CI 0.80-0.82), and 0.83 (95% CI 0.81-0.84), retrospectively. The XGBoost classifier performed the best with an AUC of 0.86 (95% CI 0.85-0.87) with excellent calibration. We then built a web-based application that provides predictions based on the XGBoost model.

CONCLUSION

We derived the multi-task XGBoost classifier based on 101 features to predict different combinations of all-cause death, ischemic CVE and major bleeding. Such models may be useful in benchmarking and risk prediction using routinely collected administrative data.

Estimating emergency department crowding with stochastic population models

Environments such as shopping malls, airports, or hospital emergency-departments often experience crowding, with many people simultaneously requesting service. Crowding highly fluctuates, with sudden overcrowding "spikes". Past research has either focused on average behavior, used context-specific models with a large number of parameters, or machine-learning models that are hard to interpret. Here we show that a stochastic population model, previously applied to a broad range of natural phenomena, can aptly describe hospital emergency-department crowding. We test the model using data from five-year minute-by-minute emergency-department records. The model provides reliable forecasting of the crowding distribution. Overcrowding is highly sensitive to the patient arrival-flux and length-of-stay: a 10% increase in arrivals triples the probability of overcrowding events. Expediting patient exit-rate to shorten the typical length-of-stay by just 20 minutes (8.5%) cuts the probability of severe overcrowding events by 50%. Such forecasting is critical in prevention and mitigation of breakdown events. Our results demonstrate that despite its high volatility, crowding follows a dynamic behavior common to many systems in nature.

Implementation of Prediction Models in the Emergency Department from an Implementation Science Perspective-Determinants, Outcomes, and Real-World Impact: A Scoping Review

STUDY OBJECTIVE

Prediction models offer a promising form of clinical decision support in the complex and fast-paced environment of the emergency department (ED). Despite significant advancements in model development and validation, implementation of such models in routine clinical practice remains elusive. This scoping review aims to survey the current state of prediction model implementation in the ED and to provide insights on contributing factors and outcomes from an implementation science perspective.

METHODS

We searched 4 databases from their inception to May 20, 2022: MEDLINE (through PubMed), Embase, Scopus, and CINAHL. Articles that reported implementation outcomes and/or contextual determinants under the Reach, Effectiveness, Adoption, Implementation Maintenance (RE-AIM)/Practical, Robust, Implementation, and Sustainability Model (PRISM) framework were included. Characteristics of studies, models, and results of the RE-AIM/PRISM domains were summarized narratively.

RESULTS

Thirty-six reports on 31 implementations were included. The most common prediction models implemented were early warning scores. The most common implementation strategies used were training stakeholders, infrastructural changes, and using evaluative or iterative strategies. Only one report examined ED patients' perspectives, whereas the rest were focused on the experience of health care workers or organizational stakeholders. Key determinants of successful implementation include strong stakeholder engagement, codevelopment of workflows and implementation strategies, education, and usability.

CONCLUSION

Examining ED prediction models from an implementation science perspective can provide valuable insights and help guide future implementations.

Handling Class Imbalance in Machine Learning-based Prediction Models: A Case Study in Asthma Management

A data-driven prediction tool has the potential to provide early warning of an asthma attack and improve asthma management and outcomes. Most previous machine learning (ML)-based studies for asthma attack prediction have reported a severe class imbalance, with major implications for model performance. We aimed to undertake a systematic comparison of several class imbalance handling techniques in the context of risk prediction models for asthma prognosis. We used data from 9,835 asthma patients extracted from the Medical Information Mart for Intensive Care (MIMIC) IV database and deployed five class imbalance handling methods based on synthetic minority oversampling technique (SMOTE) and cost function customisation. We then compared their performances in improving two-class classifier models developed using logistic regression (LR) and extreme gradient boosting (XGBoost) for three different prediction tasks with varying severity of class imbalance (proportion of majority class ranging from 90.86% to 98.98%). The cost function customisation technique substantially outperformed the SMOTE-based methods in all tasks. XGBoost combined with cost function customisation achieved the highest prediction performance for the outcome with the most extreme class imbalance ratio (AUC = 0.72). Our findings suggest that the cost function customisation-based approach to tackle class imbalance provides substantially better performance compared to oversampling in the context of asthma management.Clinical Relevance- This study underscores the challenge of class imbalance in the context of prediction tools to improve asthma management and outcomes and provides a methodological solution that addresses the challenge. Accurate asthma prediction tools can provide early warning and potentially prevent deterioration thereby improving the quality of life of patients with asthma.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),¹ Xie et al. (2022)², Saffari et al. (2022)³ and the online tutorial https://nliulab.github.io/AutoScore/.