Assessment of forecasting models for patients arrival at Emergency Department

Enhanced forecasting of emergency department patient arrivals using feature engineering approach and machine learning

BACKGROUND

Emergency department (ED) overcrowding is an important problem in many countries. Accurate predictions of ED patient arrivals can help management to better allocate staff and medical resources. In this study, we investigate the use of calendar and meteorological predictors, as well as feature-engineered variables, to predict daily patient arrivals using datasets from eleven different EDs across three countries.

METHODS

Six machine learning (ML) algorithms were tested on forecasting horizons of 7 and 45 days. Three of them - Light Gradient Boosting Machine (LightGBM), Support Vector Machine with Radial Basis Function (SVM-RBF), and Neural Network Autoregression (NNAR) - were never before reported for predicting ED patient arrivals. Algorithms' hyperparameters were tuned through a grid-search with cross-validation. Prediction performance was assessed using fivefold cross-validation and four performance metrics.

RESULTS

The eXtreme Gradient Boosting (XGBoost) was the best-performing model on both prediction horizons, also outperforming results reported in past studies on ED arrival prediction. XGBoost and NNAR achieved the best performance in nine out of the eleven analyzed datasets, with MAPE values ranging from 5.03% to 14.1%. Feature engineering (FE) improved the performance of the ML algorithms.

CONCLUSION

Accuracy in predicting ED arrivals, achieved through the FE approach, is key for managing human and material resources, as well as reducing patient waiting times and lengths of stay.

Combining machine learning and optimization for the operational patient-bed assignment problem

Assigning inpatients to hospital beds impacts patient satisfaction and the workload of nurses and doctors. The assignment is subject to unknown inpatient arrivals, in particular for emergency patients. Hospitals, therefore, need to deal with uncertainty on actual bed requirements and potential shortage situations as bed capacities are limited. This paper develops a model and solution approach for solving the patient bed-assignment problem that is based on a machine learning (ML) approach to forecasting emergency patients. First, it contributes by improving the anticipation of emergency patients using ML approaches, incorporating weather data, time and dates, important local and regional events, as well as current and historical occupancy levels. Drawing on real-life data from a large case hospital, we were able to improve forecasting accuracy for emergency inpatient arrivals. We achieved up to 17% better root mean square error (RMSE) when using ML methods compared to a baseline approach relying on averages for historical arrival rates. We further show that the ML methods outperform time series forecasts. Second, we develop a new hyper-heuristic for solving real-life problem instances based on the pilot method and a specialized greedy look-ahead (GLA) heuristic. When applying the hyper-heuristic in test sets we were able to increase the objective function by up to 5.3% in comparison to the benchmark approach in [40]. A benchmark with a Genetic Algorithm shows also the superiority of the hyper-heuristic. Third, the combination of ML for emergency patient admission forecasting with advanced optimization through the hyper-heuristic allowed us to obtain an improvement of up to 3.3% on a real-life problem.

Predicting hospital emergency department visits accurately: A systematic review

OBJECTIVES

The emergency department (ED) is a very important healthcare entrance point, known for its challenging organisation and management due to demand unpredictability. An accurate forecast system of ED visits is crucial to the implementation of better management strategies that optimise resources utilization, reduce costs and improve public confidence. The aim of this review is to investigate the different factors that affect the ED visits forecasting outcomes, in particular the predictive variables and type of models applied.

METHODS

A systematic search was conducted in PubMed, Web of Science and Scopus. The review methodology followed the PRISMA statement guidelines.

RESULTS

Seven studies were selected, all exploring predictive models to forecast ED daily visits for general care. MAPE and RMAE were used to measure models' accuracy. All models displayed good accuracy, with errors below 10%.

CONCLUSIONS

Model selection and accuracy was found to be particularly sensitive to the ED dimension. While ARIMA-based and other linear models have good performance for short-time forecast, some machine learning methods proved to be more stable when forecasting multiple horizons. The inclusion of exogenous variables was found to be advantageous only in bigger EDs.

Forecasting patient flows with pandemic induced concept drift using explainable machine learning

Accurately forecasting patient arrivals at Urgent Care Clinics (UCCs) and Emergency Departments (EDs) is important for effective resourcing and patient care. However, correctly estimating patient flows is not straightforward since it depends on many drivers. The predictability of patient arrivals has recently been further complicated by the COVID-19 pandemic conditions and the resulting lockdowns. This study investigates how a suite of novel quasi-real-time variables like Google search terms, pedestrian traffic, the prevailing incidence levels of influenza, as well as the COVID-19 Alert Level indicators can both generally improve the forecasting models of patient flows and effectively adapt the models to the unfolding disruptions of pandemic conditions. This research also uniquely contributes to the body of work in this domain by employing tools from the eXplainable AI field to investigate more deeply the internal mechanics of the models than has previously been done. The Voting ensemble-based method combining machine learning and statistical techniques was the most reliable in our experiments. Our study showed that the prevailing COVID-19 Alert Level feature together with Google search terms and pedestrian traffic were effective at producing generalisable forecasts. The implications of this study are that proxy variables can effectively augment standard autoregressive features to ensure accurate forecasting of patient flows. The experiments showed that the proposed features are potentially effective model inputs for preserving forecast accuracies in the event of future pandemic outbreaks.

Accurate Forecasting of Emergency Department Arrivals With Internet Search Index and Machine Learning Models: Model Development and Performance Evaluation

BACKGROUND

Emergency department (ED) overcrowding is a concerning global health care issue, which is mainly caused by the uncertainty of patient arrivals, especially during the pandemic. Accurate forecasting of patient arrivals can allow health resource allocation in advance to reduce overcrowding. Currently, traditional data, such as historical patient visits, weather, holiday, and calendar, are primarily used to create forecasting models. However, data from an internet search engine (eg, Google) is less studied, although they can provide pivotal real-time surveillance information. The internet data can be employed to improve forecasting performance and provide early warning, especially during the epidemic. Moreover, possible nonlinearities between patient arrivals and these variables are often ignored.

OBJECTIVE

This study aims to develop an intelligent forecasting system with machine learning models and internet search index to provide an accurate prediction of ED patient arrivals, to verify the effectiveness of the internet search index, and to explore whether nonlinear models can improve the forecasting accuracy.

METHODS

Data on ED patient arrivals were collected from July 12, 2009, to June 27, 2010, the period of the 2009 H1N1 pandemic. These included 139,910 ED visits in our collaborative hospital, which is one of the biggest public hospitals in Hong Kong. Traditional data were also collected during the same period. The internet search index was generated from 268 search queries on Google to comprehensively capture the information about potential patients. The relationship between the index and patient arrivals was verified by Pearson correlation coefficient, Johansen cointegration, and Granger causality. Linear and nonlinear models were then developed with the internet search index to predict patient arrivals. The accuracy and robustness were also examined.

RESULTS

All models could accurately predict patient arrivals. The causality test indicated internet search index as a strong predictor of ED patient arrivals. With the internet search index, the mean absolute percentage error (MAPE) and the root mean square error (RMSE) of the linear model reduced from 5.3% to 5.0% and from 24.44 to 23.18, respectively, whereas the MAPE and RMSE of the nonlinear model decreased even more, from 3.5% to 3% and from 16.72 to 14.55, respectively. Compared with each other, the experimental results revealed that the forecasting system with extreme learning machine, as well as the internet search index, had the best performance in both forecasting accuracy and robustness analysis.

CONCLUSIONS

The proposed forecasting system can make accurate, real-time prediction of ED patient arrivals. Compared with the static traditional variables, the internet search index significantly improves forecasting as a reliable predictor monitoring continuous behavior trend and sudden changes during the epidemic (P=.002). The nonlinear model performs better than the linear counterparts by capturing the dynamic relationship between the index and patient arrivals. Thus, the system can facilitate staff planning and workflow monitoring.

Forecasting patient arrivals at emergency department using calendar and meteorological information

Overcrowding in emergency departments (EDs) is a serious problem in many countries. Accurate ED patient arrival forecasts can serve as a management baseline to better allocate ED personnel and medical resources. We combined calendar and meteorological information and used ten modern machine learning methods to forecast patient arrivals. For daily patient arrival forecasting, two feature selection methods are proposed. One uses kernel principal component analysis(KPCA) to reduce the dimensionality of all of the features, and the other is to use the maximal information coefficient(MIC) method to select the features related to the daily data first and then perform KPCA dimensionality reduction. The current study focuses on a public hospital ED in Hefei, China. We used the data November 1, 2019 to August 31, 2020 for model training; and patient arrival data September 1, 2020 to November 31, 2020 for model validation. The results show that for hourly patient arrival forecasting, each machine learning model has better forecasting results than the traditional autoRegressive integrated moving average (ARIMA) model, especially long short-term memory (LSTM) model. For daily patient arrival forecasting, the feature selection method based on MIC-KPCA has a better forecasting effect, and the simpler models are better than the ensemble models. The method we proposed could be used for better planning of ED personnel resources.

Forecasting emergency department admissions

The emergency department of a hospital plays an extremely important role in the healthcare of patients. To maintain a high quality service, clinical professionals need information on how patient flow will evolve in the immediate future. With accurate emergency department forecasts it is possible to better manage available human resources by allocating clinical staff before peak periods, thus preventing service congestion, or releasing clinical staff at less busy times. This paper describes a solution developed for the presentation of hourly, four-hour, eight-hour and daily number of admissions to a hospital’s emergency department. A 10-year history (2009–2018) of the number of emergency admissions in a Portuguese hospital was used. To create the models several methods were tested, including exponential smoothing, SARIMA, autoregressive and recurrent neural network, XGBoost and ensemble learning. The models that generated the most accurate hourly time predictions were the recurrent neural network with one-layer (sMAPE = 23.26%) and with three layers (sMAPE = 23.12%) and XGBoost (sMAPE = 23.70%). In terms of efficiency, the XGBoost method has by far outperformed all others. The success of the recurrent neuronal network and XGBoost machine learning methods applied to the prediction of the number of emergency department admissions has been demonstrated here, with an accuracy that surpasses the models found in the literature.

Monitoring patient flow in a hospital emergency department: ARMA-based nonparametric GLRT scheme

Overcrowding in emergency departments (EDs) is a primary concern for hospital administration. They aim to efficiently manage patient demands and reducing stress in the ED. Detection of abnormal ED demands (patient flows) in hospital systems aids ED managers to obtain appropriate decisions by optimally allocating the available resources following patient attendance. This paper presents a monitoring strategy that provides an early alert in an ED when an abnormally high patient influx occurs. Anomaly detection using this strategy involves the amalgamation of autoregressive-moving-average (ARMA) time series models with the generalized likelihood ratio (GLR) chart. A nonparametric procedure based on kernel density estimation is employed to determine the detection threshold of the ARMA-GLR chart. The developed ARMA-based GLR has been validated through practical data from the ED at Lille Hospital, France. Then, the ARMA-based GLR method's performance was compared to that of other commonly used charts, including a Shewhart chart and an exponentially weighted moving average chart; it proved more accurate.

Performance evaluation of Emergency Department patient arrivals forecasting models by including meteorological and calendar information: A comparative study

The volume of daily patient arrivals at Emergency Departments (EDs) is unpredictable and is a significant reason of ED crowding in hospitals worldwide. Timely forecast of patients arriving at ED can help the hospital management in early planning and avoiding of overcrowding. Many different ED patient arrivals forecasting models have previously been proposed by using time series analysis methods. Even though the time series methods such as Linear and Logistic Regression, Autoregressive Integrated Moving Average (ARIMA), Seasonal ARIMA (SARIMA), Exponential Smoothing (ES), and Artificial Neural Network (ANN) have been explored extensively for the ED forecasting model development, the few significant limitations of these methods associated in the analysis of time series data make the models inadequate in many practical situations. Therefore, in this paper, Machine Learning (ML)-based Random Forest (RF) regressor, and Deep Neural Network (DNN)-based Long Short-Term Memory (LSTM) and Convolutional Neural network (CNN) methods, which have not been explored to the same extent as the other time series techniques, are implemented by incorporating meteorological and calendar parameters for the development of forecasting models. The performances of the developed three models in forecasting ED patient arrivals are evaluated. Among the three models, CNN outperformed for short-term (3 days in advance) patient arrivals prediction with Mean Absolute Percentage Error (MAPE) of 9.24% and LSTM performed better for moderate-term (7 days in advance) patient arrivals prediction with MAPE of 8.91% using weather forecast information. Whereas, LSTM model outperformed with MAPE of 8.04% compared to 9.53% by CNN and 10.10% by RF model for current day prediction of patient arrivals using 3 days past weather information. Thus, for short-term ED patient arrivals forecasting, DNN-based model performed better compared to RF regressor ML-based model.

Modeling patient-related workload in the emergency department using electronic health record data

INTRODUCTION

Understanding and managing clinician workload is important for clinician (nurses, physicians and advanced practice providers) occupational health as well as patient safety. Efforts have been made to develop strategies for managing clinician workload by improving patient assignment. The goal of the current study is to use electronic health record (EHR) data to predict the amount of work that individual patients contribute to clinician workload (patient-related workload).

METHODS

One month of EHR data was retrieved from an emergency department (ED). A list of workload indicators and five potential workload proxies were extracted from the data. Linear regression and four machine learning classification algorithms were utilized to model the relationship between the indicators and the proxies.

RESULTS

Linear regression proved that the indicators explained a substantial amount of variance of the proxies (four out of five proxies were modeled with R² > 0.80). Classification algorithms also showed success in classifying a patient as having high or low task demand based on data from early in the ED visit (e.g. 80 % accurate binary classification with data from the first hour).

CONCLUSION

The main contribution of this study is demonstrating the potential of using EHR data to predict patient-related workload automatically in the ED. The predicted workload can potentially help in managing clinician workload by supporting decisions around the assignment of new patients to providers. Future work should focus on identifying the relationship between workload proxies and actual workload, as well as improving prediction performance of regression and multi-class classification.

Effects of weather, air pollution and Oktoberfest on ambulance-transported emergency department admissions in Munich, Germany

BACKGROUND

Climate change and increasing risks of extreme weather events affect human health and lead to changes in the emergency department (ED) admissions and the emergency medical services (EMS) operations. For a better allocation of resources in the healthcare system, it is essential to predict ED numbers based on environmental variables. This publication aims to quantify weather, air pollution and calendar-related effects on daily ED admissions.

METHODS

Analyses were based on 575,725 admissions from the web-based IVENA system recording all patients in the greater Munich area with pre-hospital emergency care in ambulance operations during 2014-2018. Linear models were used to identify statistically significant associations between daily ED admissions and calendar, meteorological and pollution factors, allowing for lag effects of one to three days. Separate analyses were performed for seasons, with additional subset analyses by sex, age and surgical versus internal department.

RESULTS

ED admissions were exceptionally high during the three-week Oktoberfest, particularly for males and on the weekends, as well as during the New Year holiday. Admissions significantly increased during the years of study, decreased in spring and summer holidays, and were lower on Sundays while higher on Mondays. In the warmer seasons, admissions were significantly associated with higher temperature, adjusting for the effects of sunshine and humidity in all age groups except for the elderly. Adverse weather conditions in non-summer seasons were either linked to increasing ED admissions (from storms, gust) or decreasing them from rain. Mostly, but not exclusively, in winter, increasing ED admissions were associated with colder minimum temperatures as well as with higher NO and PM₁₀ concentrations.

CONCLUSIONS

In addition to standard calendar-related factors, incorporating seasonal weather, air pollutant and interactions with patient demographics into resource planning models can improve the daily allocation of resources and staff of EMS operations at hospital and city levels.

Forecasting emergency department overcrowding: A deep learning framework

As the demand for medical cares has considerably expanded, the issue of managing patient flow in hospitals and especially in emergency departments (EDs) is certainly a key issue to be carefully mitigated. This can lead to overcrowding and the degradation of the quality of the provided medical services. Thus, the accurate modeling and forecasting of ED visits are critical for efficiently managing the overcrowding problems and enable appropriate optimization of the available resources. This paper proposed an effective method to forecast daily and hourly visits at an ED using Variational AutoEncoder (VAE) algorithm. Indeed, the VAE model as a deep learning-based model has gained special attention in features extraction and modeling due to its distribution-free assumptions and superior nonlinear approximation. Two types of forecasting were conducted: one- and multi-step-ahead forecasting. To the best of our knowledge, this is the first time that the VAE is investigated to improve forecasting of patient arrivals time-series data. Data sets from the pediatric emergency department at Lille regional hospital center, France, are employed to evaluate the forecasting performance of the introduced method. The VAE model was evaluated and compared with seven methods namely Recurrent Neural Network (RNN), Long short-term memory (LSTM), Bidirectional LSTM (BiLSTM), Convolutional LSTM Network (ConvLSTM), restricted Boltzmann machine (RBM), Gated recurrent units (GRUs), and convolutional neural network (CNN). The results clearly show the promising performance of these deep learning models in forecasting ED visits and emphasize the better performance of the VAE in comparison to the other models.

Forecasting emergency admissions due to respiratory diseases in high variability scenarios using time series: A case study in Chile

Respiratory diseases are ranked in the top ten group of the most frequent illness in the globe. Emergency admissions are proof of this issue, especially in the winter season. For this study, the city of Santiago de Chile was chosen because of the high variability of the time series for admissions, the quality of data collected in the governmental repository DEIS (selected period: 2014-2018), and the poor ventilation conditions of the city, which in winter contributes to increase the pollution level, and therefore, respiratory emergency admissions. Different forecasting models were reviewed using the Akaike Information Criteria (AIC) with other error estimators, such as the Root Mean Square Error (RMSE), for selecting the best approach. At the end, Seasonal Autoregressive Integrated Moving Average (SARIMA) model, with parameters (p,d,q)(P,D,Q)_s=(2,1,3)(3,0,2)₇, was selected. The Mean Average Percentage Error (MAPE) for this model was 7.81%. After selection, an investigation of its performance was made using a cross-validation through a rolling window analysis, forecasting up to 30 days ahead (testing period of one year). The results showed that error do not exceed a MAPE of 20%. This allows taking better resource managing decisions in real scenarios: reactive staff hiring is avoided given the reduction of uncertainty for the medium term forecast, which translates into lower costs. Finally, a methodology for the selection of forecasting models is proposed, which includes other constraints from resource management, as well as the different impacts for social well-being.

Petri Nets Validation of Markovian Models of Emergency Department Arrivals

Modeling of hospital’s Emergency Departments (ED) is vital for optimisation of health services offered to patients that shows up at an ED requiring treatments with different level of emergency. In this paper we present a modeling study whose contribution is twofold: first, based on a dataset relative to the ED of an Italian hospital, we derive different kinds of Markovian models capable to reproduce, at different extents, the statistical character of dataset arrivals; second, we validate the derived arrivals model by interfacing it with a Petri net model of the services an ED patient undergoes. The empirical assessment of a few key performance indicators allowed us to validate some of the derived arrival process model, thus confirming that they can be used for predicting the performance of an ED.

Emergency Department Capacity Planning: A Recurrent Neural Network and Simulation Approach

Emergency departments (EDs) play a vital role in the whole healthcare system as they are the first point of care in hospitals for urgent and critically ill patients. Therefore, effective management of hospital's ED is crucial in improving the quality of the healthcare service. The effectiveness depends on how efficiently the hospital resources are used, particularly under budget constraints. Simulation modeling is one of the best methods to optimize resources and needs inputs such as patients' arrival time, patient's length of stay (LOS), and the route of patients in the ED. This study develops a simulation model to determine the optimum number of beds in an ED by minimizing the patients' LOS. The hospital data are analyzed, and patients' LOS and the route of patients in the ED are determined. To determine patients' arrival times, the features associated with patients' arrivals at ED are identified. Mean arrival rate is used as a feature in addition to climatic and temporal variables. The exhaustive feature-selection method has been used to determine the best subset of the features, and the mean arrival rate is determined as one of the most significant features. This study is executed using the one-year ED arrival data together with five-year (43.824 study hours) ED arrival data to improve the accuracy of predictions. Furthermore, ten different machine learning (ML) algorithms are used utilizing the same best subset of these features. After a tenfold cross-validation experiment, based on mean absolute percentage error (MAPE), the stateful long short-term memory (LSTM) model performed better than other models with an accuracy of 47%, followed by the decision tree and random forest methods. Using the simulation method, the LOS has been minimized by 7% and the number of beds at the ED has been optimized.

Patient classification based on volume and case-mix in the emergency department and their association with performance

Predicting daily patient volume is necessary for emergency department (ED) strategic and operational decisions, such as resource planning and workforce scheduling. For these purposes, forecast accuracy requires understanding the heterogeneity among patients with respect to their characteristics and reasons for visits. To capture the heterogeneity among ED patients (case-mix), we present a patient coding and classification scheme (PCCS) based on patient demographics and diagnostic information. The proposed PCCS allows us to mathematically formalize the arrival patterns of the patient population as well as each class of patients. We can then examine the volume and case-mix of patients presenting to an ED and investigate their relationship to the ED's quality and time-based performance metrics. We use data from five hospitals in February, July and November for the years of 2007, 2012, and 2017 in the city of Calgary, Alberta, Canada. We find meaningful arrival time patterns of the patient population as well as classes of patients in EDs. The regression results suggest that patient volume is the main predictor of time-based ED performance measures. Case-mix is, however, the key predictor of quality of care in EDs. We conclude that considering both patient volume and the mix of patients are necessary for more accurate strategic and operational planning in EDs.

Real-time forecasting of emergency department arrivals using prehospital data

Background

Crowding in emergency departments (EDs) is a challenge globally. To counteract crowding in day-to-day operations, better tools to improve monitoring of the patient flow in the ED is needed. The objective of this study was the development of a continuously updated monitoring system to forecast emergency department (ED) arrivals on a short time-horizon incorporating data from prehospital services.

Methods

Time of notification and ED arrival was obtained for all 191,939 arrivals at the ED of a Norwegian university hospital from 2010 to 2018. An arrival notification was an automatically captured time stamp which indicated the first time the ED was notified of an arriving patient, typically by a call from an ambulance to the emergency service communication center. A Poisson time-series regression model for forecasting the number of arrivals on a 1-, 2- and 3-h horizon with continuous weekly and yearly cyclic effects was implemented. We incorporated time of arrival notification by modelling time to arrival as a time varying hazard function. We validated the model on the last full year of data.

Results

In our data, 20% of the arrivals had been notified more than 1 hour prior to arrival. By incorporating time of notification into the forecasting model, we saw a substantial improvement in forecasting accuracy, especially on a one-hour horizon. In terms of mean absolute prediction error, we observed around a six percentage-point decrease compared to a simplified prediction model. The increase in accuracy was particularly large for periods with large inflow.

Conclusions

The proposed model shows increased predictability in ED patient inflow when incorporating data on patient notifications. This approach to forecasting arrivals can be a valuable tool for logistic, decision making and ED resource management.

Electronic supplementary material

The online version of this article (10.1186/s12873-019-0256-z) contains supplementary material, which is available to authorized users.