Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding

Artificial intelligence-driven forecasting and shift optimization for pediatric emergency department crowding

Objective

This study aimed to develop and evaluate an artificial intelligence (AI)-driven system for forecasting Pediatric Emergency Department (PED) overcrowding and optimizing physician shift schedules using machine learning operations (MLOps).

Materials and Methods

Data from 352 843 PED admissions between January 2018 and May 2023 were analyzed. Twenty time-series forecasting models-including classical methods and advanced deep learning architectures like Temporal Convolutional Network, Time-series Dense Encoder and Reversible Instance Normalization, Neural High-order Time Series model, and Neural Basis Expansion Analysis-were developed and compared using Python 3.8. Starting in January 2023, an MLOps simulation automated data updates and model retraining. Shift schedules were optimized based on forecasted patient volumes using integer linear programming.

Results

Advanced deep learning models outperformed traditional models, achieving initial R2 scores up to 75%. Throughout the simulation, the median R2 score for all models was 44% after MLOps-based model selection, the median R2 improved to 60%. The MLOps architecture facilitated continuous model updates, enhancing forecast accuracy. Shift optimization adjusted staffing in 69 out of 84 shifts, increasing physician allocation by up to 30.4% during peak hours. This adjustment reduced the patient-to-physician ratio by an average of 4.32 patients during the 8-16 shift and 4.40 patients during the 16-24 shift.

Discussion

The integration of advanced deep learning models with MLOps architecture allowed for continuous model updates, enhancing the accuracy of PED overcrowding forecasts and outperforming traditional methods. The AI-driven system demonstrated resilience against data drift caused by events like the COVID-19 pandemic, adapting to changing conditions. Optimizing physician shifts based on these forecasts improved workforce distribution without increasing staff numbers, reducing patient load per physician during peak hours. However, limitations include the single-center design and a fixed staffing model, indicating the need for multicenter validation and implementation in settings with dynamic staffing practices. Future research should focus on expanding datasets through multicenter collaborations and developing forecasting models that provide longer lead times without compromising accuracy.

Conclusions

The AI-driven forecasting and shift optimization system demonstrated the efficacy of integrating AI and MLOps in predicting PED overcrowding and optimizing physician shifts. This approach outperformed traditional methods, highlighting its potential for managing overcrowding in emergency departments. Future research should focus on multicenter validation and real-world implementation to fully leverage the benefits of this innovative system.

Being a patient in a crowded emergency department: a qualitative service evaluation

BACKGROUND

Emergency department (ED) crowding causes increased mortality. Professionals working in crowded departments feel unable to provide high-quality care and are predisposed to burnout. Awareness of the impact on patients, however, is limited to metrics and surveys rather than understanding perspectives. This project investigated patients' experiences and identified mitigating interventions.

METHODS

A qualitative service evaluation was undertaken in a large UK ED. Adults were recruited during periods of high occupancy or delayed transfers. Semi-structured interviews explored experience during these attendances. Participants shared potential mitigating interventions. Analysis was based on the interpretative phenomenological approach. Verbatim transcripts were read, checked for accuracy, re-read and discussed during interviewer debriefing. Reflections about positionality informed the interpretative process.

RESULTS

Seven patients and three accompanying partners participated. They were aged 24-87 with characteristics representing the catchment population. Participants' experiences were characterised by 'loss of autonomy', 'unmet expectations' and 'vulnerability'. Potential mitigating interventions centred around information provision and better identification of existing ED facilities for personal needs.

CONCLUSION

Participants attending a crowded ED experienced uncertainty, helplessness and discomfort. Recommendations included process and environmental orientation.

Measuring the crowding of emergency departments: an assessment of the NEDOCS in Lombardy, Italy, and the development of a new objective indicator based on the waiting time for the first clinical assessment

BACKGROUND

There is no ubiquitous definition of Emergency Department (ED) crowding and several indicators have been proposed to measure it. The National ED Overcrowding Study (NEDOCS) score is among the most popular, even though it has been severely criticised. We used the waiting time for the physician's initial assessment to evaluate the performance of the NEDOCS and proposed a new crowding indicator based on this objective measure.

METHODS

To evaluate the NEDOCS, we used the 2022 data of all the Lombardy EDs and compared the distribution of waiting times across the five levels of the NEDOCS at ED arrival. To construct the new indicator, we estimated the centre-specific relationship between the total number of ED patients and the waiting time of those with minor or deferrable urgency. We defined seven classes of waiting times and calculated how many patients corresponded to an average waiting time in the classes. These centre-specific cutoffs were used to define the 7-level crowding indicator. The indicator was then compared to the NEDOCS score and validated on the first six months of 2023 data.

RESULTS

Patients' waiting time did not increase at the increase of the NEDOCS score, suggesting the absence of a relationship between this score and the effect of ED crowding on the ED capacity of evaluating new patients. The indicator we propose is easy to estimate in real-time and based on centre-specific cutoffs, which depend on the volume of yearly accesses. We observed minimal agreement between the proposed indicator and the NEDOCS in most EDs, both in the development and validation datasets.

CONCLUSIONS

We proposed to quantify ED crowding using the waiting time for physician's initial assessment of patients with minor or deferrable urgency, which increases in crowding situations due to the prioritization of urgent patients. The centre-specific cutoffs avoid the problem of the heterogeneity of the volume of accesses and organization among EDs, while enabling a fair comparison between centres.

Emergency Nurses' Competency in the Emergency Department Context: A Qualitative Study

Background

The availability of clear emergency nurses' competencies is critical for safe and effective emergency health care services. The study regarding emergency nurses' competencies remained virtually limited.

Purpose

This study aimed to explore the emergency nurses' competencies in the clinical emergency department (ED) context as needed by society.

Methods

This qualitative study involved focus group discussions in six groups of 54 participants from three EDs. The data were analysed using grounded theory approach including the constant comparative, interpretations, and coding procedures; initial coding, focused coding and categories.

Results

This study revealed 8 core competencies of emergency nurses: Shifting the nursing practice, Caring for acute critical patients, Communicating and coordinating, Covering disaster nursing roles, Reflecting on the ethical and legal standards, Researching competency, Teaching competencies and Leadership competencies. The interconnection of the 8 core competencies has resulted in 2 concepts of extending the ED nursing practice and demanding the advanced ED nursing role.

Conclusion

The finding reflected the community needs of nurses who work in ED settings and the need for competency development of emergency nurses.

Pediatric ED departmental complexity: a different approach to the concept of ED crowding

OBJECTIVE

Emergency department (ED) crowding is a significant problem in Canada and internationally and is associated with the potential for patient harm. Although pediatric patients represent a significant proportion of overall ED visits, there is limited research on pediatric ED crowding. The Canadian Association of Emergency Physicians defines department crowding as a mismatch between the required and available resources to provide timely emergency care. We propose that rather than crowding, it is better to think of ED patient populations as being more or less "complex" as defined by proxies of the human and physical resources needed for patient management. The study objectives are to explore the utility of a simple and easily available retrospective metric of ED complexity, and to assess the relationship this measure has on patient outcomes in a pediatric ED.

METHODS

Using administrative data from a tertiary care pediatric ED, we developed a departmental complexity score based on patient registration number, triage acuity, and departmental length of stay as a proxy for the resources necessary to provide ED care. We then explored the relationship between this departmental complexity score and clinical care indices.

RESULTS

The score shows a strong relationship with the number of patients who left without being seen by a physician, as well as time to initial MD assessment, both measures which have been used to represent ED crowding in previous research. We found no association between our departmental complexity score and adverse impacts on patient care outcomes of hospital admission, pediatric ICU admission, or patients returning to the ED within 72 h of leaving.

CONCLUSIONS

The departmental complexity score has promise as a retrospective measure of departmental resource requirement and may have a role in the ongoing assessment of patient flow.

Wilhelms D. Differentiating properties of occupancy rate and workload to estimate crowding: A Swedish national cross-sectional study

BACKGROUND

Emergency department (ED) crowding causes increased patient morbidity and mortality. ED occupancy rate (OR; patients by treatment beds) is a common measure of crowding, but the comparability of ORs between EDs is unknown. The objective of this investigation was to investigate differences in ORs between EDs using staff-perceived workload as reference.

METHODS

This was a national cross-sectional study in Sweden. EDs provided data on census, treatment beds, staffing, and workload (1-6) at 5 time points. A baseline patient turnover was calculated as the average daily census by treatment beds, denoted turnover per treatment bed (TTB), for each ED. A census ratio (CR), current by daily census, was calculated to adjust for differences in the number of treatment beds.

RESULTS

Data were returned from 37 (51%) EDs. TTB varied considerably (mean = 4, standard deviation = 1.6; range, 2.1-9.2), and the OR was higher in EDs with TTB >4 compared with ≤4, 0.86 versus 0.43 (0.43; 95% confidence interval [CI], 0.27-0.59), but not workload, 2.75 versus 2.52 (0.23; 95% CI, -0.19 to 0.64). After adjusting for confounders, both TTB (k = -0.3; 95% CI, -0.49 to -0.14) and OR (k = 3.4; 95% CI, 1.76-5.03) affected workload. Correlation with workload was better for CR than for OR (r = 0.75 vs 0.60, respectively).

CONCLUSION

OR is affected by patient-to-treatment bed ratios that differ significantly between EDs and should be accounted for when measuring crowding. CR is not affected by baseline treatment beds and is a better comparable measure of crowding compared with OR in this national comparator study.

Real-Time Measurement of Crowding in Pediatric Emergency Department: Derivation and Validation Using Consensual Perception of Crowding (SOTU-PED)

ABSTRACT

Our study aimed to develop and validate a real-time crowding composite scale for pediatric emergency department (PED). The study took place in one teaching PED for 2 months. The outcome was the perception of crowding evaluated by triage nurses and pediatricians on a 10-level Likert scale. Triage nurses evaluated crowding at each moment of a child's admission and pediatrician at each moment of a child's discharge. The outcome was the hourly mean of all evaluations of crowding (hourly crowding perception). For analysis, originally, we only selected hours during which more than 2 nurses and more than 2 pediatricians evaluated crowding and, moreover, during which evaluations were the most consensual. As predictors, we used hourly means of 10 objective crowding indicators previously selected as consensual in a published French national Delphi study and collected automatically in our software system. The model (SOTU-PED) was developed over a 1-month data set using a backward multivariable linear regression model. Then, we applied the SOTU-PED model on a 1-month validation data set. During the study period, 7341 children were admitted in the PED. The outcome was available for 1352/1392 hours, among which 639 were included in the analysis as "consensual hours." Five indicators were included in the final model, the SOTU-PED (R2 = 0.718). On the validation data set, the correlation between the outcome (perception of crowding) and the SOTU-PED was 0.824. To predict crowded hours (hourly crowding perception >5), the area under the curve was 0.957 (0.933-0.980). The positive and negative likelihood ratios were 8.16 (3.82-17.43) and 0.153 (0.111-0.223), respectively. Using a simple model, it is possible to estimate in real time how crowded a PED is.

Validation of the modified Skåne emergency department assessment of patient load (mSEAL) model for emergency department crowding and comparison with international models; an observational study

Background

Emergency Department crowding is associated with increased morbidity and mortality but no measure of crowding has been validated in Sweden. We have previously derived and internally validated the Skåne Emergency Department Assessment of Patient Load (SEAL) score as a measure of crowding in Emergency Departments (ED) in a large regional healthcare system in Sweden. Due to differences in electronic health records (EHRs) between health care systems in Sweden, all variables in the original SEAL-score could not be measured reliably nationally. We aimed to derive and validate a modified SEAL (mSEAL) model and to compare it with established international measures of crowding.

Methods

This was an observational cross sectional study at four EDs in Sweden. All clinical staff assessed their workload (1–6 where 6 is the highest workload) at 5 timepoints each day. We used linear regression with stepwise backward elimination on the original SEAL dataset to derive and internally validate the mSEAL score against staff workload assessments. We externally validated the mSEAL at four hospitals and compared it with the National Emergency Department Overcrowding Score (NEDOCS), the simplified International Crowding Measure in Emergency Department (sICMED), and Occupancy Rate. Area under the receiver operating curve (AuROC) and coefficient of determination was used to compare crowding models. Crowding was defined as an average workload of 4.5 or higher.

Results

The mSEAL score contains the variables Patient Hours and Time to physician and showed strong correlation with crowding in the derivation (r² = 0.47), internal validation (r² = 0.64 and 0.69) and in the external validation (r² = 0.48 to 0.60). AuROC scores for crowding in the external validation were 0.91, 0.90, 0.97 and 0.80 for mSEAL, Occupancy Rate, NEDOCS and sICMED respectively.

Conclusions

The mSEAL model can measure crowding based on workload in Swedish EDs with good discriminatory capacity and has the potential to systematically evaluate crowding and help policymakers and researchers target its causes and effects. In Swedish EDs, Occupancy Rate and NEDOCS are good alternatives to measure crowding based on workload.

Validation of the National Emergency Department Overcrowding Score (NEDOCS) in a UK non-specialist emergency department

INTRODUCTION

Emergency department (ED) crowding has significant adverse consequences, however, there is no widely accepted tool to measure it. This study validated the National Emergency Department Overcrowding score (NEDOCS) (range 0-200 points), which uses routinely collected ED data.

METHODS

This prospective single-centre study sampled data during four periods of 2018. The outcome against which NEDOCS performance was assessed was a composite of clinician opinion of crowding (physician and nurse in charge). Area under the receiver operating characteristic curves (AUROCs) and calibration plots were produced. Six-hour stratified sampling was added to adjust for temporal correlation of clinician opinion. Staff inter-rater agreement and NEDOCS association with opinion of risk, safety and staffing levels were collected.

RESULTS

From 905 sampled hours, 448 paired observations were obtained, with the ED deemed crowded 18.5% of the time. Inter-rater agreement between staff was moderate (weighted kappa 0.57 (95% CI 0.56 to 0.60)). AUROC for NEDOCS was 0.81 (95% CI 0.77 to 0.86). Adjusted for temporal correlation, AUROC was 0.80 (95% CI 0.73 to 0.88). At a cut-off of 100 points sensitivity was 75.9% (95% CI 65.3% to 84.6%), specificity 72.1% (95% CI 67.1% to 76.6%), positive predictive value 38.2% (95% CI 30.7% to 46.1%) and negative predictive value 92.9% (95% CI 89.3% to 95.6%). NEDOCS underpredicted clinical opinion on Calibration assessment, only partially correcting with intercept updating. For perceived risk of harm, safety and insufficient staffing, NEDOCS AUROCs were 0.71 (95% CI 0.61 to 0.82), 0.71 (95% CI 0.63 to 0.80) and 0.70 (95% CI 0.64 to 0.76), respectively.

CONCLUSIONS

NEDOCS demonstrated good discriminatory power for clinical perception of crowding. Prior to implementation, determining individual unit ED cut-off point(s) would be important as published thresholds may not be generalisable. Future studies could explore refinement of existing variables or addition of new variables, including acute physiological data, which may improve performance.

Prevalence of crowding, boarding and staffing levels in Swedish emergency departments - a National Cross Sectional Study

Background

Emergency Department (ED) crowding occurs when demand for care exceeds the available resources. Crowding has been associated with decreased quality of care and increased mortality, but the prevalence on a national level is unknown in most countries.

Method

We performed a national, cross-sectional study on staffing levels, staff workload, occupancy rate and patients waiting for an in-hospital bed (boarding) at five time points during 24 h in Swedish EDs.

Results

Complete data were collected from 37 (51% of all) EDs in Sweden. High occupancy rate indicated crowding at 12 hospitals (37.5%) at 31 out of 170 (18.2%) time points. Mean workload (measured on a scale from 1, no workload to 6, very high workload) was moderate at 2.65 (±1.25). Boarding was more prevalent in academic EDs than rural EDs (median 3 vs 0). There were an average of 2.6, 4.6 and 3.2 patients per registered nurse, enrolled nurse and physician, respectively.

Conclusion

ED crowding based on occupancy rate was prevalent on a national level in Sweden and comparable with international data. Staff workload, boarding and patient to staff ratios were generally lower than previously described.

The effect of emergency department delays on 30-day mortality in Central Norway

OBJECTIVE

To assess whether prolonged length of stay in the emergency department was associated with risk of death.

METHODS

We analysed data from 165,183 arrivals at St. Olav's University Hospital's emergency department from 2011 to 2018, using an instrumental variable method. As instruments for prolonged length of emergency department stay, we used indicators measured before arrival of the patient. These indicators were used to study the association between prolonged length of emergency department stay and risk of death, being discharged from the emergency department and length of hospitalisation for those who were hospitalised.

RESULTS

Mean length of stay in the emergency department was 2.9 hours, and 30-day risk of death was 3.4%. Per hour prolonged length of stay in the emergency department, the overall change in risk of death was close to zero, with a narrow 95% confidence interval of -0.5 to 0.7 percentage points. Prolonged emergency department stay was associated with a higher probability of being discharged from the emergency department without admission to the hospital. We found no substantial differences in length of hospitalisation for patients who were admitted.

CONCLUSION

In this study, prolonged emergency department stay was not associated with increased risk of death.

Validation of the short form of the International Crowding Measure in Emergency Departments: an international study

OBJECTIVE

There is little consensus on the best way to measure emergency department (ED) crowding. We have previously developed a consensus-based measure, the International Crowding Measure in Emergency Departments. We aimed to externally validate a short form of the International Crowding Measure in Emergency Department (sICMED) against emergency physician's perceptions of crowding and danger.

METHODS

We performed an observational validation study in seven EDs in five different countries. We recorded sICMED observations and the most senior available emergency physician's perceptions of crowding and danger at the same time. We performed a times series regression model.

RESULTS

A total of 397 measurements were analysed. The sICMED showed moderate positive correlations with emergency physician's perceptions of crowding, r = 0.4110, P < 0.05) and safety (r = 0.4566, P < 0.05). There was considerable variation in the performance of the sICMED between different EDs. The sICMED was only slightly better than measuring occupancy or ED boarding time.

CONCLUSION

The sICMED has moderate face validity at predicting clinician's concerns about crowding and safety, but the strength of this validity varies between different EDs and different countries.

Emergency nurses' perceptions of their roles and practices during epidemics: a qualitative study

BACKGROUND

although emergency nurses have a pivotal role in the public health response to epidemics, little is known about their responsibilities and practice in terms of epidemic management.

AIMS

this study aimed to explore how emergency nurses understand and perform their professional roles and practice during epidemics.

METHODS

a qualitative descriptive study design was used. Data were collected from 24 participants through semistructured interviews and subjected to thematic analysis.

FINDINGS

the analysis yielded two overarching themes: expansion in the practice of emergency care; and the altered role of emergency nurses.

CONCLUSION

emergency nurses perceive their practice during the management of an epidemic expanded in that they shouldered a greater responsibility in the control of infectious diseases. This expansion led to role ambiguity among them.

Emergency overcrowding and access block: A smaller problem than we think

OBJECTIVES

Emergency department (ED) access block, the inability to provide timely care for high acuity patients, is the leading safety concern in First World EDs. The main cause of ED access block is hospital access block with prolonged boarding of inpatients in emergency stretchers. Cumulative emergency access gap, the product of the number of arriving high acuity patients and their average delay to reach a care space, is a novel access measure that provides a facility-level estimate of total emergency care delays. Many health leaders believe these delays are too large to be solved without substantial increases in hospital capacity. Our objective was to quantify cumulative emergency access blocks (the problem) as a fraction of inpatient capacity (the potential solution) at a large sample of Canadian hospitals.

METHODS

In this cross-sectional study, we collated 2015 administrative data from 25 Canadian hospitals summarizing patient inflow and delays to ED care space. Cumulative access gap for high acuity patients was calculated by multiplying the number of Canadian Triage Acuity Scale (CTAS) 1-3 patients by their average delay to reach a care space. We compared cumulative ED access gap to available inpatient bed hours to estimate fractional access gap.

RESULTS

Study sites included 16 tertiary and 9 community EDs in 12 cities, representing 1.79 million patient visits. Median ED census (interquartile range) was 66,300 visits per year (58,700-80,600). High acuity patients accounted for 70.7% of visits (60.9%-79.0%). The mean (SD) cumulative ED access gap was 46,000 stretcher hours per site per year (± 19,900), which was 1.14% (± 0.45%) of inpatient capacity.

CONCLUSION

ED access gaps are large and jeopardize care for high acuity patients, but they are small relative to hospital operating capacity. If access block were viewed as a "whole hospital" problem, capacity or efficiency improvements in the range of 1% to 3% could profoundly mitigate emergency care delays.

Developing an emergency department crowding dashboard: A design science approach

BACKGROUND

As an emergency department (ED) is a complex adaptive system, the analysis of continuously gathered data is valuable to gain insight in the real-time patient flow. To support the analysis and management of ED operations, relevant data should be provided in an intuitive way.

AIM

Within this context, this paper outlines the development of a dashboard which provides real-time information regarding ED crowding.

METHODS

The research project underlying this paper follows the principles of design science research, which involves the development and study of artifacts which aim to solve a generic problem. To determine the crowding indicators that are desired in the dashboard, a modified Delphi study is used. The dashboard is implemented using the open source Shinydashboard package in R.

RESULTS

A dashboard is developed containing the desired crowding indicators, together with general patient flow characteristics. It is demonstrated using a dataset of a Flemish ED and fulfills the requirements which are defined a priori.

CONCLUSIONS

The developed dashboard provides real-time information on ED crowding. This information enables ED staff to judge whether corrective actions are required in an effort to avoid the adverse effects of ED crowding.

Skåne Emergency Department Assessment of Patient Load (SEAL)-A Model to Estimate Crowding Based on Workload in Swedish Emergency Departments

Objectives

Emergency department (ED) crowding is an increasing problem in many countries. The purpose of this study was to develop a quantitative model that estimates the degree of crowding based on workload in Swedish EDs.

Methods

At five different EDs, the head nurse and physician assessed the workload on a scale from 1 to 6 at randomized time points during a three week period in 2013. Based on these assessments, a regression model was created using data from the computerized patient log system to estimate the level of crowding based on workload. The final model was prospectively validated at the two EDs with the largest census.

Results

Workload assessments and data on 14 variables in the patient log system were collected at 233 time points. The variables Patient hours, Occupancy, Time waiting for the physician and Fraction of high priority (acuity) patients all correlated significantly with the workload assessments. A regression model based on these four variables correlated well with the assessed workload in the initial dataset (r2 = 0.509, p < 0.001) and with the assessments in both EDs during validation (r2 = 0.641; p < 0.001 and r2 = 0.624; p < 0.001).

Conclusions

It is possible to estimate the level of crowding based on workload in Swedish EDs using data from the patient log system. Our model may be applicable to EDs with different sizes and characteristics, and may be used for continuous monitoring of ED workload. Before widespread use, additional validation of the model is needed.