Validity of ED: Addressing heterogeneous definitions of over-triage and under-triage

Human intelligence versus Chat-GPT: who performs better in correctly classifying patients in triage?

INTRODUCTION

Chat-GPT is rapidly emerging as a promising and potentially revolutionary tool in medicine. One of its possible applications is the stratification of patients according to the severity of clinical conditions and prognosis during the triage evaluation in the emergency department (ED).

METHODS

Using a randomly selected sample of 30 vignettes recreated from real clinical cases, we compared the concordance in risk stratification of ED patients between healthcare personnel and Chat-GPT. The concordance was assessed with Cohen's kappa, and the performance was evaluated with the area under the receiver operating characteristic curve (AUROC) curves. Among the outcomes, we considered mortality within 72 h, the need for hospitalization, and the presence of a severe or time-dependent condition.

RESULTS

The concordance in triage code assignment between triage nurses and Chat-GPT was 0.278 (unweighted Cohen's kappa; 95% confidence intervals: 0.231-0.388). For all outcomes, the ROC values were higher for the triage nurses. The most relevant difference was found in 72-h mortality, where triage nurses showed an AUROC of 0.910 (0.757-1.000) compared to only 0.669 (0.153-1.000) for Chat-GPT.

CONCLUSIONS

The current level of Chat-GPT reliability is insufficient to make it a valid substitute for the expertise of triage nurses in prioritizing ED patients. Further developments are required to enhance the safety and effectiveness of AI for risk stratification of ED patients.

Delays in treatment and disposition attributable to undertriage of pediatric emergency medicine patients

BACKGROUND

Triage, the initial assessment and sorting of patients in the Emergency Department (ED), determines priority of evaluation and treatment. Little is known about the impact of undertriage, the underestimation of disease severity at triage, on clinical care in pediatric ED patients. We evaluate the impact of undertriage on time to disposition and treatment decisions in pediatric ED patients.

METHODS

This was a case control study of ED visits for patients <22 years of age, with an assigned Emergency Severity Index (ESI) score of 4 or 5, and associated hospital admission, nebulized treatment, supplemental oxygen, and/or intravenous (IV) line placement, between January 1, 2018, to June 30, 2022. Controls were sampled from a pool of patient visits with an ESI score of 3, matched by intervention, disposition, and date and hour of arrival. Primary outcome measures were time to order of intervention (nebulized treatment, oxygen administration, or IV placement) and time to disposition decision. A secondary outcome measure was return visits requiring admission or emergency intervention within 14 days of the index visit. Continuous variables (time to orders) were analyzed using Wilcoxon rank sum test and dichotomous outcomes (return visits) were compared using odds ratios with 95% confidence intervals. Analysis was performed with Python v3.10.

RESULTS

The final analysis included 7245 undertriaged patients. Undertriaged patients had longer times to orders for nebulized treatments, (p < 0.001) IV placement, (p < 0.001) and admission (p < 0.001) when compared to controls. There were no significant differences in time to supplemental oxygen delivery and time to discharge compared to controls. Undertriaged patients were more likely to experience a return visit requiring admission or emergency intervention (OR 3.74, 95% CI 3.32,4.22).

CONCLUSIONS

Undertriage in the pediatric ED is associated with delays in care and disposition decisions and increases likelihood of return visits.

Clinical support system for triage based on federated learning for the Korea triage and acuity scale

Background and aims

This study developed a clinical support system based on federated learning to predict the need for a revised Korea Triage Acuity Scale (KTAS) to facilitate triage.

Methods

This was a retrospective study that used data from 11,952,887 patients in the Korean National Emergency Department Information System (NEDIS) from 2016 to 2018 for model development. Separate cohorts were created based on the emergency medical center level in the NEDIS: regional emergency medical center (REMC), local emergency medical center (LEMC), and local emergency medical institution (LEMI). External and temporal validation used data from emergency department (ED) of the study site from 2019 to 2021. Patient features obtained during the triage process and the initial KTAS scores were used to develop the prediction model. Federated learning was used to rectify the disparity in data quality between EDs. The patient's demographic information, vital signs in triage, mental status, arrival information, and initial KTAS were included in the input feature.

Results

3,626,154 patients' visits were included in the regional emergency medical center cohort; 8,278,081 patients' visits were included in the local emergency medical center cohort; and 48,652 patients' visits were included in the local emergency medical institution cohort. The study site cohort, which is used for external and temporal validation, included 135,780 patients visits. Among the patients in the REMC and study site cohorts, KTAS level 3 patients accounted for the highest proportion at 42.4% and 45.1%, respectively, whereas in the LEMC and LEMI cohorts, KTAS level 4 patients accounted for the highest proportion. The area under the receiver operating characteristic curve for the prediction model was 0.786, 0.750, and 0.770 in the external and temporal validation. Patients with revised KTAS scores had a higher admission rate and ED mortality rate than those with unaltered KTAS scores.

Conclusions

This novel system might accurately predict the likelihood of KTAS acuity revision and support clinician-based triage.

Validation of a modified South African triage scale in a high-resource setting: a retrospective cohort study

BACKGROUND

Triage systems are widely used in emergency departments, but are not always validated. The South African Triage Scale (SATS) has mainly been studied in resource-limited settings. The aim of this study was to determine the validity of a modified version of the SATS for the general population of patients admitted to an ED at a tertiary hospital in a high-income country. The secondary objective was to study the triage performance according to age and patient categories.

METHODS

We conducted a retrospective cohort study of patients presenting to the Emergency Department of Haukeland University Hospital in Norway during a four-year period. We used short-term mortality, ICU admission, and the need for immediate surgery and other interventions as the primary endpoints.

RESULTS

A total of 162,034 emergency department visits were included in the analysis. The negative predictive value of a low triage level to exclude severe illness was 99.1% (95% confidence interval: 99.0-99.2%). The level of overtriage, defined as the proportion of patients assigned to a high triage level who were not admitted to the hospital, was 4.1% (3.9-4.2%). Receiver operating characteristic (ROC) curves showed an area under the ROC for the detection of severe illness of 0.874 (95% confidence interval: 0.870-0.879) for all patients and 0.856 (0.837-0.875), 0.884 (0.878-0.890) and 0.869 (0.862-0.876) for children, adults and elderly individuals respectively.

CONCLUSION

We found that the modified SATS had a good sensitivity to identify short-term mortality, ICU admission, and the need for rapid surgery and other interventions. The sensitivity was higher in adults than in children and higher in medical patients than in surgical patients. The over- and undertriage rates were acceptable.

Evaluation of the Emergency Severity Index in US Emergency Departments for the Rate of Mistriage

Importance

Accurate emergency department (ED) triage is essential to prioritize the most critically ill patients and distribute resources appropriately. The most used triage system in the US is the Emergency Severity Index (ESI).

Objectives

To derive and validate an algorithm to assess the rate of mistriage and to identify characteristics associated with mistriage.

Design, Setting, and Participants

This retrospective cohort study created operational definitions for each ESI level that use ED visit electronic health record data to classify encounters as undertriaged, overtriaged, or correctly triaged. These definitions were applied to a retrospective cohort to assess variation in triage accuracy by facility and patient characteristics in 21 EDs within the Kaiser Permanente Northern California (KPNC) health care system. All ED encounters by patients 18 years and older between January 1, 2016, and December 31, 2020, were assessed for eligibility. Encounters with missing ESI or incomplete ED time variables and patients who left against medical advice or without being seen were excluded. Data were analyzed between January 1, 2021, and November 30, 2022.

Exposures

Assigned ESI level.

Main Outcomes and Measures

Rate of undertriage and overtriage by assigned ESI level based on a mistriage algorithm and patient and visit characteristics associated with undertriage and overtriage.

Results

A total of 5 315 176 ED encounters were included. The mean (SD) patient age was 52 (21) years; 44.3% of patients were men and 55.7% were women. In terms of race and ethnicity, 11.1% of participants were Asian, 15.1% were Black, 21.4% were Hispanic, 44.0% were non-Hispanic White, and 8.5% were of other (includes American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, and multiple races or ethnicities), unknown, or missing race or ethnicity. Mistriage occurred in 1 713 260 encounters (32.2%), of which 176 131 (3.3%) were undertriaged and 1 537 129 (28.9%) were overtriaged. The sensitivity of ESI to identify a patient with high-acuity illness (correctly assigning ESI I or II among patients who had a life-stabilizing intervention) was 65.9%. In adjusted analyses, Black patients had a 4.6% (95% CI, 4.3%-4.9%) greater relative risk of overtriage and an 18.5% (95% CI, 16.9%-20.0%) greater relative risk of undertriage compared with White patients, while Black male patients had a 9.9% (95% CI, 9.8%-10.0%) greater relative risk of overtriage and a 41.0% (95% CI, 40.0%-41.9%) greater relative risk of undertriage compared with White female patients. High relative risk of undertriage was found among patients taking high-risk medications (30.3% [95% CI, 28.3%-32.4%]) and those with a greater comorbidity burden (22.4% [95% CI, 20.1%-24.4%]) and recent intensive care unit utilization (36.7% [95% CI, 30.5%-41.4%]).

Conclusions and Relevance

In this retrospective cohort study of over 5 million ED encounters, mistriage with ESI was common. Quality improvement should focus on limiting critical undertriage, optimizing resource allocation by patient need, and promoting equity.

Daily triage audit can improve nurses' triage stratification: A pre-post study

AIMS

The objective was to evaluate whether the error rate in the application of the triage system decreased after the introduction of daily auditing, and it was also evaluated if the agreement rate between physician and nurse on triage priority levels increased after the introduction of daily auditing and if the error-related variables in the pre-intervention period changed in the post-intervention period.

DESIGN

A quasi-experimental study was performed with a pre-post design, between June 2019 and June 2021 in one emergency department.

METHODS

The accuracy and error rate of triage in the pre- and post-intervention period were compared. Univariate and multivariate logistic regression analyses were performed to explore the relationships between the variables related to the error. The comparison between the priority level assigned by the physician and the triage nurse was analysed using Cohen's K.

RESULTS

Nine hundred four patients were enrolled in the pre-intervention period and 869 in the post-intervention period. The error rate in the pre-intervention period was 23.3% and in the post-intervention period was 9.7%. The concordance between the degree of priority expressed by the physician and the nurse varied from a quadratically weighted Cohen's K of 0.447 in the pre-intervention period to 0.881 in the post-intervention period.

CONCLUSION

Daily auditing is a clinical procedure that improves the nurse's application of the triage system. Daily auditing has reduced errors by the nurse, improving performance and concordance with the physician.

IMPACT

Triage systems are a key point for the stratification of the priority level of patients and it is therefore evident that they maintain high-quality standards. Through the practice of daily auditing, not only a reduction in the error rate, which ensures patient safety, but also an improvement in triage performance has been demonstrated.

NO PATIENT OR PUBLIC CONTRIBUTION

The study did not involve any patients during its conduction.

Accuracy of nurses’ performance in triage using the emergency severity index and its relationship with clinical outcome measures

Emergency department (ED) triage in hospitals is part of their emergency structure carried out by nurses in emergency units. There have not been many studies available on nurses' performance in triage based on the emergency severity index (ESI). This present study aimed to investigate the nurses’ performance in triage with regard to the emergency severity index and its relation to clinical outcome measures in the emergency department. This was a cross-sectional study. The hospitalization record of 600 patients who arrived at the emergency department of Sanandaj Social Security Hospital was randomly assessed based on the accuracy of triage performed by nurses. The data analysis procedure was done by employing STATA software version 12, as well as Fisher's exact test, independent t-test, and one-way ANOVA. Findings of the study revealed that nurses’ overall performance in triage showed that 82.67% of nurses had perfect triage accuracy, 12.17% had low-level triage accuracy and 5.17% had high-level triage accuracy. There was an association between nurses' performance in triage with the disposition of patients (p=0.029) and length of stay (p=0.009). Results of the study highlighted the importance of theoretical and practical triage training courses for nurses and provided a foundation for identifying effective factors for decreasing the length of stay and disposition of patients in emergency care units.

Assessment of patient mobility improves the risk stratification of triage with the Emergency Severity Index: a prospective cohort study

BACKGROUND AND IMPORTANCE

Formal triage may assign a low acuity to patients at high risk of deterioration and mortality. A patient's mobility can be easily assessed at triage.

OBJECTIVE

To investigate if a simple assessment of mobility at triage can improve the Emergency Severity Index's (ESI) prediction of adverse outcomes.

DESIGN, SETTING AND PARTICIPANTS

Prospective observational study of all patients attending the emergency department (ED) of a single academic hospital in Switzerland over a period of 3 weeks.

OUTCOME MEASURES AND ANALYSIS

Triage clinicians classified participants as having normal or impaired mobility at triage. Impaired mobility was defined as the lack of a stable independent gait, regardless of its cause or duration (e.g. any patient who needed help to walk). The primary outcome was 30-day mortality. We performed a survival analysis stratified by mobility and ESI level. We compared the performance of regression models including the ESI alone or in combination with mobility as predictors of mortality using the Bayesian information criterion (BIC).

MAIN RESULTS

2523 patients were included in the study and 880 (34.9%) had impaired mobility. Patients with impaired mobility had a lower median 30-day survival in ESI levels 1-3. Survival of patients with normal mobility was similar regardless of their ESI level.

CONCLUSION

The assessment of mobility at triage improves the ESI algorithm's risk stratification.

Validity and reliability of the triage scale in older people in a regional emergency department in Hong Kong

Background:

Older people (⩾65 years) present a unique challenge in emergency department triage. Hong Kong’s Hospital Authority adopts a five-level emergency department triage system, with no special considerations for older people. We evaluated the validity and reliability of this triage scale in older people in a regional Hong Kong emergency department.

Methods:

In total, 295 cases stratified by triage category were randomly selected for review from November 2016 to January 2017. Validity was established by comparing the real emergency department patients’ triage category against (1) that of an expert panel and (2) the need for life-saving intervention. Triage notes were extracted to make case scenarios to evaluate inter- and intra-rater reliabilities. Emergency department nurses (n = 8) were randomly selected and grouped into <5 and ⩾5 years emergency department experience. All nurses independently rated all 295 scenarios, blinded to clinical outcomes.

Results:

The percentage agreement between the real emergency department patients’ triage category and the expert panel’s assignment was 68.5%, with 16.3% and 15.3% over-triage and under-triage, respectively. Quadratic weighting kappa for agreement with the expert panel was 0.72 (95% confidence interval: 0.53–0.91). The sensitivity, specificity and positive likelihood ratio for the need for life-saving interventions were 75.0% (95% confidence interval: 47.6%–92.7%), 97.1% (95% confidence interval: 94.4%–98.8%) and 26.2 (95% confidence interval: 12.5%–54.8%), respectively. The Fleiss kappa value for inter-rater reliability was 0.50 (95% confidence interval: 0.47–0.54) for junior and senior nurse groups, respectively.

Conclusion:

The current triage scale demonstrates reasonable validity and reliability for use in our older people. Considerations highlighting the unique characteristics of older people emergency department presentations are recommended.

Dispatcher Triage Accuracy in the Western Cape Government Emergency Medical Services System, Cape Town, South Africa

INTRODUCTION

Triage - the sorting of patients according to urgency of need for clinical care - is an essential part of delivering effective and efficient emergency care. But when frequent over- or under-triaging occurs, finite time and resources are diverted away from those in greatest need of care and the entire Emergency Medical Services (EMS) system is strained. In resource-constrained settings, such as South Africa, poor triage in EMS only serves to compound other contextual challenges. This study examined the accuracy of dispatcher triage over a one-year period in the Western Cape Government (WCG) EMS system in South Africa.

METHODS

A retrospective analysis of existing dispatch and EMS data to assess the accuracy of dispatch-assigned priorities was conducted. The mismatch between dispatcher-assigned call priority and triage levels determined by EMS personnel was analyzed via over- and under-triage rates, sensitivity and specificity, and positive and negative predictive values (PPVs and NPVs, respectively).

RESULTS

A total of 185,166 records from December 2016 through November 2017 were analyzed. Across all dispatch complaints, the over-triage rate was 67.6% (95% CI, 66.34-68.76) and the under-triage rate was 16.2% (95% CI, 15.44-16.90). Dispatch triage sensitivity for all included records was 49.2% (95% CI, 48.10-50.38), specificity 71.9% (95% CI, 71.00-72.92), PPV 32.5% (95% CI, 30.02-34.88), and NPV 83.8% (95% CI, 81.93-85.73).

CONCLUSION

This study provides the first evaluation of dispatch triage accuracy in the WCG EMS system, identifying that the system is suffering from both under- and over-triage. Despite variance across dispatch complaints, both under- and over-triage remained higher than widely accepted norms, and all rates were significantly above acceptable target metrics described in similar studies. Results of this study will be used to motivate the development of more rigorous training programs and resources for WCG EMS dispatchers, including improved dispatch protocols for conditions suffering from high over- and under-triage.

Why Is Treatment Urgency Often Overestimated? An Experimental Study on the Phenomenon of Over-triage

In the 19th century, triage emerged as an administrative concept to overcome the unjust and medically unreasonable consequences of an unsystematic adhoc selection of casualties. Until today, however, triage concepts are often applied incorrectly. High over-triage rates are a well-known phenomenon, which increase mortality rates. In order to examine their frequent occurrences, the article discusses different reasons and presents results of an experimental study. Two triage exercises were conducted: a paper-based triage exercise and a real-world simulation. Both exercises used the same case-vignettes consisting of 5 pairs. Each pair described a patient with the same injury pattern and vital parameters but with differing behaviour (calm/highly excited). Different behavior has a minor but no significant effect on over-triage rates. Over-triage is significantly higher in the real-world simulation than in the paper exercise. This is explained by the characteristics of face-to-face situations themselves: they are more complex and ambiguous, and hold more normative power. Accordingly, over-triage is understood as a means to resolve unclear situations ("better to over- than to under-triage") and to comply with normative demands "within" the strict margins of an administrative concept.

Older adults are prioritized in terms of waiting time under the emergency triage system in Guangzhou, China

AIM

To assess whether elderly patients are prioritized under the emergency triage system in Guangzhou, China.

METHODS

This was a cross-sectional survey of clinical data from adult visitors to the emergency department of the Third Affiliated Hospital of Sun Yat-sen University between 1 August 2015 and 31 December 2017. The primary end-point was receiving the first medical service within the target waiting time, which varied according to the triage level of the patient. Multivariate logistic regression was used to determine whether age was an independent predictor of a shorter waiting time.

RESULTS

Data from 262 282 emergency patients were analyzed. The mean age of patients was 35.97 years, and 7.5% were aged ≥65 years. In total, 88.3% of patients received medical service within the target waiting time, and 87.4% of elderly patients received medical service within the target waiting time. Multivariate logistic regression analysis showed that advanced age was independently associated with receiving medical service within the target waiting time (adjusted odds ratio 1.258, 95% confidence interval 1.198-1.321; P < 0.001). The triage level, type of emergency subdivision, availability of outpatient services and time of day were also associated with receiving medical service within the target waiting time.

CONCLUSION

Under the emergency triage system of the hospital, older adults are more likely to receive medical service within the target waiting time than younger patients. Geriatr Gerontol Int 2019; 19: 786-791.

Implementation and performance of the South African Triage Scale at Kenyatta National Hospital in Nairobi, Kenya

INTRODUCTION

Triage protocols standardize and improve patient care in accident and emergency departments (A&Es). Kenyatta National Hospital (KNH), the largest public tertiary hospital in East Africa, is resource-limited and was without A&E-specific triage protocols.

OBJECTIVES

We sought to standardize patient triage through implementation of the South African Triage Scale (SATS). We aimed to (1) assess the reliability of triage decisions among A&E healthcare workers following an educational intervention and (2) analyze the validity of the SATS in KNH's A&E.

METHODS

Part 1 was a prospective, before and after trial utilizing an educational intervention and assessing triage reliability using previously validated vignettes administered to 166 healthcare workers. Part 2 was a triage chart review wherein we assessed the validity of the SATS in predicting patient disposition outcomes by inclusion of 2420 charts through retrospective, systematic sampling.

RESULTS

Healthcare workers agreed with an expert defined triage standard for 64% of triage scenarios following an educational intervention, and had a 97% agreement allowing for a one-level discrepancy in the SATS score. There was "good" inter-rater agreement based on an intraclass correlation coefficient and quadratic weighted kappa. We analyzed 1209 pre-SATS and 1211 post-SATS patient charts and found a non-significant difference in undertriage and statistically significant decrease in overtriage rates between the pre- and post-SATS cohorts (undertriage 3.8 and 7.8%, respectively, p = 0.2; overtriage 70.9 and 62.3%, respectively, p < 0.05). The SATS had a sensitivity of 92.2% and specificity of 37.7% for predicting admission, death, or discharge in the A&E.

CONCLUSION

Healthcare worker triage decisions using the SATS were more consistent with expert opinion following an educational intervention. The SATS also performed well in predicting outcomes with high sensitivity and satisfactory levels of both undertriage and overtriage, confirming the SATS as a contextually appropriate triage system at a major East African A&E.