Efficiency of electronic signout for ED-to-inpatient admission at a non-teaching hospital

Developing and Evaluating Large Language Model-Generated Emergency Medicine Handoff Notes

Importance

An emergency medicine (EM) handoff note generated by a large language model (LLM) has the potential to reduce physician documentation burden without compromising the safety of EM-to-inpatient (IP) handoffs.

Objective

To develop LLM-generated EM-to-IP handoff notes and evaluate their accuracy and safety compared with physician-written notes.

Design, Setting, and Participants

This cohort study used EM patient medical records with acute hospital admissions that occurred in 2023 at NewYork-Presbyterian/Weill Cornell Medical Center. A customized clinical LLM pipeline was trained, tested, and evaluated to generate templated EM-to-IP handoff notes. Using both conventional automated methods (ie, recall-oriented understudy for gisting evaluation [ROUGE], bidirectional encoder representations from transformers score [BERTScore], and source chunking approach for large-scale inconsistency evaluation [SCALE]) and a novel patient safety-focused framework, LLM-generated handoff notes vs physician-written notes were compared. Data were analyzed from October 2023 to March 2024.

Exposure

LLM-generated EM handoff notes.

Main Outcomes and Measures

LLM-generated handoff notes were evaluated for (1) lexical similarity with respect to physician-written notes using ROUGE and BERTScore; (2) fidelity with respect to source notes using SCALE; and (3) readability, completeness, curation, correctness, usefulness, and implications for patient safety using a novel framework.

Results

In this study of 1600 EM patient records (832 [52%] female and mean [SD] age of 59.9 [18.9] years), LLM-generated handoff notes, compared with physician-written ones, had higher ROUGE (0.322 vs 0.088), BERTScore (0.859 vs 0.796), and SCALE scores (0.691 vs 0.456), indicating the LLM-generated summaries exhibited greater similarity and more detail. As reviewed by 3 board-certified EM physicians, a subsample of 50 LLM-generated summaries had a mean (SD) usefulness score of 4.04 (0.86) out of 5 (compared with 4.36 [0.71] for physician-written) and mean (SD) patient safety scores of 4.06 (0.86) out of 5 (compared with 4.50 [0.56] for physician-written). None of the LLM-generated summaries were classified as a critical patient safety risk.

Conclusions and Relevance

In this cohort study of 1600 EM patient medical records, LLM-generated EM-to-IP handoff notes were determined superior compared with physician-written summaries via conventional automated evaluation methods, but marginally inferior in usefulness and safety via a novel evaluation framework. This study suggests the importance of a physician-in-loop implementation design for this model and demonstrates an effective strategy to measure preimplementation patient safety of LLM models.

Evaluation of Process Improvement Interventions on Handoff Times between the Emergency Department and Observation Unit

BACKGROUND

Mitigating hospital crowding requires judicious use of inpatient resources, making Emergency Department Observation Units (EDOUs) an increasingly vital destination for patients that are not suitable for discharge. Maximizing the utility of the EDOU hinges on efficient patient transfers and safe provider communication, which may be accomplished with asynchronous handoff and an emphasis on pull-through operations.

OBJECTIVE

The purpose of this study was to assess the impact of an electronic, asynchronous handoff replacing verbal handoff on transfer times from the Emergency Department (ED) to the EDOU.

METHODS

A retrospective observational study was performed with patients transferred to the EDOU throughout several process improvement measures focused on asynchronous handoff. Multivariable linear regression analysis was used to determine the effect that these process improvements had on the time from EDOU bed assignment to patient transfer.

RESULTS

There were 14,996 EDOU stays during the 20-month period included in the analysis. Time from EDOU bed assignment to patient transfer decreased significantly with all three interventions studied. An auto-page to the clinicians notifying them of a ready bed reduced the mean time to transfer by 10.1 min (p < 0.0001), asynchronous nursing handoff reduced it by 3.57 min (p = 0.0299), and asynchronous clinician handoff reduced it by 14.67 min (p < 0.0001).

CONCLUSION

Introducing automatic pages regarding bed status and converting the handoff process from a verbal model to an asynchronous, electronic handoff were effective ways to reduce the time from bed assignment to transfer out of the ED for patients being sent to the EDOU.

What is the effect of electronic clinical handovers on patient outcomes? A systematic review

Handover between physicians is a high-risk event for communication errors. Using electronic handover platforms has potential to improve the quality of informational transfer and therefore minimise this risk. This systematic review sought to compare the effectiveness of electronic handover methods on patient outcomes. Articles were identified by searching MEDLINE, EMbase, Scopus and CINAHL databases. Studies involving electronic handover between two healthcare personnel or teams, and which described patientspecific outcomes, were included. This search yielded 390 articles, with a total of nine publications included in the analysis. Outcomes reported in studies included length of stay, adverse event rates, time to procedure and handover completeness. This review suggests that e-handover may improve the handover completeness; however, it is unclear at this time if that translates to an improvement in patient care. The lack of reliable evidence highlights the need for further research exploring the effect of e-handovers on patient care.