Consensus Development of a Modern Ontology of Emergency Department Presenting Problems-The Hierarchical Presenting Problem Ontology (HaPPy)

Development and validation of a machine learning framework for improved resource allocation in the emergency department

OBJECTIVE

The Emergency Severity Index (ESI) is the most commonly used system in over 70% of all U.S. emergency departments (ED) that uses predicted resource utilization as a means to triage [1], Mistriage, which includes both undertriage and overtriage has been a persistent issue, affecting 32.2% of total ED visits [2]. Our goal is to develop a machine learning framework that predicts patients' resource needs, thereby improving resource allocation during triage.

METHODS

This retrospective study analyzed ED visits from the Medical Information Mart for Intensive Care IV, dividing the data into training (80%) and testing (20%) cohorts. We utilized data available during triage, including patient vital signs, age, gender, mode of arrival, medication history, and chief complaint. Azure AutoML was used to create different machine learning models trained to predict the 144 target columns including laboratory panels and imaging modalities as well as medications required during patients' ED visits. The 144 models' performance was evaluated using the area under the receiver operating characteristic curve (AUROC), F1 score, accuracy, precision and recall.

RESULTS

A total of 391,472 ED visits were analyzed. 144 Voting ensemble models were created for each target. All frameworks achieved on average an AUC score of 0.82 and accuracy of 0.76. We gathered the feature importance for each target and observed that 'chief complaint', among others, had a high aggregate feature importance across different targets.

CONCLUSION

This study shows the high accuracy in predicting resource needs for patients in the ED using a machine learning model. This can greatly improve patient flow and resource allocation in already resource limited emergency departments.

Near Real-Time Syndromic Surveillance of Emergency Department Triage Texts Using Natural Language Processing: Case Study in Febrile Convulsion Detection

BACKGROUND

Collecting information on adverse events following immunization from as many sources as possible is critical for promptly identifying potential safety concerns and taking appropriate actions. Febrile convulsions are recognized as an important potential reaction to vaccination in children aged <6 years.

OBJECTIVE

The primary aim of this study was to evaluate the performance of natural language processing techniques and machine learning (ML) models for the rapid detection of febrile convulsion presentations in emergency departments (EDs), especially with respect to the minimum training data requirements to obtain optimum model performance. In addition, we examined the deployment requirements for a ML model to perform real-time monitoring of ED triage notes.

METHODS

We developed a pattern matching approach as a baseline and evaluated ML models for the classification of febrile convulsions in ED triage notes to determine both their training requirements and their effectiveness in detecting febrile convulsions. We measured their performance during training and then compared the deployed models' result on new incoming ED data.

RESULTS

Although the best standard neural networks had acceptable performance and were low-resource models, transformer-based models outperformed them substantially, justifying their ongoing deployment.

CONCLUSIONS

Using natural language processing, particularly with the use of large language models, offers significant advantages in syndromic surveillance. Large language models make highly effective classifiers, and their text generation capacity can be used to enhance the quality and diversity of training data.

Clinical decision support system in emergency telephone triage: A scoping review of technical design, implementation and evaluation

OBJECTIVES

Emergency department overcrowding could be improved by upstream telephone triage. Emergency telephone triage aims at managing and orientating adequately patients as early as possible and distributing limited supply of staff and materials. This complex task could be improved with the use of Clinical decision support systems (CDSS). The aim of this scoping review was to identify literature gaps for the future development and evaluation of CDSS for Emergency telephone triage.

MATERIALS AND METHODS

We present here a scoping review of CDSS designed for emergency telephone triage, and compared them in terms of functional characteristics, technical design, health care implementation and methodologies used for evaluation, following the PRISMA-ScR guidelines.

RESULTS

Regarding design, 19 CDSS were retrieved: 12 were knowledge based CDSS (decisional algorithms built according to guidelines or clinical expertise) and 7 were data driven (statistical, machine learning, or deep learning models). Most of them aimed at assisting nurses or non-medical staff by providing patient orientation and/or severity/priority assessment. Eleven were implemented in real life, and only three were connected to the Electronic Health Record. Regarding evaluation, CDSS were assessed through various aspects: intrinsic characteristics, impact on clinical practice or user apprehension. Only one pragmatic trial and one randomized controlled trial were conducted.

CONCLUSION

This review highlights the potential of a hybrid system, user tailored, flexible, connected to the electronic health record, which could work with oral, video and digital data; and the need to evaluate CDSS on intrinsic characteristics and impact on clinical practice, iteratively at each distinct stage of the IT lifecycle.

Machine Learning Methods for Predicting Patient-Level Emergency Department Workload

BACKGROUND

Work Relative Value Units (wRVUs) are a component of many compensation models, and a proxy for the effort required to care for a patient. Accurate prediction of wRVUs generated per patient at triage could facilitate real-time load balancing between physicians and provide many practical operational and clinical benefits.

OBJECTIVE

We examined whether deep-learning approaches could predict the wRVUs generated by a patient's visit using data commonly available at triage.

METHODS

Adult patients presenting to an urban, academic emergency department from July 1, 2016-March 1, 2020 were included. Deidentified triage information included structured data (age, sex, vital signs, Emergency Severity Index score, language, race, standardized chief complaint) and unstructured data (free-text chief complaint) with wRVUs as outcome. Five models were examined: average wRVUs per chief complaint, linear regression, neural network and gradient-boosted tree on structured data, and neural network on unstructured textual data. Models were evaluated using mean absolute error.

RESULTS

We analyzed 204,064 visits between July 1, 2016 and March 1, 2020. The median wRVUs were 3.80 (interquartile range 2.56-4.21), with significant effects of age, gender, and race. Models demonstrated lower error as complexity increased. Predictions using averages from chief complaints alone demonstrated a mean error of 2.17 predicted wRVUs per visit (95% confidence interval [CI] 2.07-2.27), the linear regression model: 1.00 wRVUs (95% CI 0.97-1.04), gradient-boosted tree: 0.85 wRVUs (95% CI 0.84-0.86), neural network with structured data: 0.86 wRVUs (95% CI 0.85-0.87), and neural network with unstructured data: 0.78 wRVUs (95% CI 0.76-0.80).

CONCLUSIONS

Chief complaints are a poor predictor of the effort needed to evaluate a patient; however, deep-learning techniques show promise. These algorithms have the potential to provide many practical applications, including balancing workloads and compensation between emergency physicians, quantify crowding and mobilizing resources, and reducing bias in the triage process.

Risk of malignancy in patients with chronic kidney disease

Background

Fifteen percent of US adults have chronic kidney disease (CKD). The effect of CKD on the development of different malignancies is unknown. Understanding the effect of CKD on the risk of development of cancer could have important implications for screening and early detection of cancer in these patients.

Methods

Adult CKD patients [estimated GFR (eGFR) <60ml/min/1.73m²] between January 2001 and December 2020 were identified in this single institution study. Patients were divided into four stages of CKD by eGFR. The incidence of cancer and time to development of the first cancer were identified. Multivariable models were used to compare the overall cancer incidence while considering death as a competing risk event and adjusting for relevant covariates (sex, race, diabetes, hypertension, CAD, smoking or not, BMI, and CKD stages). Separate multivariable models of the incidence of cancers were conducted in each age group. Multivariable Cox models were used to fit the overall death adjusting for relevant covariates. Patients were censored at the conclusion of the study period (December 31, 2020). Statistical analysis was performed with SAS software (version 9.4).

Results

Of the 13,750 patients with a diagnosis of CKD in this cohort, 2,758 (20.1%) developed a malignancy. The median time to development of cancer following a diagnosis of CKD was 8.5 years. Factors associated with the risk of developing cancer in CKD patients included increasing age, male sex and worsening chronic kidney disease, while diabetes was associated with a lower risk of malignancy. On multivariate analysis, the factors associated with increased mortality in patients who developed cancer included increasing age, diabetes and lower eGFR.

Conclusion

CKD is an increased risk factor for the development of various malignancies. Age appropriate cancer screening should be aggressively pursued in those with progressive CKD.

Design and Evaluation of a Postpartum Depression Ontology

OBJECTIVE

Postpartum depression (PPD) remains an understudied research area despite its high prevalence. The goal of this study is to develop an ontology to aid in the identification of patients with PPD and to enable future analyses with electronic health record (EHR) data.

METHODS

We used Protégé-OWL to construct a postpartum depression ontology (PDO) of relevant comorbidities, symptoms, treatments, and other items pertinent to the study and treatment of PPD.

RESULTS

The PDO identifies and visualizes the risk factor status of variables for PPD, including comorbidities, confounders, symptoms, and treatments. The PDO includes 734 classes, 13 object properties, and 4,844 individuals. We also linked known and potential risk factors to their respective codes in the International Classification of Diseases versions 9 and 10 that would be useful in structured EHR data analyses. The representation and usefulness of the PDO was assessed using a task-based patient case study approach, involving 10 PPD case studies. Final evaluation of the ontology yielded 86.4% coverage of PPD symptoms, treatments, and risk factors. This demonstrates strong coverage of the PDO for the PPD domain.

CONCLUSION

The PDO will enable future researchers to study PPD using EHR data as it contains important information with regard to structured (e.g., billing codes) and unstructured data (e.g., synonyms of symptoms not coded in EHRs). The PDO is publicly available through the National Center for Biomedical Ontology (NCBO) BioPortal ( https://bioportal.bioontology.org/ontologies/PARTUMDO ) which will enable other informaticists to utilize the PDO to study PPD in other populations.

Calibrating a chief complaint list for low resource settings: a methodologic case study

BACKGROUND

The chief or presenting complaint is the reason for seeking health care, often in the patient's own words. In limited resource settings, a diagnosis-based approach to quantifying burden of disease is not possible, partly due to limited availability of an established lexicon or coding system. Our group worked with colleagues from the African Federation of Emergency Medicine building on the existing literature to create a pilot symptom list representing an attempt to standardize undifferentiated chief complaints in emergency and acute care settings. An ideal list for any setting is one that strikes a balance between ease of use and length, while covering the vast majority of diseases with enough detail to permit epidemiologic surveillance and make informed decisions about resource needs.

METHODS

This study was incorporated as a part of a larger prospective observational study on human immunodeficiency virus testing in Emergency Departments in South Africa. The pilot symptom list was used for chief complaint coding in three Emergency Departments. Data was collected on 3357 patients using paper case report forms. Chief complaint terms were reviewed by two study team members to determine the frequency of concordance between the coded chief complaint term and the selected symptom(s) from the pilot symptom list.

RESULTS

Overall, 3537 patients' chief complaints were reviewed, of which 640 were identified as 'potential mismatches.' When considering the 191 confirmed mismatches (29.8%), the Delphi process identified 6 (3.1%) false mismatches and 185 (96.9%) true mismatches. Significant chief-complaint clustering was identified with 9 sets of complaints frequently selected together for the same patient. "Pain" was used 2076 times for 58.7% of all patients. A combination of user feedback and expert-panel modified Delphi analysis of mismatched complaints and clustered complaints resulted in several substantial changes to the pilot symptom list.

CONCLUSIONS

This study presented a systematic methodology for calibrating a chief complaint list for the local context. Our revised list removed/reworded symptoms that frequently clustered together or were misinterpreted by health professionals. Recommendations for additions, modifications, and/or deletions from the pilot chief complaint list we believe will improve the functionality of the list in low resource environments.

Generating contextual embeddings for emergency department chief complaints

Objective

We learn contextual embeddings for emergency department (ED) chief complaints using Bidirectional Encoder Representations from Transformers (BERT), a state-of-the-art language model, to derive a compact and computationally useful representation for free-text chief complaints.

Materials and methods

Retrospective data on 2.1 million adult and pediatric ED visits was obtained from a large healthcare system covering the period of March 2013 to July 2019. A total of 355 497 (16.4%) visits from 65 737 (8.9%) patients were removed for absence of either a structured or unstructured chief complaint. To ensure adequate training set size, chief complaint labels that comprised less than 0.01%, or 1 in 10 000, of all visits were excluded. The cutoff threshold was incremented on a log scale to create seven datasets of decreasing sparsity. The classification task was to predict the provider-assigned label from the free-text chief complaint using BERT, with Long Short-Term Memory (LSTM) and Embeddings from Language Models (ELMo) as baselines. Performance was measured as the Top-k accuracy from k = 1:5 on a hold-out test set comprising 5% of the samples. The embedding for each free-text chief complaint was extracted as the final 768-dimensional layer of the BERT model and visualized using t-distributed stochastic neighbor embedding (t-SNE).

Results

The models achieved increasing performance with datasets of decreasing sparsity, with BERT outperforming both LSTM and ELMo. The BERT model yielded Top-1 accuracies of 0.65 and 0.69, Top-3 accuracies of 0.87 and 0.90, and Top-5 accuracies of 0.92 and 0.94 on datasets comprised of 434 and 188 labels, respectively. Visualization using t-SNE mapped the learned embeddings in a clinically meaningful way, with related concepts embedded close to each other and broader types of chief complaints clustered together.

Discussion

Despite the inherent noise in the chief complaint label space, the model was able to learn a rich representation of chief complaints and generate reasonable predictions of their labels. The learned embeddings accurately predict provider-assigned chief complaint labels and map semantically similar chief complaints to nearby points in vector space.

Conclusion

Such a model may be used to automatically map free-text chief complaints to structured fields and to assist the development of a standardized, data-driven ontology of chief complaints for healthcare institutions.

Data Model Requirements for a Digital Cognitive Aid for Anesthesia to Support Intraoperative Crisis Management

Objective  The aim of this study is to define data model requirements supporting the development of a digital cognitive aid (CA) for intraoperative crisis management in anesthesia, including medical emergency text modules (text elements) and branches or loops within emergency instructions (control structures) as well as their properties, data types, and value ranges.

Methods  The analysis process comprised three steps: reviewing the structure of paper-based CAs to identify common text elements and control structures, identifying requirements derived from content, design, and purpose of a digital CA, and validating requirements by loading exemplary emergency checklist data into the resulting prototype data model.

Results  The analysis of paper-based CAs identified 19 general text elements and two control structures. Aggregating these elements and analyzing the content, design and purpose of a digital CA revealed 20 relevant data model requirements. These included checklist tags to enable different search options, structured checklist action steps (items) in groups and subgroups, and additional information on each item. Checklist and Item were identified as two main classes of the prototype data model. A data object built according to this model was successfully integrated into a digital CA prototype.

Conclusion  To enable consistent design and interactivity with the content, presentation of critical medical information in a digital CA for crisis management requires a uniform structure. So far it has not been investigated which requirements need to be met by a data model for this purpose. The results of this study define the requirements and structure that enable the presentation of critical medical information. Further research is needed to develop a comprehensive data model for a digital CA for crisis management in anesthesia, including supplementation of requirements resulting from simulation studies and feasibility analyses regarding existing data models. This model may also be a useful template for developing data models for CAs in other medical domains.

Predicting Intensive Care Unit admission among patients presenting to the emergency department using machine learning and natural language processing

The risk stratification of patients in the emergency department begins at triage. It is vital to stratify patients early based on their severity, since undertriage can lead to increased morbidity, mortality and costs. Our aim was to present a new approach to assist healthcare professionals at triage in the stratification of patients and in identifying those with higher risk of ICU admission. Adult patients assigned Manchester Triage System (MTS) or Emergency Severity Index (ESI) 1 to 3 from a Portuguese and a United States Emergency Departments were analyzed. Variables routinely collected at triage were used and natural language processing was applied to the patient chief complaint. Stratified random sampling was applied to split the data in train (70%) and test (30%) sets and 10-fold cross validation was performed for model training. Logistic regression, random forests, and a random undersampling boosting algorithm were used. We compared the performance obtained with the reference model—using only triage priorities—with the models using additional variables. For both hospitals, a logistic regression model achieved higher overall performance, yielding areas under the receiver operating characteristic and precision-recall curves of 0.91 (95% CI 0.90-0.92) and 0.30 (95% CI 0.27-0.33) for the United States hospital and of 0.85 (95% CI 0.83-0.86) and 0.06 (95% CI 0.05-0.07) for the Portuguese hospital. Heart rate, pulse oximetry, respiratory rate and systolic blood pressure were the most important predictors of ICU admission. Compared to the reference models, the models using clinical variables and the chief complaint presented higher recall for patients assigned MTS/ESI 3 and can identify patients assigned MTS/ESI 3 who are at risk for ICU admission.