Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients

Machine Learning Multimodal Model for Delirium Risk Stratification

Importance

Automating the identification of risk for developing hospital delirium with models that use machine learning (ML) could facilitate more rapid prevention, identification, and treatment of delirium. However, there are very few reports on the performance of ML models for delirium risk stratification in live clinical practice.

Objective

To report on development, operationalization, and validation of a multimodal ML model for delirium risk stratification in live clinical practice and its associations with workflow and clinical outcomes.

Design, Setting, and Participants

This quality improvement study developed an ML model supported by automated electronic medical records to stratify the risk of non-intensive care unit delirium in live clinical practice using the Confusion Assessment Method as the diagnostic reference standard, with an iterative model update method. Data from patients aged at least 60 years admitted to non-intensive care units at Mount Sinai Hospital between January 2016 and January 2020 were used to train and test the ML model presented. The model was validated in live clinical practice from March 2023 to March 2024. Analysis of the model's associations with workflow and clinical outcomes was conducted retrospectively in 2024, comparing hospitalized patients prior to deployment of any model version (pre-ML cohort) and during model clinical deployment (post-ML cohort).

Main Outcomes and Measures

Outcomes of interest were area under the receiver operating characteristic curve, monthly delirium detection rates, median length of hospital stay, and daily doses of opiate, benzodiazepine, and antipsychotic medications administered.

Results

The overall sample included 32 284 inpatient admissions (mean [SD] age, 73.56 (9.67) years, 15 157 [46.9%] women). A total of 25 261 inpatient admissions of older patients with both medical and surgical primary diagnoses represented the combined model testing and training cohort (median age, 73.37 [66.42-81.36] years) and live clinical deployment validation cohort (median [IQR] age, 72.11 [62.26-78.97] years), while 7023 inpatient admissions of older patients with both medical and surgical primary diagnoses represented the combined pre-ML (median [IQR] age, 74.00 [68.00-81.00] years) and post-ML (median [IQR] age, 75.33 [68.34-82.91] years) cohorts. The model presented is a fusion of electronic medical record patient data features and clinical note features processed by natural language processing. The results of model validation in live clinical practice included an area under the curve of 0.94 (95% CI, 0.93-0.95). Median (IQR) monthly delirium detection rates of inpatients assessed for delirium with the Confusion Assessment Method increased from 4.42% (95% CI, 3.70%-5.14%) in the pre-ML cohort to 17.17% (95% CI, 15.54%-18.80%) in the post-ML cohort (P < .001). Post-ML vs pre-ML cohorts received lower daily doses of benzodiazepines (median [IQR] 0.93 [0.42-2.28] diazepam dose equivalents vs 1.60 [0.66-4.27] diazepam dose equivalents; P < .001) and olanzapine (median [IQR], 1.09 [0.38-2.46] mg vs 2.50 [1.17-6.65] mg; P < .001).

Conclusions and Relevance

This quality improvement study demonstrates the feasibility of a novel multimodal ML model to automate delirium risk stratification in live clinical practice. The model demonstrated acceptable performance in live clinical practice and may facilitate resource allocation to enhance delirium identification and care.

Diagnostic yield of laboratory testing in hospitalized older adults with altered mental status

BACKGROUND

Altered mental status (AMS) is a common cause of hospitalization among older adults, with a wide range of potential etiologies. However, the diagnostic and therapeutic yield of routine laboratory testing in such patients is unknown.

METHODS

In a retrospective cohort of inpatient hospital admissions to a large academic medical center from 2017 to 2022 of patients 65 years and older for whom the admitting diagnosis was AMS, we assessed laboratory testing for thyroid stimulating hormone (TSH), syphilis, vitamin B12, folate, vitamin C, vitamin D, zinc, niacin, and thiamine. We calculated the frequency of testing, rate of abnormal results, and rate of follow-up treatment.

RESULTS

Of the 3169 patients, 2312 (73 %) received at least one designated lab, and overall, 12 % of labs were abnormal. Labs varied in frequency of use (0.2 % for niacin-66 % for TSH) and rate of abnormality (0 % for niacin-71 % for zinc). 16 % of abnormal index labs led to a new prescription at discharge. The most common tests - TSH, folate, and B12- were of relatively low diagnostic and therapeutic utility. Tests that were less common-zinc, vitamin D, and vitamin C-were more commonly abnormal. 3.8 % of patients tested for syphilis had abnormal results, and 72 % of patients with an abnormal result received treatment with penicillin during the index hospitalization.

CONCLUSIONS

These analyses suggest that commonly obtained labs in the workup of AMS have varied diagnostic and therapeutic utility. The contribution of observed laboratory abnormalities to a patients' AMS warrants further study to improve the delivery of high-value care.

The Multidimensional Prognostic Index predicts incident delirium among hospitalized older patients with COVID-19: a multicenter prospective European study

PURPOSE

Incident delirium is a frequent complication among hospitalized older people with COVID-19, associated with increased length of hospital stay, higher morbidity and mortality rates. Although delirium is preventable with early detection, systematic assessment methods and predictive models are not universally defined, thus delirium is often underrated. In this study, we tested the role of the Multidimensional Prognostic Index (MPI), a prognostic tool based on Comprehensive Geriatric Assessment, to predict the risk of incident delirium.

METHODS

Hospitalized older patients (≥ 65 years) with COVID-19 infection were enrolled (n = 502) from ten centers across Europe. At hospital admission, the MPI was administered to all the patients and two already validated delirium prediction models were computed (AWOL delirium risk-stratification score and Martinez model). Delirium occurrence during hospitalization was ascertained using the 4A's Test (4AT). Accuracy of the MPI and the other delirium predictive models was assessed through logistic regression models and the area under the curve (AUC).

RESULTS

We analyzed 293 patients without delirium at hospital admission. Of them 33 (11.3%) developed delirium during hospitalization. Higher MPI score at admission (higher multidimensional frailty) was associated with higher risk of incident delirium also adjusting for the other delirium predictive models and COVID-19 severity (OR = 12.72, 95% CI = 2.11-76.86 for MPI-2 vs MPI-1, and OR = 33.44, 95% CI = 4.55-146.61 for MPI-3 vs MPI-1). The MPI showed good accuracy in predicting incident delirium (AUC = 0.71) also superior to AWOL tool, (AUC = 0.63) and Martinez model (AUC = 0.61) (p < 0.0001 for both comparisons).

CONCLUSIONS

The MPI is a sensitive tool for early identification of older patients with incident delirium.

Identification of risk factors for the onset of delirium associated with COVID-19 by mining nursing records

COVID-19 has a range of complications, from no symptoms to severe pneumonia. It can also affect multiple organs including the nervous system. COVID-19 affects the brain, leading to neurological symptoms such as delirium. Delirium, a sudden change in consciousness, can increase the risk of death and prolong the hospital stay. However, research on delirium prediction in patients with COVID-19 is insufficient. This study aimed to identify new risk factors that could predict the onset of delirium in patients with COVID-19 using machine learning (ML) applied to nursing records. This retrospective cohort study used natural language processing and ML to develop a model for classifying the nursing records of patients with delirium. We extracted the features of each word from the model and grouped similar words. To evaluate the usefulness of word groups in predicting the occurrence of delirium in patients with COVID-19, we analyzed the temporal changes in the frequency of occurrence of these word groups before and after the onset of delirium. Moreover, the sensitivity, specificity, and odds ratios were calculated. We identified (1) elimination-related behaviors and conditions and (2) abnormal patient behavior and conditions as risk factors for delirium. Group 1 had the highest sensitivity (0.603), whereas group 2 had the highest specificity and odds ratio (0.938 and 6.903, respectively). These results suggest that these parameters may be useful in predicting delirium in these patients. The risk factors for COVID-19-associated delirium identified in this study were more specific but less sensitive than the ICDSC (Intensive Care Delirium Screening Checklist) and CAM-ICU (Confusion Assessment Method for the Intensive Care Unit). However, they are superior to the ICDSC and CAM-ICU because they can predict delirium without medical staff and at no cost.

Use of a Data Repository to Identify Delirium as a Presenting Symptom of COVID-19 Infection in Hospitalized Adults: Cross-Sectional Cohort Pilot Study

BACKGROUND

Delirium, an acute confusional state highlighted by inattention, has been reported to occur in 10% to 50% of patients with COVID-19. People hospitalized with COVID-19 have been noted to present with or develop delirium and neurocognitive disorders. Caring for patients with delirium is associated with more burden for nurses, clinicians, and caregivers. Using information in electronic health record data to recognize delirium and possibly COVID-19 could lead to earlier treatment of the underlying viral infection and improve outcomes in clinical and health care systems cost per patient. Clinical data repositories can further support rapid discovery through cohort identification tools, such as the Informatics for Integrating Biology and the Bedside tool.

OBJECTIVE

The specific aim of this research was to investigate delirium in hospitalized older adults as a possible presenting symptom in COVID-19 using a data repository to identify neurocognitive disorders with a novel group of International Classification of Diseases, Tenth Revision (ICD-10) codes.

METHODS

We analyzed data from 2 catchment areas with different demographics. The first catchment area (7 counties in the North-Central Florida) is predominantly rural while the second (1 county in North Florida) is predominantly urban. The Integrating Biology and the Bedside data repository was queried for patients with COVID-19 admitted to inpatient units via the emergency department (ED) within the health center from April 1, 2020, and April 1, 2022. Patients with COVID-19 were identified by having a positive COVID-19 laboratory test or a diagnosis code of U07.1. We identified neurocognitive disorders as delirium or encephalopathy, using ICD-10 codes.

RESULTS

Less than one-third (1437/4828, 29.8%) of patients with COVID-19 were diagnosed with a co-occurring neurocognitive disorder. A neurocognitive disorder was present on admission for 15.8% (762/4828) of all patients with COVID-19 admitted through the ED. Among patients with both COVID-19 and a neurocognitive disorder, 56.9% (817/1437) were aged ≥65 years, a significantly higher proportion than those with no neurocognitive disorder (P<.001). The proportion of patients aged <65 years was significantly higher among patients diagnosed with encephalopathy only than patients diagnosed with delirium only and both delirium and encephalopathy (P<.001). Most (1272/4828, 26.3%) patients with COVID-19 admitted through the ED during our study period were admitted during the Delta variant peak.

CONCLUSIONS

The data collected demonstrated that an increased number of older patients with neurocognitive disorder present on admission were infected with COVID-19. Knowing that delirium increases the staffing, nursing care needs, hospital resources used, and the length of stay as previously noted, identifying delirium early may benefit hospital administration when planning for newly anticipated COVID-19 surges. A robust and accessible data repository, such as the one used in this study, can provide invaluable support to clinicians and clinical administrators in such resource reallocation and clinical decision-making.

Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models

OBJECTIVES

Early identification of inpatients at risk of developing delirium and implementing preventive measures could avoid up to 40% of delirium cases. Machine learning (ML)-based prediction models may enable risk stratification and targeted intervention, but establishing their current evolutionary status requires a scoping review of recent literature.

METHODS

We searched ten databases up to June 2022 for studies of ML-based delirium prediction models. Eligible criteria comprised: use of at least one ML prediction method in an adult hospital inpatient population; published in English; reporting at least one performance measure (area under receiver-operator curve (AUROC), sensitivity, specificity, positive or negative predictive value). Included models were categorised by their stage of maturation and assessed for performance, utility and user acceptance in clinical practice.

RESULTS

Among 921 screened studies, 39 met eligibility criteria. In-silico performance was consistently high (median AUROC: 0.85); however, only six articles (15.4%) reported external validation, revealing degraded performance (median AUROC: 0.75). Three studies (7.7%) of models deployed within clinical workflows reported high accuracy (median AUROC: 0.92) and high user acceptance.

DISCUSSION

ML models have potential to identify inpatients at risk of developing delirium before symptom onset. However, few models were externally validated and even fewer underwent prospective evaluation in clinical settings.

CONCLUSION

This review confirms a rapidly growing body of research into using ML for predicting delirium risk in hospital settings. Our findings offer insights for both developers and clinicians into strengths and limitations of current ML delirium prediction applications aiming to support but not usurp clinician decision-making.

Delirium and Healthcare Utilization in Patients Undergoing Hematopoietic Stem Cell Transplantation

Delirium, a common neuropsychiatric syndrome among hospitalized patients, has been associated with significant morbidity and mortality in patients undergoing hematopoietic stem cell transplantation (HSCT). Although delirium is often reversible with prompt diagnosis and appropriate management, timely screening of hospitalized patients, including HSCT recipients at risk for delirium, is lacking. The association between delirium symptoms and healthcare utilization among HSCT recipients is also limited. We conducted a retrospective analysis of 502 hospitalized patients admitted for allogeneic or autologous HSCT at 2 tertiary care hospitals between April 2016 and April 2021. We used Natural Language Processing (NLP) to identify patients with delirium symptoms, as defined by an NLP-assisted chart review of the electronic health record (EHR). We used multivariable regression models to examine the associations between delirium symptoms, clinical outcomes, and healthcare utilization, adjusting for patient-, disease-, and transplantation-related factors. Overall, 44.4% (124 of 279) of patients undergoing allogeneic HSCT and 39.0% (87 of 223) of those undergoing autologous HSCT were identified as having delirium symptoms during their index hospitalization. Two-thirds (139 of 211) of the patients with delirium symptoms were prescribed treatment with antipsychotic medications. Among allogeneic HSCT recipients, delirium symptoms were associated with longer hospital length of stay (β = 7.960; P < .001), fewer days alive and out of the hospital (β = -23.669; P < .001), and more intensive care unit admissions (odds ratio, 2.854; P = .002). In autologous HSCT recipients, delirium symptoms were associated with longer hospital length of stay (β = 2.204; P < .001). NLP-assisted EHR review is a feasible approach to identifying hospitalized patients, including HSCT recipients at risk for delirium. Because delirium symptoms are negatively associated with health care utilization during and after HSCT, our findings underscore the need to efficiently identify patients hospitalized for HSCT who are at risk of delirium to improve their outcomes. © 2023 American Society for Transplantation and Cellular Therapy. Published by Elsevier Inc.

The occurrence of delirium diagnosis among youth hospitalizations in the United States: A Kids' Inpatient Database analysis

OBJECTIVES

Delirium is an acute neuropsychiatric condition associated with increased morbidity and mortality. There is increasing recognition of delirium as a substantial health burden in younger patients, although few studies have characterized its occurrence. This study analyzes the occurrence of delirium diagnosis, its comorbidities, and cost among youth hospitalized in the United States.

METHODS

The Kids' Inpatient Database, a national all-payers sample of pediatric hospitalizations in general hospitals, was examined for the year 2019. Hospitalizations with a discharge diagnosis of delirium among patients aged 1-20 years were included in the analysis.

RESULTS

Delirium was diagnosed in 43,138 hospitalizations (95% CI: 41,170-45,106), or 2.3% of studied hospitalizations. Delirium was diagnosed in a broad range of illnesses, with suicide and self-inflicted injury as the most common primary discharge diagnosis among patients with delirium. In-hospital mortality was seven times greater in hospitalizations caring a delirium diagnosis. The diagnosis of delirium was associated with an adjusted increased hospital cost of $8648 per hospitalization, or $373 million in aggregate cost.

CONCLUSIONS

Based on a large national claims database, delirium was diagnosed in youth at a lower rate than expected based on prospective studies. The relative absence of delirium diagnosis in claims data may reflect underdiagnosis, a failure to code, and/or a lower rate of delirium in general hospitals compared with other settings. Further research is needed to better characterize the incidence and prevalence of delirium in young people in the hospital setting.