The development of an automated ward independent delirium risk prediction model

Assessing the Risk of Developing Delirium on Admission to Inpatient Rehabilitation: A Clinical Prediction Model

OBJECTIVES

To develop a clinical model to predict the risk of an individual patient developing delirium during inpatient rehabilitation, based on patient characteristics and clinical data available on admission.

DESIGN

Retrospective observational study based on electronic health record data.

SETTING AND PARTICIPANTS

We studied a previously validated data set of inpatients including incident delirium episodes during rehabilitation. These patients were admitted to ZURZACH Care, Rehaklinik Bad Zurzach, a Swiss inpatient rehabilitation clinic, between January 1, 2015, and December 31, 2018.

METHODS

We performed logistic regression analysis using backward and forward selection with alpha = 0.01 to remove any noninformative potential predictor. We subsequentially used the Akaike information criterion (AIC) to select the final model among the resulting "intermediate" models. Discrimination of the final prediction model was evaluated using the C-statistic.

RESULTS

Of the 20 candidate predictor variables, 6 were included in the final prediction model: a linear spline of age with 1 knot at 60 years and a linear spline of the functional independence measure (FIM), a measure of the functional degree of patients independency, with 1 knot at 64 points, diagnosis of disorders of fluid, electrolyte, and acid-base balance (E87), use of other analgesic and antipyretics (N02B), use of anti-parkinson drugs (N04B), and an anticholinergic burden score (ACB) of ≥3 points.

CONCLUSIONS AND IMPLICATIONS

Our clinical prediction model could, upon validation, identify patients at risk of incident delirium at admission to inpatient rehabilitation, and thus enable targeted prevention strategies.

Predicting delirium in older non-intensive care unit inpatients: development and validation of the DELIrium risK Tool (DELIKT)

BACKGROUND

Effective delirium prevention could benefit from automatic risk stratification of older inpatients using routinely collected clinical data.

AIM

Primary aim was to develop and validate a delirium prediction model (DELIKT) suitable for implementation in hospitals. Secondary aim was to select an anticholinergic burden scale as a predictor.

METHOD

We used one cohort for model development and another for validation with electronically available data collected within the first 24 h of admission. Included were patients aged ≥ 65, hospitalised ≥ 48 h with no stay > 24 h in an intensive care unit. Predictors, such as administrative and laboratory variables or an anticholinergic burden scale, were selected using a combination of feature selection filter method and forward/backward selection. The final model was based on logistic regression and the DELIKT was derived from the β-coefficients. We report the following performance measures: area under the curve, sensitivity, specificity and odds ratio.

RESULTS

Both cohorts were similar and included over 10,000 patients each (mean age 77.6 ± 7.6 years) with 11% experiencing delirium. The model included nine variables: age, medical department, dementia, hemi-/paraplegia, catheterisation, potassium, creatinine, polypharmacy and the anticholinergic burden measured with the Clinician-rated Anticholinergic Scale (CrAS). The external validation yielded an AUC of 0.795. With a cut-off at 20 points in the DELIKT, we received a sensitivity of 79.7%, specificity of 62.3% and an odds ratio of 5.9 (95% CI 5.2, 6.7).

CONCLUSION

The DELIKT is a potentially automatic tool with predictors from standard care including the CrAS to identify patients at high risk for delirium.

External validity of an automated delirium prediction model (DEMO) and comparison to the manual VMS-questions: a retrospective cohort study

BACKGROUND

It is estimated that one-third of delirium cases in hospitals could be prevented with appropriate interventions. In Dutch hospitals a manual instrument (VMS-questions) is used to identify patients at-risk for delirium. Delirium Model (DEMO) is an automated model which could support delirium prevention more efficiently. However, it has not been validated beyond the hospital it was developed in.

AIM

To externally validate the DEMO and compare its performance to the VMS-questions.

METHOD

A retrospective cohort study between July and December 2018 was conducted. Delirium cases were identified through a chart review, and the VMS-questions were extracted from the electronic health records. The DEMO was validated in patients ≥ 60 years, and a comparison with the VMS-questions was made in patients ≥ 70 years.

RESULTS

In total 1,345 admissions were included. The DEMO predicted 59 out of 75 delirium cases (sensitivity 0.79, 95% CI = 0.68-0.87; specificity 0.75, 95% CI = 0.72-0.77). Compared to the VMS-questions, the DEMO showed a lower specificity (0.64 vs. 0.72; p < 0.001) and a comparable sensitivity (0.83 vs. 0.80; p = 0.56). The VMS-questions were missing in 20% of admissions, in which the DEMO correctly predicted 10 of 12 delirium cases.

CONCLUSION

The DEMO showed acceptable performance for delirium prediction. Overall the DEMO predicted more delirium cases because the VMS-questions were missing in 20% of admissions. This study shows that automated instruments such as DEMO could play a key role in the efficient and timely deployment of measures to prevent delirium.

Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium

BACKGROUND

Prognostic models of hospital-induced delirium, that include potential predisposing and precipitating factors, may be used to identify vulnerable patients and inform the implementation of tailored preventive interventions. It is recommended that, in prediction model development studies, candidate predictors are selected on the basis of existing knowledge, including knowledge from clinical practice. The purpose of this article is to describe the process of identifying and operationalizing candidate predictors of hospital-induced delirium for application in a prediction model development study using a practice-based approach.

METHODS

This study is part of a larger, retrospective cohort study that is developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. First, we conducted a review of the literature to identify clinical concepts that had been used as candidate predictors in prognostic model development-and-validation studies of hospital-induced delirium. Then, we consulted a multidisciplinary task force of nine members who independently judged whether each clinical concept was associated with hospital-induced delirium. Finally, we mapped the clinical concepts to the administrative and electronic health records and operationalized our candidate predictors.

RESULTS

In the review of 34 studies, we identified 504 unique clinical concepts. Two-thirds of the clinical concepts (337/504) were used as candidate predictors only once. The most common clinical concepts included age (31/34), sex (29/34), and alcohol use (22/34). 96% of the clinical concepts (484/504) were judged to be associated with the development of hospital-induced delirium by at least two members of the task force. All of the task force members agreed that 47 or 9% of the 504 clinical concepts were associated with hospital-induced delirium.

CONCLUSIONS

Heterogeneity among candidate predictors of hospital-induced delirium in the literature suggests a still evolving list of factors that contribute to the development of this complex phenomenon. We demonstrated a practice-based approach to variable selection for our model development study of hospital-induced delirium. Expert judgement of variables enabled us to categorize the variables based on the amount of agreement among the experts and plan for the development of different models, including an expert-model and data-driven model.

Risk of bias in prognostic models of hospital-induced delirium for medical-surgical units: A systematic review

PURPOSE

The purpose of this systematic review was to assess risk of bias in existing prognostic models of hospital-induced delirium for medical-surgical units.

METHODS

APA PsycInfo, CINAHL, MEDLINE, and Web of Science Core Collection were searched on July 8, 2022, to identify original studies which developed and validated prognostic models of hospital-induced delirium for adult patients who were hospitalized in medical-surgical units. The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies was used for data extraction. The Prediction Model Risk of Bias Assessment Tool was used to assess risk of bias. Risk of bias was assessed across four domains: participants, predictors, outcome, and analysis.

RESULTS

Thirteen studies were included in the qualitative synthesis, including ten model development and validation studies and three model validation only studies. The methods in all of the studies were rated to be at high overall risk of bias. The methods of statistical analysis were the greatest source of bias. External validity of models in the included studies was tested at low levels of transportability.

CONCLUSIONS

Our findings highlight the ongoing scientific challenge of developing a valid prognostic model of hospital-induced delirium for medical-surgical units to tailor preventive interventions to patients who are at high risk of this iatrogenic condition. With limited knowledge about generalizable prognosis of hospital-induced delirium in medical-surgical units, existing prognostic models should be used with caution when creating clinical practice policies. Future research protocols must include robust study designs which take into account the perspectives of clinicians to identify and validate risk factors of hospital-induced delirium for accurate and generalizable prognosis in medical-surgical units.

Delirium screening in an acute care setting with a machine learning classifier based on routinely collected nursing data: A model development study

Delirium screening in acute care settings is a resource intensive process with frequent deviations from screening protocols. A predictive model relying only on daily collected nursing data for delirium screening could expand the populations covered by such screening programs. Here, we present the results of the development and validation of a series of machine-learning based delirium prediction models. For this purpose, we used data of all patients 18 years or older which were hospitalized for more than a day between January 1, 2014, and December 31, 2018, at a single tertiary teaching hospital in Zurich, Switzerland. A total of 48,840 patients met inclusion criteria. 18,873 (38.6%) were excluded due to missing data. Mean age (SD) of the included 29,967 patients was 71.1 (12.2) years and 12,231 (40.8%) were women. Delirium was assessed with the Delirium Observation Scale (DOS) with a total score of 3 or greater indicating that a patient is at risk for delirium. Additional measures included structured data collected for nursing process planning and demographic characteristics. The performance of the machine learning models was assessed using the area under the receiver operating characteristic curve (AUC). The training set consisted of 21,147 patients (mean age 71.1 (12.1) years; 8,630 (40.8%) women|) including 233,024 observations with 16,167 (6.9%) positive DOS screens. The test set comprised 8,820 patients (median age 71.1 (12.4) years; 3,601 (40.8%) women) with 91,026 observations with 5,445 (6.0%) positive DOS screens. Overall, the gradient boosting machine model performed best with an AUC of 0.933 (95% CI, 0.929 - 0.936). In conclusion, machine learning models based only on structured nursing data can reliably predict patients at risk for delirium in an acute care setting. Prediction models, using existing data collection processes, could reduce the resources required for delirium screening procedures in clinical practice.

A quality improvement project addressing the underreporting of delirium in hip fracture patients

INTRODUCTION

After discovering a low incidence of delirium for hip fracture patients at our institution, we evaluated if this was due to underreporting and, if so, where process errors occurred.

METHODS

Hip fracture patients aged ≥60 with a diagnosis of delirium were identified. Chart-Based Delirium Identification Instrument (CHART-DEL) identified missed diagnoses of delirium. Process maps were created based off staff interviews and observations.

RESULTS

The incidence of delirium was 15.3% (N = 176). Within a random sample (n = 98), 15 patients (15.5%) were diagnosed, while 20 (24.7%) went undiagnosed despite evidence of delirium. Including missed diagnoses, delirium prevalence was higher in the sample compared to all patients (35.7% vs 15.3%, p < 0.001). Most missed diagnoses were due to failure in identifying delirium (60%) or failure in documenting/coding diagnosis (20%). The prevalence of baseline cognitive impairment was higher in undiagnosed delirium patients versus correctly diagnosed patients (80% vs 20%, p = 0.001).

CONCLUSIONS

Our institution significantly underreports delirium among hip fracture patients mainly due to; (1) failure to identify delirium by the clinical staff, and (2) failure to document/code diagnosis despite correct identification. Baseline cognitive impairment can render delirium diagnosis challenging. These serve as targets for quality improvement and hip fracture care enhancement.

Head-to-head comparison of 14 prediction models for postoperative delirium in elderly non-ICU patients: an external validation study

OBJECTIVES

Delirium is associated with increased morbidity, mortality, prolonged hospitalisation and increased healthcare costs. The number of clinical prediction models (CPM) to predict postoperative delirium has increased exponentially. Our goal is to perform a head-to-head comparison of CPMs predicting postoperative delirium in non-intensive care unit (non-ICU) elderly patients to identify the best performing models.

SETTING

Single-site university hospital.

DESIGN

Secondary analysis of prospective cohort study.

PARTICIPANTS AND INCLUSION

CPMs published within the timeframe of 1 January 1990 to 1 May 2020 were checked for eligibility (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). For the time period of 1 January 1990 to 1 January 2017, included CPMs were identified in systematic reviews based on prespecified inclusion and exclusion criteria. An extended literature search for original studies was performed independently by two authors, including CPMs published between 1 January 2017 and 1 May 2020. External validation was performed using a surgical cohort consisting of 292 elderly non-ICU patients.

PRIMARY OUTCOME MEASURES

Discrimination, calibration and clinical usefulness.

RESULTS

14 CPMs were eligible for analysis out of 366 full texts reviewed. External validation was previously published for 8/14 (57%) CPMs. C-indices ranged from 0.52 to 0.74, intercepts from -0.02 to 0.34, slopes from -0.74 to 1.96 and scaled Brier from -1.29 to 0.088. Based on predefined criteria, the two best performing models were those of Dai et al (c-index: 0.739; (95% CI: 0.664 to 0.813); intercept: -0.018; slope: 1.96; scaled Brier: 0.049) and Litaker et al (c-index: 0.706 (95% CI: 0.590 to 0.823); intercept: -0.015; slope: 0.995; scaled Brier: 0.088). For the remaining CPMs, model discrimination was considered poor with corresponding c-indices <0.70.

CONCLUSION

Our head-to-head analysis identified 2 out of 14 CPMs as best-performing models with a fair discrimination and acceptable calibration. Based on our findings, these models might assist physicians in postoperative delirium risk estimation and patient selection for preventive measures.

Machine Learning to Develop and Internally Validate a Predictive Model for Post-operative Delirium in a Prospective, Observational Clinical Cohort Study of Older Surgical Patients

BACKGROUND

Our objective was to assess the performance of machine learning methods to predict post-operative delirium using a prospective clinical cohort.

METHODS

We analyzed data from an observational cohort study of 560 older adults (≥ 70 years) without dementia undergoing major elective non-cardiac surgery. Post-operative delirium was determined by the Confusion Assessment Method supplemented by a medical chart review (N = 134, 24%). Five machine learning algorithms and a standard stepwise logistic regression model were developed in a training sample (80% of participants) and evaluated in the remaining hold-out testing sample. We evaluated three overlapping feature sets, restricted to variables that are readily available or minimally burdensome to collect in clinical settings, including interview and medical record data. A large feature set included 71 potential predictors. A smaller set of 18 features was selected by an expert panel using a consensus process, and this smaller feature set was considered with and without a measure of pre-operative mental status.

RESULTS

The area under the receiver operating characteristic curve (AUC) was higher in the large feature set conditions (range of AUC, 0.62-0.71 across algorithms) versus the selected feature set conditions (AUC range, 0.53-0.57). The restricted feature set with mental status had intermediate AUC values (range, 0.53-0.68). In the full feature set condition, algorithms such as gradient boosting, cross-validated logistic regression, and neural network (AUC = 0.71, 95% CI 0.58-0.83) were comparable with a model developed using traditional stepwise logistic regression (AUC = 0.69, 95% CI 0.57-0.82). Calibration for all models and feature sets was poor.

CONCLUSIONS

We developed machine learning prediction models for post-operative delirium that performed better than chance and are comparable with traditional stepwise logistic regression. Delirium proved to be a phenotype that was difficult to predict with appreciable accuracy.

Derivation, Validation, Sustained Performance, and Clinical Impact of an Electronic Medical Record-Based Perioperative Delirium Risk Stratification Tool

BACKGROUND

Postoperative delirium is an important problem for surgical inpatients and was the target of a multidisciplinary quality improvement project at our institution. We developed and tested a semiautomated delirium risk stratification instrument, Age, WORLD backwards, Orientation, iLlness severity, Surgery-specific risk (AWOL-S), in 3 independent cohorts from our tertiary care hospital and describe its performance characteristics and impact on clinical care.

METHODS

The risk stratification instrument was derived with elective surgical patients who were admitted at least overnight and received at least 1 postoperative delirium screen (Nursing Delirium Screening Scale [NuDESC] or Confusion Assessment Method for the Intensive Care Unit [CAM-ICU]) and preoperative cognitive screening tests (orientation to place and ability to spell WORLD backward). Using data pragmatically collected between December 7, 2016, and June 15, 2017, we derived a logistic regression model predicting probability of delirium in the first 7 postoperative hospital days. A priori predictors included age, cognitive screening, illness severity or American Society of Anesthesiologists physical status, and surgical delirium risk. We applied model odds ratios to 2 subsequent cohorts ("validation" and "sustained performance") and assessed performance using area under the receiver operator characteristic curves (AUC-ROC). A post hoc sensitivity analysis assessed performance in emergency and preadmitted patients. Finally, we retrospectively evaluated the use of benzodiazepines and anticholinergic medications in patients who screened at high risk for delirium.

RESULTS

The logistic regression model used to derive odds ratios for the risk prediction tool included 2091 patients. Model AUC-ROC was 0.71 (0.67-0.75), compared with 0.65 (0.58-0.72) in the validation (n = 908) and 0.75 (0.71-0.78) in the sustained performance (n = 3168) cohorts. Sensitivity was approximately 75% in the derivation and sustained performance cohorts; specificity was approximately 59%. The AUC-ROC for emergency and preadmitted patients was 0.71 (0.67-0.75; n = 1301). After AWOL-S was implemented clinically, patients at high risk for delirium (n = 3630) had 21% (3%-36%) lower relative risk of receiving an anticholinergic medication perioperatively after controlling for secular trends.

CONCLUSIONS

The AWOL-S delirium risk stratification tool has moderate accuracy for delirium prediction in a cohort of elective surgical patients, and performance is largely unchanged in emergent/preadmitted surgical patients. Using AWOL-S risk stratification as a part of a multidisciplinary delirium reduction intervention was associated with significantly lower rates of perioperative anticholinergic but not benzodiazepine, medications in those at high risk for delirium. AWOL-S offers a feasible starting point for electronic medical record-based postoperative delirium risk stratification and may serve as a useful paradigm for other institutions.

Delirium prediction in the intensive care unit: a temporal approach

The incidence of delirium in intensive care units is high and associated with poor outcomes; therefore, its prediction is desirable to establish preventive treatments. This retrospective study proposes a novel approach for delirium prediction. We analyzed static and temporal data from 10,475 patients admitted to one of 15 intensive care units (ICUs) in Alberta, Canada between January 1, 2014 and June 30, 2016. We tested 168 different combinations of study design parameters and five different predictive models (logistic regression, support vector machines, random forests, adaptive boosting and neural networks). The area under the receiver operating characteristic curve (AUROC) ranged from 0.754 (CI 95% ± 0.018) to 0.852 (± 0.033), with sensitivity and specificity respectively ranging from 0.739 (CI 95% ± 0.047) to 0.840 (CI 95% ± 0.064), and 0.770 (CI 95% ± 0.030) to 0.865 (CI 95% ± 0.038). These results are similar to previous studies; however, our approach allows for continuous updates and short-term prediction horizons which might provide major advantages.

Development of a Risk Score to Predict Postoperative Delirium in Patients With Hip Fracture

BACKGROUND

Post-hip fracture surgery delirium (PHFD) is a significant clinical problem in older patients, but an adequate, simple risk prediction model for use in the preoperative period has not been developed.

METHODS

The 2016 American College of Surgeons National Surgical Quality Improvement Program Hip Fracture Procedure Targeted Participant Use Data File was used to obtain a cohort of patients ≥60 years of age who underwent hip fracture surgery (n = 8871; randomly assigned to derivation [70%] or validation [30%] cohorts). A parsimonious prediction model for PHFD was developed in the derivation cohort using stepwise multivariable logistic regression with further removal of variables by evaluating changes in the area under the receiver operator characteristic curve (AUC). A risk score was developed from the final multivariable model.

RESULTS

Of 6210 patients in the derivation cohort, PHFD occurred in 1816 (29.2%). Of 32 candidate variables, 9 were included in the final model: (1) preoperative delirium (adjusted odds ratio [aOR], 8.32 [95% confidence interval {CI}, 6.78-10.21], 8 risk score points); (2) preoperative dementia (aOR, 2.38 [95% CI, 2.05-2.76], 3 points); (3) age (reference, 60-69 years of age) (age 70-79: aOR, 1.60 [95% CI, 1.20-2.12], 2 points; age 80-89: aOR, 2.09 [95% CI, 1.59-2.74], 2 points; and age ≥90: aOR, 2.43 [95% CI, 1.82-3.23], 3 points); (4) medical comanagement (aOR, 1.43 [95% CI, 1.13-1.81], 1 point); (5) American Society of Anesthesiologists (ASA) physical status III-V (aOR, 1.40 [95% CI, 1.14-1.73], 1 point); (6) functional dependence (aOR, 1.37 [95% CI, 1.17-1.61], 1 point); (7) smoking (aOR, 1.36 [95% CI, 1.07-1.72], 1 point); (8) systemic inflammatory response syndrome/sepsis/septic shock (aOR, 1.34 [95% CI, 1.09-1.65], 1 point); and (9) preoperative use of mobility aid (aOR, 1.32 [95% CI, 1.14-1.52], 1 point), resulting in a risk score ranging from 0 to 20 points. The AUCs of the logistic regression and risk score models were 0.77 (95% CI, 0.76-0.78) and 0.77 (95% CI, 0.76-0.78), respectively, with similar results in the validation cohort.

CONCLUSIONS

A risk score based on 9 preoperative risk factors can predict PHFD in older adult patients with fairly good accuracy.

Predicting medication-associated altered mental status in hospitalized patients: Development and validation of a risk model

PURPOSE

This study presents a medication-associated altered mental status (AMS) risk model for real-time implementation in inpatient electronic health record (EHR) systems.

METHODS

We utilized a retrospective cohort of patients admitted to 2 large hospitals between January 2012 and October 2013. The study population included admitted patients aged ≥18 years with exposure to an AMS risk-inducing medication within the first 5 hospitalization days. AMS events were identified by a measurable mental status change documented in the EHR in conjunction with the administration of an atypical antipsychotic or haloperidol. AMS risk factors and AMS risk-inducing medications were identified from the literature, drug information databases, and expert opinion. We used multivariate logistic regression with a full and backward eliminated set of risk factors to predict AMS. The final model was validated with 100 bootstrap samples.

RESULTS

During 194,156 at-risk days for 66,875 admissions, 262 medication-associated AMS events occurred (an event rate of 0.13%). The strongest predictors included a history of AMS (odds ratio [OR], 9.55; 95% confidence interval [CI], 5.64-16.17), alcohol withdrawal (OR, 3.34; 95% CI, 2.18-5.13), history of delirium or psychosis (OR, 3.25; 95% CI, 2.39-4.40), presence in the intensive care unit (OR, 2.53; 95% CI, 1.89-3.39), and hypernatremia (OR, 2.40; 95% CI, 1.61-3.56). With a C statistic of 0.85, among patients scoring in the 90th percentile, our model captured 159 AMS events (60.7%).

CONCLUSION

The risk model was demonstrated to have good predictive ability, with all risk factors operationalized from discrete EHR fields. The real-time identification of higher-risk patients would allow pharmacists to prioritize surveillance, thus allowing early management of precipitating factors.

Delirium risk in non-surgical patients: systematic review of predictive tools

OBJECTIVE

Delirium is a common, serious condition associated with poor hospital outcomes. Guidelines recommend screening for delirium risk to target diagnostic and/or prevention strategies. This study critically reviews multicomponent delirium risk prediction tools in adult non-surgical inpatients.

STUDY DESIGN

Systematic review of studies incorporating at least two clinical factors in a multicomponent tool predicting risk of delirium during hospital admission. Derivation and validation studies were included. Study design, risk factors and tool performance were extracted and tabulated, and study quality was assessed by CHARMS criteria.

DATA SOURCES

PubMed, Embase, PsycINFO, and Cumulative Index to Nursing Health Literature (CINAHL) to 11^th March 2018.

DATA SYNTHESIS

22 derivation studies enrolling 38,874 participants (9 with a validation component) and 4 additional validation studies were identified, from a range of ward types. All studies had at least moderate risk of bias. Older age and cognitive, functional and sensory impairment were important predisposing factors. Precipitating risk factors included infection, illness severity, renal and electrolyte disturbances. Tools mostly did not differentiate between predisposing and precipitating risk factors mathematically or conceptually Most tools showed fair to good discrimination, and identified more than half of older inpatients at risk.

CONCLUSIONS

Several validated delirium risk prediction tools can identify patients at increased risk of delirium, but do not provide clear advice for clinical application. Most recommended cut-points are sensitive but have low specificity. Implementation studies demonstrating how risk screening can better direct clinical interventions in specific clinical settings are needed to define the potential value of these tools.

Development and Validation of an Electronic Health Record-Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment

IMPORTANCE

Current methods for identifying hospitalized patients at increased risk of delirium require nurse-administered questionnaires with moderate accuracy.

OBJECTIVE

To develop and validate a machine learning model that predicts incident delirium risk based on electronic health data available on admission.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective cohort study evaluating 5 machine learning algorithms to predict delirium using 796 clinical variables identified by an expert panel as relevant to delirium prediction and consistently available in electronic health records within 24 hours of admission. The training set comprised 14 227 adult patients with non-intensive care unit hospital stays and no delirium on admission who were discharged between January 1, 2016, and August 31, 2017, from UCSF Health, a large academic health institution. The test set comprised 3996 patients with hospital stays who were discharged between August 1, 2017, and November 30, 2017.

EXPOSURES

Patient demographic characteristics, diagnoses, nursing records, laboratory results, and medications available in electronic health records during hospitalization.

MAIN OUTCOMES AND MEASURES

Delirium was defined as a positive Nursing Delirium Screening Scale or Confusion Assessment Method for the Intensive Care Unit score. Models were assessed using the area under the receiver operating characteristic curve (AUC) and compared against the 4-point scoring system AWOL (age >79 years, failure to spell world backward, disorientation to place, and higher nurse-rated illness severity), a validated delirium risk-assessment tool routinely administered in this cohort.

RESULTS

The training set included 14 227 patients (5113 [35.9%] aged >64 years; 7335 [51.6%] female; 687 [4.8%] with delirium), and the test set included 3996 patients (1491 [37.3%] aged >64 years; 1966 [49.2%] female; 191 [4.8%] with delirium). In total, the analysis included 18 223 hospital admissions (6604 [36.2%] aged >64 years; 9301 [51.0%] female; 878 [4.8%] with delirium). The AWOL system achieved a baseline AUC of 0.678. The gradient boosting machine model performed best, with an AUC of 0.855. Setting specificity at 90%, the model had a 59.7% (95% CI, 52.4%-66.7%) sensitivity, 23.1% (95% CI, 20.5%-25.9%) positive predictive value, 97.8% (95% CI, 97.4%-98.1%) negative predictive value, and a number needed to screen of 4.8. Penalized logistic regression and random forest models also performed well, with AUCs of 0.854 and 0.848, respectively.

CONCLUSIONS AND RELEVANCE

Machine learning can be used to estimate hospital-acquired delirium risk using electronic health record data available within 24 hours of hospital admission. Such a model may allow more precise targeting of delirium prevention resources without increasing the burden on health care professionals.

Systematic review of prediction models for delirium in the older adult inpatient

OBJECTIVE

To identify existing prognostic delirium prediction models and evaluate their validity and statistical methodology in the older adult (≥60 years) acute hospital population.

DESIGN

Systematic review.

DATA SOURCES AND METHODS

PubMed, CINAHL, PsychINFO, SocINFO, Cochrane, Web of Science and Embase were searched from 1 January 1990 to 31 December 2016. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses and CHARMS Statement guided protocol development.

INCLUSION CRITERIA

age >60 years, inpatient, developed/validated a prognostic delirium prediction model.

EXCLUSION CRITERIA

alcohol-related delirium, sample size ≤50. The primary performance measures were calibration and discrimination statistics. Two authors independently conducted search and extracted data. The synthesis of data was done by the first author. Disagreement was resolved by the mentoring author.

RESULTS

The initial search resulted in 7,502 studies. Following full-text review of 192 studies, 33 were excluded based on age criteria (<60 years) and 27 met the defined criteria. Twenty-three delirium prediction models were identified, 14 were externally validated and 3 were internally validated. The following populations were represented: 11 medical, 3 medical/surgical and 13 surgical. The assessment of delirium was often non-systematic, resulting in varied incidence. Fourteen models were externally validated with an area under the receiver operating curve range from 0.52 to 0.94. Limitations in design, data collection methods and model metric reporting statistics were identified.

CONCLUSIONS

Delirium prediction models for older adults show variable and typically inadequate predictive capabilities. Our review highlights the need for development of robust models to predict delirium in older inpatients. We provide recommendations for the development of such models.

Prediction and early detection of delirium in the intensive care unit by using heart rate variability and machine learning

OBJECTIVE

Delirium is an important syndrome found in patients in the intensive care unit (ICU), however, it is usually under-recognized during treatment. This study was performed to investigate whether delirious patients can be successfully distinguished from non-delirious patients by using heart rate variability (HRV) and machine learning.

APPROACH

Electrocardiography data of 140 patients was acquired during daily ICU care, and HRV data were analyzed. Delirium, including its type, severity, and etiologies, was evaluated daily by trained psychiatrists. HRV data and various machine learning algorithms including linear support vector machine (SVM), SVM with radial basis function (RBF) kernels, linear extreme learning machine (ELM), ELM with RBF kernels, linear discriminant analysis, and quadratic discriminant analysis were utilized to distinguish delirium patients from non-delirium patients.

MAIN RESULTS

HRV data of 4797 ECGs were included, and 39 patients had delirium at least once during their ICU stay. The maximum classification accuracy was acquired using SVM with RBF kernels. Our prediction method based on HRV with machine learning was comparable to previous delirium prediction models using massive amounts of clinical information.

SIGNIFICANCE

Our results show that autonomic alterations could be a significant feature of patients with delirium in the ICU, suggesting the potential for the automatic prediction and early detection of delirium based on HRV with machine learning.

Validation of an automated delirium prediction model (DElirium MOdel (DEMO)): an observational study

OBJECTIVES

Delirium is an underdiagnosed, severe and costly disorder, and 30%-40% of cases can be prevented. A fully automated model to predict delirium (DEMO) in older people has been developed, and the objective of this study is to validate the model in a hospital setting.

SETTING

Secondary care, one hospital with two locations.

DESIGN

Observational study.

PARTICIPANTS

The study included 450 randomly selected patients over 60 years of age admitted to Zuyderland Medical Centre. Patients who presented with delirium on admission were excluded.

PRIMARY OUTCOME MEASURES

Development of delirium through chart review.

RESULTS

A total of 383 patients were included in this study. The analysis was performed for delirium within 1, 3 and 5 days after a DEMO score was obtained. Sensitivity was 87.1% (95% CI 0.756 to 0.939), 84.2% (95% CI 0.732 to 0.915) and 82.7% (95% CI 0.734 to 0.893) for 1, 3 and 5 days, respectively, after obtaining the DEMO score. Specificity was 77.9% (95% CI 0.729 to 0.882), 81.5% (95% CI 0.766 to 0.856) and 84.5% (95% CI 0.797 to 0.884) for 1, 3 and 5 days, respectively, after obtaining the DEMO score.

CONCLUSION

DEMO is a satisfactory prediction model but needs further prospective validation with in-person delirium confirmation. In the future, DEMO will be applied in clinical practice so that physicians will be aware of when a patient is at an increased risk of developing delirium, which will facilitate earlier recognition and diagnosis, and thus will allow the implementation of prevention measures.