MEWS++: Enhancing the Prediction of Clinical Deterioration in Admitted Patients through a Machine Learning Model

Remote Monitoring and Artificial Intelligence: Outlook for 2050

Remote monitoring and artificial intelligence will become common and intertwined in anesthesiology by 2050. In the intraoperative period, technology will lead to the development of integrated monitoring systems that will integrate multiple data streams and allow anesthesiologists to track patients more effectively. This will free up anesthesiologists to focus on more complex tasks, such as managing risk and making value-based decisions. This will also enable the continued integration of remote monitoring and control towers having profound effects on coverage and practice models. In the PACU and ICU, the technology will lead to the development of early warning systems that can identify patients who are at risk of complications, enabling early interventions and more proactive care. The integration of augmented reality will allow for better integration of diverse types of data and better decision-making. Postoperatively, the proliferation of wearable devices that can monitor patient vital signs and track their progress will allow patients to be discharged from the hospital sooner and receive care at home. This will require increased use of telemedicine, which will allow patients to consult with doctors remotely. All of these advances will require changes to legal and regulatory frameworks that will enable new workflows that are different from those familiar to today's providers.

Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain

OBJECTIVE

To identify factors influencing implementation of machine learning algorithms (MLAs) that predict clinical deterioration in hospitalized adult patients and relate these to a validated implementation framework.

MATERIALS AND METHODS

A systematic review of studies of implemented or trialed real-time clinical deterioration prediction MLAs was undertaken, which identified: how MLA implementation was measured; impact of MLAs on clinical processes and patient outcomes; and barriers, enablers and uncertainties within the implementation process. Review findings were then mapped to the SALIENT end-to-end implementation framework to identify the implementation stages at which these factors applied.

RESULTS

Thirty-seven articles relating to 14 groups of MLAs were identified, each trialing or implementing a bespoke algorithm. One hundred and seven distinct implementation evaluation metrics were identified. Four groups reported decreased hospital mortality, 1 significantly. We identified 24 barriers, 40 enablers, and 14 uncertainties and mapped these to the 5 stages of the SALIENT implementation framework.

DISCUSSION

Algorithm performance across implementation stages decreased between in silico and trial stages. Silent plus pilot trial inclusion was associated with decreased mortality, as was the use of logistic regression algorithms that used less than 39 variables. Mitigation of alert fatigue via alert suppression and threshold configuration was commonly employed across groups.

CONCLUSIONS

: There is evidence that real-world implementation of clinical deterioration prediction MLAs may improve clinical outcomes. Various factors identified as influencing success or failure of implementation can be mapped to different stages of implementation, thereby providing useful and practical guidance for implementers.

Development and Validation of a Machine Learning Algorithm Using Clinical Pages to Predict Imminent Clinical Deterioration

BACKGROUND

Early detection of clinical deterioration among hospitalized patients is a clinical priority for patient safety and quality of care. Current automated approaches for identifying these patients perform poorly at identifying imminent events.

OBJECTIVE

Develop a machine learning algorithm using pager messages sent between clinical team members to predict imminent clinical deterioration.

DESIGN

We conducted a large observational study using long short-term memory machine learning models on the content and frequency of clinical pages.

PARTICIPANTS

We included all hospitalizations between January 1, 2018 and December 31, 2020 at Vanderbilt University Medical Center that included at least one page message to physicians. Exclusion criteria included patients receiving palliative care, hospitalizations with a planned intensive care stay, and hospitalizations in the top 2% longest length of stay.

MAIN MEASURES

Model classification performance to identify in-hospital cardiac arrest, transfer to intensive care, or Rapid Response activation in the next 3-, 6-, and 12-hours. We compared model performance against three common early warning scores: Modified Early Warning Score, National Early Warning Score, and the Epic Deterioration Index.

KEY RESULTS

There were 87,783 patients (mean [SD] age 54.0 [18.8] years; 45,835 [52.2%] women) who experienced 136,778 hospitalizations. 6214 hospitalized patients experienced a deterioration event. The machine learning model accurately identified 62% of deterioration events within 3-hours prior to the event and 47% of events within 12-hours. Across each time horizon, the model surpassed performance of the best early warning score including area under the receiver operating characteristic curve at 6-hours (0.856 vs. 0.781), sensitivity at 6-hours (0.590 vs. 0.505), specificity at 6-hours (0.900 vs. 0.878), and F-score at 6-hours (0.291 vs. 0.220).

CONCLUSIONS

Machine learning applied to the content and frequency of clinical pages improves prediction of imminent deterioration. Using clinical pages to monitor patient acuity supports improved detection of imminent deterioration without requiring changes to clinical workflow or nursing documentation.

Machine learning as a tool to identify inpatients who are not at risk of adverse drug events in a large dataset of a tertiary care hospital in the USA

AIMS

Adverse drug events (ADEs) are a major threat to inpatients in the United States of America (USA). It is unknown how well machine learning (ML) is able to predict whether or not a patient will suffer from an ADE during hospital stay based on data available at hospital admission for emergency department patients of all ages (binary classification task). It is further unknown whether ML is able to outperform logistic regression (LR) in doing so, and which variables are the most important predictors.

METHODS

In this study, 5 ML models- namely a random forest, gradient boosting machine (GBM), ridge regression, least absolute shrinkage and selection operator (LASSO) regression, and elastic net regression-as well as a LR were trained and tested for the prediction of inpatient ADEs identified using ICD-10-CM codes based on comprehensive previous work in a diverse population. In total, 210 181 observations from patients who were admitted to a large tertiary care hospital after emergency department stay between 2011 and 2019 were included. The area under the receiver operating characteristics curve (AUC) and AUC-precision-recall (AUC-PR) were used as primary performance indicators.

RESULTS

Tree-based models performed best with respect to AUC and AUC-PR. The gradient boosting machine (GBM) reached an AUC of 0.747 (95% confidence interval (CI): 0.735 to 0.759) and an AUC-PR of 0.134 (95% CI: 0.131 to 0.137) on unforeseen test data, while the random forest reached an AUC of 0.743 (95% CI: 0.731 to 0.755) and an AUC-PR of 0.139 (95% CI: 0.135 to 0.142), respectively. ML statistically significantly outperformed LR both on AUC and AUC-PR. Nonetheless, overall, models did not differ much with respect to their performance. Most important predictors were admission type, temperature and chief complaint for the best performing model (GBM).

CONCLUSIONS

The study demonstrated a first application of ML to predict inpatient ADEs based on ICD-10-CM codes, and a comparison with LR. Future research should address concerns arising from low precision and related problems.

Evaluation of machine learning-based models for prediction of clinical deterioration: A systematic literature review

BACKGROUND AND OBJECTIVE

Early identification of patients at risk of deterioration can prevent life-threatening adverse events and shorten length of stay. Although there are numerous models applied to predict patient clinical deterioration, most are based on vital signs and have methodological shortcomings that are not able to provide accurate estimates of deterioration risk. The aim of this systematic review is to examine the effectiveness, challenges, and limitations of using machine learning (ML) techniques to predict patient clinical deterioration in hospital settings.

METHODS

A systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and meta-Analyses (PRISMA) guidelines using EMBASE, MEDLINE Complete, CINAHL Complete, and IEEExplore databases. Citation searching was carried out for studies that met inclusion criteria. Two reviewers used the inclusion/exclusion criteria to independently screen studies and extract data. To address any discrepancies in the screening process, the two reviewers discussed their findings and a third reviewer was consulted as needed to reach a consensus. Studies focusing on use of ML in predicting patient clinical deterioration that were published from inception to July 2022 were included.

RESULTS

A total of 29 primary studies that evaluated ML models to predict patient clinical deterioration were identified. After reviewing these studies, we found that 15 types of ML techniques have been employed to predict patient clinical deterioration. While six studies used a single technique exclusively, several others utilised a combination of classical techniques, unsupervised and supervised learning, as well as other novel techniques. Depending on which ML model was applied and the type of input features, ML models predicted outcomes with an area under the curve from 0.55 to 0.99.

CONCLUSIONS

Numerous ML methods have been employed to automate the identification of patient deterioration. Despite these advancements, there is still a need for further investigation to examine the application and effectiveness of these methods in real-world situations.

A protocol for a systematic review of electronic early warning/track-and-trigger systems (EW/TTS) to predict clinical deterioration: Focus on automated features, technologies, and algorithms

BACKGROUND

This is a systematic review protocol to identify automated features, applied technologies, and algorithms in the electronic early warning/track and triage system (EW/TTS) developed to predict clinical deterioration (CD).

METHODOLOGY

This study will be conducted using PubMed, Scopus, and Web of Science databases to evaluate the features of EW/TTS in terms of their automated features, technologies, and algorithms. To this end, we will include any English articles reporting an EW/TTS without time limitation. Retrieved records will be independently screened by two authors and relevant data will be extracted from studies and abstracted for further analysis. The included articles will be evaluated independently using the JBI critical appraisal checklist by two researchers.

DISCUSSION

This study is an effort to address the available automated features in the electronic version of the EW/TTS to shed light on the applied technologies, automated level of systems, and utilized algorithms in order to smooth the road toward the fully automated EW/TTS as one of the potential solutions of prevention CD and its adverse consequences.

TRIAL REGISTRATION

Systematic review registration: PROSPERO CRD42022334988.

Similarity of expert clinicians’ rank order of differential diagnoses in a newborn resuscitation context

Background

Methods

Results

Conclusions

Early Prediction of All-Cause Clinical Deterioration in General Wards Patients: Development and Validation of a Biomarker-Based Machine Learning Model Derived From Rapid Response Team Activations

OBJECTIVE

The aim of the study is to evaluate the performance of a biomarker-based machine learning (ML) model (not including vital signs) derived from reviewed rapid response team (RRT) activations in predicting all-cause deterioration in general wards patients.

DESIGN

This is a retrospective single-institution study. All consecutive adult patients' cases on noncritical wards identified by RRT calls occurring at least 24 hours after patient admission, between April 2018 and June 2020, were included. The cases were reviewed and labeled for clinical deterioration by a multidisciplinary expert consensus panel. A supervised learning approach was adopted based on a set of biomarkers and demographic data available in the patient's electronic medical record (EMR).

SETTING

The setting is a 250-bed tertiary university hospital with a basic EMR, with adult (>18 y) patients on general wards.

PATIENTS

The study analyzed the cases of 514 patients for which the RRT was activated. Rapid response teams were extracted from the hospital telephone log data. Two hundred eighteen clinical deterioration cases were identified in these patients after expert chart review and complemented by 146 "nonevent" cases to build the training and validation data set.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The best performance was achieved with the random forests algorithm, with a maximal area under the receiver operating curve of 0.90 and F1 score of 0.85 obtained at prediction time T0-6h, slightly decreasing but still acceptable (area under the receiver operating curve, >0.8; F1 score, >0.75) at T0-42h. The system outperformed most classical track-and-trigger systems both in terms of prediction performance and prediction horizon.

CONCLUSIONS

In hospitals with a basic EMR, a biomarker-based ML model could be used to predict clinical deterioration in general wards patients earlier than classical track-and-trigger systems, thus enabling appropriate clinical interventions for patient safety and improved outcomes.

Implementing a Machine Learning Screening Tool for Malnutrition: Insights from Qualitative Research Applicable to Other ML-Based CDSS (Preprint)

BACKGROUND

Machine learning (ML)-based clinical decision support systems (CDSS) are popular in clinical practice settings but are often criticized for being limited in usability, interpretability, and effectiveness. Evaluating the implementation of ML-based CDSS is critical to ensure CDSS is acceptable and useful to clinicians and helps them deliver high-quality health care. Malnutrition is a common and underdiagnosed condition among hospital patients, which can have serious adverse impacts. Early identification and treatment of malnutrition are important.

OBJECTIVE

This study aims to evaluate the implementation of an ML tool, Malnutrition Universal Screening Tool (MUST)-Plus, that predicts hospital patients at high risk for malnutrition and identify best implementation practices applicable to this and other ML-based CDSS.

METHODS

We conducted a qualitative postimplementation evaluation using in-depth interviews with registered dietitians (RDs) who use MUST-Plus output in their everyday work. After coding the data, we mapped emergent themes onto select domains of the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework.

RESULTS

We interviewed 17 of the 24 RDs approached (71%), representing 37% of those who use MUST-Plus output. Several themes emerged: (1) enhancements to the tool were made to improve accuracy and usability; (2) MUST-Plus helped identify patients that would not otherwise be seen; perceived usefulness was highest in the original site; (3) perceived accuracy varied by respondent and site; (4) RDs valued autonomy in prioritizing patients; (5) depth of tool understanding varied by hospital and level; (6) MUST-Plus was integrated into workflows and electronic health records; and (7) RDs expressed a desire to eventually have 1 automated screener.

CONCLUSIONS

Our findings suggest that continuous involvement of stakeholders at new sites given staff turnover is vital to ensure buy-in. Qualitative research can help identify the potential bias of ML tools and should be widely used to ensure health equity. Ongoing collaboration among CDSS developers, data scientists, and clinical providers may help refine CDSS for optimal use and improve the acceptability of CDSS in the clinical context.

Technologies used by nursing to predict clinical deterioration in hospitalized adults: a scoping review

OBJECTIVE

to map the early clinical deterioration technologies used in nurses' professional practice in the care of hospitalized adult patients.

METHODS

this is a scoping review, according to Joanna Briggs Institute Reviewer's Manual, which seeks to map the main technologies for detecting early clinical deterioration of hospitalized patients available for use by nurses, summarizing them and indicating gaps in knowledge to be investigated.

RESULTS

twenty-seven studies were found. The most present variables in the technologies were vital signs, urinary output, awareness and risk scales, clinical examination and nurses' judgment. The main outcomes were activation of rapid response teams, death, cardiac arrest and admission to critical care units.

FINAL CONSIDERATIONS

the study emphasizes the most accurate variables in patient clinical assessment, so that indicative signs of potential severity can be prioritized to guide health conducts aiming to intervene early in the face of ongoing clinical deterioration.

Patient Deterioration on General Care Units: A Concept Analysis

Patient deterioration is a phenomenon that occurs from the inability to recognize it or respond to a change in condition. Despite the published reports on recognizing a deteriorating patient on general care floors, a gap remains in the ability of nurses to describe the concept, affecting patient outcomes. Walker and Avant's approach was applied to analyze patient deterioration. The aim of this article was to explore and clarify the meaning of patient deterioration and identify attributes, antecedents, and consequences. The defining attributes were compared to early warning scores. An operational definition was developed and its value to nurses established.

Tecnologias utilizadas pela enfermagem para predição de deterioração clínica em adultos hospitalizados: revisão de escopo

RESUMO Objetivo: mapear as tecnologias de deterioração clínica precoce utilizadas na prática profissional do enfermeiro na assistência a pacientes adultos hospitalizados. Métodos: trata-se de scoping review, segundo Joanna Briggs Institute Reviewer’s Manual, que busca o mapeamento das principais tecnologias para detecção de deterioração clínica precoce de pacientes hospitalizados disponíveis de uso do enfermeiro, sumarizando-as e indicando lacunas no conhecimento a serem investigadas. Resultados: foram encontrados 27 estudos. As variáveis mais presentes nas tecnologias foram sinais vitais, débito urinário, escalas de consciência e riscos, exame clínico e julgamento do enfermeiro. Os principais desfechos foram acionamento de times de resposta rápida, morte, parada cardiorrespiratória e admissão em unidades de cuidados críticos. Considerações finais: o estudo enfatiza as variáveis mais acuradas na avaliação clínica do paciente, para que se possam priorizar sinais indicativos de potencial gravidade para guiar condutas em saúde visando intervir precocemente diante da deterioração clínica em curso.

Plasma host protein biomarkers correlating with increasing Mycobacterium tuberculosis infection activity prior to tuberculosis diagnosis in people living with HIV

BACKGROUND

Biomarkers correlating with Mycobacterium tuberculosis infection activity/burden in asymptomatic individuals are urgently needed to identify and treat those at highest risk for developing active tuberculosis (TB). Our main objective was to identify plasma host protein biomarkers that change over time prior to developing TB in people living with HIV (PLHIV).

METHODS

Using multiplex MRM-MS, we investigated host protein expressions from 2 years before until time of TB diagnosis in longitudinally collected (every 3-6 months) and stored plasma from PLHIV with incident TB, identified within a South African (SA) and US cohort. We performed temporal trend and discriminant analyses for proteins, and, to assure clinical relevance, we further compared protein levels at TB diagnosis to interferon-gamma release assay (IGRA; SA) or tuberculin-skin test (TST; US) positive and negative cohort subjects without TB. SA and US exploratory data were analyzed separately.

FINDINGS

We identified 15 proteins in the SA (n=30) and 10 in the US (n=24) incident TB subjects which both changed from 2 years prior until time of TB diagnosis after controlling for 10% false discovery rate, and were significantly different at time of TB diagnosis compared to non-TB subjects (p<0.01). Five proteins, CD14, A2GL, NID1, SCTM1, and A1AG1, overlapped between both cohorts. Furthermore, after cross-validation, panels of 5 - 12 proteins were able to predict TB up to two years before diagnosis.

INTERPRETATION

Host proteins can be biomarkers for increasing Mycobacterium tuberculosis infection activity/burden, incipient TB, and predict TB development in PLHIV.

FUNDING

NIH/NIAID AI117927, AI146329, and AI127173 to JMA.

Explainable machine learning model for predicting spontaneous bacterial peritonitis in cirrhotic patients with ascites

Spontaneous bacterial peritonitis (SBP) is a life-threatening complication in patients with cirrhosis. We aimed to develop an explainable machine learning model to achieve the early prediction and outcome interpretation of SBP. We used CatBoost algorithm to construct MODEL-1 with 46 variables. After dimensionality reduction, we constructed MODEL-2. We calculated and compared the sensitivity and negative predictive value (NPV) of MODEL-1 and MODEL-2. Finally, we used the SHAP (SHapley Additive exPlanations) method to provide insights into the model’s outcome or prediction. MODEL-2 (AUROC: 0.822; 95% confidence interval [CI] 0.783–0.856), liked MODEL-1 (AUROC: 0.822; 95% CI 0.784–0.856), could well predict the risk of SBP in cirrhotic ascites patients. The 6 most influential predictive variables were total protein, C-reactive protein, prothrombin activity, cholinesterase, lymphocyte ratio and apolipoprotein A1. For binary classifier, the sensitivity and NPV of MODEL-1 were 0.894 and 0.885, respectively, while for MODEL-2 they were 0.927 and 0.904, respectively. We applied CatBoost algorithm to establish a practical and explainable prediction model for risk of SBP in cirrhotic patients with ascites. We also identified 6 important variables closely related to the occurrence of SBP.

Predicting Patient Deterioration: A Review of Tools in the Digital Hospital Setting

BACKGROUND

Early warning tools identify patients at risk of deterioration in hospitals. Electronic medical records in hospitals offer real-time data and the opportunity to automate early warning tools and provide real-time, dynamic risk estimates.

OBJECTIVE

This review describes published studies on the development, validation, and implementation of tools for predicting patient deterioration in general wards in hospitals.

METHODS

An electronic database search of peer reviewed journal papers from 2008-2020 identified studies reporting the use of tools and algorithms for predicting patient deterioration, defined by unplanned transfer to the intensive care unit, cardiac arrest, or death. Studies conducted solely in intensive care units, emergency departments, or single diagnosis patient groups were excluded.

RESULTS

A total of 46 publications were eligible for inclusion. These publications were heterogeneous in design, setting, and outcome measures. Most studies were retrospective studies using cohort data to develop, validate, or statistically evaluate prediction tools. The tools consisted of early warning, screening, or scoring systems based on physiologic data, as well as more complex algorithms developed to better represent real-time data, deal with complexities of longitudinal data, and warn of deterioration risk earlier. Only a few studies detailed the results of the implementation of deterioration warning tools.

CONCLUSIONS

Despite relative progress in the development of algorithms to predict patient deterioration, the literature has not shown that the deployment or implementation of such algorithms is reproducibly associated with improvements in patient outcomes. Further work is needed to realize the potential of automated predictions and update dynamic risk estimates as part of an operational early warning system for inpatient deterioration.

Predicting in-hospital mortality in adult non-traumatic emergency department patients: a retrospective comparison of the Modified Early Warning Score (MEWS) and machine learning approach

Background

A feasible and accurate risk prediction systems for emergency department (ED) patients is urgently required. The Modified Early Warning Score (MEWS) is a wide-used tool to predict clinical outcomes in ED. Literatures showed that machine learning (ML) had better predictability in specific patient population than traditional scoring system. By analyzing a large multicenter dataset, we aim to develop a ML model to predict in-hospital morality of the adult non traumatic ED patients for different time stages, and comparing performance with other ML models and MEWS.

Methods

A retrospective observational cohort study was conducted in five Taiwan EDs including two tertiary medical centers and three regional hospitals. All consecutively adult (>17 years old) non-traumatic patients admit to ED during a 9-year period (January first, 2008 to December 31th, 2016) were included. Exclusion criteria including patients with (1) out-of-hospital cardiac arrest and (2) discharge against medical advice and transferred to other hospital (3) missing collect variables. The primary outcome was in-hospital mortality and were categorized into 6, 24, 72, 168 hours mortality. MEWS was calculated by systolic blood pressure, pulse rate, respiratory rate, body temperature, and level of consciousness. An ensemble supervised stacking ML model was developed and compared to sensitive and unsensitive Xgboost, Random Forest, and Adaboost. We conducted a performance test and examine both the area under the receiver operating characteristic (AUROC) and the area under the precision and recall curve (AUPRC) as the comparative measures.

Result

After excluding 182,001 visits (7.46%), study group was consisted of 24,37,326 ED visits. The dataset was split into 67% training data and 33% test data for ML model development. There was no statistically difference found in the characteristics between two groups. For the prediction of 6, 24, 72, 168 hours in-hospital mortality, the AUROC of MEW and ML mode was 0.897, 0.865, 0.841, 0.816 and 0.939, 0.928, 0.913, 0.902 respectively. The stacking ML model outperform other ML model as well. For the prediction of in-hospital mortality over 48-hours, AUPRC performance of MEWS drop below 0.1, while the AUPRC of ML mode was 0.317 in 6 hours and 0.2150 in 168 hours. For each time frame, ML model achieved statistically significant higher AUROC and AUPRC than MEWS (all P < 0.001). Both models showed decreasing prediction ability as time elapse, but there was a trend that the gap of AUROC values between two model increases gradually (P < 0.001). Three MEWS thresholds (score >3, >4, and >5) were determined as baselines for comparison, ML mode consistently showed improved or equally performance in sensitivity, PPV, NPV, but not in specific.

Conclusion

Stacking ML methods improve predicted in-hospital mortality than MEWS in adult non-traumatic ED patients, especially in the prediction of delayed mortality.

Using machine learning to improve the accuracy of patient deterioration predictions: Mayo Clinic Early Warning Score (MC-EWS)

OBJECTIVE

We aimed to develop a model for accurate prediction of general care inpatient deterioration.

MATERIALS AND METHODS

Training and internal validation datasets were built using 2-year data from a quaternary hospital in the Midwest. Model training used gradient boosting and feature engineering (clinically relevant interactions, time-series information) to predict general care inpatient deterioration (resuscitation call, intensive care unit transfer, or rapid response team call) in 24 hours. Data from a tertiary care hospital in the Southwest were used for external validation. C-statistic, sensitivity, positive predictive value, and alert rate were calculated for different cutoffs and compared with the National Early Warning Score. Sensitivity analysis evaluated prediction of intensive care unit transfer or resuscitation call.

RESULTS

Training, internal validation, and external validation datasets included 24 500, 25 784 and 53 956 hospitalizations, respectively. The Mayo Clinic Early Warning Score (MC-EWS) demonstrated excellent discrimination in both the internal and external validation datasets (C-statistic = 0.913, 0.937, respectively), and results were consistent in the sensitivity analysis (C-statistic = 0.932 in external validation). At a sensitivity of 73%, MC-EWS would generate 0.7 alerts per day per 10 patients, 45% less than the National Early Warning Score.

DISCUSSION

Low alert rates are important for implementation of an alert system. Other early warning scores developed for the general care ward have achieved lower discrimination overall compared with MC-EWS, likely because MC-EWS includes both nursing assessments and extensive feature engineering.

CONCLUSIONS

MC-EWS achieved superior prediction of general care inpatient deterioration using sophisticated feature engineering and a machine learning approach, reducing alert rate.

Validation of National Early Warning Score for predicting 30-day mortality after rapid response system activation in Japan

Aim

Although rapid response systems (RRS) are used to prevent adverse events, Japan reportedly has low activation rates and high mortality rates. The National Early Warning Score (NEWS) could provide a solution, but it has not been validated in Japan. We aimed to validate NEWS for Japanese patients.

Methods

This retrospective observational study included data of 2,255 adult patients from 33 facilities registered in the In‐Hospital Emergency Registry in Japan between January 2014 and March 2018. The primary evaluated outcome was mortality rate 30 days after RRS activation. Accuracy of NEWS was analyzed with the correlation coefficient and area under the receiver operating characteristic curve. Prediction weights of NEWS parameters were then analyzed using multiple logistic regression and a machine learning method, classification and regression trees.

Results

The correlation coefficient of NEWS for 30‐day mortality rate was 0.95 (95% confidence interval [CI], 0.88–0.98) and the area under the receiver operating characteristic curve was 0.668 (95% CI, 0.642–0.693). Sensitivity and specificity values with a cut‐off score of 7 were 89.8% and 45.1%, respectively. Regarding prediction values of each parameter, oxygen saturation showed the highest odds ratio of 1.36 (95% CI, 1.25–1.48), followed by altered mental status 1.23 (95% CI, 1.14–1.32), heart rate 1.21 (95% CI, 1.09–1.34), systolic blood pressure 1.12 (95% CI, 1.04–1.22), and respiratory rate 1.03 (95% CI, 1.05–1.26). Body temperature and oxygen supplementation were not significantly associated. Classification and regression trees showed oxygen saturation as the most heavily weighted parameter, followed by altered mental status and respiratory rate.

Conclusions

National Early Warning Score could stratify 30‐day mortality risk following RRS activation in Japanese patients.

Comparison of within 7 Day All-Cause Mortality among HDU Patients with Modified Early Warning Score of ≥5 with those with Score of <5

Objective

To compare 7-Day All-Cause Mortality among HDU Patients with Modified Early Warning Score of ≥5 with Those with Score of <5.

Methods

All patients of age more than 18 years, of either gender admitted in HDU of Medical Unit-II, CHK between September 2019 to February 2020 were included. MEWS was calculated for each patient at time of admission. Patients with MEWS score of ≥5 were allocated to Group-A and those with score of <5 were allocated to Group-B. Patients were followed for seven days and outcome status of alive, expired or discharged was noted.

Results

Total of 336 patients were selected out of which 168 patients was inducted in Group-A and 168 patients in Group-B. MEWS Score in patients who expired was significantly higher (Mdn=11) than in those who survived (Mdn=4), p <.001. 7-day mortality in Group-A was 62 (39.9%) while in Group-B was 40 (23.8%). ROC was plotted of MEWS Score for mortality, it showed significant area under curve of 68.4% (p <.001, 95% CI = .62 to .75). MEWS Score of 3.5 showed sensitivity of 89.2% and specificity of 65%.

Conclusion

Our results show that MEWS has a positive trend to predict mortality. MEWS score of 3.5 is suggested cut off based on ROC in our study.

Developing an Institute for Health Care Delivery Science: successes, challenges, and solutions in the first five years

Medical knowledge is increasing at an exponential rate. At the same time, unexplained variations in practice and patient outcomes and unacceptable rates of medical errors and inefficiencies in health care delivery have emerged. Our Institute for Health Care Delivery Science (I-HDS) began in 2014 as a novel platform to conduct multidisciplinary healthcare delivery research. We followed ten strategies to develop a successful institute with excellence in methodology and strong understanding of the value of team science. Our work was organized around five hubs: 1) Quality/Process Improvement and Systematic Review, 2) Comparative Effectiveness Research, Pragmatic Clinical Trials, and Predictive Analytics, 3) Health Economics and Decision Modeling, 4) Qualitative, Survey, and Mixed Methods, and 5) Training and Mentoring. In the first 5 years of the I-HDS, we have identified opportunities for change in clinical practice through research using our health system's electronic health record (EHR) data, and designed programs to educate clinicians in the value of research to improve patient care and recognize efficiencies in processes. Testing the value of several model interventions has guided prioritization of evidence-based quality improvements. Some of the changes in practice have already been embedded in the EHR workflow successfully. Development and sustainability of the I-HDS has been fostered by a mix of internal and external funding, including philanthropic foundations. Challenges remain due to the highly competitive funding environment and changes needed to adapt the EHR to healthcare delivery research. Further stakeholder engagement and culture change working with hospital leadership and I-HDS core and affiliate members continues.

Evaluation of ML-Based Clinical Decision Support Tool to Replace an Existing Tool in an Academic Health System: Lessons Learned

There is increasing application of machine learning tools to problems in healthcare, with an ultimate goal to improve patient safety and health outcomes. When applied appropriately, machine learning tools can augment clinical care provided to patients. However, even if a model has impressive performance characteristics, prospectively evaluating and effectively implementing models into clinical care remains difficult. The primary objective of this paper is to recount our experiences and challenges in comparing a novel machine learning-based clinical decision support tool to legacy, non-machine learning tools addressing potential safety events in the hospitals and to summarize the obstacles which prevented evaluation of clinical efficacy of tools prior to widespread institutional use. We collected and compared safety events data, specifically patient falls and pressure injuries, between the standard of care approach and machine learning (ML)-based clinical decision support (CDS). Our assessment was limited to performance of the model rather than the workflow due to challenges in directly comparing both approaches. We did note a modest improvement in falls with ML-based CDS; however, it was not possible to determine that overall improvement was due to model characteristics.

Using Machine Learning to Predict ICU Transfer in Hospitalized COVID-19 Patients

OBJECTIVES

Approximately 20-30% of patients with COVID-19 require hospitalization, and 5-12% may require critical care in an intensive care unit (ICU). A rapid surge in cases of severe COVID-19 will lead to a corresponding surge in demand for ICU care. Because of constraints on resources, frontline healthcare workers may be unable to provide the frequent monitoring and assessment required for all patients at high risk of clinical deterioration. We developed a machine learning-based risk prioritization tool that predicts ICU transfer within 24 h, seeking to facilitate efficient use of care providers' efforts and help hospitals plan their flow of operations.

METHODS

A retrospective cohort was comprised of non-ICU COVID-19 admissions at a large acute care health system between 26 February and 18 April 2020. Time series data, including vital signs, nursing assessments, laboratory data, and electrocardiograms, were used as input variables for training a random forest (RF) model. The cohort was randomly split (70:30) into training and test sets. The RF model was trained using 10-fold cross-validation on the training set, and its predictive performance on the test set was then evaluated.

RESULTS

The cohort consisted of 1987 unique patients diagnosed with COVID-19 and admitted to non-ICU units of the hospital. The median time to ICU transfer was 2.45 days from the time of admission. Compared to actual admissions, the tool had 72.8% (95% CI: 63.2-81.1%) sensitivity, 76.3% (95% CI: 74.7-77.9%) specificity, 76.2% (95% CI: 74.6-77.7%) accuracy, and 79.9% (95% CI: 75.2-84.6%) area under the receiver operating characteristics curve.

CONCLUSIONS

A ML-based prediction model can be used as a screening tool to identify patients at risk of imminent ICU transfer within 24 h. This tool could improve the management of hospital resources and patient-throughput planning, thus delivering more effective care to patients hospitalized with COVID-19.

Development and external-validation of a nomogram for predicting the survival of hospitalised HIV/AIDS patients based on a large study cohort in western China

The aim of this study was to develop and externally validate a simple-to-use nomogram for predicting the survival of hospitalised human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) patients (hospitalised person living with HIV/AIDS (PLWHAs)). Hospitalised PLWHAs (n = 3724) between January 2012 and December 2014 were enrolled in the training cohort. HIV-infected inpatients (n = 1987) admitted in 2015 were included as the external-validation cohort. The least absolute shrinkage and selection operator method was used to perform data dimension reduction and select the optimal predictors. The nomogram incorporated 11 independent predictors, including occupation, antiretroviral therapy, pneumonia, tuberculosis, Talaromyces marneffei, hypertension, septicemia, anaemia, respiratory failure, hypoproteinemia and electrolyte disturbances. The Likelihood χ² statistic of the model was 516.30 (P = 0.000). Integrated Brier Score was 0.076 and Brier scores of the nomogram at the 10-day and 20-day time points were 0.046 and 0.071, respectively. The area under the curves for receiver operating characteristic were 0.819 and 0.828, and precision-recall curves were 0.242 and 0.378 at two time points. Calibration plots and decision curve analysis in the two sets showed good performance and a high net benefit of nomogram. In conclusion, the nomogram developed in the current study has relatively high calibration and is clinically useful. It provides a convenient and useful tool for timely clinical decision-making and the risk management of hospitalised PLWHAs.

Development and Validation of a Quick Sepsis-Related Organ Failure Assessment-Based Machine-Learning Model for Mortality Prediction in Patients with Suspected Infection in the Emergency Department

The quick sepsis-related organ failure assessment (qSOFA) score has been introduced to predict the likelihood of organ dysfunction in patients with suspected infection. We hypothesized that machine-learning models using qSOFA variables for predicting three-day mortality would provide better accuracy than the qSOFA score in the emergency department (ED). Between January 2016 and December 2018, the medical records of patients aged over 18 years with suspected infection were retrospectively obtained from four EDs in Korea. Data from three hospitals (n = 19,353) were used as training-validation datasets and data from one (n = 4234) as the test dataset. Machine-learning algorithms including extreme gradient boosting, light gradient boosting machine, and random forest were used. We assessed the prediction ability of machine-learning models using the area under the receiver operating characteristic (AUROC) curve, and DeLong's test was used to compare AUROCs between the qSOFA scores and qSOFA-based machine-learning models. A total of 447,926 patients visited EDs during the study period. We analyzed 23,587 patients with suspected infection who were admitted to the EDs. The median age of the patients was 63 years (interquartile range: 43-78 years) and in-hospital mortality was 4.0% (n = 941). For predicting three-day mortality among patients with suspected infection in the ED, the AUROC of the qSOFA-based machine-learning model (0.86 [95% CI 0.85-0.87]) for three -day mortality was higher than that of the qSOFA scores (0.78 [95% CI 0.77-0.79], p < 0.001). For predicting three-day mortality in patients with suspected infection in the ED, the qSOFA-based machine-learning model was found to be superior to the conventional qSOFA scores.