AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records

Machine learning risk-prediction model for in-hospital mortality in Takotsubo cardiomyopathy

BACKGROUND

Takotsubo cardiomyopathy (TC) is an acute heart failure syndrome characterized by transient left ventricular dysfunction, often triggered by stress. Data on risk scores predicting mortality in TC is sparse. We developed a machine-learning risk score model to predict in-hospital mortality in patients with TC.

METHODS

The National Inpatient Sample (NIS) database 2016-2020 was queried to identify adult patients (≥18 years) with TC using ICD-10 code I51.81. The primary outcome was in-hospital mortality. The dataset was randomly split into training (70 %), validation (20 %), and testing (10 %) dataset. Model performance was assessed using the area under the curve (AUC) with 95 % confidence intervals (95 % CI).

RESULTS

Amongst 38,662 TC patients identified [mean age 67.15 ± 14.17 years, female 32,089 (83 %)], 2499 (6.5 %) died. A novel risk score (0-127) was developed on age, race, Elixhauser comorbidity burden, history of hypertension, history of cardiac arrhythmia, presentation of cardiac arrest, cardiogenic shock, and acute kidney injury. Model AUCs (95 % CI) in the training, validation, and testing datasets were 0.809 (0.781-0.838), 0.809 (0.780-0.837), and 0.838 (0.820-0.856), respectively.

CONCLUSION

TC carries high morbidity and mortality. Our novel machine learning-based risk score is an important tool for risk stratification. External validation is needed to confirm these findings.

Patients Presenting to the Emergency Department with Bleeding in Early Pregnancy: Comparing Different Models to Predict Pregnancy Success

OBJECTIVES

Bleeding in early pregnancy is a common obstetric presentation in the emergency department (ED), and the outcome is difficult to predict. We developed and compared random forest machine learning (Live Birth Risk Score [LiBRisk]) and nomogram models for predicting the likelihood of a live birth among women presenting at 3 Canadian EDs with bleeding in early pregnancy.

METHODS

Data were prospectively collected on 200 patients with bleeding in early pregnancy using a structured questionnaire, medical record review, and administrative databases. We developed the nomogram with variables selected via multivariable logistic regression analysis. LiBRisk was built using the Shapley variable importance cloud (ShapleyVIC) to derive a simple point-based clinical risk scoring system.

RESULTS

Overall, 115 (55%) patients experienced a miscarriage. We excluded duration of vaginal bleeding and pain score, which did not enhance model performance, and constructed LiBRisk with the 8 most important variables (β-human chorionic gonadotrophin level, age, gestational age, gravidity, parity, proportionality of uterine size to gestational age, abdominal cramping, and number of prior spontaneous abortions). All 10 variables were included in the nomogram. The area under the receiver operating characteristic curve of LiBRisk in the test and validation sets were 0.913 (95% CI 0.907-0.919) and 0.900 (95% CI 0.887-0.913), respectively. The C-index of the nomogram was 0.720 (95% CI 0.714-0.726) and 0.860 (95% CI 0.853-0.867) in the test and validation sets, respectively. LiBRisk outperformed the nomogram in all metrics.

CONCLUSIONS

We developed and compared LiBRisk and nomogram models for determining the probability of eventual pregnancy success/failure in women presenting to the ED with bleeding in early pregnancy. LiBRisk was more parsimonious, incorporating only 8 variables, and outperformed the nomogram in all metrics. Given these promising results, further testing seems warranted.

A point-based cognitive impairment scoring system for southeast Asian adults

BACKGROUND

Cognitive impairment is a growing concern in Southeast Asian populations, where the burden of cerebrovascular disease (CeVD) is high. Currently, there is no point-based scoring system for identifying cognitive impairment in these populations.

OBJECTIVE

To develop and validate a simple point-based Cognitive Impairment Scoring System (CISS) for identifying individuals with cognitive impairment no dementia (CIND) and concomitant CeVD in Southeast Asian populations.

DESIGN

A cross-sectional study using data from two population-based studies.

SETTING

Community-based setting in Southeast Asia.

PARTICIPANTS

1,511 Southeast Asian adults (664 with CIND, 44.0 %).

MEASURES

Two CISS measures were developed: a basic measure including 11 easily assessable risk factors, and an extended measure incorporating seven additional neuroimaging markers. Performance was evaluated using receiver operating characteristic analysis (AUC) and calibration plots.

RESULTS

The AUC for CISS-basic and CISS-extended were 0.81 (95 %CI, 0.76-0.86) and 0.85 (95 %CI, 0.81-0.89), respectively. Calibration plots indicated satisfactory fit for both the basic measure (p=0.82) and the extended measure (p=0.17). The basic measure included age, gender, ethnicity, education, systolic blood pressure, BMI, smoking history, diabetes, hyperlipidemia, stroke history, and mild/moderate depression. The extended measure added neuroimaging markers of CeVD and brain atrophy.

CONCLUSION

The CISS provides a quick, objective, and clinically relevant tool for assessing cognitive impairment risk in Southeast Asian populations. The basic measure is suitable for initial community-based screenings, while the extended measure offers higher specificity for probable diagnosis. This point-based system enables rapid estimation of cognitive status without requiring complex calculations, potentially improving early detection and management of cognitive impairment in clinical practice.

Fast and interpretable mortality risk scores for critical care patients

OBJECTIVE

Prediction of mortality in intensive care unit (ICU) patients typically relies on black box models (that are unacceptable for use in hospitals) or hand-tuned interpretable models (that might lead to the loss in performance). We aim to bridge the gap between these 2 categories by building on modern interpretable machine learning (ML) techniques to design interpretable mortality risk scores that are as accurate as black boxes.

MATERIAL AND METHODS

We developed a new algorithm, GroupFasterRisk, which has several important benefits: it uses both hard and soft direct sparsity regularization, it incorporates group sparsity to allow more cohesive models, it allows for monotonicity constraint to include domain knowledge, and it produces many equally good models, which allows domain experts to choose among them. For evaluation, we leveraged the largest existing public ICU monitoring datasets (MIMIC III and eICU).

RESULTS

Models produced by GroupFasterRisk outperformed OASIS and SAPS II scores and performed similarly to APACHE IV/IVa while using at most a third of the parameters. For patients with sepsis/septicemia, acute myocardial infarction, heart failure, and acute kidney failure, GroupFasterRisk models outperformed OASIS and SOFA. Finally, different mortality prediction ML approaches performed better based on variables selected by GroupFasterRisk as compared to OASIS variables.

DISCUSSION

GroupFasterRisk's models performed better than risk scores currently used in hospitals, and on par with black box ML models, while being orders of magnitude sparser. Because GroupFasterRisk produces a variety of risk scores, it allows design flexibility-the key enabler of practical model creation.

CONCLUSION

GroupFasterRisk is a fast, accessible, and flexible procedure that allows learning a diverse set of sparse risk scores for mortality prediction.

Machine Learning-Based Explainable Automated Nonlinear Computation Scoring System for Health Score and an Application for Prediction of Perioperative Stroke: Retrospective Study

BACKGROUND

Machine learning (ML) has the potential to enhance performance by capturing nonlinear interactions. However, ML-based models have some limitations in terms of interpretability.

OBJECTIVE

This study aimed to develop and validate a more comprehensible and efficient ML-based scoring system using SHapley Additive exPlanations (SHAP) values.

METHODS

We developed and validated the Explainable Automated nonlinear Computation scoring system for Health (EACH) framework score. We developed a CatBoost-based prediction model, identified key features, and automatically detected the top 5 steepest slope change points based on SHAP plots. Subsequently, we developed a scoring system (EACH) and normalized the score. Finally, the EACH score was used to predict perioperative stroke. We developed the EACH score using data from the Seoul National University Hospital cohort and validated it using data from the Boramae Medical Center, which was geographically and temporally different from the development set.

RESULTS

When applied for perioperative stroke prediction among 38,737 patients undergoing noncardiac surgery, the EACH score achieved an area under the curve (AUC) of 0.829 (95% CI 0.753-0.892). In the external validation, the EACH score demonstrated superior predictive performance with an AUC of 0.784 (95% CI 0.694-0.871) compared with a traditional score (AUC=0.528, 95% CI 0.457-0.619) and another ML-based scoring generator (AUC=0.564, 95% CI 0.516-0.612).

CONCLUSIONS

The EACH score is a more precise, explainable ML-based risk tool, proven effective in real-world data. The EACH score outperformed traditional scoring system and other prediction models based on different ML techniques in predicting perioperative stroke.

Machine learning-based scoring model for predicting mortality in ICU-admitted ischemic stroke patients with moderate to severe consciousness disorders

Background

Stroke is a leading cause of mortality and disability globally. Among ischemic stroke patients, those with moderate to severe consciousness disorders constitute a particularly high-risk subgroup. Accurate predictive models are essential for guiding clinical decisions in this population. This study aimed to develop and validate an automated scoring system using machine learning algorithms for predicting short-term (3- and 7-day) and relatively long-term (30- and 90-day) mortality in this population.

Methods

This retrospective observational study utilized data from the MIMIC-IV database, including 648 ischemic stroke patients with Glasgow Coma Scale (GCS) scores ≤12, admitted to the ICU between 2008 and 2019. Patients with GCS scores indicating speech dysfunction but clear consciousness were excluded. A total of 47 candidate variables were evaluated, and the top six predictors for each mortality model were identified using the AutoScore framework. Model performance was assessed using the area under the curve (AUC) from receiver operating characteristic (ROC) analyses.

Results

The median age of the cohort was 76.8 years (IQR, 64.97-86.34), with mortality rates of 8.02% at 3 days, 18.67% at 7 days, 33.49% at 30 days, and 38.89% at 90 days. The AUCs for the test cohort's 3-, 7-, 30-, and 90-day mortality prediction models were 0.698, 0.678, 0.724, and 0.730, respectively.

Conclusion

We developed and validated a novel machine learning-based scoring tool that effectively predicts both short-term and relatively long-term mortality in ischemic stroke patients with moderate to severe consciousness disorders. This tool has the potential to enhance clinical decision-making and resource allocation for these patients in the ICU.

Prediction of contrast-associated acute kidney injury with machine-learning in patients undergoing contrast-enhanced computed tomography in emergency department

Radiocontrast media is a major cause of nephrotoxic acute kidney injury(AKI). Contrast-enhanced CT(CE-CT) is commonly performed in emergency departments(ED). Predicting individualized risks of contrast-associated AKI(CA-AKI) in ED patients is challenging due to a narrow time window and rapid patient turnover. We aimed to develop machine-learning(ML) models to predict CA-AKI in ED patients. Adult ED patients who underwent CE-CT between 2016 and 2020 at an academic, tertiary, referral hospital were included. Demographic, clinical, and laboratory data were collected from electronic medical records. Five ML models based on logistic regression; random forest; extreme gradient boosting; light gradient boosting; and multilayer perceptron were developed, using 42 features. Among 22,984 ED patients who underwent CE-CT; 1,862(8.1%) developed CA-AKI. The LGB model performed the best (AUROC = 0.731). Its top 10 features, in order of importance for predicting CA-AKI, were baseline serum creatinine; systolic blood pressure; serum albumin; estimated glomerular filtration rate; blood urea nitrogen; body weight; serum uric acid; hemoglobin; triglyceride; and body temperature. Given the difficulty of predicting risk of CA-AKI in ED, this model can help clinicians with early recognition of AKI and nephroprotective point-of-care interventions.

The Scoring Model to Predict ICU Stay and Mortality After Emergency Admissions in Atrial Fibrillation: A Retrospective Study of 30 366 Patients

BACKGROUND

The rapid assessment of the conditions is crucial for the prognosis of atrial fibrillation (AF) patients admitted to the emergency department (ED). We aim to derive and validate a more accurate and simplified scoring model to optimize the triage of AF patients in the ED.

MATERIALS AND METHODS

We conducted a retrospective study using data from the Medical Information Mart for Intensive Care (MIMIC-IV) database and developed scoring models employing the Random Forest algorithm. The area under the receiver operating characteristic (ROC) curve (AUC) was used to measure the performance of the prediction for intensive care unit (ICU) stay, and the death likelihood within 3, 7, and 30 days following the ED admission.

RESULTS

The study included 30 366 AF patients, randomly divided into training, validation, and testing cohorts at a 7:1:2 ratio. The training set consisted of 21 257 patients, the validation set included 3036 patients, and the remaining 6073 patients were classified as the validation set. Among the cohorts, 9594 patients (32%) required ICU transfers, with mortality rates of 1% at 3 days, 3% at 7 days, and 6% at 30 days. In the testing set, the scoring models demonstrated strong discriminative ability with AUCs of 0.724 for ICU stay, 0.782 for 3-day mortality, 0.755 for 7-day mortality, and 0.767 for 30-day mortality.

CONCLUSION

We derived and validated novel simplified scoring models with good discriminative performance to predict the likelihood of ICU stay, 3-day, 7-day, and 30-day death in AF patients after ED admission.

SymScore: Machine learning accuracy meets transparency in a symbolic regression-based clinical score generator

Self-report questionnaires play a crucial role in healthcare for assessing disease risks, yet their extensive length can be burdensome for respondents, potentially compromising data quality. To address this, machine learning-based shortened questionnaires have been developed. While these questionnaires possess high levels of accuracy, their practical use in clinical settings is hindered by a lack of transparency and the need for specialized machine learning expertise. This makes their integration into clinical workflows challenging and also decreases trust among healthcare professionals who prefer interpretable tools for decision-making. To preserve both predictive accuracy and interpretability, this study introduces the Symbolic Regression-Based Clinical Score Generator (SymScore). SymScore produces score tables for shortened questionnaires, which enable clinicians to estimate the results that reflect those of the original questionnaires. SymScore generates the score tables by optimally grouping responses, assigning weights based on predictive importance, imposing necessary constraints, and fitting models via symbolic regression. We compared SymScore's performance with the machine learning-based shortened questionnaires MCQI-6 (n=310) and SLEEPS (n=4257), both renowned for their high accuracy in assessing sleep disorders. SymScore's questionnaire demonstrated comparable performance (MAE = 10.73, R2 = 0.77) to that of the MCQI-6 (MAE = 9.94, R2 = 0.82) and achieved AUROC values of 0.85-0.91 for various sleep disorders, closely matching those of SLEEPS (0.88-0.94). By generating accurate and interpretable score tables, SymScore ensures that healthcare professionals can easily explain and trust its results without specialized machine learning knowledge. Thus, SymScore advances explainable AI for healthcare by offering a user-friendly and resource-efficient alternative to machine learning-based questionnaires, supporting improved patient outcomes and workflow efficiency.

Artificial intelligence in obstetric anaesthesiology - the future of patient care?

The use of artificial intelligence (AI) in obstetric anaesthesiology shows great potential in enhancing our practice and delivery of care. In this narrative review, we summarise the current applications of AI in four key areas of obstetric anaesthesiology (perioperative care, neuraxial procedures, labour analgesia and obstetric critical care), where AI has been employed to varying degrees for decision support, event prediction, risk stratification and procedural assistance. We also identify gaps in current practice and propose areas for further research. While promising, AI cannot replace the expertise and clinical judgement of a trained obstetric anaesthesiologist. It should, instead, be viewed as a valuable tool to facilitate and support our practice.

Assessing Risk in Implementing New Artificial Intelligence Triage Tools-How Much Risk is Reasonable in an Already Risky World?

Risk prediction in emergency medicine (EM) holds unique challenges due to issues surrounding urgency, blurry research-practise distinctions, and the high-pressure environment in emergency departments (ED). Artificial intelligence (AI) risk prediction tools have been developed with the aim of streamlining triaging processes and mitigating perennial issues affecting EDs globally, such as overcrowding and delays. The implementation of these tools is complicated by the potential risks associated with over-triage and under-triage, untraceable false positives, as well as the potential for the biases of healthcare professionals toward technology leading to the incorrect usage of such tools. This paper explores risk surrounding these issues in an analysis of a case study involving a machine learning triage tool called the Score for Emergency Risk Prediction (SERP) in Singapore. This tool is used for estimating mortality risk in presentation at the ED. After two successful retrospective studies demonstrating SERP's strong predictive accuracy, researchers decided that the pre-implementation randomised controlled trial (RCT) would not be feasible due to how the tool interacts with clinical judgement, complicating the blinded arm of the trial. This led them to consider other methods of testing SERP's real-world capabilities, such as ongoing-evaluation type studies. We discuss the outcomes of a risk-benefit analysis to argue that the proposed implementation strategy is ethically appropriate and aligns with improvement-focused and systemic approaches to implementation, especially the learning health systems framework (LHS) to ensure safety, efficacy, and ongoing learning.

Frailty Modeling Using Machine Learning Methodologies: A Systematic Review With Discussions on Outstanding Questions

Studying frailty is crucial for enhancing the health and quality of life among older adults, refining healthcare delivery methods, and tackling the obstacles linked to an aging demographic. Approaches to frailty modeling often utilise simple analytic techniques rather than available advanced machine learning methods, which may be sub-optimal. There is no large-scale systematic review on applications of machine learning methods on frailty modeling. In this study we explore the use of machine learning methods to predict or classify frailty in older persons in routinely collected data. We reviewed 181 research articles, and categorised analytic methods into three categories: generalised linear models, survival models, and non-linear models. These methods have a moderate agreement with existing frailty scores and predictive validity for adverse outcomes. Limited evidence suggests that non-linear methods outperform generalised linear methods. The top-three predictor/input variables are specific diagnosis or groups of diagnoses, functional performance (e.g., ADLs), and impaired cognition. Mortality, hospital admissions and prolonged hospital stay are the mainly predicted outcomes. Most studies utilise classical machine learning methods with cross-sectional data. Longitudinal data collected by wearable sensors have been used for frailty modeling. We also discuss the opportunities to use more advanced machine learning methods with high dimensional longitudinal data for more personalised and accessible frailty tools.

The MOST (Mortality Score for TBI): A novel prediction model beyond CRASH-Basic and IMPACT-Core for isolated traumatic brain injury

BACKGROUND

Due to significant injury heterogeneity, outcome prediction following traumatic brain injury (TBI) is challenging. This study aimed to develop a simple model for high-accuracy mortality risk prediction after TBI.

STUDY DESIGN

Data from the American College of Surgeons (ACS) Trauma Quality Program (TQP) from 2019 to 2021 was used to develop a summary score based on age, the Glasgow Coma Scale (GCS) component subscores, and pupillary reactivity data. We then compared the predictive accuracy to that of the Corticosteroid Randomisation After Significant Head Injury Trial (CRASH)-Basic and International Mission for Prognosis and Analysis of Clinical Trial in TBI (IMPACT)-Core models. Two separate series of sensitivity analyses were conducted to further assess our model's generalizability. We evaluated predictive performance of the models with discrimination [the area under the receiver-operating characteristic curves (AUC), sensitivity, specificity] and calibration (Brier score). Discriminative ability was compared with DeLong tests.

RESULTS

259,404 patients were included in the present study (mean age, 60 years; 93,495 (36 %) female). The mortality score after TBI (MOST) model (AUC = 0.875) had better discrimination (DeLong test p values < 0.00001) than CRASH-Basic (AUC = 0.837) and IMPACT-Core (AUC = 0.821) models, and superior calibration (MOST = 0.02729, CRASH-Basic = 0.02962, IMPACT-Core = 0.02962) in predicting in-hospital mortality. The MOST model similarly outperformed in predicting 3-, 7-, 14-, and 30-day mortality.

CONCLUSION

The MOST model can be rapidly calculated and outperforms two widely used models for predicting mortality in TBI patients. It utilizes a larger, contemporaneous dataset that reflects modern trauma care.

Screening for frequent hospitalization risk among community-dwelling older adult between 2016 and 2023: machine learning-driven item selection, scoring system development, and prospective validation

BACKGROUND

Screening for frequent hospitalizations in the community can help prevent super-utilizers from growing in the inpatient population. However, the determinants of frequent hospitalizations have not been systematically examined, their operational definitions have been inconsistent, and screening among community members lacks tools. Nor do we know if what determined frequent hospitalizations before COVID-19 continued to be the determinant of frequent hospitalizations at the height of the pandemic. Hence, the current study aims to identify determinants of frequent hospitalization and their screening items developed from the Comprehensive Geriatric Assessment (CGA), as our 273-item CGA is too lengthy to administer in full in community or primary care settings. The stability of the identified determinants will be examined in terms of the prospective validity of pre-COVID-selected items administered at the height of the pandemic.

METHODS

Comprehensive Geriatric Assessments (CGAs) were administered between 2016 and 2018 in the homes of 1,611 older adults aged 65+ years. Learning models were deployed to select CGA items to maximize the classification of different operational definitions of frequent hospitalizations, ranging from the most inclusive definition, wherein two or more hospitalizations over 2 years, to the most exclusive, wherein two or more hospitalizations must appear during year two, reflecting different care needs. In addition, the CGA items selected by the best-performing learning model were then developed into a random-forest-based scoring system for assessing frequent hospitalization risk, the validity of which was tested during 2018 and again prospectively between 2022 and 2023 in a sample of 329 older adults recruited from a district adjacent to where the CGAs were initially performed.

RESULTS

Seventeen items were selected from the CGA by our best-performing algorithm (DeepBoost), achieving 0.90 AUC in classifying operational definitions of frequent hospitalizations differing in temporal distributions and care needs. The number of medications prescribed and the need for assistance with emptying the bowel, housekeeping, transportation, and laundry were selected using the DeepBoost algorithm under the supervision of all operational definitions of frequent hospitalizations. On the other hand, reliance on walking aids, ability to balance on one's own, history of chronic obstructive pulmonary disease (COPD), and usage of social services were selected in the top 10 by all but the operational definitions that reflect the greatest care needs. The prospective validation of the original risk-scoring system using a sample recruited from a different district during the COVID-19 pandemic achieved an AUC of 0.82 in differentiating those rehospitalized twice or more over 2 years from those who were not.

CONCLUSION

A small subset of CGA items representing one's independence in aspects of (instrumental) activities of daily living, mobility, history of COPD, and social service utilization are sufficient for community members at risk of frequent hospitalization. The determinants of frequent hospitalization represented by the subset of CGA items remain relevant over the course of COVID-19 pandemic and across sociogeography.

Early prediction of sepsis in emergency department patients using various methods and scoring systems

BACKGROUND

Early recognition of sepsis, a common life-threatening condition in intensive care units (ICUs), is beneficial for improving patient outcomes. However, most sepsis prediction models were trained and assessed in the ICU, which might not apply to emergency department (ED) settings.

AIMS

To establish an early predictive model based on basic but essential information collected upon ED presentation for the follow-up diagnosis of sepsis observed in the ICU.

STUDY DESIGN

This study developed and validated a reliable model of sepsis prediction among ED patients by comparing 10 different methods based on retrospective electronic health record data from the MIMIC-IV database. In-ICU sepsis was identified as the primary outcome. The potential predictors encompassed baseline demographics, vital signs, pain scale, chief complaints and Emergency Severity Index (ESI). 80% and 20% of the total of 425 737 ED visit records were randomly selected for the train set and the test set for model development and validation, respectively.

RESULTS

Among the methods evaluated, XGBoost demonstrated an optimal predictive performance with an area under the curve (AUC) of 0.90 (95% CI: 0.90-0.91). Logistic regression exhibited a comparable predictive ability to XGBoost, with an AUC of 0.89 (95% CI: 0.89-0.90), along with a sensitivity and specificity of 85% (95% CI: 0.83-0.86) and 78% (95% CI: 0.77-0.80), respectively. Neither of the five commonly used severity scoring systems demonstrated satisfactory performance for sepsis prediction. The predictive ability of using ESI as the sole predictor (AUC: 0.79, 95% CI: 0.78-0.80) was also inferior to the model integrating ESI and other basic information.

CONCLUSIONS

The use of ESI combined with basic clinical information upon ED presentation accurately predicted sepsis among ED patients, strengthening its application in ED.

RELEVANCE TO CLINICAL PRACTICE

The proposed model may assist nurses in risk stratification management and prioritize interventions for potential sepsis patients, even in low-resource settings.

Development and validation of the Florey Dementia Risk Score web-based tool to screen for Alzheimer's disease in primary care

Background

It is estimated that ∼60% of people with Alzheimer's disease (AD) are undetected or undiagnosed, with higher rates of underdiagnosis in low-to middle-income areas with limited medical resources. To promote health equity, we have developed a web-based tool that utilizes easy-to-collect clinical data to enhance AD detection rate in primary care settings.

Methods

This study was leveraged on the data collected from participants of the Australian Imaging, Biomarker & Lifestyle (AIBL) study and the Religious Orders Study and Memory and Aging Project (ROSMAP). The study included three phases: (1) constructing and evaluating a model on retrospective cohort data (1407 AIBL participants), (2) performing simulated trials to assess model accuracy (30 AIBL participants) and missing data tolerability (30 AIBL participants), and (3) external evaluation using a non-Australian dataset (500 ROSMAP participants). The auto-score machine learning algorithm was employed to develop the Florey Dementia Risk Score (FDRS). All the simulated trials and evaluation were performed using a web-based FDRS tool.

Findings

FDRS achieved an area under the curve (AUC) of approximately 0.82 [95% CI, 0.75-0.88], with a sensitivity of 0.74 [0.60-0.86] and a specificity of 0.73 [0.70-0.79]. The accuracy of the simulated pilot trial for 30 AIBL participants with complete record was 87% (26/30 correct), while it only slightly decreased (80.0-83.3%, depending on imputation methods) for another 30 AIBL participants with one or two missing data. FDRS achieved an AUC of 0.82 [0.77-0.86] of 500 ROSMAP participants.

Interpretation

The FDRS tool offers a potential low-cost solution to AD screening in primary care. The present study warrants future trials of FDRS for optimization and to confirm its generalizability across a more diverse population, especially people in low-income countries.

Funding

National Health and Medical Research Council, Australia (GNT2007912) and Alzheimer's Association, USA (23AARF-1020292).

A Machine Learning-Based Risk Score for Prediction of Infective Endocarditis Among Patients With Staphylococcus aureus Bacteremia-The SABIER Score

BACKGROUND

Early risk assessment is needed to stratify Staphylococcus aureus infective endocarditis (SA-IE) risk among patients with S. aureus bacteremia (SAB) to guide clinical management. The objective of the current study was to develop a novel risk score that is independent of subjective clinical judgment and can be used early, at the time of blood culture positivity.

METHODS

We conducted a retrospective big data analysis from territory-wide electronic data and included hospitalized patients with SAB between 2009 and 2019. We applied a random forest risk scoring model to select variables from an array of parameters, according to the statistical importance in predicting SA-IE outcome. The data were divided into derivation and validation cohorts. The areas under the curve of the receiver operating characteristic (AUCROCs) were determined.

RESULTS

We identified 15 741 SAB patients, among them 658 (4.18%) had SA-IE. The AUCROC was 0.74 (95%CI 0.70-0.76), with a negative predictive value of 0.980 (95%CI 0.977-0.983). The four most discriminatory features were age, history of infective endocarditis, valvular heart disease, and community onset.

CONCLUSIONS

We developed a novel risk score with performance comparable with existing scores, which can be used at the time of SAB and prior to subjective clinical judgment.

A novel hyperparameter search approach for accuracy and simplicity in disease prediction risk scoring

OBJECTIVE

Develop a novel technique to identify an optimal number of regression units corresponding to a single risk point, while creating risk scoring systems from logistic regression-based disease predictive models. The optimal value of this hyperparameter balances simplicity and accuracy, yielding risk scores of small scale and high accuracy for patient risk stratification.

MATERIALS AND METHODS

The proposed technique applies an adapted line search across all potential hyperparameter values. Additionally, DeLong test is integrated to ensure the selected value produces an accuracy insignificantly different from the best achievable risk score accuracy. We assessed the approach through two case studies predicting diabetic retinopathy (DR) within six months and hip fracture readmissions (HFR) within 30 days, involving cohorts of 90 400 diabetic patients and 18 065 hip fracture patients.

RESULTS

Our scores achieve accuracies insignificantly different from those obtained by existing approaches, reaching AUROCs of 0.803 and 0.645 for DR and HFR predictions, respectively. Regarding the scale, our scores ranged 0-53 for DR and 0-15 for HFR, while scores produced by existing methods frequently spanned hundreds or thousands.

DISCUSSION

According to the assessment, our risk scores offer simple and accurate predictions for diseases. Furthermore, our new DR score provides a competitive alternative to state-of-the-art risk scores for DR, while our HFR case study presents the first risk score for this condition.

CONCLUSION

Our technique offers a generalizable framework for crafting precise risk scores of compact scales, addressing the demand for user-friendly and effective risk stratification tool in healthcare.

Interpretable Machine Learning-Based Risk Score for Predicting 10-Year Corneal Graft Survival After Penetrating Keratoplasty and Deep Anterior Lamellar Keratoplasty in Asian Eyes

PURPOSE

To predict 10-year graft survival after deep anterior lamellar keratoplasty (DALK) and penetrating keratoplasty (PK) using a machine learning (ML)-based interpretable risk score.

METHODS

Singapore Corneal Transplant Registry patients (n = 1687) who underwent DALK (n = 524) or PK (n = 1163) for optical indications (excluding endothelial diseases) were followed up for 10 years. Variable importance scores from random survival forests were used to identify variables associated with graft survival. Parsimonious analysis using nested Cox models selected the top factors. An ML-based clinical score generator (AutoScore) converted identified variables into an interpretable risk score. Predictive performance was evaluated using Kaplan-Meier (KM) curves and time-integrated AUC (iAUC) on an independent testing set.

RESULTS

Mean recipient age was 51.8 years, 54.1% were male, and majority were Chinese (60.0%). Surgical indications included corneal scar (46.5%), keratoconus (18.3%), and regraft (16.2%). Five-year and ten-year KM survival was 93.4% and 92.3% for DALK, compared with 67.6% and 56.6% for PK (log-rank P < 0.001). Five factors were identified by ML algorithm as predictors of 10-year graft survival: recipient sex, preoperative visual acuity, choice of procedure, surgical indication, and active inflammation. AutoScore stratified participants into low-risk and high-risk groups-with KM survival of 73.6% and 39.0%, respectively (log-rank P < 0.001). ML analysis outperformed traditional Cox regression in predicting graft survival beyond 5 years (iAUC 0.75 vs. 0.69).

CONCLUSIONS

A combination of ML and traditional techniques identified factors associated with graft failure to derive a clinically interpretable risk score to stratify PK and DALK patients-a technique that may be replicated in other corneal transplant programs.

A machine learning clinic scoring system for hepatocellular carcinoma based on the Surveillance, Epidemiology, and End Results database

BACKGROUND

Hepatocellular carcinoma (HCC) poses a global threat to life; however, numerical tools to predict the clinical prognosis of these patients remain scarce. The primary objective of this study is to establish a clinical scoring system for evaluating the overall survival (OS) rate and cancer-specific survival (CSS) rate in HCC patients.

METHODS

From the Surveillance, Epidemiology, and End Results (SEER) Program, we identified 45,827 primary HCC patients. These cases were randomly allocated to a training cohort (22,914 patients) and a validation cohort (22,913 patients). Univariate and multivariate Cox regression analyses, coupled with Kaplan-Meier methods, were employed to evaluate prognosis-related clinical and demographic features. Factors demonstrating prognostic significance were used to construct the model. The model's stability and accuracy were assessed through C-index, receiver operating characteristic (ROC) curves, calibration curves, and clinical decision curve analysis (DCA), while comparisons were made with the American Joint Committee on Cancer (AJCC) staging. Ultimately, machine learning (ML) quantified the variables in the model to establish a clinical scoring system.

RESULTS

Univariate and multivariate Cox regression analyses identified 11 demographic and clinical-pathological features as independent prognostic indicators for both CSS and OS using. Two models, each incorporating the 11 features, were developed, both of which demonstrated significant prognostic relevance. The C-index for predicting CSS and OS surpassed that of the AJCC staging system. The area under the curve (AUC) in time-dependent ROC consistently exceeded 0.74 in both the training and validation sets. Furthermore, internal and external calibration plots indicated that the model predictions aligned closely with observed outcomes. Additionally, DCA demonstrated the superiority of the model over the AJCC staging system, yielding greater clinical net benefit. Ultimately, the quantified clinical scoring system could efficiently discriminate between high and low-risk patients.

CONCLUSIONS

A ML clinical scoring system trained on a large-scale dataset exhibits good predictive and risk stratification performance in the cohorts. Such a clinical scoring system is readily integrable into clinical practice and will be valuable in enhancing the accuracy and efficiency of HCC management.

Investigating Machine Learning Techniques for Predicting Risk of Asthma Exacerbations: A Systematic Review

Asthma, a common chronic respiratory disease among children and adults, affects more than 200 million people worldwide and causes about 450,000 deaths each year. Machine learning is increasingly applied in healthcare to assist health practitioners in decision-making. In asthma management, machine learning excels in performing well-defined tasks, such as diagnosis, prediction, medication, and management. However, there remain uncertainties about how machine learning can be applied to predict asthma exacerbation. This study aimed to systematically review recent applications of machine learning techniques in predicting the risk of asthma attacks to assist asthma control and management. A total of 860 studies were initially identified from five databases. After the screening and full-text review, 20 studies were selected for inclusion in this review. The review considered recent studies published from January 2010 to February 2023. The 20 studies used machine learning techniques to support future asthma risk prediction by using various data sources such as clinical, medical, biological, and socio-demographic data sources, as well as environmental and meteorological data. While some studies considered prediction as a category, other studies predicted the probability of exacerbation. Only a group of studies applied prediction windows. The paper proposes a conceptual model to summarise how machine learning and available data sources can be leveraged to produce effective models for the early detection of asthma attacks. The review also generated a list of data sources that other researchers may use in similar work. Furthermore, we present opportunities for further research and the limitations of the preceding studies.

LC risk score - development and evaluation of a scale for assessing the risk of developing long COVID

INTRODUCTION

The aim of the study was to create a valuable practical tool for evaluating the risk of developing long COVID.

MATERIAL AND METHODS

1150 patients from the Polish STOP-COVID registry (PoLoCOV study) were used to develop the risk score. The patients were ill between 03/2020 and 04/2022. To develop a clinically useful scoring model, the LC risk score was generated using the machine learning-based framework AutoScore. Patient data were first randomised into a training (70% of output) and a test (30% of output) cohort. Due to the relatively small study group, cross-validation was used. Model predictive ability was evaluated based on the ROC curve and the AUC value. The result of the risk score for a given patient was the total value of points assigned to selected variables.

RESULTS

To create the LC risk score, eight variables were ultimately selected due to their significance and clinical value. Female gender significantly contributed to higher final outcome values, with age range 40-49, body mass index < 18.5 kg/m2, hospitalisation during active disease, arthralgia, myalgia as well as loss of taste and smell during infection, COVID-19 symptoms lasting at least 14 days, and unvaccinated status. The final predictive value of the developed LC risk score for a cut-off of 58 points was AUC = 0.630 (95% CI: 0.571-0.688) with sensitivity 39.80%, specificity 85.1%, positive predictive value 80.8%, and negative predictive value 47.3%.

CONCLUSIONS

The LC risk score may be a practical and undemanding utility that employs basic sociodemographic data, vaccination status, and symptoms during COVID-19 to assess the risk of long COVID.

Inter hospital external validation of interpretable machine learning based triage score for the emergency department using common data model

Emergency departments (ED) are complex, triage is a main task in the ED to prioritize patient with limited medical resources who need them most. Machine learning (ML) based ED triage tool, Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable ML framework with single center. We aimed to develop SERP with 3 Korean multicenter cohorts based on common data model (CDM) without data sharing and compare performance with inter-hospital validation design. This retrospective cohort study included all adult emergency visit patients of 3 hospitals in Korea from 2016 to 2017. We adopted CDM for the standardized multicenter research. The outcome of interest was 2-day mortality after the patients' ED visit. We developed each hospital SERP using interpretable ML framework and validated inter-hospital wisely. We accessed the performance of each hospital's score based on some metrics considering data imbalance strategy. The study population for each hospital included 87,670, 83,363 and 54,423 ED visits from 2016 to 2017. The 2-day mortality rate were 0.51%, 0.56% and 0.65%. Validation results showed accurate for inter hospital validation which has at least AUROC of 0.899 (0.858-0.940). We developed multicenter based Interpretable ML model using CDM for 2-day mortality prediction and executed Inter-hospital external validation which showed enough high accuracy.

Blood metabolomic and transcriptomic signatures stratify patient subgroups in multiple sclerosis according to disease severity

There are no blood-based biomarkers distinguishing patients with relapsing-remitting (RRMS) from secondary progressive multiple sclerosis (SPMS) although evidence supports metabolomic changes according to MS disease severity. Here machine learning analysis of serum metabolomic data stratified patients with RRMS from SPMS with high accuracy and a putative score was developed that stratified MS patient subsets. The top differentially expressed metabolites between SPMS versus patients with RRMS included lipids and fatty acids, metabolites enriched in pathways related to cellular respiration, notably, elevated lactate and glutamine (gluconeogenesis-related) and acetoacetate and bOHbutyrate (ketone bodies), and reduced alanine and pyruvate (glycolysis-related). Serum metabolomic changes were recapitulated in the whole blood transcriptome, whereby differentially expressed genes were also enriched in cellular respiration pathways in patients with SPMS. The final gene-metabolite interaction network demonstrated a potential metabolic shift from glycolysis toward increased gluconeogenesis and ketogenesis in SPMS, indicating metabolic stress which may trigger stress response pathways and subsequent neurodegeneration.

Impact of Environment on Pain among the Working Poor: Making Use of Random Forest-Based Stratification Tool to Study the Socioecology of Pain Interference

Pain interferes with one's work and social life and, at a personal level, daily activities, mood, and sleep quality. However, little research has been conducted on pain interference and its socioecological determinants among the working poor. Noting the clinical/policy decision needs and the technical challenges of isolating the intricately interrelated socioecological factors' unique contributions to pain interference and quantifying the relative contributions of each factor in an interpretable manner to inform clinical and policy decision-making, we deployed a novel random forest algorithm to model and quantify the unique contribution of a diverse ensemble of environmental, sociodemographic, and clinical factors to pain interference. Our analyses revealed that features representing the internal built environment of the working poor, such as the size of the living space, air quality, access to light, architectural design conducive to social connection, and age of the building, were assigned greater statistical importance than other more commonly examined predisposing factors for pain interference, such as age, occupation, the severity and locations of pain, BMI, serum blood sugar, and blood pressure. The findings were discussed in the context of their benefit in informing community pain screening to target residential areas whose built environment contributed most to pain interference and informing the design of intervention programs to minimize pain interference among those who suffered from chronic pain and showed specific characteristics. The findings support the call for good architecture to provide the spirit and value of buildings in city development.

Development and Validation of a Machine Learning Risk-Prediction Model for 30-Day Readmission for Heart Failure Following Transcatheter Aortic Valve Replacement (TAVR-HF Score)

Transcatheter aortic valve replacement (TAVR) is the treatment of choice for patients with severe aortic stenosis across the spectrum of surgical risk. About one-third of 30-day readmissions following TAVR are related to heart failure (HF). Hence, we aim to develop an easy-to-use clinical predictive model to identify patients at risk for HF readmission. We used data from the National Readmission Database (2015-2018) utilizing ICD-10 codes to identify TAVR procedures. Readmission was defined as the first unplanned HF readmission within 30-day of discharge. A machine learning framework was used to develop a 30-day TAVR-HF readmission score. The receiver operator characteristic curve was used to evaluate the predictive power of the model. A total of 92,363 cases of TAVR were included in the analysis. Of the included patients, 3299 (3.6%) were readmitted within 30 days of discharge with HF. Individuals who got readmitted, vs those without readmission, had more emergent admissions during index procedure (33.4% vs 19.8%), electrolyte abnormalities (38% vs 16.7%), chronic kidney disease (34.8% vs 21.2%), and atrial fibrillation (60.1% vs 40.7%). Candidate variables were ranked by importance using a parsimony plot. A total of 7 variables were selected based on predictive ability as well as clinical relevance: HF with reduced ejection fraction (25 points), HF preserved EF (20 points), electrolyte abnormalities (17 points), atrial fibrillation (12 points), Charlson comorbidity index (<6 = 0, 6-8 = 9, 9-10 = 13, >10 = 14 points), chronic kidney disease (7 points), and emergent index admission (5 points). On performance evaluation using the testing dataset, an area under the curve of 0.761 (95% CI 0.744-0.778) was achieved. Thirty-day TAVR-HF readmission score is an easy-to-use risk prediction tool. The score can be incorporated into electronic health record systems to identify at-risk individuals for readmissions with HF following TAVR. However, further external validation studies are needed.

DeepBiomarker2: Prediction of Alcohol and Substance Use Disorder Risk in Post-Traumatic Stress Disorder Patients Using Electronic Medical Records and Multiple Social Determinants of Health

Prediction of high-risk events amongst patients with mental disorders is critical for personalized interventions. We developed DeepBiomarker2 by leveraging deep learning and natural language processing to analyze lab tests, medication use, diagnosis, social determinants of health (SDoH) parameters, and psychotherapy for outcome prediction. To increase the model's interpretability, we further refined our contribution analysis to identify key features by scaling with a factor from a reference feature. We applied DeepBiomarker2 to analyze the EMR data of 38,807 patients from the University of Pittsburgh Medical Center diagnosed with post-traumatic stress disorder (PTSD) to determine their risk of developing alcohol and substance use disorder (ASUD). DeepBiomarker2 predicted whether a PTSD patient would have a diagnosis of ASUD within the following 3 months with an average c-statistic (receiver operating characteristic AUC) of 0.93 and average F1 score, precision, and recall of 0.880, 0.895, and 0.866 in the test sets, respectively. Our study found that the medications clindamycin, enalapril, penicillin, valacyclovir, Xarelto/rivaroxaban, moxifloxacin, and atropine and the SDoH parameters access to psychotherapy, living in zip codes with a high normalized vegetative index, Gini index, and low-income segregation may have potential to reduce the risk of ASUDs in PTSD. In conclusion, the integration of SDoH information, coupled with the refined feature contribution analysis, empowers DeepBiomarker2 to accurately predict ASUD risk. Moreover, the model can further identify potential indicators of increased risk along with medications with beneficial effects.

The application of explainable artificial intelligence (XAI) in electronic health record research: A scoping review

Machine Learning (ML) and Deep Learning (DL) models show potential in surpassing traditional methods including generalised linear models for healthcare predictions, particularly with large, complex datasets. However, low interpretability hinders practical implementation. To address this, Explainable Artificial Intelligence (XAI) methods are proposed, but a comprehensive evaluation of their effectiveness is currently limited. The aim of this scoping review is to critically appraise the application of XAI methods in ML/DL models using Electronic Health Record (EHR) data. In accordance with PRISMA scoping review guidelines, the study searched PUBMED and OVID/MEDLINE (including EMBASE) for publications related to tabular EHR data that employed ML/DL models with XAI. Out of 3220 identified publications, 76 were included. The selected publications published between February 2017 and June 2023, demonstrated an exponential increase over time. Extreme Gradient Boosting and Random Forest models were the most frequently used ML/DL methods, with 51 and 50 publications, respectively. Among XAI methods, Shapley Additive Explanations (SHAP) was predominant in 63 out of 76 publications, followed by partial dependence plots (PDPs) in 11 publications, and Locally Interpretable Model-Agnostic Explanations (LIME) in 8 publications. Despite the growing adoption of XAI methods, their applications varied widely and lacked critical evaluation. This review identifies the increasing use of XAI in tabular EHR research and highlights a deficiency in the reporting of methods and a lack of critical appraisal of validity and robustness. The study emphasises the need for further evaluation of XAI methods and underscores the importance of cautious implementation and interpretation in healthcare settings.

Explainable artificial intelligence in emergency medicine: an overview

Artificial intelligence (AI) and machine learning (ML) have potential to revolutionize emergency medical care by enhancing triage systems, improving diagnostic accuracy, refining prognostication, and optimizing various aspects of clinical care. However, as clinicians often lack AI expertise, they might perceive AI as a "black box," leading to trust issues. To address this, "explainable AI," which teaches AI functionalities to end-users, is important. This review presents the definitions, importance, and role of explainable AI, as well as potential challenges in emergency medicine. First, we introduce the terms explainability, interpretability, and transparency of AI models. These terms sound similar but have different roles in discussion of AI. Second, we indicate that explainable AI is required in clinical settings for reasons of justification, control, improvement, and discovery and provide examples. Third, we describe three major categories of explainability: pre-modeling explainability, interpretable models, and post-modeling explainability and present examples (especially for post-modeling explainability), such as visualization, simplification, text justification, and feature relevance. Last, we show the challenges of implementing AI and ML models in clinical settings and highlight the importance of collaboration between clinicians, developers, and researchers. This paper summarizes the concept of "explainable AI" for emergency medicine clinicians. This review may help clinicians understand explainable AI in emergency contexts.

Artificial intelligence in emergency medicine. A systematic literature review

Motivation and objective: Emergency medicine is becoming a popular application area for artificial intelligence methods but remains less investigated than other healthcare branches. The need for time-sensitive decision-making on the basis of high data volumes makes the use of quantitative technologies inevitable. However, the specifics of healthcare regulations impose strict requirements for such applications. Published contributions cover separate parts of emergency medicine and use disparate data and algorithms. This study aims to systematize the relevant contributions, investigate the main obstacles to artificial intelligence applications in emergency medicine, and propose directions for further studies.

METHODS

The contributions selection process was conducted with systematic electronic databases querying and filtering with respect to established exclusion criteria. Among the 380 papers gathered from IEEE Xplore, ACM Digital Library, Springer Library, ScienceDirect, and Nature databases 116 were considered to be a part of the survey. The main features of the selected papers are the focus on emergency medicine and the use of machine learning or deep learning algorithms.

FINDINGS AND DISCUSSION

The selected papers were classified into two branches: diagnostics-specific and triage-specific. The former ones are focused on either diagnosis prediction or decision support. The latter covers such applications as mortality, outcome, admission prediction, condition severity estimation, and urgent care prediction. The observed contributions are highly specialized within a single disease or medical operation and often use privately collected retrospective data, making them incomparable. These and other issues can be addressed by creating an end-to-end solution based on human-machine interaction.

CONCLUSION

Artificial intelligence applications are finding their place in emergency medicine, while most of the corresponding studies remain isolated and lack higher generalization and more sophisticated methodology, which can be a matter of forthcoming improvements.

FedScore: A privacy-preserving framework for federated scoring system development

OBJECTIVE

We propose FedScore, a privacy-preserving federated learning framework for scoring system generation across multiple sites to facilitate cross-institutional collaborations.

MATERIALS AND METHODS

The FedScore framework includes five modules: federated variable ranking, federated variable transformation, federated score derivation, federated model selection and federated model evaluation. To illustrate usage and assess FedScore's performance, we built a hypothetical global scoring system for mortality prediction within 30 days after a visit to an emergency department using 10 simulated sites divided from a tertiary hospital in Singapore. We employed a pre-existing score generator to construct 10 local scoring systems independently at each site and we also developed a scoring system using centralized data for comparison.

RESULTS

We compared the acquired FedScore model's performance with that of other scoring models using the receiver operating characteristic (ROC) analysis. The FedScore model achieved an average area under the curve (AUC) value of 0.763 across all sites, with a standard deviation (SD) of 0.020. We also calculated the average AUC values and SDs for each local model, and the FedScore model showed promising accuracy and stability with a high average AUC value which was closest to the one of the pooled model and SD which was lower than that of most local models.

CONCLUSION

This study demonstrates that FedScore is a privacy-preserving scoring system generator with potentially good generalizability.

Febrile infants risk score at triage (FIRST) for the early identification of serious bacterial infections

We aimed to derive the Febrile Infants Risk Score at Triage (FIRST) to quantify risk for serious bacterial infections (SBIs), defined as bacteremia, meningitis and urinary tract infections. We performed a prospective observational study on febrile infants < 3 months old at a tertiary hospital in Singapore between 2018 and 2021. We utilized machine learning and logistic regression to derive 2 models: FIRST, based on patient demographics, vital signs and history, and FIRST + , adding laboratory results to the same variables. SBIs were diagnosed in 224/1002 (22.4%) infants. Among 994 children with complete data, age (adjusted odds ratio [aOR] 1.01 95%CI 1.01-1.02, p < 0.001), high temperature (aOR 2.22 95%CI 1.69-2.91, p < 0.001), male sex (aOR 2.62 95%CI 1.86-3.70, p < 0.001) and fever of ≥ 2 days (aOR 1.79 95%CI 1.18-2.74, p = 0.007) were independently associated with SBIs. For FIRST + , abnormal urine leukocyte esterase (aOR 16.46 95%CI 10.00-27.11, p < 0.001) and procalcitonin (aOR 1.05 95%CI 1.01-1.09, p = 0.009) were further identified. A FIRST + threshold of ≥ 15% predicted risk had a sensitivity of 81.8% (95%CI 70.5-91.0%) and specificity of 65.6% (95%CI 57.8-72.7%). In the testing dataset, FIRST + had an area under receiver operating characteristic curve of 0.87 (95%CI 0.81-0.94). These scores can potentially guide triage and prioritization of febrile infants.

Development and Validation Nomogram for Predicting the Survival of Patients with Thrombocytopenia in Intensive Care Units

Background

The number of patients with thrombocytopenia (TCP) is relatively high in intensive care units (ICUs). It is therefore necessary to evaluate the prognostic risk of such patients.

Aim

This study investigated the risk factors affecting the survival of patients with TCP in the ICU. Using the findings of this investigation, we developed and validated a risk prediction model.

Methods

We evaluated patients admitted to the ICU who presented with TCP. We used LASSO regression to identify important clinical indicators. Based on these indicators, we developed a prediction model complete with a nomogram for the development cohort set. We then evaluated the mode's accuracy using a receiver operating characteristic (ROC) curve, calibration curves, and decision curve analysis (DCA) in a validation cohort.

Results

A total of 141 cases of ICU TCP were included in the sample, of which 47 involved death of the patient. Clinical results were as follows: N (HR 0.91, 95% CI 0.86-0.97, P=0.003); TBIL (HR 1.98, 95% CI 1.02-1.99, P=0.048); APACHE II (HR 1.94, 95% CI 1.39, 2.48, P=0.045); WPRN (HR 6.22, 95% CI 2.86-13.53, P<0.001); WTOST (HR 0.56, 95% CI 0.21-1.46, P<0.001); and DMV [HR1.87, 95% CI 1.12-2.33]. The prediction model yielded an area under the curve (AUC) of 0.918 (95% CI 0.863-0.974) in the development cohort and 0.926 (95% CI 0.849-0.994) in the validation cohort. Application of the nomogram in the validation cohort gave good discrimination (C-index 0.853, 95% CI 0.810-0.922) and good calibration. DCA indicated that the nomogram was clinically useful.

Conclusion

The individualized nomogram developed through our analysis demonstrated effective prognostic prediction for patients with TCP in ICUs. Use of this prediction metric may reduce TCP-related morbidity and mortality in ICUs.

Prediction tool for renal adaptation after living kidney donation using interpretable machine learning

Introduction

Post-donation renal outcomes are a crucial issue for living kidney donors considering young donors' high life expectancy and elderly donors' comorbidities that affect kidney function. We developed a prediction model for renal adaptation after living kidney donation using interpretable machine learning.

Methods

The study included 823 living kidney donors who underwent nephrectomy in 2009-2020. AutoScore, a machine learning-based score generator, was used to develop a prediction model. Fair and good renal adaptation were defined as post-donation estimated glomerular filtration rate (eGFR) of ≥ 60 mL/min/1.73 m² and ≥ 65% of the pre-donation values, respectively.

Results

The mean age was 45.2 years; 51.6% were female. The model included pre-donation demographic and laboratory variables, GFR measured by diethylenetriamine pentaacetate scan, and computed tomography kidney volume/body weight of both kidneys and the remaining kidney. The areas under the receiver operating characteristic curve were 0.846 (95% confidence interval, 0.762-0.930) and 0.626 (0.541-0.712), while the areas under the precision-recall curve were 0.965 (0.944-0.978) and 0.709 (0.647-0.788) for fair and good renal adaptation, respectively. An interactive clinical decision support system was developed.

Conclusion

The prediction tool for post-donation renal adaptation showed good predictive capability and may help clinical decisions through an easy-to-use web-based application.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.

Development and Asian-wide validation of the Grade for Interpretable Field Triage (GIFT) for predicting mortality in pre-hospital patients using the Pan-Asian Trauma Outcomes Study (PATOS)

Background

Field triage is critical in injury patients as the appropriate transport of patients to trauma centers is directly associated with clinical outcomes. Several prehospital triage scores have been developed in Western and European cohorts; however, their validity and applicability in Asia remains unclear. Therefore, we aimed to develop and validate an interpretable field triage scoring systems based on a multinational trauma registry in Asia.

Methods

This retrospective and multinational cohort study included all adult transferred injury patients from Korea, Malaysia, Vietnam, and Taiwan between 2016 and 2018. The outcome of interest was a death in the emergency department (ED) after the patients' ED visit. Using these results, we developed the interpretable field triage score with the Korea registry using an interpretable machine learning framework and validated the score externally. The performance of each country's score was assessed using the area under the receiver operating characteristic curve (AUROC). Furthermore, a website for real-world application was developed using R Shiny.

Findings

The study population included 26,294, 9404, 673 and 826 transferred injury patients between 2016 and 2018 from Korea, Malaysia, Vietnam, and Taiwan, respectively. The corresponding rates of a death in the ED were 0.30%, 0.60%, 4.0%, and 4.6% respectively. Age and vital sign were found to be the significant variables for predicting mortality. External validation showed the accuracy of the model with an AUROC of 0.756-0.850.

Interpretation

The Grade for Interpretable Field Triage (GIFT) score is an interpretable and practical tool to predict mortality in field triage for trauma.

Funding

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant Number: HI19C1328).

Application of an Interpretable Machine Learning for Estimating Severity of Graves’ Orbitopathy Based on Initial Finding

(1) Background: We constructed scores for moderate-to-severe and muscle-predominant types of Graves’ orbitopathy (GO) risk prediction based on initial ophthalmic findings. (2) Methods: 400 patients diagnosed with GO and followed up at both endocrinology and ophthalmology clinics with at least 6 months of follow-up. The Score for Moderate-to-Severe type of GO risk Prediction (SMSGOP) and the Score for Muscle-predominant type of GO risk Prediction (SMGOP) were constructed using the machine learning-based automatic clinical score generation algorithm. (3) Results: 55.3% were classified as mild type and 44.8% were classified as moderate-to-severe type. In the moderate-to-severe type group, 32.3% and 12.5% were classified as fat-predominant and muscle-predominant type, respectively. SMSGOP included age, central diplopia, thyroid stimulating immunoglobulin, modified NOSPECS classification, clinical activity score and ratio of the inferior rectus muscle cross-sectional area to total orbit in initial examination. SMGOP included age, central diplopia, amount of eye deviation, serum FT4 level and the interval between diagnosis of GD and GO in initial examination. Scores ≥46 and ≥49 had predictive value, respectively. (4) Conclusions: This is the first study to analyze factors in initial findings that can predict the severity of GO and to construct scores for risk prediction for Korean. We set the predictive scores using initial findings.

Human-Centered Design to Address Biases in Artificial Intelligence

The potential of artificial intelligence (AI) to reduce health care disparities and inequities is recognized, but it can also exacerbate these issues if not implemented in an equitable manner. This perspective identifies potential biases in each stage of the AI life cycle, including data collection, annotation, machine learning model development, evaluation, deployment, operationalization, monitoring, and feedback integration. To mitigate these biases, we suggest involving a diverse group of stakeholders, using human-centered AI principles. Human-centered AI can help ensure that AI systems are designed and used in a way that benefits patients and society, which can reduce health disparities and inequities. By recognizing and addressing biases at each stage of the AI life cycle, AI can achieve its potential in health care.

An artificial intelligence approach for predicting death or organ failure after hospitalization for COVID-19: development of a novel risk prediction tool and comparisons with ISARIC-4C, CURB-65, qSOFA, and MEWS scoring systems

BACKGROUND

We applied machine learning (ML) algorithms to generate a risk prediction tool [Collaboration for Risk Evaluation in COVID-19 (CORE-COVID-19)] for predicting the composite of 30-day endotracheal intubation, intravenous administration of vasopressors, or death after COVID-19 hospitalization and compared it with the existing risk scores.

METHODS

This is a retrospective study of adults hospitalized with COVID-19 from March 2020 to February 2021. Patients, each with 92 variables, and one composite outcome underwent feature selection process to identify the most predictive variables. Selected variables were modeled to build four ML algorithms (artificial neural network, support vector machine, gradient boosting machine, and Logistic regression) and an ensemble model to generate a CORE-COVID-19 model to predict the composite outcome and compared with existing risk prediction scores. The net benefit for clinical use of each model was assessed by decision curve analysis.

RESULTS

Of 1796 patients, 278 (15%) patients reached primary outcome. Six most predictive features were identified. Four ML algorithms achieved comparable discrimination (P > 0.827) with c-statistics ranged 0.849-0.856, calibration slopes 0.911-1.173, and Hosmer-Lemeshow P > 0.141 in validation dataset. These 6-variable fitted CORE-COVID-19 model revealed a c-statistic of 0.880, which was significantly (P < 0.04) higher than ISARIC-4C (0.751), CURB-65 (0.735), qSOFA (0.676), and MEWS (0.674) for outcome prediction. The net benefit of the CORE-COVID-19 model was greater than that of the existing risk scores.

CONCLUSION

The CORE-COVID-19 model accurately assigned 88% of patients who potentially progressed to 30-day composite events and revealed improved performance over existing risk scores, indicating its potential utility in clinical practice.

Development and assessment of scoring model for ICU stay and mortality prediction after emergency admissions in ischemic heart disease: a retrospective study of MIMIC-IV databases

Ischemic heart disease (IHD) is the leading cause of death and emergency department (ED) admission. We aimed to develop more accurate and straightforward scoring models to optimize the triaging of IHD patients in ED. This was a retrospective study based on the MIMIC-IV database. Scoring models were established by AutoScore formwork based on machine learning algorithm. The predictive power was measured by the area under the curve in the receiver operating characteristic analysis, with the prediction of intensive care unit (ICU) stay, 3d-death, 7d-death, and 30d-death after emergency admission. A total of 8381 IHD patients were included (median patient age, 71 years, 95% CI 62-81; 3035 [36%] female), in which 5867 episodes were randomly assigned to the training set, 838 to validation set, and 1676 to testing set. In total cohort, there were 2551 (30%) patients transferred into ICU; the mortality rates were 1% at 3 days, 3% at 7 days, and 7% at 30 days. In the testing cohort, the areas under the curve of scoring models for shorter and longer term outcomes prediction were 0.7551 (95% CI 0.7297-0.7805) for ICU stay, 0.7856 (95% CI 0.7166-0.8545) for 3d-death, 0.7371 (95% CI 0.6665-0.8077) for 7d-death, and 0.7407 (95% CI 0.6972-0.7842) for 30d-death. This newly accurate and parsimonious scoring models present good discriminative performance for predicting the possibility of transferring to ICU, 3d-death, 7d-death, and 30d-death in IHD patients visiting ED.

Machine learning-based infection prediction model for newly diagnosed multiple myeloma patients

Objective

To understand the infection characteristics and risk factors for infection by analyzing multicenter clinical data of newly diagnosed multiple myeloma (NDMM) patients.

Methods

This study reviewed 564 NDMM patients from 2 large tertiary hospitals from January 2018 to December 2021, of whom 395 comprised the training set and 169 comprised the validation set. Thirty-eight variables from first admission records were collected, including patient demographic characteristics, clinical scores and characteristics, laboratory indicators, complications, and medication history, and key variables were screened using the Lasso method. Multiple machine learning algorithms were compared, and the best performing algorithm was used to build a machine learning prediction model. The model performance was evaluated using the AUC, accuracy, and Youden's index. Finally, the SHAP package was used to assess two cases and demonstrate the application of the model.

Results

In this study, 15 important key variables were selected, namely, age, ECOG, osteolytic disruption, VCD, neutrophils, lymphocytes, monocytes, hemoglobin, platelets, albumin, creatinine, lactate dehydrogenase, affected globulin, β2 microglobulin, and preventive medicine. The predictive performance of the XGBoost model was significantly better than that of the other models (AUROC: 0.8664), and it also performed well for the expected dataset (accuracy: 68.64%).

Conclusion

A machine learning algorithm was used to establish an infection prediction model for NDMM patients that was simple, convenient, validated, and performed well in reducing the incidence of infection and improving the prognosis of patients.

Explainable machine learning approach to predict extubation in critically ill ventilated patients: a retrospective study in central Taiwan

Background

Weaning from mechanical ventilation (MV) is an essential issue in critically ill patients, and we used an explainable machine learning (ML) approach to establish an extubation prediction model.

Methods

We enrolled patients who were admitted to intensive care units during 2015–2019 at Taichung Veterans General Hospital, a referral hospital in central Taiwan. We used five ML models, including extreme gradient boosting (XGBoost), categorical boosting (CatBoost), light gradient boosting machine (LightGBM), random forest (RF) and logistic regression (LR), to establish the extubation prediction model, and the feature window as well as prediction window was 48 h and 24 h, respectively. We further employed feature importance, Shapley additive explanations (SHAP) plot, partial dependence plot (PDP) and local interpretable model-agnostic explanations (LIME) for interpretation of the model at the domain, feature, and individual levels.

Results

We enrolled 5,940 patients and found the accuracy was comparable among XGBoost, LightGBM, CatBoost and RF, with the area under the receiver operating characteristic curve using XGBoost to predict extubation was 0.921. The calibration and decision curve analysis showed well applicability of models. We also used the SHAP summary plot and PDP plot to demonstrate discriminative points of six key features in predicting extubation. Moreover, we employed LIME and SHAP force plots to show predicted probabilities of extubation and the rationale of the prediction at the individual level.

Conclusions

We developed an extubation prediction model with high accuracy and visualised explanations aligned with clinical workflow, and the model may serve as an autonomous screen tool for timely weaning.

AutoScore-Ordinal: an interpretable machine learning framework for generating scoring models for ordinal outcomes

Background

Risk prediction models are useful tools in clinical decision-making which help with risk stratification and resource allocations and may lead to a better health care for patients. AutoScore is a machine learning–based automatic clinical score generator for binary outcomes. This study aims to expand the AutoScore framework to provide a tool for interpretable risk prediction for ordinal outcomes.

Methods

The AutoScore-Ordinal framework is generated using the same 6 modules of the original AutoScore algorithm including variable ranking, variable transformation, score derivation (from proportional odds models), model selection, score fine-tuning, and model evaluation. To illustrate the AutoScore-Ordinal performance, the method was conducted on electronic health records data from the emergency department at Singapore General Hospital over 2008 to 2017. The model was trained on 70% of the data, validated on 10% and tested on the remaining 20%.

Results

This study included 445,989 inpatient cases, where the distribution of the ordinal outcome was 80.7% alive without 30-day readmission, 12.5% alive with 30-day readmission, and 6.8% died inpatient or by day 30 post discharge. Two point-based risk prediction models were developed using two sets of 8 predictor variables identified by the flexible variable selection procedure. The two models indicated reasonably good performance measured by mean area under the receiver operating characteristic curve (0.758 and 0.793) and generalized c-index (0.737 and 0.760), which were comparable to alternative models.

Conclusion

AutoScore-Ordinal provides an automated and easy-to-use framework for development and validation of risk prediction models for ordinal outcomes, which can systematically identify potential predictors from high-dimensional data.

Benchmarking emergency department prediction models with machine learning and public electronic health records

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.

An external validation study of the Score for Emergency Risk Prediction (SERP), an interpretable machine learning-based triage score for the emergency department

Emergency departments (EDs) are experiencing complex demands. An ED triage tool, the Score for Emergency Risk Prediction (SERP), was previously developed using an interpretable machine learning framework. It achieved a good performance in the Singapore population. We aimed to externally validate the SERP in a Korean cohort for all ED patients and compare its performance with Korean triage acuity scale (KTAS). This retrospective cohort study included all adult ED patients of Samsung Medical Center from 2016 to 2020. The outcomes were 30-day and in-hospital mortality after the patients' ED visit. We used the area under the receiver operating characteristic curve (AUROC) to assess the performance of the SERP and other conventional scores, including KTAS. The study population included 285,523 ED visits, of which 53,541 were after the COVID-19 outbreak (2020). The whole cohort, in-hospital, and 30 days mortality rates were 1.60%, and 3.80%. The SERP achieved an AUROC of 0.821 and 0.803, outperforming KTAS of 0.679 and 0.729 for in-hospital and 30-day mortality, respectively. SERP was superior to other scores for in-hospital and 30-day mortality prediction in an external validation cohort. SERP is a generic, intuitive, and effective triage tool to stratify general patients who present to the emergency department.

External validation of the Survival After ROSC in Cardiac Arrest (SARICA) score for predicting survival after return of spontaneous circulation using multinational pan-asian cohorts

Aim

Accurate and timely prognostication of patients with out-of-hospital cardiac arrest (OHCA) who attain return of spontaneous circulation (ROSC) is crucial in clinical decision-making, resource allocation, and communication with family. A clinical decision tool, Survival After ROSC in Cardiac Arrest (SARICA), was recently developed, showing excellent performance on internal validation. We aimed to externally validate SARICA in multinational cohorts within the Pan-Asian Resuscitation Outcomes Study.

Materials and methods

This was an international, retrospective cohort study of patients who attained ROSC after OHCA in the Asia Pacific between January 2009 and August 2018. Pediatric (age <18 years) and traumatic arrests were excluded. The SARICA score was calculated for each patient. The primary outcome was survival. We used receiver operating characteristics (ROC) analysis to calculate the model performance of the SARICA score in predicting survival. A calibration belt plot was used to assess calibration.

Results

Out of 207,450 cases of OHCA, 24,897 cases from Taiwan, Japan and South Korea were eligible for inclusion. Of this validation cohort, 30.4% survived. The median SARICA score was 4. Area under the ROC curve (AUC) was 0.759 (95% confidence interval, CI 0.753–0.766) for the total population. A higher AUC was observed in subgroups that received bystander CPR (AUC 0.791, 95% CI 0.782–0.801) and of presumed cardiac etiology (AUC 0.790, 95% CI 0.782–0.797). The model was well-calibrated.

Conclusion

This external validation study of SARICA demonstrated high model performance in a multinational Pan-Asian cohort. Further modification and validation in other populations can be performed to assess its readiness for clinical translation.

Identifying surrogate markers in real-world comparative effectiveness research

In comparative effectiveness research (CER), leveraging short-term surrogates to infer treatment effects on long-term outcomes can guide policymakers evaluating new treatments. Numerous statistical procedures for identifying surrogates have been proposed for randomized clinical trials (RCTs), but no methods currently exist to evaluate the proportion of treatment effect (PTE) explained by surrogates in real-world data (RWD), which have become increasingly common. To address this knowledge gap, we propose inverse probability weighted (IPW) and doubly robust (DR) estimators of an optimal transformation of the surrogate and the corresponding PTE measure. We demonstrate that the proposed estimators are consistent and asymptotically normal, and the DR estimator is consistent when either the propensity score model or outcome regression model is correctly specified. Our proposed estimators are evaluated through extensive simulation studies. In two RWD settings, we show that our method can identify and validate surrogate markers for inflammatory bowel disease (IBD).

Predicting Amyloid Positivity in Cognitively Unimpaired Older Adults: A Machine Learning Approach Using A4 Data

BACKGROUND AND OBJECTIVES

To develop and test the performance of the Positive Aβ Risk Score (PARS) for prediction of β-amyloid (Aβ) positivity in cognitively unimpaired individuals for use in clinical research. Detecting Aβ positivity is essential for identifying at-risk individuals who are candidates for early intervention with amyloid targeted treatments.

METHODS

We used data from 4,134 cognitively normal individuals from the Anti-Amyloid Treatment in Asymptomatic Alzheimer's (A4) Study. The sample was divided into training and test sets. A modified version of AutoScore, a machine learning-based software tool, was used to develop a scoring system using the training set. Three risk scores were developed using candidate predictors in various combinations from the following categories: demographics (age, sex, education, race, family history, body mass index, marital status, and ethnicity), subjective measures (Alzheimer's Disease Cooperative Study Activities of Daily Living-Prevention Instrument, Geriatric Depression Scale, and Memory Complaint Questionnaire), objective measures (free recall, Mini-Mental State Examination, immediate recall, digit symbol substitution, and delayed logical memory scores), and APOE4 status. Performance of the risk scores was evaluated in the independent test set.

RESULTS

PARS model 1 included age, body mass index (BMI), and family history and had an area under the curve (AUC) of 0.60 (95% CI 0.57-0.64). PARS model 2 included free recall in addition to the PARS model 1 variables and had an AUC of 0.61 (0.58-0.64). PARS model 3, which consisted of age, BMI, and APOE4 information, had an AUC of 0.73 (0.70-0.76). PARS model 3 showed the highest, but still moderate, performance metrics in comparison with other models with sensitivity of 72.0% (67.6%-76.4%), specificity of 62.1% (58.8%-65.4%), accuracy of 65.3% (62.7%-68.0%), and positive predictive value of 48.1% (44.1%-52.1%).

DISCUSSION

PARS models are a set of simple and practical risk scores that may improve our ability to identify individuals more likely to be amyloid positive. The models can potentially be used to enrich trials and serve as a screening step in research settings. This approach can be followed by the use of additional variables for the development of improved risk scores.

CLASSIFICATION OF EVIDENCE

This study provides Class II evidence that in cognitively unimpaired individuals PARS models predict Aβ positivity with moderate accuracy.