AutoScore-Survival: Developing interpretable machine learning-based time-to-event scores with right-censored survival data

An interpretable machine learning scoring tool for estimating time to recurrence readmissions in stroke patients

BACKGROUND

Stroke recurrence readmission poses an additional burden on both patients and healthcare systems. Risk stratification aims to accurately divide patients into groups to provide targeted interventions at reducing readmission. To accurately predict short and intermediate-term risks of readmission and provide information for further temporal risk stratification, we developed and validated an interpretable machine learning risk scoring tool.

METHODS

In this retrospective study, all stroke admission episodes from January 1st 2015 to December 31st 2019 were obtained from the Shanghai Health and Health Development Research Centre database, which covers medical records of all patients hospitalized in 436 medical institutes in Shanghai. The outcome was time to stroke recurrence readmission within 90 days post discharge. The Score for Stroke Recurrence Readmission Prediction (SSRRP) tool was derived via an interpretable machine learning-based system for time-to-event outcomes. SSRRP as six-variable survival score includes sequelae, length of stay, type of stroke, random plasma glucose, medical expense payment, and number of hospitalizations.

RESULTS

A total of 339,212 S admission episodes were finally included in the whole cohort. Among them, 217,393 episodes were included in the training dataset, 54,347 episodes in the internal validation dataset, and 67,472 in the temporal validation dataset. Readmission within 90 days was documented in 33922(9.97 %) episodes, with a median time to emergency readmission of 19 days (Interquartile range: 8-43). In the temporal validation dataset, the SSRRP achieved an integrated area under the curve of 0.730(95 % CI, 0.724-0.737). In addition, SSRRP demonstrated good calibration and clinical benefit rate.

CONCLUSIONS

In this retrospective cohort study, the SSRRP, a parsimonious and point-based scoring tool, was developed to predict the risk of recurrent readmission for stroke. It also provided accurate information on the time to stroke readmission, enabling further temporal risk stratification and informed clinical decision-making.

Frailty Modeling Using Machine Learning Methodologies: A Systematic Review With Discussions on Outstanding Questions

Studying frailty is crucial for enhancing the health and quality of life among older adults, refining healthcare delivery methods, and tackling the obstacles linked to an aging demographic. Approaches to frailty modeling often utilise simple analytic techniques rather than available advanced machine learning methods, which may be sub-optimal. There is no large-scale systematic review on applications of machine learning methods on frailty modeling. In this study we explore the use of machine learning methods to predict or classify frailty in older persons in routinely collected data. We reviewed 181 research articles, and categorised analytic methods into three categories: generalised linear models, survival models, and non-linear models. These methods have a moderate agreement with existing frailty scores and predictive validity for adverse outcomes. Limited evidence suggests that non-linear methods outperform generalised linear methods. The top-three predictor/input variables are specific diagnosis or groups of diagnoses, functional performance (e.g., ADLs), and impaired cognition. Mortality, hospital admissions and prolonged hospital stay are the mainly predicted outcomes. Most studies utilise classical machine learning methods with cross-sectional data. Longitudinal data collected by wearable sensors have been used for frailty modeling. We also discuss the opportunities to use more advanced machine learning methods with high dimensional longitudinal data for more personalised and accessible frailty tools.

Point-Based Prediction Model for Bladder Cancer Risk in Diabetes: A Random Survival Forest-Guided Approach

Background: Previous epidemiological studies have shown that diabetes is associated with an increased risk of several cancers, including bladder cancer. However, prediction models for bladder cancer among diabetes patients remain scarce. This study aims to develop a scoring system for bladder cancer risk prediction among diabetes patients who receive routine care in general outpatient clinics using a machine learning-guided approach. Methods: A territory-wide retrospective cohort study was conducted using electronic health records of Hong Kong. Patients who received diabetes care in public general outpatient clinics between 2010 and 2019 without a history of malignancy were identified and followed up until December 2019. To develop a scoring system for bladder cancer risk prediction, random survival forest was employed to guide variable selection, and Cox regression was subsequently applied for weight assignment. Results: Of the 382,770 patients identified, 644 patients developed bladder cancer during follow-up (median: 6.2 years). The incidence rate was 0.29 per 1000 person-years. In the final time-to-event scoring system, age, serum creatinine, sex, and smoking were included as predictors. Serum creatinine ≥94 µmol/L appeared to be associated with an increased risk of developing bladder cancer. The 2-year and 5-year AUCs on test set were 0.88 (95%CI: 0.84-0.92) and 0.86 (95%CI: 0.80-0.92) respectively. Conclusions: Renal dysfunction could be a potential predictor of bladder cancer among diabetes patients. The proposed scoring system could be potentially useful for providing individualized risk prediction among diabetes patients.

Scoring System for Predicting the Risk of Liver Cancer among Diabetes Patients: A Random Survival Forest-Guided Approach

BACKGROUND

Most liver cancer scoring systems focus on patients with preexisting liver diseases such as chronic viral hepatitis or liver cirrhosis. Patients with diabetes are at higher risk of developing liver cancer than the general population. However, liver cancer scoring systems for patients in the absence of liver diseases or those with diabetes remain rare. This study aims to develop a risk scoring system for liver cancer prediction among diabetes patients and a sub-model among diabetes patients without cirrhosis/chronic viral hepatitis.

METHODS

A retrospective cohort study was performed using electronic health records of Hong Kong. Patients who received diabetes care in general outpatient clinics between 2010 and 2019 without cancer history were included and followed up until December 2019. The outcome was diagnosis of liver cancer during follow-up. A risk scoring system was developed by applying random survival forest in variable selection, and Cox regression in weight assignment.

RESULTS

The liver cancer incidence was 0.92 per 1000 person-years. Patients who developed liver cancer (n = 1995) and those who remained free of cancer (n = 1969) during follow-up (median: 6.2 years) were selected for model building. In the final time-to-event scoring system, presence of chronic hepatitis B/C, alanine aminotransferase, age, presence of cirrhosis, and sex were included as predictors. The concordance index was 0.706 (95%CI: 0.676-0.741). In the sub-model for patients without cirrhosis/chronic viral hepatitis, alanine aminotransferase, age, triglycerides, and sex were selected as predictors.

CONCLUSIONS

The proposed scoring system may provide a parsimonious score for liver cancer risk prediction among diabetes patients.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.

Development and Asian-wide validation of the Grade for Interpretable Field Triage (GIFT) for predicting mortality in pre-hospital patients using the Pan-Asian Trauma Outcomes Study (PATOS)

Background

Field triage is critical in injury patients as the appropriate transport of patients to trauma centers is directly associated with clinical outcomes. Several prehospital triage scores have been developed in Western and European cohorts; however, their validity and applicability in Asia remains unclear. Therefore, we aimed to develop and validate an interpretable field triage scoring systems based on a multinational trauma registry in Asia.

Methods

This retrospective and multinational cohort study included all adult transferred injury patients from Korea, Malaysia, Vietnam, and Taiwan between 2016 and 2018. The outcome of interest was a death in the emergency department (ED) after the patients' ED visit. Using these results, we developed the interpretable field triage score with the Korea registry using an interpretable machine learning framework and validated the score externally. The performance of each country's score was assessed using the area under the receiver operating characteristic curve (AUROC). Furthermore, a website for real-world application was developed using R Shiny.

Findings

The study population included 26,294, 9404, 673 and 826 transferred injury patients between 2016 and 2018 from Korea, Malaysia, Vietnam, and Taiwan, respectively. The corresponding rates of a death in the ED were 0.30%, 0.60%, 4.0%, and 4.6% respectively. Age and vital sign were found to be the significant variables for predicting mortality. External validation showed the accuracy of the model with an AUROC of 0.756-0.850.

Interpretation

The Grade for Interpretable Field Triage (GIFT) score is an interpretable and practical tool to predict mortality in field triage for trauma.

Funding

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant Number: HI19C1328).

Benchmarking emergency department prediction models with machine learning and public electronic health records

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.

A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study

Risk scores are widely used for clinical decision making and commonly generated from logistic regression models. Machine-learning-based methods may work well for identifying important predictors to create parsimonious scores, but such 'black box' variable selection limits interpretability, and variable importance evaluated from a single model can be biased. We propose a robust and interpretable variable selection approach using the recently developed Shapley variable importance cloud (ShapleyVIC) that accounts for variability in variable importance across models. Our approach evaluates and visualizes overall variable contributions for in-depth inference and transparent variable selection, and filters out non-significant contributors to simplify model building steps. We derive an ensemble variable ranking from variable contributions across models, which is easily integrated with an automated and modularized risk score generator, AutoScore, for convenient implementation. In a study of early death or unplanned readmission after hospital discharge, ShapleyVIC selected 6 variables from 41 candidates to create a well-performing risk score, which had similar performance to a 16-variable model from machine-learning-based ranking. Our work contributes to the recent emphasis on interpretability of prediction models for high-stakes decision making, providing a disciplined solution to detailed assessment of variable importance and transparent development of parsimonious clinical risk scores.

AutoScore-Imbalance: An Interpretable Machine Learning Tool for Development of Clinical Scores with Rare Events Data

BACKGROUND

Medical decision-making impacts both individual and public health. Clinical scores are commonly used among various decision-making models to determine the degree of disease deterioration at the bedside. AutoScore was proposed as a useful clinical score generator based on machine learning and a generalized linear model. However, its current framework still leaves room for improvement when addressing unbalanced data of rare events.

METHODS

Using machine intelligence approaches, we developed AutoScore-Imbalance, which comprises three components: training dataset optimization, sample weight optimization, and adjusted AutoScore. Baseline techniques for performance comparison included the original AutoScore, full logistic regression, stepwise logistic regression, least absolute shrinkage and selection operator (LASSO), full random forest, and random forest with a reduced number of variables. These models were evaluated based on their area under the curve (AUC) in the receiver operating characteristic analysis and balanced accuracy (i.e., mean value of sensitivity and specificity). By utilizing a publicly accessible dataset from Beth Israel Deaconess Medical Center, we assessed the proposed model and baseline approaches to predict inpatient mortality.

RESULTS

AutoScore-Imbalance outperformed baselines in terms of AUC and balanced accuracy. The nine-variable AutoScore-Imbalance sub-model achieved the highest AUC of 0.786 (0.732-0.839), while the eleven-variable original AutoScore obtained an AUC of 0.723 (0.663-0.783), and the logistic regression with 21 variables obtained an AUC of 0.743 (0.685-0.800). The AutoScore-Imbalance sub-model (using a down-sampling algorithm) yielded an AUC of 0.771 (0.718-0.823) with only five variables, demonstrating a good balance between performance and variable sparsity. Furthermore, AutoScore-Imbalance obtained the highest balanced accuracy of 0.757 (0.702-0.805), compared to 0.698 (0.643-0.753) by the original AutoScore and the maximum of 0.720 (0.664-0.769) by other baseline models.

CONCLUSIONS

We have developed an interpretable tool to handle clinical data imbalance, presented its structure, and demonstrated its superiority over baselines. The AutoScore-Imbalance tool can be applied to highly unbalanced datasets to gain further insight into rare medical events and facilitate real-world clinical decision-making.

Development and validation of an interpretable machine learning scoring tool for estimating time to emergency readmissions

BACKGROUND

Emergency readmission poses an additional burden on both patients and healthcare systems. Risk stratification is the first step of transitional care interventions targeted at reducing readmission. To accurately predict the short- and intermediate-term risks of readmission and provide information for further temporal risk stratification, we developed and validated an interpretable machine learning risk scoring system.

METHODS

In this retrospective study, all emergency admission episodes from January 1st 2009 to December 31st 2016 at a tertiary hospital in Singapore were assessed. The primary outcome was time to emergency readmission within 90 days post discharge. The Score for Emergency ReAdmission Prediction (SERAP) tool was derived via an interpretable machine learning-based system for time-to-event outcomes. SERAP is six-variable survival score, and takes the number of emergency admissions last year, age, history of malignancy, history of renal diseases, serum creatinine level, and serum albumin level during index admission into consideration.

FINDINGS

A total of 293,589 ED admission episodes were finally included in the whole cohort. Among them, 203,748 episodes were included in the training cohort, 50,937 episodes in the validation cohort, and 38,904 in the testing cohort. Readmission within 90 days was documented in 80,213 (27.3%) episodes, with a median time to emergency readmission of 22 days (Interquartile range: 8-47). For different time points, the readmission rates observed in the whole cohort were 6.7% at 7 days, 10.6% at 14 days, 13.6% at 21 days, 16.4% at 30 days, and 23.0% at 60 days. In the testing cohort, the SERAP achieved an integrated area under the curve of 0.737 (95% confidence interval: 0.730-0.743). For a specific 30-day readmission prediction, SERAP outperformed the LACE index (Length of stay, Acuity of admission, Charlson comorbidity index, and Emergency department visits in past six months) and the HOSPITAL score (Hemoglobin at discharge, discharge from an Oncology service, Sodium level at discharge, Procedure during the index admission, Index Type of admission, number of Admissions during the last 12 months, and Length of stay). Besides 30-day readmission, SERAP can predict readmission rates at any time point during the 90-day period.

INTERPRETATION

Better performance in risk prediction was achieved by the SERAP than other existing scores, and accurate information about time to emergency readmission was generated for further temporal risk stratification and clinical decision-making. In the future, external validation studies are needed to evaluate the SERAP at different settings and assess their real-world performance.

FUNDING

This study was supported by the Singapore National Medical Research Council under the PULSES Center Grant, and Duke-NUS Medical School.