1
|
Xie F, Ning Y, Liu M, Li S, Saffari SE, Yuan H, Volovici V, Ting DSW, Goldstein BA, Ong MEH, Vaughan R, Chakraborty B, Liu N. A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes. STAR Protoc 2023; 4:102302. [PMID: 37178115 DOI: 10.1016/j.xpro.2023.102302] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 03/13/2023] [Accepted: 04/21/2023] [Indexed: 05/15/2023] Open
Abstract
The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.
Collapse
Affiliation(s)
- Feng Xie
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Yilin Ning
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Mingxuan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Siqi Li
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Seyed Ehsan Saffari
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Han Yuan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Victor Volovici
- Department of Neurosurgery, Erasmus MC University Medical Center, 3015 GD Rotterdam, the Netherlands; Department of Public Health, Erasmus MC, 3015 GD Rotterdam, the Netherlands
| | - Daniel Shu Wei Ting
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 168751, Singapore; SingHealth AI Office, Singapore Health Services, Singapore 168582, Singapore
| | - Benjamin Alan Goldstein
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27710, USA
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Health Services Research Centre, Singapore Health Services, Singapore 169856, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore 169608, Singapore
| | - Roger Vaughan
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Bibhas Chakraborty
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27710, USA; Department of Statistics and Data Science, National University of Singapore, Singapore 117546, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore 169857, Singapore; Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore 169857, Singapore; SingHealth AI Office, Singapore Health Services, Singapore 168582, Singapore; Institute of Data Science, National University of Singapore, Singapore 117602, Singapore.
| |
Collapse
|
2
|
Yu JY, Heo S, Xie F, Liu N, Yoon SY, Chang HS, Kim T, Lee SU, Hock Ong ME, Ng YY, Do shin S, Kajino K, Cha WC. Development and Asian-wide validation of the Grade for Interpretable Field Triage (GIFT) for predicting mortality in pre-hospital patients using the Pan-Asian Trauma Outcomes Study (PATOS). THE LANCET REGIONAL HEALTH. WESTERN PACIFIC 2023; 34:100733. [PMID: 37283981 PMCID: PMC10240358 DOI: 10.1016/j.lanwpc.2023.100733] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2022] [Revised: 01/24/2023] [Accepted: 02/19/2023] [Indexed: 03/07/2023]
Abstract
Background Field triage is critical in injury patients as the appropriate transport of patients to trauma centers is directly associated with clinical outcomes. Several prehospital triage scores have been developed in Western and European cohorts; however, their validity and applicability in Asia remains unclear. Therefore, we aimed to develop and validate an interpretable field triage scoring systems based on a multinational trauma registry in Asia. Methods This retrospective and multinational cohort study included all adult transferred injury patients from Korea, Malaysia, Vietnam, and Taiwan between 2016 and 2018. The outcome of interest was a death in the emergency department (ED) after the patients' ED visit. Using these results, we developed the interpretable field triage score with the Korea registry using an interpretable machine learning framework and validated the score externally. The performance of each country's score was assessed using the area under the receiver operating characteristic curve (AUROC). Furthermore, a website for real-world application was developed using R Shiny. Findings The study population included 26,294, 9404, 673 and 826 transferred injury patients between 2016 and 2018 from Korea, Malaysia, Vietnam, and Taiwan, respectively. The corresponding rates of a death in the ED were 0.30%, 0.60%, 4.0%, and 4.6% respectively. Age and vital sign were found to be the significant variables for predicting mortality. External validation showed the accuracy of the model with an AUROC of 0.756-0.850. Interpretation The Grade for Interpretable Field Triage (GIFT) score is an interpretable and practical tool to predict mortality in field triage for trauma. Funding This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant Number: HI19C1328).
Collapse
Affiliation(s)
- Jae Yong Yu
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
- Digital & Smart Health Office, Tan Tock Seng Hospital, Singapore
| | - Sejin Heo
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
| | - Feng Xie
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Biomedical Data Science, Stanford University, Stanford, USA
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University, Stanford, USA
| | - Nan Liu
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Health Service Research Centre, Singapore Health Services, Singapore
- Institute of Data Science, National University of Singapore, Singapore
| | - Sun Yung Yoon
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
| | - Han Sol Chang
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
| | - Taerim Kim
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
| | - Se Uk Lee
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke–National University of Singapore Medical School, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| | - Yih Yng Ng
- Digital & Smart Health Office, Tan Tock Seng Hospital, Singapore
| | - Sang Do shin
- Department of Emergency Medicine, Seoul National University College of Medicine, Seoul, South Korea
| | - Kentaro Kajino
- Department of Emergency and Critical Care Medicine, Kansai Medical University, Moriguchi, Japan
| | - Won Chul Cha
- Department of Digital Health, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea
- Digital Innovation Center, Samsung Medical Center, Seoul, South Korea
| |
Collapse
|
3
|
Benchmarking emergency department prediction models with machine learning and public electronic health records. Sci Data 2022; 9:658. [PMID: 36302776 PMCID: PMC9610299 DOI: 10.1038/s41597-022-01782-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 10/14/2022] [Indexed: 11/26/2022] Open
Abstract
The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.
Collapse
|
4
|
Ning Y, Li S, Ong MEH, Xie F, Chakraborty B, Ting DSW, Liu N. A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study. PLOS DIGITAL HEALTH 2022; 1:e0000062. [PMID: 36812536 PMCID: PMC9931273 DOI: 10.1371/journal.pdig.0000062] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Accepted: 05/10/2022] [Indexed: 01/19/2023]
Abstract
Risk scores are widely used for clinical decision making and commonly generated from logistic regression models. Machine-learning-based methods may work well for identifying important predictors to create parsimonious scores, but such 'black box' variable selection limits interpretability, and variable importance evaluated from a single model can be biased. We propose a robust and interpretable variable selection approach using the recently developed Shapley variable importance cloud (ShapleyVIC) that accounts for variability in variable importance across models. Our approach evaluates and visualizes overall variable contributions for in-depth inference and transparent variable selection, and filters out non-significant contributors to simplify model building steps. We derive an ensemble variable ranking from variable contributions across models, which is easily integrated with an automated and modularized risk score generator, AutoScore, for convenient implementation. In a study of early death or unplanned readmission after hospital discharge, ShapleyVIC selected 6 variables from 41 candidates to create a well-performing risk score, which had similar performance to a 16-variable model from machine-learning-based ranking. Our work contributes to the recent emphasis on interpretability of prediction models for high-stakes decision making, providing a disciplined solution to detailed assessment of variable importance and transparent development of parsimonious clinical risk scores.
Collapse
Affiliation(s)
- Yilin Ning
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Siqi Li
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore,Health Services Research Centre, Singapore Health Services, Singapore, Singapore,Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore
| | - Feng Xie
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore,Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore
| | - Bibhas Chakraborty
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore,Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore,Department of Statistics and Data Science, National University of Singapore, Singapore, Singapore,Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, United States of America
| | - Daniel Shu Wei Ting
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore,Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore,SingHealth AI Health Program, Singapore Health Services, Singapore, Singapore
| | - Nan Liu
- Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, Singapore,Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore, Singapore,Health Services Research Centre, Singapore Health Services, Singapore, Singapore,SingHealth AI Health Program, Singapore Health Services, Singapore, Singapore,Institute of Data Science, National University of Singapore, Singapore, Singapore,* E-mail:
| |
Collapse
|
5
|
Yuan H, Xie F, Eng Hock Ong M, Ning Y, Lucas Chee M, Ehsan Saffari S, Rizal Abdullah H, Alan Goldstein B, Chakraborty B, Liu N. AutoScore-Imbalance: An Interpretable Machine Learning Tool for Development of Clinical Scores with Rare Events Data. J Biomed Inform 2022; 129:104072. [PMID: 35421602 DOI: 10.1016/j.jbi.2022.104072] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2021] [Revised: 03/10/2022] [Accepted: 04/07/2022] [Indexed: 02/06/2023]
Abstract
BACKGROUND Medical decision-making impacts both individual and public health. Clinical scores are commonly used among various decision-making models to determine the degree of disease deterioration at the bedside. AutoScore was proposed as a useful clinical score generator based on machine learning and a generalized linear model. However, its current framework still leaves room for improvement when addressing unbalanced data of rare events. METHODS Using machine intelligence approaches, we developed AutoScore-Imbalance, which comprises three components: training dataset optimization, sample weight optimization, and adjusted AutoScore. Baseline techniques for performance comparison included the original AutoScore, full logistic regression, stepwise logistic regression, least absolute shrinkage and selection operator (LASSO), full random forest, and random forest with a reduced number of variables. These models were evaluated based on their area under the curve (AUC) in the receiver operating characteristic analysis and balanced accuracy (i.e., mean value of sensitivity and specificity). By utilizing a publicly accessible dataset from Beth Israel Deaconess Medical Center, we assessed the proposed model and baseline approaches to predict inpatient mortality. RESULTS AutoScore-Imbalance outperformed baselines in terms of AUC and balanced accuracy. The nine-variable AutoScore-Imbalance sub-model achieved the highest AUC of 0.786 (0.732-0.839), while the eleven-variable original AutoScore obtained an AUC of 0.723 (0.663-0.783), and the logistic regression with 21 variables obtained an AUC of 0.743 (0.685-0.800). The AutoScore-Imbalance sub-model (using a down-sampling algorithm) yielded an AUC of 0.771 (0.718-0.823) with only five variables, demonstrating a good balance between performance and variable sparsity. Furthermore, AutoScore-Imbalance obtained the highest balanced accuracy of 0.757 (0.702-0.805), compared to 0.698 (0.643-0.753) by the original AutoScore and the maximum of 0.720 (0.664-0.769) by other baseline models. CONCLUSIONS We have developed an interpretable tool to handle clinical data imbalance, presented its structure, and demonstrated its superiority over baselines. The AutoScore-Imbalance tool can be applied to highly unbalanced datasets to gain further insight into rare medical events and facilitate real-world clinical decision-making.
Collapse
Affiliation(s)
- Han Yuan
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Feng Xie
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Marcus Eng Hock Ong
- Duke-NUS Medical School, National University of Singapore, Singapore; Department of Emergency Medicine, Singapore General Hospital, Singapore; Health Services Research Centre, Singapore Health Services, Singapore
| | - Yilin Ning
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Marcel Lucas Chee
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Australia
| | | | - Hairil Rizal Abdullah
- Duke-NUS Medical School, National University of Singapore, Singapore; Department of Anaesthesiology, Singapore General Hospital, Singapore
| | - Benjamin Alan Goldstein
- Duke-NUS Medical School, National University of Singapore, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, United States
| | - Bibhas Chakraborty
- Duke-NUS Medical School, National University of Singapore, Singapore; Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, United States; Department of Statistics and Data Science, National University of Singapore, Singapore
| | - Nan Liu
- Duke-NUS Medical School, National University of Singapore, Singapore; Health Services Research Centre, Singapore Health Services, Singapore; Institute of Data Science, National University of Singapore, Singapore.
| |
Collapse
|
6
|
Xie F, Liu N, Yan L, Ning Y, Lim KK, Gong C, Kwan YH, Ho AFW, Low LL, Chakraborty B, Ong MEH. Development and validation of an interpretable machine learning scoring tool for estimating time to emergency readmissions. EClinicalMedicine 2022; 45:101315. [PMID: 35284804 PMCID: PMC8904223 DOI: 10.1016/j.eclinm.2022.101315] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/23/2021] [Revised: 01/22/2022] [Accepted: 02/07/2022] [Indexed: 01/06/2023] Open
Abstract
BACKGROUND Emergency readmission poses an additional burden on both patients and healthcare systems. Risk stratification is the first step of transitional care interventions targeted at reducing readmission. To accurately predict the short- and intermediate-term risks of readmission and provide information for further temporal risk stratification, we developed and validated an interpretable machine learning risk scoring system. METHODS In this retrospective study, all emergency admission episodes from January 1st 2009 to December 31st 2016 at a tertiary hospital in Singapore were assessed. The primary outcome was time to emergency readmission within 90 days post discharge. The Score for Emergency ReAdmission Prediction (SERAP) tool was derived via an interpretable machine learning-based system for time-to-event outcomes. SERAP is six-variable survival score, and takes the number of emergency admissions last year, age, history of malignancy, history of renal diseases, serum creatinine level, and serum albumin level during index admission into consideration. FINDINGS A total of 293,589 ED admission episodes were finally included in the whole cohort. Among them, 203,748 episodes were included in the training cohort, 50,937 episodes in the validation cohort, and 38,904 in the testing cohort. Readmission within 90 days was documented in 80,213 (27.3%) episodes, with a median time to emergency readmission of 22 days (Interquartile range: 8-47). For different time points, the readmission rates observed in the whole cohort were 6.7% at 7 days, 10.6% at 14 days, 13.6% at 21 days, 16.4% at 30 days, and 23.0% at 60 days. In the testing cohort, the SERAP achieved an integrated area under the curve of 0.737 (95% confidence interval: 0.730-0.743). For a specific 30-day readmission prediction, SERAP outperformed the LACE index (Length of stay, Acuity of admission, Charlson comorbidity index, and Emergency department visits in past six months) and the HOSPITAL score (Hemoglobin at discharge, discharge from an Oncology service, Sodium level at discharge, Procedure during the index admission, Index Type of admission, number of Admissions during the last 12 months, and Length of stay). Besides 30-day readmission, SERAP can predict readmission rates at any time point during the 90-day period. INTERPRETATION Better performance in risk prediction was achieved by the SERAP than other existing scores, and accurate information about time to emergency readmission was generated for further temporal risk stratification and clinical decision-making. In the future, external validation studies are needed to evaluate the SERAP at different settings and assess their real-world performance. FUNDING This study was supported by the Singapore National Medical Research Council under the PULSES Center Grant, and Duke-NUS Medical School.
Collapse
Affiliation(s)
- Feng Xie
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
| | - Nan Liu
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
- Health Services Research Centre, Singapore Health Services, Singapore
- Institute of Data Science, National University of Singapore, Singapore
- Corresponding author at: Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore.
| | - Linxuan Yan
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
| | - Yilin Ning
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
| | - Ka Keat Lim
- School of Population Health and Environmental Sciences, Faculty of Life Sciences and Medicine, King's College London, London, United Kingdom
- National Institute for Health Research (NIHR) Biomedical Research Centre, Guy's and St Thomas' NHS Foundation Trust and King's College London, London, United Kingdom
| | - Changlin Gong
- Department of Internal Medicine, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Yu Heng Kwan
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
| | - Andrew Fu Wah Ho
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| | - Lian Leng Low
- Department of Family Medicine and Continuing Care, Singapore General Hospital, Singapore
- Department of Post-Acute and Continuing Care, Outram Community Hospital, Singapore
- SingHealth Duke-NUS Family Medicine Academic Clinical Program, Duke-NUS Medical School, Singapore
| | - Bibhas Chakraborty
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
- Department of Statistics and Data Science, National University of Singapore, Singapore
- Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, United States
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, 8 College Road, 169857, Singapore
- Health Services Research Centre, Singapore Health Services, Singapore
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| |
Collapse
|