AutoScore-Ordinal: an interpretable machine learning framework for generating scoring models for ordinal outcomes

Machine learning risk-prediction model for in-hospital mortality in Takotsubo cardiomyopathy

BACKGROUND

Takotsubo cardiomyopathy (TC) is an acute heart failure syndrome characterized by transient left ventricular dysfunction, often triggered by stress. Data on risk scores predicting mortality in TC is sparse. We developed a machine-learning risk score model to predict in-hospital mortality in patients with TC.

METHODS

The National Inpatient Sample (NIS) database 2016-2020 was queried to identify adult patients (≥18 years) with TC using ICD-10 code I51.81. The primary outcome was in-hospital mortality. The dataset was randomly split into training (70 %), validation (20 %), and testing (10 %) dataset. Model performance was assessed using the area under the curve (AUC) with 95 % confidence intervals (95 % CI).

RESULTS

Amongst 38,662 TC patients identified [mean age 67.15 ± 14.17 years, female 32,089 (83 %)], 2499 (6.5 %) died. A novel risk score (0-127) was developed on age, race, Elixhauser comorbidity burden, history of hypertension, history of cardiac arrhythmia, presentation of cardiac arrest, cardiogenic shock, and acute kidney injury. Model AUCs (95 % CI) in the training, validation, and testing datasets were 0.809 (0.781-0.838), 0.809 (0.780-0.837), and 0.838 (0.820-0.856), respectively.

CONCLUSION

TC carries high morbidity and mortality. Our novel machine learning-based risk score is an important tool for risk stratification. External validation is needed to confirm these findings.

A systematic review of prediction models on arteriovenous fistula: Risk scores and machine learning approaches

OBJECTIVE

Failure-to-mature and early stenosis remains the Achille's heel of hemodialysis arteriovenous fistula (AVF) creation. The maturation and patency of an AVF can be influenced by a variety of demographic, comorbidity, and anatomical factors. This study aims to review the prediction models of AVF maturation and patency with various risk scores and machine learning models.

DATA SOURCES AND REVIEW METHODS

Literature search was performed on PubMed, Scopus, and Embase to identify eligible articles. The quality of the studies was assessed using the Prediction model Risk Of Bias ASsessment (PROBAST) Tool. The performance (discrimination and calibration) of the included studies were extracted.

RESULTS

Fourteen studies (seven studies used risk score approaches; seven studies used machine learning approaches) were included in the review. Among them, 12 studies were rated as high or unclear "risk of bias." Six studies were rated as high concern or unclear for "applicability." C-statistics (Model discrimination metric) was reported in five studies using risk score approach (0.70-0.886) and three utilized machine learning methods (0.80-0.85). Model calibration was reported in three studies. Failure-to-mature risk score developed by one of the studies has been externally validated in three different patient populations, however the model discrimination degraded significantly (C-statistics: 0.519-0.53).

CONCLUSION

The performance of existing predictive models for AVF maturation/patency is underreported. They showed satisfactory performance in their own study population. However, there was high risk of bias in methodology used to build some of the models. The reviewed models also lack external validation or had reduced performance in external cohort.

A point-based cognitive impairment scoring system for southeast Asian adults

BACKGROUND

Cognitive impairment is a growing concern in Southeast Asian populations, where the burden of cerebrovascular disease (CeVD) is high. Currently, there is no point-based scoring system for identifying cognitive impairment in these populations.

OBJECTIVE

To develop and validate a simple point-based Cognitive Impairment Scoring System (CISS) for identifying individuals with cognitive impairment no dementia (CIND) and concomitant CeVD in Southeast Asian populations.

DESIGN

A cross-sectional study using data from two population-based studies.

SETTING

Community-based setting in Southeast Asia.

PARTICIPANTS

1,511 Southeast Asian adults (664 with CIND, 44.0 %).

MEASURES

Two CISS measures were developed: a basic measure including 11 easily assessable risk factors, and an extended measure incorporating seven additional neuroimaging markers. Performance was evaluated using receiver operating characteristic analysis (AUC) and calibration plots.

RESULTS

The AUC for CISS-basic and CISS-extended were 0.81 (95 %CI, 0.76-0.86) and 0.85 (95 %CI, 0.81-0.89), respectively. Calibration plots indicated satisfactory fit for both the basic measure (p=0.82) and the extended measure (p=0.17). The basic measure included age, gender, ethnicity, education, systolic blood pressure, BMI, smoking history, diabetes, hyperlipidemia, stroke history, and mild/moderate depression. The extended measure added neuroimaging markers of CeVD and brain atrophy.

CONCLUSION

The CISS provides a quick, objective, and clinically relevant tool for assessing cognitive impairment risk in Southeast Asian populations. The basic measure is suitable for initial community-based screenings, while the extended measure offers higher specificity for probable diagnosis. This point-based system enables rapid estimation of cognitive status without requiring complex calculations, potentially improving early detection and management of cognitive impairment in clinical practice.

Developing and validating a prediction tool for cerebral amyloid angiopathy neuropathological severity

INTRODUCTION

Cerebral amyloid angiopathy (CAA) is a cerebrovascular condition, the severity of which can only be determined post mortem. Here, we developed machine learning models, the Florey CAA Score (FCAAS), to predict CAA severity (none/mild/moderate/severe).

METHODS

Building on an auto-score-ordinal algorithm, the FCAAS models were developed and validated using data collected by three cohort studies of aging and dementia. The developed FCAAS models were digitized as a web-based tool. A pilot trial was conducted using this web-based tool.

RESULTS

The FCAAS-4 achieved a mean area under the receiver operating characteristic curve (AUC-ROC) of 0.74 (95% confidence interval: 0.71-0.77) and a Harrell generalized c-index of 0.72 (0.70-0.75). Pilot trial results obtained a mean AUC-ROC of 0.82 (0.71-0.85) and Harrell generalized c-index 0.79 (0.73-0.82).

DISCUSSION

The FCAAS models demonstrate a promising performance in predicting CAA severity. This framework holds the potential for predicting development of amyloid-related imaging abnormalities (ARIAs), given the CAA-ARIAs link.

HIGHLIGHTS

The severity of cerebral amyloid angiopathy (CAA) can only be determined post mortem. A web tool, the Florey CAA Score (FCAAS), was developed to predict CAA severity. The FCAAS holds the potential to be used for CAA risk stratification in clinics. CAA is linked to increased risk of amyloid-related imaging abnormalities (ARIAs). The framework used by FCAAS can possibly be adapted to predict ARIAs risk.

Machine Learning for Early Detection of Cognitive Decline in Parkinson's Disease Using Multimodal Biomarker and Clinical Data

Background: Parkinson's disease (PD) is the second most common neurodegenerative disease, primarily affecting the middle-aged to elderly population. Among its nonmotor symptoms, cognitive decline (CD) is a precursor to dementia and represents a critical target for early risk assessment and diagnosis. Accurate CD prediction is crucial for timely intervention and tailored management of at-risk patients. This study used machine learning (ML) techniques to predict the CD risk over five-year in early-stage PD. Methods: Data from the Early Parkinson's Disease Longitudinal Singapore (2014 to 2018) was used to predict CD defined as a one-unit annual decrease or a one-unit decline in Montreal Cognitive Assessment over two consecutive years. Four ML methods-AutoScore, Random Forest, K-Nearest Neighbors and Neural Network-were applied using baseline demographics, clinical assessments and blood biomarkers. Results: Variable selection identified key predictors of CD, including education year, diastolic lying blood pressure, diastolic standing blood pressure, systolic lying blood pressure, Hoehn and Yahr scale, body mass index, phosphorylated tau at threonine 181, total tau, Neurofilament light chain and suppression of tumorigenicity 2. Random Forest was the most effective, achieving an AUC of 0.93 (95% CI: 0.89, 0.97), using 10-fold cross-validation. Conclusions: Here, we demonstrate that ML-based models can identify early-stage PD patients at high risk for CD, supporting targeted interventions and improved PD management.

LC risk score - development and evaluation of a scale for assessing the risk of developing long COVID

INTRODUCTION

The aim of the study was to create a valuable practical tool for evaluating the risk of developing long COVID.

MATERIAL AND METHODS

1150 patients from the Polish STOP-COVID registry (PoLoCOV study) were used to develop the risk score. The patients were ill between 03/2020 and 04/2022. To develop a clinically useful scoring model, the LC risk score was generated using the machine learning-based framework AutoScore. Patient data were first randomised into a training (70% of output) and a test (30% of output) cohort. Due to the relatively small study group, cross-validation was used. Model predictive ability was evaluated based on the ROC curve and the AUC value. The result of the risk score for a given patient was the total value of points assigned to selected variables.

RESULTS

To create the LC risk score, eight variables were ultimately selected due to their significance and clinical value. Female gender significantly contributed to higher final outcome values, with age range 40-49, body mass index < 18.5 kg/m2, hospitalisation during active disease, arthralgia, myalgia as well as loss of taste and smell during infection, COVID-19 symptoms lasting at least 14 days, and unvaccinated status. The final predictive value of the developed LC risk score for a cut-off of 58 points was AUC = 0.630 (95% CI: 0.571-0.688) with sensitivity 39.80%, specificity 85.1%, positive predictive value 80.8%, and negative predictive value 47.3%.

CONCLUSIONS

The LC risk score may be a practical and undemanding utility that employs basic sociodemographic data, vaccination status, and symptoms during COVID-19 to assess the risk of long COVID.

Development and Validation of a Machine Learning Risk-Prediction Model for 30-Day Readmission for Heart Failure Following Transcatheter Aortic Valve Replacement (TAVR-HF Score)

Transcatheter aortic valve replacement (TAVR) is the treatment of choice for patients with severe aortic stenosis across the spectrum of surgical risk. About one-third of 30-day readmissions following TAVR are related to heart failure (HF). Hence, we aim to develop an easy-to-use clinical predictive model to identify patients at risk for HF readmission. We used data from the National Readmission Database (2015-2018) utilizing ICD-10 codes to identify TAVR procedures. Readmission was defined as the first unplanned HF readmission within 30-day of discharge. A machine learning framework was used to develop a 30-day TAVR-HF readmission score. The receiver operator characteristic curve was used to evaluate the predictive power of the model. A total of 92,363 cases of TAVR were included in the analysis. Of the included patients, 3299 (3.6%) were readmitted within 30 days of discharge with HF. Individuals who got readmitted, vs those without readmission, had more emergent admissions during index procedure (33.4% vs 19.8%), electrolyte abnormalities (38% vs 16.7%), chronic kidney disease (34.8% vs 21.2%), and atrial fibrillation (60.1% vs 40.7%). Candidate variables were ranked by importance using a parsimony plot. A total of 7 variables were selected based on predictive ability as well as clinical relevance: HF with reduced ejection fraction (25 points), HF preserved EF (20 points), electrolyte abnormalities (17 points), atrial fibrillation (12 points), Charlson comorbidity index (<6 = 0, 6-8 = 9, 9-10 = 13, >10 = 14 points), chronic kidney disease (7 points), and emergent index admission (5 points). On performance evaluation using the testing dataset, an area under the curve of 0.761 (95% CI 0.744-0.778) was achieved. Thirty-day TAVR-HF readmission score is an easy-to-use risk prediction tool. The score can be incorporated into electronic health record systems to identify at-risk individuals for readmissions with HF following TAVR. However, further external validation studies are needed.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),1 Xie et al. (2022)2, Saffari et al. (2022)3 and the online tutorial https://nliulab.github.io/AutoScore/.