AutoScore-Imbalance: An Interpretable Machine Learning Tool for Development of Clinical Scores with Rare Events Data

Improved interpretable machine learning emergency department triage tool addressing class imbalance

Objective

The Score for Emergency Risk Prediction (SERP) is a novel mortality risk prediction score which leverages machine learning in supporting triage decisions. In its derivation study, SERP-2d, SERP-7d and SERP-30d demonstrated good predictive performance for 2-day, 7-day and 30-day mortality. However, the dataset used had significant class imbalance. This study aimed to determine if addressing class imbalance can improve SERP's performance, ultimately improving triage accuracy.

Methods

The Singapore General Hospital (SGH) emergency department (ED) dataset was used, which contains 1,833,908 ED records between 2008 and 2020. Records between 2008 and 2017 were randomly split into a training set (80%) and validation set (20%). The 2019 and 2020 records were used as test sets. To address class imbalance, we used random oversampling and random undersampling in the AutoScore-Imbalance framework to develop SERP+-2d, SERP+-7d, and SERP+-30d scores. The performance of SERP+, SERP, and the commonly used triage risk scores was compared.

Results

The developed SERP+ scores had five to six variables. The AUC of SERP+ scores (0.874 to 0.905) was higher than that of the corresponding SERP scores (0.859 to 0.894) on both test sets. This superior performance was statistically significant for SERP+-7d (2019: Z = -5.843, p < 0.001, 2020: Z = -4.548, p < 0.001) and SERP+-30d (2019: Z = -3.063, p = 0.002, 2020: Z = -3.256, p = 0.001). SERP+ outperformed SERP marginally on sensitivity, specificity, balanced accuracy, and positive predictive value measures. Negative predictive value was the same for SERP+ and SERP. Additionally, SERP+ showed better performance compared to the commonly used triage risk scores.

Conclusions

Accounting for class imbalance during training improved score performance for SERP+. Better stratification of even a small number of patients can be meaningful in the context of the ED triage. Our findings reiterate the potential of machine learning-based scores like SERP+ in supporting accurate, data-driven triage decisions at the ED.

A universal AutoScore framework to develop interpretable scoring systems for predicting common types of clinical outcomes

The AutoScore framework can automatically generate data-driven clinical scores in various clinical applications. Here, we present a protocol for developing clinical scoring systems for binary, survival, and ordinal outcomes using the open-source AutoScore package. We describe steps for package installation, detailed data processing and checking, and variable ranking. We then explain how to iterate through steps for variable selection, score generation, fine-tuning, and evaluation to generate understandable and explainable scoring systems using data-driven evidence and clinical knowledge. For complete details on the use and execution of this protocol, please refer to Xie et al. (2020),¹ Xie et al. (2022)², Saffari et al. (2022)³ and the online tutorial https://nliulab.github.io/AutoScore/.

Development and Asian-wide validation of the Grade for Interpretable Field Triage (GIFT) for predicting mortality in pre-hospital patients using the Pan-Asian Trauma Outcomes Study (PATOS)

Background

Field triage is critical in injury patients as the appropriate transport of patients to trauma centers is directly associated with clinical outcomes. Several prehospital triage scores have been developed in Western and European cohorts; however, their validity and applicability in Asia remains unclear. Therefore, we aimed to develop and validate an interpretable field triage scoring systems based on a multinational trauma registry in Asia.

Methods

This retrospective and multinational cohort study included all adult transferred injury patients from Korea, Malaysia, Vietnam, and Taiwan between 2016 and 2018. The outcome of interest was a death in the emergency department (ED) after the patients' ED visit. Using these results, we developed the interpretable field triage score with the Korea registry using an interpretable machine learning framework and validated the score externally. The performance of each country's score was assessed using the area under the receiver operating characteristic curve (AUROC). Furthermore, a website for real-world application was developed using R Shiny.

Findings

The study population included 26,294, 9404, 673 and 826 transferred injury patients between 2016 and 2018 from Korea, Malaysia, Vietnam, and Taiwan, respectively. The corresponding rates of a death in the ED were 0.30%, 0.60%, 4.0%, and 4.6% respectively. Age and vital sign were found to be the significant variables for predicting mortality. External validation showed the accuracy of the model with an AUROC of 0.756-0.850.

Interpretation

The Grade for Interpretable Field Triage (GIFT) score is an interpretable and practical tool to predict mortality in field triage for trauma.

Funding

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (Grant Number: HI19C1328).

Patterns of risk-Using machine learning and structural neuroimaging to identify pedophilic offenders

Background

Child sexual abuse (CSA) has become a focal point for lawmakers, law enforcement, and mental health professionals. With high prevalence rates around the world and far-reaching, often chronic, individual, and societal implications, CSA and its leading risk factor, pedophilia, have been well investigated. This has led to a wide range of clinical tools and actuarial instruments for diagnosis and risk assessment regarding CSA. However, the neurobiological underpinnings of pedosexual behavior, specifically regarding hands-on pedophilic offenders (PO), remain elusive. Such biomarkers for PO individuals could potentially improve the early detection of high-risk PO individuals and enhance efforts to prevent future CSA.

Aim

To use machine learning and MRI data to identify PO individuals.

Methods

From a single-center male cohort of 14 PO individuals and 15 matched healthy control (HC) individuals, we acquired diffusion tensor imaging data (anisotropy, diffusivity, and fiber tracking) in literature-based regions of interest (prefrontal cortex, anterior cingulate cortex, amygdala, and corpus callosum). We trained a linear support vector machine to discriminate between PO and HC individuals using these WM microstructure data. Post hoc, we investigated the PO model decision scores with respect to sociodemographic (age, education, and IQ) and forensic characteristics (psychopathy, sexual deviance, and future risk of sexual violence) in the PO subpopulation. We assessed model specificity in an external cohort of 53 HC individuals.

Results

The classifier discriminated PO from HC individuals with a balanced accuracy of 75.5% (sensitivity = 64.3%, specificity = 86.7%, P ₅₀₀₀ = 0.018) and an out-of-sample specificity to correctly identify HC individuals of 94.3%. The predictive brain pattern contained bilateral fractional anisotropy in the anterior cingulate cortex, diffusivity in the left amygdala, and structural prefrontal cortex-amygdala connectivity in both hemispheres. This brain pattern was associated with the number of previous child victims, the current stance on sexuality, and the professionally assessed risk of future sexual violent reoffending.

Conclusion

Aberrant white matter microstructure in the prefronto-temporo-limbic circuit could be a potential neurobiological correlate for PO individuals at high-risk of reoffending with CSA. Although preliminary and exploratory at this point, our findings highlight the general potential of MRI-based biomarkers and particularly WM microstructure patterns for future CSA risk assessment and preventive efforts.

Benchmarking emergency department prediction models with machine learning and public electronic health records

The demand for emergency department (ED) services is increasing across the globe, particularly during the current COVID-19 pandemic. Clinical triage and risk assessment have become increasingly challenging due to the shortage of medical resources and the strain on hospital infrastructure caused by the pandemic. As a result of the widespread use of electronic health records (EHRs), we now have access to a vast amount of clinical data, which allows us to develop prediction models and decision support systems to address these challenges. To date, there is no widely accepted clinical prediction benchmark related to the ED based on large-scale public EHRs. An open-source benchmark data platform would streamline research workflows by eliminating cumbersome data preprocessing, and facilitate comparisons among different studies and methodologies. Based on the Medical Information Mart for Intensive Care IV Emergency Department (MIMIC-IV-ED) database, we created a benchmark dataset and proposed three clinical prediction benchmarks. This study provides future researchers with insights, suggestions, and protocols for managing data and developing predictive tools for emergency care.