Comparing Machine Learning to Regression Methods for Mortality Prediction Using Veterans Affairs Electronic Health Record Clinical Data

Predictive Modeling of Drug-Related Adverse Events with Real-World Data: A Case Study of Linezolid Hematologic Outcomes

Electronic health records (EHRs) provide meaningful knowledge of drug-related adverse events (AEs) that are not captured in standard drug development and postmarketing surveillance. Using variables obtained from EHR data in the University of California San Francisco de-identified Clinical Data Warehouse, we aimed to evaluate the potential of machine learning to predict two hematological AEs, thrombocytopenia and anemia, in a cohort of patients treated with linezolid for 3 or more days. Features for model input were extracted at linezolid initiation (index), and outcomes were characterized from index to 14 days post-treatment. Random forest classification (RFC) was used for AE prediction, and reduced feature models were evaluated using cumulative importance (cImp) for feature selection. Grade 3+ thrombocytopenia and anemia occurred in 31% of 2,171 and 56% of 2,170 evaluable patients, respectively. Of the total 53 features, as few as 7 contributed at least 50% cImp, resulting in prediction accuracies of 70% or higher and area under the receiver operating characteristic curves of 0.886 for grade 3+ thrombocytopenia and 0.759 for grade 3+ anemia. Sensitivity analyses in strictly defined patient subgroups revealed similarly high predictive performance in full and reduced feature models. A logistic regression model with the same 50% cImp features showed similar predictive performance as RFC and good concordance with RFC probability predictions after isotonic calibration, adding interpretability. Collectively, this work demonstrates potential for machine learning prediction of AE risk in real-world patients using few variables regularly available in EHRs, which may aid in clinical decision making and/or monitoring.

Development and validation of a stacking ensemble model for death prediction in the Chinese Longitudinal Healthy Longevity Survey (CLHLS)

OBJECTIVE

This study aimed to develop and validate a mortality risk prediction model for older people based on the Chinese Longitudinal Healthy Longevity Survey using the stacking ensemble strategy.

MATERIAL AND METHODS

A total of 12,769 participants aged 65 or more at baseline were included. Ensemble machine learning models were applied to develop a mortality prediction model. We selected three base learners, including logistic regression, eXtreme Gradient Boosting, and Categorical + Boosting, and used logistic regression as the meta-learner. The primary outcome was five-year survival. Variable importance was evaluated by the SHapley Additive exPlanations method.

RESULTS

The mean age at baseline was 88, and 57.8 % of participants were women. The CatBoost model performed the best among the three base learners, the area under the receiver operating characteristics curve (AUC) reached 0.8469 (95%CI: 0.8345-0.8593), and the stacking ensemble model further improved the discrimination ability (AUC = 0.8486, 95%CI: 0.8367-0.8612, P = 0.046). Conventional logistic regression had comparable performance (AUC = 0.8470, 95 % CI: 0.8346-0.8595). Older age, higher scores for self-care activities of daily living, being male, higher objective physical performance capacity scores, not undertaking housework, and lower scores on the Mini-Mental State Examination contributed to higher risk.

CONCLUSIONS

We successfully constructed and validated a few death risk prediction models for a Chinese population of older adults. While the stacking ensemble approach had the best prediction performance, the improvement over conventional logistic regression was insubstantial.

Do functional status and Medicare claims data improve the predictive accuracy of an electronic health record mortality index? Findings from a national Veterans Affairs cohort

Background

Electronic health record (EHR) prediction models may be easier to use in busy clinical settings since EHR data can be auto-populated into models. This study assessed whether adding functional status and/or Medicare claims data (which are often not available in EHRs) improves the accuracy of a previously developed Veterans Affairs (VA) EHR-based mortality index.

Methods

This was a retrospective cohort study of veterans aged 75 years and older enrolled in VA primary care clinics followed from January 2014 to April 2020 (n = 62,014). We randomly split participants into development (n = 49,612) and validation (n = 12,402) cohorts. The primary outcome was all-cause mortality. We performed logistic regression with backward stepwise selection to develop a 100-predictor base model using 854 EHR candidate variables, including demographics, laboratory values, medications, healthcare utilization, diagnosis codes, and vitals. We incorporated functional measures in a base + function model by adding activities of daily living (range 0-5) and instrumental activities of daily living (range 0-7) scores. Medicare data, including healthcare utilization (e.g., emergency department visits, hospitalizations) and diagnosis codes, were incorporated in a base + Medicare model. A base + function + Medicare model included all data elements. We assessed model performance with the c-statistic, reclassification metrics, fraction of new information provided, and calibration plots.

Results

In the overall cohort, mean age was 82.6 years and 98.6% were male. At the end of follow-up, 30,263 participants (48.8%) had died. The base model c-statistic was 0.809 (95% CI 0.805-0.812) in the development cohort and 0.804 (95% CI 0.796-0.812) in the validation cohort. Validation cohort c-statistics for the base + function, base + Medicare, and base + function + Medicare models were 0.809 (95% CI 0.801-0.816), 0.811 (95% CI 0.803-0.818), and 0.814 (95% CI 0.807-0.822), respectively. Adding functional status and Medicare data resulted in similarly small improvements among other model performance measures. All models showed excellent calibration.

Conclusions

Incorporation of functional status and Medicare data into a VA EHR-based mortality index led to small but likely clinically insignificant improvements in model performance.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-022-03126-z.