Comparison of Machine Learning Techniques for Mortality Prediction in a Prospective Cohort of Older Adults

Short-term All-cause In-hospital Mortality Prediction by Machine Learning Using Numeric Laboratory Results

Introduction

A critical value (or panic value) is a laboratory test result that significantly deviates from the normal value and represents a potentially life-threatening condition requiring immediate action. Although notification of critical values by critical value list (CVL) is a well-established method, their contribution to mortality prediction is unclear.

Methods

A total of 335,430 clinical laboratory results from 92,673 patients from July 2018 to December 2019 were used. Data in the first 12 months were divided into two datasets at a ratio of 70:30, and a 7-day mortality prediction model by machine learning (eXtreme Gradient Boosting [XGB] decision tree) was created using stratified random undersampling data of the 70% dataset. Mortality predictions by the CVL and XGB model were validated using the remaining 30% of the data, as well as different 6-month datasets from July to December 2019.

Results

The true results which were the sum of correct predictions by the XGB model and CVL using the remaining 30% data were 61,535 and 61,024 tests, and the false results which were the sum of incorrect predictions were 5,492 and 6,003, respectively. Furthermore, the true results with the different datasets were 105,956 and 102,061 tests, and the false results were 6,052 and 9,947, respectively. The XGB model was significantly better than CVL (p < 0.001) in both datasets.The receiver operating characteristic-area under the curve values for the 30% and validation data by XGB were 0.9807 and 0.9646, respectively, which were significantly higher than those by CVL (0.7549 and 0.7172, respectively).

Conclusions

Mortality prediction within 7 days by machine learning using numeric laboratory results was significantly better than that by conventional CVL. The results indicate that machine learning enables timely notification to healthcare providers and may be safer than prediction by conventional CVL.

Prediction models of grip strength in adults above 65 years using Korean National Physical Fitness Award Data from 2009 to 2019

PURPOSE

We aimed to determine the best machine learning (ML) regression model for predicting grip strength in adults above 65 years using various independent variables, such as body composition, blood pressure, and physical performance.

METHODS

The data comprised 107,290 participants, of whom 33.3% were male and 66.7% were female in Korean National Fitness Award Data from 2009 to 2019. The dependent variable was grip strength, which was calculated as the mean of right and left grip strength values.

RESULTS

The results showed that the CatBoost Regressor had the lowest mean squared error (M [Formula: see text] SE:16.659 ± 0.549) and highest R2 value (M [Formula: see text] SE:0.719 ± 0.009) among the seven prediction models tested. The importance of independent variables in facilitating model learning was also determined, with the Figure-of-8 walk test being the most significant. These findings suggest that walking ability and grip strength are closely related, and the Figure-of- 8 walk test is a reasonable indicator of grip strength in older adults.

CONCLUSION

The findings of this study can be used to develop more accurate predictive models of grip strength in older adults.

Application of machine learning approaches in predicting clinical outcomes in older adults - a systematic review and meta-analysis

BACKGROUND

Machine learning-based prediction models have the potential to have a considerable positive impact on geriatric care.

DESIGN

Systematic review and meta-analyses.

PARTICIPANTS

Older adults (≥ 65 years) in any setting.

INTERVENTION

Machine learning models for predicting clinical outcomes in older adults were evaluated. A random-effects meta-analysis was conducted in two grouped cohorts, where the predictive models were compared based on their performance in predicting mortality i) under and including 6 months ii) over 6 months.

OUTCOME MEASURES

Studies were grouped into two groups by the clinical outcome, and the models were compared based on the area under the receiver operating characteristic curve metric.

RESULTS

Thirty-seven studies that satisfied the systematic review criteria were appraised, and eight studies predicting a mortality outcome were included in the meta-analyses. We could only pool studies by mortality as there were inconsistent definitions and sparse data to pool studies for other clinical outcomes. The area under the receiver operating characteristic curve from the meta-analysis yielded a summary estimate of 0.80 (95% CI: 0.76 - 0.84) for mortality within 6 months and 0.81 (95% CI: 0.76 - 0.86) for mortality over 6 months, signifying good discriminatory power.

CONCLUSION

The meta-analysis indicates that machine learning models display good discriminatory power in predicting mortality. However, more large-scale validation studies are necessary. As electronic healthcare databases grow larger and more comprehensive, the available computational power increases and machine learning models become more sophisticated; there should be an effort to integrate these models into a larger research setting to predict various clinical outcomes.

Deep-learning model for predicting physical fitness in possible sarcopenia: analysis of the Korean physical fitness award from 2010 to 2023

Introduction

Physical fitness is regarded as a significant indicator of sarcopenia. This study aimed to develop and evaluate a deep-learning model for predicting the decline in physical fitness due to sarcopenia in individuals with potential sarcopenia.

Methods

This study used the 2010-2023 Korean National Physical Fitness Award data. The data comprised exercise- and health-related measurements in Koreans aged >65 years and included body composition and physical fitness variables. Appendicular muscle mass (ASM) was calculated as ASM/height2 to define normal and possible sarcopenia. The deep-learning model was created with EarlyStopping and ModelCheckpoint to prevent overfitting and was evaluated using stratified k-fold cross-validation (k = 5). The model was trained and tested using training data and validation data from each fold. The model's performance was assessed using a confusion matrix, receiver operating characteristic curve, and area under the curve. The average performance metrics obtained from each cross-validation were determined. For the analysis of feature importance, SHAP, permutation feature importance, and LIME were employed as model-agnostic explanation methods.

Results

The deep-learning model proved effective in distinguishing from sarcopenia, with an accuracy of 87.55%, precision of 85.57%, recall of 90.34%, and F1 score of 87.89%. Waist circumference (WC, cm), absolute grip strength (kg), and body fat (BF, %) had an influence on the model output. SHAP, LIME, and permutation feature importance analyses revealed that WC and absolute grip strength were the most important variables. WC, figure-of-8 walk, BF, timed up-and-go, and sit-and-reach emerged as key factors for predicting possible sarcopenia.

Conclusion

The deep-learning model showed high accuracy and recall with respect to possible sarcopenia prediction. Considering the need for the development of a more detailed and accurate sarcopenia prediction model, the study findings hold promise for enhancing sarcopenia prediction using deep learning.

Intercontinental validation of a clinical prediction model for predicting 90-day and 2-year mortality in an Israeli cohort of 2033 patients with a femoral neck fracture aged 65 or above

PURPOSE

Mortality prediction in elderly femoral neck fracture patients is valuable in treatment decision-making. A previously developed and internally validated clinical prediction model shows promise in identifying patients at risk of 90-day and 2-year mortality. Validation in an independent cohort is required to assess the generalizability; especially in geographically distinct regions. Therefore we questioned, is the SORG Orthopaedic Research Group (SORG) femoral neck fracture mortality algorithm externally valid in an Israeli cohort to predict 90-day and 2-year mortality?

METHODS

We previously developed a prediction model in 2022 for estimating the risk of mortality in femoral neck fracture patients using a multicenter institutional cohort of 2,478 patients from the USA. The model included the following input variables that are available on clinical admission: age, male gender, creatinine level, absolute neutrophil, hemoglobin level, international normalized ratio (INR), congestive heart failure (CHF), displaced fracture, hemiplegia, chronic obstructive pulmonary disease (COPD), history of cerebrovascular accident (CVA) and beta-blocker use. To assess the generalizability, we used an intercontinental institutional cohort from the Sheba Medical Center in Israel (level I trauma center), queried between June 2008 and February 2022. Generalizability of the model was assessed using discrimination, calibration, Brier score, and decision curve analysis.

RESULTS

The validation cohort included 2,033 patients, aged 65 years or above, that underwent femoral neck fracture surgery. Most patients were female 64.8% (n = 1317), the median age was 81 years (interquartile range = 75-86), and 80.4% (n = 1635) patients sustained a displaced fracture (Garden III/IV). The 90-day mortality was 9.4% (n = 190) and 2-year mortality was 30.0% (n = 610). Despite numerous baseline differences, the model performed acceptably to the validation cohort on discrimination (c-statistic 0.67 for 90-day, 0.67 for 2-year), calibration, Brier score, and decision curve analysis.

CONCLUSIONS

The previously developed SORG femoral neck fracture mortality algorithm demonstrated good performance in an independent intercontinental population. Current iteration should not be relied on for patient care, though suggesting potential utility in assessing patients at low risk for 90-day or 2-year mortality. Further studies should evaluate this tool in a prospective setting and evaluate its feasibility and efficacy in clinical practice. The algorithm can be freely accessed: https://sorg-apps.shinyapps.io/hipfracturemortality/ .

LEVEL OF EVIDENCE

Level III, Prognostic study.

Development and validation of questionnaire-based machine learning models for predicting all-cause mortality in a representative population of China

Background

Considering that the previously developed mortality prediction models have limited applications to the Chinese population, a questionnaire-based prediction model is of great importance for its accuracy and convenience in clinical practice.

Methods

Two national cohort, namely, the China Health and Nutrition Survey (8,355 individual older than 18) and the China Health and Retirement Longitudinal Study (12,711 individuals older than 45) were used for model development and validation. One hundred and fifty-nine variables were compiled to generate predictions. The Cox regression model and six machine learning (ML) models were used to predict all-cause mortality. Finally, a simple questionnaire-based ML prediction model was developed using the best algorithm and validated.

Results

In the internal validation set, all the ML models performed better than the traditional Cox model in predicting 6-year mortality and the random survival forest (RSF) model performed best. The questionnaire-based ML model, which only included 20 variables, achieved a C-index of 0.86 (95%CI: 0.80-0.92). On external validation, the simple questionnaire-based model achieved a C-index of 0.82 (95%CI: 0.77-0.87), 0.77 (95%CI: 0.75-0.79), and 0.79 (95%CI: 0.77-0.81), respectively, in predicting 2-, 9-, and 11-year mortality.

Conclusions

In this prospective population-based study, a model based on the RSF analysis performed best among all models. Furthermore, there was no significant difference between the prediction performance of the questionnaire-based ML model, which only included 20 variables, and that of the model with all variables (including laboratory variables). The simple questionnaire-based ML prediction model, which needs to be further explored, is of great importance for its accuracy and suitability to the Chinese general population.

Study and validation of an explainable machine learning-based mortality prediction following emergency surgery in the elderly: A prospective observational study

INTRODUCTION

The heterogeneity of procedures and the variety of comorbidities of the patients undergoing surgery in an emergency setting makes perioperative risk stratification, planning, and risk mitigation crucial. In this optic, Machine Learning has the capability of deriving data-driven predictions based on multivariate interactions of thousands of instances. Our aim was to cross-validate and test interpretable models for the prediction of post-operative mortality after any surgery in an emergency setting on elderly patients.

METHODS

This study is a secondary analysis derived from the FRAILESEL study, a multi-center (N = 29 emergency care units), nationwide, observational prospective study with data collected between 06-2017 and 06-2018 investigating perioperative outcomes of elderly patients (age≥65 years) undergoing emergency surgery. Demographic and clinical data, medical and surgical history, preoperative risk factors, frailty, biochemical blood examination, vital parameters, and operative details were collected and the primary outcome was set to the 30-day mortality.

RESULTS

Of the 2570 included patients (50.66% males, median age 77 [IQR = 13] years) 238 (9.26%) were in the non-survivors group. The best performing solution (MultiLayer Perceptron) resulted in a test accuracy of 94.9% (sensitivity = 92.0%, specificity = 95.2%). Model explanations showed how non-chronic cardiac-related comorbidities reduced activities of daily living, low consciousness levels, high creatinine and low saturation increase the risk of death following surgery.

CONCLUSIONS

In this prospective observational study, a robustly cross-validated model resulted in better predictive performance than existing tools and scores in literature. By using only preoperative features and by deriving patient-specific explanations, the model provides crucial information during shared decision-making processes required for risk mitigation procedures.