Predictability of Mortality in Patients With Myocardial Injury After Noncardiac Surgery Based on Perioperative Factors via Machine Learning: Retrospective Study

A review of evaluation approaches for explainable AI with applications in cardiology

Explainable artificial intelligence (XAI) elucidates the decision-making process of complex AI models and is important in building trust in model predictions. XAI explanations themselves require evaluation as to accuracy and reasonableness and in the context of use of the underlying AI model. This review details the evaluation of XAI in cardiac AI applications and has found that, of the studies examined, 37% evaluated XAI quality using literature results, 11% used clinicians as domain-experts, 11% used proxies or statistical analysis, with the remaining 43% not assessing the XAI used at all. We aim to inspire additional studies within healthcare, urging researchers not only to apply XAI methods but to systematically assess the resulting explanations, as a step towards developing trustworthy and safe models.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10462-024-10852-w.

Prediction of Complications and Prognostication in Perioperative Medicine: A Systematic Review and PROBAST Assessment of Machine Learning Tools

BACKGROUND

The utilization of artificial intelligence and machine learning as diagnostic and predictive tools in perioperative medicine holds great promise. Indeed, many studies have been performed in recent years to explore the potential. The purpose of this systematic review is to assess the current state of machine learning in perioperative medicine, its utility in prediction of complications and prognostication, and limitations related to bias and validation.

METHODS

A multidisciplinary team of clinicians and engineers conducted a systematic review using the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) protocol. Multiple databases were searched, including Scopus, Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Cochrane Library, PubMed, Medline, Embase, and Web of Science. The systematic review focused on study design, type of machine learning model used, validation techniques applied, and reported model performance on prediction of complications and prognostication. This review further classified outcomes and machine learning applications using an ad hoc classification system. The Prediction model Risk Of Bias Assessment Tool (PROBAST) was used to assess risk of bias and applicability of the studies.

RESULTS

A total of 103 studies were identified. The models reported in the literature were primarily based on single-center validations (75%), with only 13% being externally validated across multiple centers. Most of the mortality models demonstrated a limited ability to discriminate and classify effectively. The PROBAST assessment indicated a high risk of systematic errors in predicted outcomes and artificial intelligence or machine learning applications.

CONCLUSIONS

The findings indicate that the development of this field is still in its early stages. This systematic review indicates that application of machine learning in perioperative medicine is still at an early stage. While many studies suggest potential utility, several key challenges must be first overcome before their introduction into clinical practice.

EDITOR’S PERSPECTIVE

Short-term All-cause In-hospital Mortality Prediction by Machine Learning Using Numeric Laboratory Results

Introduction

A critical value (or panic value) is a laboratory test result that significantly deviates from the normal value and represents a potentially life-threatening condition requiring immediate action. Although notification of critical values by critical value list (CVL) is a well-established method, their contribution to mortality prediction is unclear.

Methods

A total of 335,430 clinical laboratory results from 92,673 patients from July 2018 to December 2019 were used. Data in the first 12 months were divided into two datasets at a ratio of 70:30, and a 7-day mortality prediction model by machine learning (eXtreme Gradient Boosting [XGB] decision tree) was created using stratified random undersampling data of the 70% dataset. Mortality predictions by the CVL and XGB model were validated using the remaining 30% of the data, as well as different 6-month datasets from July to December 2019.

Results

The true results which were the sum of correct predictions by the XGB model and CVL using the remaining 30% data were 61,535 and 61,024 tests, and the false results which were the sum of incorrect predictions were 5,492 and 6,003, respectively. Furthermore, the true results with the different datasets were 105,956 and 102,061 tests, and the false results were 6,052 and 9,947, respectively. The XGB model was significantly better than CVL (p < 0.001) in both datasets.The receiver operating characteristic-area under the curve values for the 30% and validation data by XGB were 0.9807 and 0.9646, respectively, which were significantly higher than those by CVL (0.7549 and 0.7172, respectively).

Conclusions

Mortality prediction within 7 days by machine learning using numeric laboratory results was significantly better than that by conventional CVL. The results indicate that machine learning enables timely notification to healthcare providers and may be safer than prediction by conventional CVL.

Development and validation of a prediction model to predict major adverse cardiovascular events in elderly patients undergoing noncardiac surgery: A retrospective cohort study

BACKGROUND AND AIMS

Current existing predictive tools have limitations in predicting major adverse cardiovascular events (MACEs) in elderly patients. We will build a new prediction model to predict MACEs in elderly patients undergoing noncardiac surgery by using traditional statistical methods and machine learning algorithms.

METHODS

MACEs were defined as acute myocardial infarction (AMI), ischemic stroke, heart failure and death within 30 days after surgery. Clinical data from 45,102 elderly patients (≥65 years old), who underwent noncardiac surgery from two independent cohorts, were used to develop and validate the prediction models. A traditional logistic regression and five machine learning models (decision tree, random forest, LGBM, AdaBoost, and XGBoost) were compared by the area under the receiver operating characteristic curve (AUC). In the traditional prediction model, the calibration was assessed using the calibration curve and the patients' net benefit was measured by decision curve analysis (DCA).

RESULTS

Among 45,102 elderly patients, 346 (0.76%) developed MACEs. The AUC of this traditional model was 0.800 (95% CI, 0.708-0.831) in the internal validation set, and 0.768 (95% CI, 0.702-0.835) in the external validation set. In the best machine learning prediction model-AdaBoost model, the AUC in the internal and external validation set was 0.778 and 0.732, respectively. Besides, for the traditional prediction model, the calibration curve of model performance accurately predicted the risk of MACEs (Hosmer and Lemeshow, p = 0.573), the DCA results showed that the nomogram had a high net benefit for predicting postoperative MACEs.

CONCLUSIONS

This prediction model based on the traditional method could accurately predict the risk of MACEs after noncardiac surgery in elderly patients.

Machine learning-based prediction model for postoperative delirium in non-cardiac surgery

BACKGROUND

Postoperative delirium is a common complication that is distressing. This study aimed to demonstrate a prediction model for delirium.

METHODS

Among 203,374undergoing non-cardiac surgery between January 2011 and June 2019 at Samsung Medical Center, 2,865 (1.4%) were diagnosed with postoperative delirium. After comparing performances of machine learning algorithms, we chose variables for a prediction model based on an extreme gradient boosting algorithm. Using the top five variables, we generated a prediction model for delirium and conducted an external validation. The Kaplan-Meier and Cox survival analyses were used to analyse the difference of delirium occurrence in patients classified as a prediction model.

RESULTS

The top five variables selected for the postoperative delirium prediction model were age, operation duration, physical status classification, male sex, and surgical risk. An optimal probability threshold in this model was estimated to be 0.02. The area under the receiver operating characteristic (AUROC) curve was 0.870 with a 95% confidence interval of 0.855-0.885, and the sensitivity and specificity of the model were 0.76 and 0.84, respectively. In an external validation, the AUROC was 0.867 (0.845-0.877). In the survival analysis, delirium occurred more frequently in the group of patients predicted as delirium using an internal validation dataset (p < 0.001).

CONCLUSION

Based on machine learning techniques, we analyzed a prediction model of delirium in patients who underwent non-cardiac surgery. Screening for delirium based on the prediction model could improve postoperative care. The working model is provided online and is available for further verification among other populations.

TRIAL REGISTRATION

KCT 0006363.

Prediction model for myocardial injury after non-cardiac surgery using machine learning

Myocardial injury after non-cardiac surgery (MINS) is strongly associated with postoperative outcomes. We developed a prediction model for MINS and have provided it online. Between January 2010 and June 2019, a total of 6811 patients underwent non-cardiac surgery with normal preoperative level of cardiac troponin (cTn). We used machine learning techniques with an extreme gradient boosting algorithm to evaluate the effects of variables on MINS development. We generated two prediction models based on the top 12 and 6 variables. MINS was observed in 1499 (22.0%) patients. The top 12 variables in descending order according to the effects on MINS are preoperative cTn level, intraoperative inotropic drug infusion, operation duration, emergency operation, operation type, age, high-risk surgery, body mass index, chronic kidney disease, coronary artery disease, intraoperative red blood cell transfusion, and current alcoholic use. The prediction models are available at https://sjshin.shinyapps.io/mins_occur_prediction/ . The estimated thresholds were 0.47 in 12-variable models and 0.53 in 6-variable models. The areas under the receiver operating characteristic curves are 0.78 (95% confidence interval [CI] 0.77-0.78) and 0.77 (95% CI 0.77-0.78), respectively, with an accuracy of 0.97 for both models. Using machine learning techniques, we demonstrated prediction models for MINS. These models require further verification in other populations.

Future of Perioperative Precision Medicine: Integration of Molecular Science, Dynamic Health Care Informatics, and Implementation of Predictive Pathways in Real Time

Conceptually, precision medicine is a deep dive to discover disease origin at the molecular or genetic level, thus providing insights that allow clinicians to design corresponding individualized patient therapies. We know that a disease state is created by not only certain molecular derangements but also a biologic milieu promoting the expression of such derangements. These factors together lead to manifested symptoms. At the level of molecular definition, every average, "similar" individual stands to be "dissimilar." Hence, there is the need for customized therapy, moving away from therapy based on aggregate statistics. The perioperative state is a mix of several, simultaneously active molecular mechanisms, surgical insult, drugs, severe inflammatory response, and the body's continuous adaptation to maintain a state of homeostasis. Postoperative outcomes are a net result of several of those rapid genetic and molecular transformations that do or do not ensue. With the advent and advances of artificial intelligence, the translation from identifying these intricate mechanisms to implementing them in clinical practice has made a huge leap. Precision medicine is gaining ground with the help of personalized health recorders and personal devices that identify disease mechanics, patient-reported outcomes, adverse drug reactions, and drug-drug interaction at the individual level in a closed-loop feedback system. This phenomenon is especially true given increasing surgeries in older adults, many of whom are on multiple medications and varyingly frail. In this era of precision medicine, to provide a comprehensive remedy, the perioperative surgical home must expand, incorporating not only clinicians but also basic science experts and data scientists.