Development and Validation of Machine Learning Models to Predict Readmission After Colorectal Surgery

A novel artificial intelligence framework to quantify the impact of clinical compared with nonclinical influences on postoperative length of stay

BACKGROUND

The relative proportion of clinical compared with nonclinical influences on length of stay after colectomy has never been measured. We developed a novel machine-learning framework that quantifies the proportion of length of stay after colectomy attributable to clinical factors and infers the overall impact of nonclinical influences.

STUDY DESIGN

Patients who underwent partial colectomy, total colectomy, or low anterior resection included in American College of Surgeons National Surgical Quality Improvement were analyzed. Multivariable linear regression, random forest, and neural network models were developed to assess the impact of 56 clinical variables on length of stay. The random forest and neural network models were fine-tuned to maximize the explanatory power of clinical variables on length of stay. R2 measured the proportion of length of stay explained by clinical factors. The contribution of nonclinical factors was inferred from residual analysis. Mean absolute error was used to measure the discrepancy between actual and model-predicted length of stay.

RESULTS

Of 96,081 patients, 71% underwent partial colectomy (mean length of stay, 6.8 days; standard deviation, 5.6), 27% low anterior resection (5.4; 4.4), and 2% total colectomy (11.8; 7.1). Clinical factors in multivariable linear regression models accounted for only 29-54% of length of stay variability. The random forest and neural network models demonstrated persistent unexplained length of stay variability even when considering nonlinear interactions (R2: random forest [range, 0.46-0.55]; neural network [range, 0.44-0.57]), consistent with multivariable linear regression models. Mean absolute error showed clinical factors could not account for 2-2.5 days of length of stay after low anterior resection and partial colectomy, and 4 days after total colectomy.

CONCLUSION

This is the first study to quantify the overall influence of clinical factors on post-colectomy length of stay, revealing they explain less than 55% of variability. By maximizing clinical factors' explanatory impact using machine learning, the remaining variability is inferred to be nonclinical. Our findings provide hospitals with a novel paradigm to indirectly measure the influence of previously elusive nonclinical factors.

The best machine learning algorithm for building surgical site infection predictive models: A systematic review and network meta-analysis

BACKGROUND

Many machine learning (ML) algorithms have been used to develop surgical site infection (SSI) prediction models, but little is known about their predicting performance. We conducted a network meta-analysis to compare the performance of different ML algorithms and to explore which one may perform best.

METHODS

MEDLINE, EMBASE, CINAHL, Web of Science, and Cochrane Library were systematically searched from inception to November 25, 2023. We included diagnostic accuracy trials constructing SSI predictive model by ML. Two reviewers selected relevant studies and extracted data. The certainty of the evidence was rated using the QUADAS-2 tool. Performance statistics of the diagnostic analysis and the ranking of the different ML algorithms have been expressed in Relative Diagnostic Odds Ratio (RDOR) and superiority index (SI), respectively, using statistical software STATA and R.

RESULTS

Of 493 articles identified, 10 algorithms from 84 SSI prediction models in 40 articles were included in this review. The results of our study revealed that models based on solely surgical type outperformed models without discrimination of surgical type (RDOR 2.71, 95 % CI: 1.25-5.90, P = 0.01), and mixed-use of structured and textual data-based models outperformed models solely based on structured data (RDOR 8.70, 95 % CI: 3.65-20.75, P < 0.01). Combining the ML algorithms in different databased subgroups separately yields the sorted results: Boosted Classifiers had the best overall prediction for the mixed databased model (SI6.17, 95 % CI: 0.09, 13.00), and Support Vector Machine for the structured (SI 4.70, 95 % CI: 0.11, 13.00).

CONCLUSIONS

ML algorithms developed with structured and textual data provided optimal performance. Boosted Classifiers may be the best algorithm in SSI prediction.

Artificial Intelligence in Surgery: A Systematic Review of Use and Validation

Background: Artificial Intelligence (AI) holds promise for transforming healthcare, with AI models gaining increasing clinical use in surgery. However, new AI models are developed without established standards for their validation and use. Before AI can be widely adopted, it is crucial to ensure these models are both accurate and safe for patients. Without proper validation, there is a risk of integrating AI models into practice without sufficient evidence of their safety and accuracy, potentially leading to suboptimal patient outcomes. In this work, we review the current use and validation methods of AI models in clinical surgical settings and propose a novel classification system. Methods: A systematic review was conducted in PubMed and Cochrane using the keywords "validation", "artificial intelligence", and "surgery", following PRISMA guidelines. Results: The search yielded a total of 7627 articles, of which 102 were included for data extraction, encompassing 2,837,211 patients. A validation classification system named Surgical Validation Score (SURVAS) was developed. The primary applications of models were risk assessment and decision-making in the preoperative setting. Validation methods were ranked as high evidence in only 45% of studies, and only 14% of the studies provided publicly available datasets. Conclusions: AI has significant applications in surgery, but validation quality remains suboptimal, and public data availability is limited. Current AI applications are mainly focused on preoperative risk assessment and are suggested to improve decision-making. Classification systems such as SURVAS can help clinicians confirm the degree of validity of AI models before their application in practice.

Bias in Prediction Models to Identify Patients With Colorectal Cancer at High Risk for Readmission After Resection

PURPOSE

Machine learning algorithms are used for predictive modeling in medicine, but studies often do not evaluate or report on the potential biases of the models. Our purpose was to develop clinical prediction models for readmission after surgery in colorectal cancer (CRC) patients and to examine their potential for racial bias.

METHODS

We used the 2012-2020 American College of Surgeons' National Surgical Quality Improvement Program (ACS-NSQIP) Participant Use File and Targeted Colectomy File. Patients were categorized into four race groups - White, Black or African American, Other, and Unknown/Not Reported. Potential predictive features were identified from studies of risk factors of 30-day readmission in CRC patients. We compared four machine learning-based methods - logistic regression (LR), multilayer perceptron (MLP), random forest (RF), and XGBoost (XGB). Model bias was assessed using false negative rate (FNR) difference, false positive rate (FPR) difference, and disparate impact.

RESULTS

In all, 112,077 patients were included, 67.2% of whom were White, 9.2% Black, 5.6% Other race, and 18% with race not recorded. There were significant differences in the AUROC, FPR and FNR between race groups across all models. Notably, patients in the 'Other' race category had higher FNR compared to Black patients in all but the XGB model, while Black patients had higher FPR than White patients in some models. Patients in the 'Other' category consistently had the lowest FPR. Applying the 80% rule for disparate impact, the models consistently met the threshold for unfairness for the 'Other' race category.

CONCLUSION

Predictive models for 30-day readmission after colorectal surgery may perform unequally for different race groups, potentially propagating to inequalities in delivery of care and patient outcomes if the predictions from these models are used to direct care.

Machine Learning-based Prediction of Postoperative Pancreatic Fistula Following Pancreaticoduodenectomy

OBJECTIVE

The aim of this study was to develop a novel machine learning model to predict clinically relevant postoperative pancreatic fistula (CR-POPF) following pancreaticoduodenectomy (PD).

BACKGROUND

Accurate prognostication of CR-POPF may allow for risk stratification and adaptive treatment strategies for potential PD candidates. However, antecedent models, such as the modified Fistula Risk Score (mFRS), are limited by poor discrimination and calibration.

METHODS

All records entailing PD within the 2014 to 2018 American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) were identified. In addition, patients undergoing PD at our institution between 2013 and 2021 were queried from our local data repository. An eXtreme Gradient Boosting (XGBoost) model was developed to estimate the risk of CR-POPF using data from the ACS NSQIP and evaluated using institutional data. Model discrimination was estimated using the area under the receiver operating characteristic (AUROC) and area under the precision recall curve (AUPRC).

RESULTS

Overall, 12,281 and 445 patients undergoing PD were identified within the 2014 to 2018 ACS NSQIP and our institutional registry, respectively. Application of the XGBoost and mFRS scores to the internal validation dataset revealed that the former model had significantly greater AUROC (0.72 vs 0.68, P <0.001) and AUPRC (0.22 vs 0.18, P <0.001). Within the external validation dataset, the XGBoost model remained superior to the mFRS with an AUROC of 0.79 (95% CI: 0.74-0.84) versus 0.75 (95% CI: 0.70-0.80, P <0.001). In addition, AUPRC was higher for the XGBoost model, compared with the mFRS.

CONCLUSION

Our novel machine learning model consistently outperformed the previously validated mFRS within internal and external validation cohorts, thereby demonstrating its generalizability and utility for enhancing prediction of CR-POPF.

Meta-lasso: new insight on infection prediction after minimally invasive surgery

Surgical site infection (SSI) after minimally invasive lung cancer surgery constitutes an important factor influencing the direct and indirect economic implications, patient prognosis, and the 5-year survival rate for early-stage lung cancer patients. In the realm of predictive healthcare, machine learning algorithms have been instrumental in anticipating various surgical outcomes, including SSI. However, accurately predicting infection after minimally invasive surgery remains a clinical challenge due to the multitude of physiological and surgical factors associated with it. Furthermore, clinical patient data, in addition to being high-dimensional, often exists the long-tail problem, posing difficulties for traditional machine learning algorithms in effectively processing such data. Based on this insight, we propose a novel approach called meta-lasso for infection prediction following minimally invasive surgery. Our approach leverages the sparse learning algorithm lasso regression to select informative features and introduces a meta-learning framework to mitigate bias towards the dominant class. We conducted a retrospective cohort study on patients who had undergone minimally invasive surgery for lung cancer at Shanghai Chest Hospital between 2018 and 2020. The evaluation encompassed key performance metrics, including sensitivity, specificity, precision (PPV), negative predictive value (NPV), and accuracy. Our approach has surpassed the performance of logistic regression, random forest, Naive Bayes classifier, gradient boosting decision tree, ANN, and lasso regression, with sensitivity at 0.798, specificity at 0.779, precision at 0.789, NPV at 0.798, and accuracy at 0.788 and has greatly improved the classification performance of the inferior class.

Ambulatory Robotic Colectomy: Factors Affecting and Affected by Postoperative Opioid Use

BACKGROUND

The ongoing opioid crisis demands an investigation into the factors driving postoperative opioid use. Ambulatory robotic colectomies are an emerging concept in colorectal surgery, but concerns persist surrounding adequate pain control for these patients who are discharged very early. We sought to identify key factors affecting recovery room opioid use (ROU) and additional outpatient opioid prescriptions (AOP) after ambulatory robotic colectomies.

METHODS

This was a single-institution retrospective review of ambulatory robotic colon resections performed between 2019 and 2022. Patients were included if they discharged on the same day (SDD) or postoperative day 1 (POD1). Outcomes of interest included ROU [measured in parenteral morphine milligram equivalents (MMEs)], AOP (written between PODs 2 to 7), postoperative emergency department presentations, and readmissions.

RESULTS

Two hundred nineteen cases were examined, 48 of which underwent SDD. The mean ROU was 29.4 MME, and 8.7% of patients required AOP. Between SDD and POD1 patients, there were no differences in postoperative emergency department presentations, readmissions, recovery opioid use, or additional outpatient opioid scripts. Older age was associated with a lower ROU (-0.54 MME for each additional year). Older age, a higher body mass index, and right-sided colectomies were also more likely to use zero ROU. Readmissions were strongly associated with lower ROU. Among SDD patients, lower ROU was also associated with higher rates of AOP.

CONCLUSION

Ambulatory robotic colectomies and SDD can be performed with low opioid use and readmission rates. Notably, we found an association between low ROU and more readmission, and, in some cases, higher AOP. This suggests that adequate pain control during the postoperative recovery phase is a crucial component of reducing these negative outcomes.

Artificial intelligence algorithms for predicting post-operative ileus after laparoscopic surgery

Objective

By constructing a predictive model using machine learning and deep learning technologies, we aim to understand the risk factors for postoperative intestinal obstruction in laparoscopic colorectal cancer patients, and establish an effective artificial intelligence-based predictive model to guide individualized prevention and treatment, thus improving patient outcomes.

Methods

We constructed a model of the artificial intelligence algorithm in Python. Subjects were randomly assigned to either a training set for variable identification and model construction, or a test set for testing model performance, at a ratio of 7:3. The model was trained with ten algorithms. We used the AUC values of the ROC curves, as well as accuracy, precision, recall rate and F1 scores.

Results

The results of feature engineering composited with the GBDT algorithm showed that opioid use, anesthesia duration, and body weight were the top three factors in the development of POI. We used ten machine learning and deep learning algorithms to validate the model, and the results were as follows: the three algorithms with best accuracy were XGB (0.807), Decision Tree (0.807) and Neural DecisionTree (0.807); the two algorithms with best precision were XGB (0.500) and Decision Tree (0.500); the two algorithms with best recall rate were adab (0.243) and Decision Tree (0.135); the two algorithms with highest F1 score were adab (0.290) and Decision Tree (0.213); and the three algorithms with best AUC were Gradient Boosting (0.678), XGB (0.638) and LinearSVC (0.633).

Conclusion

This study shows that XGB and Decision Tree are the two best algorithms for predicting the risk of developing ileus after laparoscopic colon cancer surgery. It provides new insight and approaches to the field of postoperative intestinal obstruction in colorectal cancer through the application of machine learning techniques, thereby improving our understanding of the disease and offering strong support for clinical decision-making.

Using machine learning models to predict the surgical risk of children with pancreaticobiliary maljunction and biliary dilatation

PURPOSE

To develop machine learning (ML) models to predict the surgical risk of children with pancreaticobiliary maljunction (PBM) and biliary dilatation.

METHODS

The subjects of this study were 157 pediatric patients who underwent surgery for PBM with biliary dilatation between January, 2015 and August, 2022. Using preoperative data, four ML models were developed, including logistic regression (LR), random forest (RF), support vector machine classifier (SVC), and extreme gradient boosting (XGBoost). The performance of each model was assessed via the area under the receiver operator characteristic curve (AUC). Model interpretations were generated by Shapley Additive Explanations. A nomogram was used to validate the best-performing model.

RESULTS

Sixty-eight patients (43.3%) were classified as the high-risk surgery group. The XGBoost model (AUC = 0.822) outperformed the LR (AUC = 0.798), RF (AUC = 0.802) and SVC (AUC = 0.804) models. In all four models, enhancement of the choledochal cystic wall and an abnormal position of the right hepatic artery were the two most important features. Moreover, the diameter of the choledochal cyst, bile duct variation, and serum amylase were selected as key predictive factors by all four models.

CONCLUSIONS

Using preoperative data, the ML models, especially XGBoost, have the potential to predict the surgical risk of children with PBM and biliary dilatation. The nomogram may provide surgeons early warning to avoid intraoperative iatrogenic injury.

Recipient Survival after Orthotopic Liver Transplantation: Interpretable Machine Learning Survival Tree Algorithm for Patient-Specific Outcomes

BACKGROUND

Elucidating contributors affecting liver transplant survival is paramount. Current methods offer crude global group outcomes. To refine patient-specific mortality probability estimation and to determine covariate interaction using recipient and donor data, we generated a survival tree algorithm, Recipient Survival After Orthotopic Liver Transplantation (ReSOLT), using United Network Organ Sharing (UNOS) transplant data.

STUDY DESIGN

The UNOS database was queried for liver transplants in patients ≥18 years old between 2000 and 2021. Preoperative factors were evaluated with stepwise logistic regression; 43 significant factors were used in survival tree modeling. Graft survival of <7 days was excluded. The data were split into training and testing sets and further validated with 10-fold cross-validation. Survival tree pruning and model selection was achieved based on Akaike information criterion and log-likelihood values. Log-rank pairwise comparisons between subgroups and estimated survival probabilities were calculated.

RESULTS

A total of 122,134 liver transplant patients were included for modeling. Multivariable logistic regression (area under the curve = 0.742, F1 = 0.822) and survival tree modeling returned 8 significant recipient survival factors: recipient age, donor age, recipient primary payment, recipient hepatitis C status, recipient diabetes, recipient functional status at registration and at transplantation, and deceased donor pulmonary infection. Twenty subgroups consisting of combinations of these factors were identified with distinct Kaplan-Meier survival curves (p < 0.001 among all by log rank test) with 5- and 10-year survival probabilities.

CONCLUSIONS

Survival trees are a flexible and effective approach to understand the effects and interactions of covariates on survival. Individualized survival probability following liver transplant is possible with ReSOLT, allowing for more coherent patient and family counseling and prediction of patient outcome using both recipient and donor factors.