Development and validation of a machine learning ASA-score to identify candidates for comprehensive preoperative screening and risk stratification

Development and Validation of a Risk Model to Predict Intraoperative Blood Transfusion

Importance

Crossmatched packed red blood cells (pRBC) that are not transfused result in significant waste of this scarce resource. Efficient utilization should be part of a patient blood management strategy.

Objective

To develop and validate a prediction model to identify surgical patients at high risk of intraoperative pRBC transfusion.

Design, Setting, and Participants

This prognostic study used hospital registry data from 2 quaternary hospital networks from January 2016 to June 2021 (development: Montefiore Medical Center [MMC], Bronx, New York), June 2021 to February 2023 (internal validation: MMC), and January 2008 to June 2022 (external validation: Beth Israel Deaconess Medical Center [BIDMC], Boston, Massachusetts). Participants were patients aged 18 years or older undergoing surgery.

Main Outcome and Measures

The outcome was intraoperative transfusion of 1 or more pRBC units. Based on a priori-defined candidate predictors, stepwise backward regression was applied to develop a computational model of independent predictors for intraoperative pRBC transfusion.

Results

The development and validation cohorts consisted of 816 618 patients (273 654 at MMC: mean [SD], age 57.5 [17.2] years; 161 481 [59.0%] female; 542 964 at BIDMC: mean [SD] age, 56.0 [17.1] years; 310 272 [57.1%] female). Overall, 18 662 patients (2.3%) received at least 1 unit of pRBC. The final model contained 24 preoperative predictors: nonambulatory surgery; American Society of Anesthesiologists physical status; international normalized ratio; redo surgery; emergency surgery or surgery outside of regular working hours; estimated surgical duration of at least 120 minutes; surgical complexity; liver disease; hypoalbuminemia; thrombocytopenia; mild, moderate, or severe anemia; and surgery type. The area under the receiver operating characteristic curve (AUC) was 0.93 (95% CI, 0.92-0.93), suggesting high predictive accuracy and generalizability. Positive predictive value (PPV) and negative predictive value (NPV) were 8.9% (95% CI, 8.7%-9.2%) and 99.7% (95% CI, 99.7%-99.7%), respectively, with increased predictive values for operations with a higher a priori risk of pRBC transfusion. The model's performance was confirmed in internal and external validation. The prediction tool outperformed the established Transfusion Risk Understanding Scoring Tool (AUC, 0.64 [0.63-0.64]; PPV, 2.6% [95% CI, 2.5%-2.6%]; NPV, 99.2% [95% CI, 99.1%-99.3%]) (P < .001) and was noninferior to 3 machine learning-derived scores.

Conclusions and Relevance

In this prognostic study of surgical patients, the Transfusion Forecast Utility for Surgical Events (TRANSFUSE) model for predicting intraoperative pRBC transfusion was developed and validated. The instrument can be used independently of machine learning infrastructure availability to inform preoperative pRBC orders and to minimize waste of nontransfused red blood cell units.

Association of anaesthesia-directed sedation with unplanned discharge to a nursing home following non-ambulatory interventional radiology and endoscopic procedures: a retrospective cohort study

INTRODUCTION

Interventional radiology procedures and endoscopies are performed commonly worldwide, often necessitating pharmacological sedation to optimise patient comfort. It is unclear to what extent non-anaesthetists should provide procedural sedation.

METHODS

We studied adult patients who previously lived independently and underwent a non-ambulatory interventional radiology or gastroenterology procedure under anaesthetist-directed or non-anaesthetist-directed sedation at a large healthcare network. The primary outcomes were postprocedural adverse discharge to a nursing home and postprocedural duration of hospital stay.

RESULTS

Among 22,868 patients included, 15,168 (66.3%) and 7700 (33.7%) underwent anaesthetist-directed sedation and non-anaesthetist-directed sedation, respectively. Of all patients receiving anaesthetist-directed sedation, 9.2% experienced adverse discharge to a nursing home compared with 21.3% undergoing non-anaesthetist-directed sedation. Anaesthetist-directed sedation was associated with reduced risk of adverse discharge to a nursing home (adjusted relative risk 0.54, 95%CI 0.45-0.63, p < 0.001, adjusted risk difference -4.6%, 95%CI -5.8 to -3.4, p < 0.001) and a shorter postprocedural duration of hospital stay (median (IQR [range]) 2 (1-6 [0-315]) days vs. 5 (2-12 [0-268]) days; adjusted model estimate 0.84, 95%CI 0.79-0.89, p < 0.001). The lower risk of adverse discharge to a nursing home and shorter duration of hospital stay in patients undergoing anaesthetist-directed sedation was reproduced in an instrumental variable analysis (adjusted risk difference -4.3%, 95%CI -8.4 to -0.1, p = 0.043; and -1.41 days, 95%CI -1.43 to -1.41 days, p < 0.001, respectively). Among patients undergoing anaesthetist-directed sedation the mean (SD) proportion of missing blood pressure measurements was lower (0.7 (4.9) % vs. 8.0 (14.6) %, p < 0.001), which mediated the effect of anaesthetist-directed sedation on adverse discharge.

DISCUSSION

Among patients undergoing a non-ambulatory interventional radiology procedure or a gastrointestinal endoscopy, anaesthetist-directed sedation is associated with a reduced risk of adverse discharge to a nursing home and a shorter duration of hospital stay.

The anesthesiologist's guide to critically assessing machine learning research: a narrative review

Artificial Intelligence (AI), especially Machine Learning (ML), has developed systems capable of performing tasks that require human intelligence. In anesthesiology and other medical fields, AI applications can improve the precision and efficiency of daily clinical practice, and can also facilitate a personalized approach to patient care, which can lead to improved outcomes and quality of care. ML has been successfully applied in various settings of daily anesthesiology practice, such as predicting acute kidney injury, optimizing anesthetic doses, and managing postoperative nausea and vomiting. The critical evaluation of ML models in healthcare is crucial to assess their validity, safety, and clinical applicability. Evaluation metrics allow an objective statistical assessment of model performance. Tools such as Shapley Values (SHAP) help interpret how individual variables contribute to model predictions. Transparency in reporting is key in maintaining trust in these technologies and to ensure their use follows ethical principles, aiming to reduce safety concerns while also benefiting patients. Understanding evaluation metrics is essential, as they provide detailed information on model performance and their ability to discriminate between individual class rates. This article offers a comprehensive framework in assessing the validity, applicability, and limitations of models, guiding responsible and effective integration of ML technologies into clinical practice. A balance between innovation, patient safety and ethical considerations must be pursued.

Machine learning: implications and applications for ambulatory anesthesia

PURPOSE OF REVIEW

This review explores the timely and relevant applications of machine learning in ambulatory anesthesia, focusing on its potential to optimize operational efficiency, personalize risk assessment, and enhance patient care.

RECENT FINDINGS

Machine learning models have demonstrated the ability to accurately forecast case durations, Post-Anesthesia Care Unit (PACU) lengths of stay, and risk of hospital transfers based on preoperative patient and procedural factors. These models can inform case scheduling, resource allocation, and preoperative evaluation. Additionally, machine learning can standardize assessments, predict outcomes, improve handoff communication, and enrich patient education.

SUMMARY

Machine learning has the potential to revolutionize ambulatory anesthesia practice by optimizing efficiency, personalizing care, and improving quality and safety. However, limitations such as algorithmic opacity, data biases, reproducibility issues, and adoption barriers must be addressed through transparent, participatory design principles and ongoing validation to ensure responsible innovation and incremental adoption.

Factors influencing agitation during anesthesia recovery after laparoscopic hernia repair under total inhalation combined with caudal block anesthesia

BACKGROUND

Laparoscopic hernia repair is a minimally invasive surgery, but patients may experience emergence agitation (EA) during the post-anesthesia recovery period, which can increase pain and lead to complications such as wound reopening and bleeding. There is limited research on the risk factors for this agitation, and few effective tools exist to predict it. Therefore, by integrating clinical data, we have developed nomograms and random forest predictive models to help clinicians predict and potentially prevent EA.

AIM

To establish a risk nomogram prediction model for EA in patients undergoing laparoscopic hernia surgery under total inhalation combined with sacral block anesthesia.

METHODS

Based on the clinical information of 300 patients who underwent laparoscopic hernia surgery in the Nanning Tenth People's Hospital, Guangxi, from January 2020 to June 2023, the patients were divided into two groups according to their sedation-agitation scale score, i.e., the EA group (≥ 5 points) and the non-EA group (≤ 4 points), during anesthesia recovery. Least absolute shrinkage and selection operator regression was used to select the key features that predict EA, and incorporating them into logistic regression analysis to obtain potential predictive factors and establish EA nomogram and random forest risk prediction models through R software.

RESULTS

Out of the 300 patients, 72 had agitation during anesthesia recovery, with an incidence of 24.0%. American Society of Anesthesiologists classification, preoperative anxiety, solid food fasting time, clear liquid fasting time, indwelling catheter, and pain level upon awakening are key predictors of EA in patients undergoing laparoscopic hernia surgery with total intravenous anesthesia and caudal block anesthesia. The nomogram predicts EA with an area under the receiver operating characteristic curve (AUC) of 0.947, a sensitivity of 0.917, and a specificity of 0.877, whereas the random forest model has an AUC of 0.923, a sensitivity of 0.912, and a specificity of 0.877. Delong's test shows no significant difference in AUC between the two models. Clinical decision curve analysis indicates that both models have good net benefits in predicting EA, with the nomogram effective within the threshold of 0.02 to 0.96 and the random forest model within 0.03 to 0.90. In the external model validation of 50 cases of laparoscopic hernia surgery, both models predicted EA. The nomogram model had a sensitivity of 83.33%, specificity of 86.84%, and accuracy of 86.00%, while the random forest model had a sensitivity of 75.00%, specificity of 78.95%, and accuracy of 78.00%, suggesting that the nomogram model performs better in predicting EA.

CONCLUSION

Independent predictors of EA in patients undergoing laparoscopic hernia repair with total intravenous anesthesia combined with caudal block include American Society of Anesthesiologists classification, preoperative anxiety, duration of solid food fasting, duration of clear liquid fasting, presence of an indwelling catheter, and pain level upon waking. The nomogram and random forest models based on these factors can help tailor clinical decisions in the future.

A machine learning-based prediction model for delayed clinically important postoperative nausea and vomiting in high-risk patients undergoing laparoscopic gastrointestinal surgery

BACKGROUND

Delayed clinically important postoperative nausea and vomiting (CIPONV) could lead to significant consequences following surgery. We aimed to develop a prediction model for it using machine learning algorithms utilizing perioperative data from patients undergoing laparoscopic gastrointestinal surgery.

METHODS

All 1154 patients in the FDP-PONV trial were enrolled. The optimal features for model development were selected by least absolute shrinkage and selection operator and stepwise regression from 81 perioperative variables. The machine learning algorithm with the best area under the receiver operating characteristic curve (ROCAUC) was determined and assessed. The interpretation of the prediction model was performed by the SHapley Additive Explanations library.

RESULTS

Six important predictors were identified. The random forest model showed the best performance in predicting delayed CIPONV, achieving an ROCAUC of 0.737 in the validation cohort.

CONCLUSION

This study developed an interpretable model predicting personalized risk for delayed CIPONV, aiding high-risk patient identification and prevention strategies.

Comparison of NLP machine learning models with human physicians for ASA Physical Status classification

The American Society of Anesthesiologist's Physical Status (ASA-PS) classification system assesses comorbidities before sedation and analgesia, but inconsistencies among raters have hindered its objective use. This study aimed to develop natural language processing (NLP) models to classify ASA-PS using pre-anesthesia evaluation summaries, comparing their performance to human physicians. Data from 717,389 surgical cases in a tertiary hospital (October 2004-May 2023) was split into training, tuning, and test datasets. Board-certified anesthesiologists created reference labels for tuning and test datasets. The NLP models, including ClinicalBigBird, BioClinicalBERT, and Generative Pretrained Transformer 4, were validated against anesthesiologists. The ClinicalBigBird model achieved an area under the receiver operating characteristic curve of 0.915. It outperformed board-certified anesthesiologists with a specificity of 0.901 vs. 0.897, precision of 0.732 vs. 0.715, and F1-score of 0.716 vs. 0.713 (all p <0.01). This approach will facilitate automatic and objective ASA-PS classification, thereby streamlining the clinical workflow.

Evaluating the accuracy of ChatGPT-4 in predicting ASA scores: A prospective multicentric study ChatGPT-4 in ASA score prediction

BACKGROUND

This study investigates the potential of ChatGPT-4, developed by OpenAI, in enhancing medical decision-making processes, particularly in preoperative assessments using the American Society of Anesthesiologists (ASA) scoring system. The ASA score, a critical tool in evaluating patients' health status and anesthesia risks before surgery, categorizes patients from I to VI based on their overall health and risk factors. Despite its widespread use, determining accurate ASA scores remains a subjective process that may benefit from AI-supported assessments. This research aims to evaluate ChatGPT-4's capability to predict ASA scores accurately compared to expert anesthesiologists' assessments.

METHODS

In this prospective multicentric study, ethical board approval was obtained, and the study was registered with clinicaltrials.gov (NCT06321445). We included 2851 patients from anesthesiology outpatient clinics, spanning neonates to all age groups and genders, with ASA scores between I-IV. Exclusion criteria were set for ASA V and VI scores, emergency operations, and insufficient information for ASA score determination. Data on patients' demographics, health conditions, and ASA scores by anesthesiologists were collected and anonymized. ChatGPT-4 was then tasked with assigning ASA scores based on the standardized patient data.

RESULTS

Our results indicate a high level of concordance between ChatGPT-4 predictions and anesthesiologists' evaluations, with Cohen's kappa analysis showing a kappa value of 0.858 (p = 0.000). While the model demonstrated over 90% accuracy in predicting ASA scores I to III, it showed a notable variance in ASA IV scores, suggesting a potential limitation in assessing patients with more complex health conditions.

DISCUSSION

The findings suggest that ChatGPT-4 can significantly contribute to the medical field by supporting anesthesiologists in preoperative assessments. This study not only demonstrates ChatGPT-4's efficacy in medical data analysis and decision-making but also opens new avenues for AI applications in healthcare, particularly in enhancing patient safety and optimizing surgical outcomes. Further research is needed to refine AI models for complex case assessments and integrate them seamlessly into clinical workflows.

Artificial intelligence and nonoperating room anesthesia

PURPOSE OF REVIEW

The integration of artificial intelligence (AI) in nonoperating room anesthesia (NORA) represents a timely and significant advancement. As the demand for NORA services expands, the application of AI is poised to improve patient selection, perioperative care, and anesthesia delivery. This review examines AI's growing impact on NORA and how it can optimize our clinical practice in the near future.

RECENT FINDINGS

AI has already improved various aspects of anesthesia, including preoperative assessment, intraoperative management, and postoperative care. Studies highlight AI's role in patient risk stratification, real-time decision support, and predictive modeling for patient outcomes. Notably, AI applications can be used to target patients at risk of complications, alert clinicians to the upcoming occurrence of an intraoperative adverse event such as hypotension or hypoxemia, or predict their tolerance of anesthesia after the procedure. Despite these advances, challenges persist, including ethical considerations, algorithmic bias, data security, and the need for transparent decision-making processes within AI systems.

SUMMARY

The findings underscore the substantial benefits of AI in NORA, which include improved safety, efficiency, and personalized care. AI's predictive capabilities in assessing hypoxemia risk and other perioperative events, have demonstrated potential to exceed human prognostic accuracy. The implications of these findings advocate for a careful yet progressive adoption of AI in clinical practice, encouraging the development of robust ethical guidelines, continual professional training, and comprehensive data management strategies. Furthermore, AI's role in anesthesia underscores the need for multidisciplinary research to address the limitations and fully leverage AI's capabilities for patient-centered anesthesia care.

Novel Preoperative Risk Stratification Using Digital Phenotyping Applying a Scalable Machine-Learning Approach

BACKGROUND

Classification of perioperative risk is important for patient care, resource allocation, and guiding shared decision-making. Using discriminative features from the electronic health record (EHR), machine-learning algorithms can create digital phenotypes among heterogenous populations, representing distinct patient subpopulations grouped by shared characteristics, from which we can personalize care, anticipate clinical care trajectories, and explore therapies. We hypothesized that digital phenotypes in preoperative settings are associated with postoperative adverse events including in-hospital and 30-day mortality, 30-day surgical redo, intensive care unit (ICU) admission, and hospital length of stay (LOS).

METHODS

We identified all laminectomies, colectomies, and thoracic surgeries performed over a 9-year period from a large hospital system. Seventy-seven readily extractable preoperative features were first selected from clinical consensus, including demographics, medical history, and lab results. Three surgery-specific datasets were built and split into derivation and validation cohorts using chronological occurrence. Consensus k -means clustering was performed independently on each derivation cohort, from which phenotypes' characteristics were explored. Cluster assignments were used to train a random forest model to assign patient phenotypes in validation cohorts. We reconducted descriptive analyses on validation cohorts to confirm the similarity of patient characteristics with derivation cohorts, and quantified the association of each phenotype with postoperative adverse events by using the area under receiver operating characteristic curve (AUROC). We compared our approach to American Society of Anesthesiologists (ASA) alone and investigated a combination of our phenotypes with the ASA score.

RESULTS

A total of 7251 patients met inclusion criteria, of which 2770 were held out in a validation dataset based on chronological occurrence. Using segmentation metrics and clinical consensus, 3 distinct phenotypes were created for each surgery. The main features used for segmentation included urgency of the procedure, preoperative LOS, age, and comorbidities. The most relevant characteristics varied for each of the 3 surgeries. Low-risk phenotype alpha was the most common (2039 of 2770, 74%), while high-risk phenotype gamma was the rarest (302 of 2770, 11%). Adverse outcomes progressively increased from phenotypes alpha to gamma, including 30-day mortality (0.3%, 2.1%, and 6.0%, respectively), in-hospital mortality (0.2%, 2.3%, and 7.3%), and prolonged hospital LOS (3.4%, 22.1%, and 25.8%). When combined with the ASA score, digital phenotypes achieved higher AUROC than the ASA score alone (hospital mortality: 0.91 vs 0.84; prolonged hospitalization: 0.80 vs 0.71).

CONCLUSIONS

For 3 frequently performed surgeries, we identified 3 digital phenotypes. The typical profiles of each phenotype were described and could be used to anticipate adverse postoperative events.

The effect of a bundle intervention for ambulatory otorhinolaryngology procedures on same-day case cancellation rate and associated costs

Cancellations within 24 h of planned elective surgical procedures reduce operating theatre efficiency, add unnecessary costs and negatively affect patient experience. We implemented a bundle intervention that aimed to reduce same-day case cancellations. This consisted of communication tools to improve patient engagement and new screening instruments (automated estimation of ASA physical status and case cancellation risk score plus four screening questions) to identify patients in advance (ideally before case booking) who needed comprehensive pre-operative risk stratification. We studied patients scheduled for ambulatory surgery with the otorhinolaryngology service at a single centre from April 2021 to December 2022. Multivariable logistic regression and interrupted time-series analyses were used to analyse the effects of this intervention on case cancellations within 24 h and costs. We analysed 1548 consecutive scheduled cases. Cancellation within 24 h occurred in 114 of 929 (12.3%) cases pre-intervention and 52 of 619 (8.4%) cases post-intervention. The cancellation rate decreased by 2.7% (95%CI 1.6-3.7%, p < 0.01) during the first month, followed by a monthly decrease of 0.2% (95%CI 0.1-0.4%, p < 0.01). This resulted in an estimated $150,200 (£118,755; €138,370) or 35.3% cost saving (p < 0.01). Median (IQR [range]) number of days between case scheduling and day of surgery decreased from 34 (21-61 [0-288]) pre-intervention to 31 (20-51 [1-250]) post-intervention (p < 0.01). Patient engagement via the electronic health record patient portal or text messaging increased from 75.9% at baseline to 90.8% (p < 0.01) post-intervention. The primary reason for case cancellation was patients' missed appointment on the day of surgery, which decreased from 7.2% pre-intervention to 4.5% post-intervention (p = 0.03). An anaesthetist-driven, clinical informatics-based bundle intervention decreases same-day case cancellation rate and associated costs in patients scheduled for ambulatory otorhinolaryngology surgery.

Development and validation of an instrument to predict prolonged length of stay in the postanesthesia care unit following ambulatory surgery

PURPOSE

We sought to develop and validate an Anticipated Surveillance Requirement Prediction Instrument (ASRI) for prediction of prolonged postanesthesia care unit length of stay (PACU-LOS, more than four hours) after ambulatory surgery.

METHODS

We analyzed hospital registry data from patients who received anesthesia care in ambulatory surgery centres (ASCs) of university-affiliated hospital networks in New York, USA (development and internal validation cohort [n = 183,711]) and Massachusetts, USA (validation cohort [n = 148,105]). We used stepwise backwards elimination to create ASRI.

RESULTS

The model showed discriminatory ability in the development, internal, and external validation cohorts with areas under the receiver operating characteristic curve of 0.82 (95% confidence interval [CI], 0.82 to 0.83), 0.82 (95% CI, 0.81 to 0.83), and 0.80 (95% CI, 0.79 to 0.80), respectively. In cases started in the afternoon, ASRI scores ≥ 43 had a total predicted risk for PACU stay past 8 p.m. of 32% (95% CI, 31.1 to 33.3) vs 8% (95% CI, 7.9 to 8.5) compared with low score values (P-for-interaction < 0.001), which translated to a higher direct PACU cost of care of USD 207 (95% CI, 194 to 2,019; model estimate, 1.68; 95% CI, 1.64 to 1.73; P < 0.001) The effects of using the ASRI score on PACU use efficiency were greater in a free-standing ASC with no limitations on PACU bed availability.

CONCLUSION

We developed and validated a preoperative prediction tool for prolonged PACU-LOS after ambulatory surgery that can be used to guide scheduling in ambulatory surgery to optimize PACU use during normal work hours, particularly in settings without limitation of PACU bed availability.

Artificial intelligence software can generate residency application personal statements that program directors find acceptable and difficult to distinguish from applicant compositions

STUDY OBJECTIVE

Create personal statements using an artificial intelligence program for anesthesiology residency applications that residency program directors rate as acceptable.

STUDY DESIGN

Generate two personal statements and survey program directors.

SETTING

Anesthesiology residency training programs.

INTERVENTIONS

We instructed ChatGPT, a new artificial-intelligence software program, to generate two 400-word personal statements using the common applicant experiences of involvement in athletics or gourmet cooking.

METHODS

We sent the generated personal statements to anesthesia program directors and asked them if the statements were acceptable for application to their individual programs, to rate them as poor, good, or excellent, and determine if they could detect anything in the statements that indicated they were not written by an applicant.

MEASUREMENTS

Ninety-four program directors received and opened the survey, and 31 responded. Twenty-eight (90%) responding directors found the personal statement with athletic experience acceptable, with 22 (74%) rating it as good or excellent. Nineteen (61%) program directors did not detect anything in the statement to distinguish it from an applicant-written composition. Twenty-nine (97%) program directors found the personal statement with cooking experience acceptable, with 19 (63%) finding it good or excellent. Twenty-four (80%) directors did not detect anything in the statement to distinguish it from an applicant-written composition.

CONCLUSIONS

ChatGPT can create personal statements for residency applications that program directors find acceptable and difficult to differentiate from personally crafted statements. Applicants may stop using expensive contractor application services and start using artificial intelligence software to create their personal statements because of its quickness, low cost, and high quality.