1
|
Mason EM, Henderson WG, Bronsert MR, Colborn KL, Dyas AR, Madsen HJ, Lambert-Kerzner A, Meguid RA. Preoperative Prediction of Unplanned Reoperation in a Broad Surgical Population. J Surg Res 2023; 285:1-12. [PMID: 36640606 PMCID: PMC9975057 DOI: 10.1016/j.jss.2022.12.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2022] [Revised: 11/07/2022] [Accepted: 12/24/2022] [Indexed: 01/15/2023]
Abstract
INTRODUCTION Unplanned reoperation is an undesirable outcome with considerable risks and an increasingly assessed quality of care metric. There are no preoperative prediction models for reoperation after an index surgery in a broad surgical population in the literature. The Surgical Risk Preoperative Assessment System (SURPAS) preoperatively predicts 12 postoperative adverse events using 8 preoperative variables, but its ability to predict unplanned reoperation has not been assessed. This study's objective was to determine whether the SURPAS model could accurately predict unplanned reoperation. METHODS This was a retrospective analysis of the American College of Surgeons' National Surgical Quality Improvement Program adult database, 2012-2018. An unplanned reoperation was defined as any unintended operation within 30 d of an initial scheduled operation. The 8-variable SURPAS model and a 29-variable "full" model, incorporating all available American College of Surgeons' National Surgical Quality Improvement Program nonlaboratory preoperative variables, were developed using multiple logistic regression and compared using discrimination and calibration metrics: C-indices (C), Hosmer-Lemeshow observed-to-expected plots, and Brier scores (BSs). The internal chronological validation of the SURPAS model was conducted using "training" (2012-2017) and "test" (2018) datasets. RESULTS Of 5,777,108 patients, 162,387 (2.81%) underwent an unplanned reoperation. The SURPAS model's C-index of 0.748 was 99.20% of that for the full model (C = 0.754). Hosmer-Lemeshow plots showed good calibration for both models and BSs were similar (BS = 0.0264, full; BS = 0.0265, SURPAS). Internal chronological validation results were similar for the training (C = 0.749, BS = 0.0268) and test (C = 0.748, BS = 0.0250) datasets. CONCLUSIONS The SURPAS model accurately predicted unplanned reoperation and was internally validated. Unplanned reoperation can be integrated into the SURPAS tool to provide preoperative risk assessment of this outcome, which could aid patient risk education.
Collapse
Affiliation(s)
- Emily M Mason
- Clinical Science Program, University of Colorado Anschutz Medical Campus, Graduate School, Colorado Clinical and Translational Sciences Institute, Aurora, Colorado; Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado
| | - William G Henderson
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, Colorado School of Public Health, Aurora, Colorado
| | - Michael R Bronsert
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado
| | - Kathryn L Colborn
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, Colorado School of Public Health, Aurora, Colorado
| | - Adam R Dyas
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado
| | - Helen J Madsen
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado
| | - Anne Lambert-Kerzner
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, Colorado School of Public Health, Aurora, Colorado
| | - Robert A Meguid
- Department of Surgery, Surgical Outcomes and Applied Research Program, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado; Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus, School of Medicine, Aurora, Colorado.
| |
Collapse
|
2
|
Vernooij JEM, Koning NJ, Geurts JW, Holewijn S, Preckel B, Kalkman CJ, Vernooij LM. Performance and usability of pre-operative prediction models for 30-day peri-operative mortality risk: a systematic review. Anaesthesia 2023; 78:607-619. [PMID: 36823388 DOI: 10.1111/anae.15988] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/06/2023] [Indexed: 02/25/2023]
Abstract
Estimating pre-operative mortality risk may inform clinical decision-making for peri-operative care. However, pre-operative mortality risk prediction models are rarely implemented in routine clinical practice. High predictive accuracy and clinical usability are essential for acceptance and clinical implementation. In this systematic review, we identified and appraised prediction models for 30-day postoperative mortality in non-cardiac surgical cohorts. PubMed and Embase were searched up to December 2022 for studies investigating pre-operative prediction models for 30-day mortality. We assessed predictive performance in terms of discrimination and calibration. Risk of bias was evaluated using a tool to assess the risk of bias and applicability of prediction model studies. To further inform potential adoption, we also assessed clinical usability for selected models. In all, 15 studies evaluating 10 prediction models were included. Discrimination ranged from a c-statistic of 0.82 (MySurgeryRisk) to 0.96 (extreme gradient boosting machine learning model). Calibration was reported in only six studies. Model performance was highest for the surgical outcome risk tool (SORT) and its external validations. Clinical usability was highest for the surgical risk pre-operative assessment system. The SORT and risk quantification index also scored high on clinical usability. We found unclear or high risk of bias in the development of all models. The SORT showed the best combination of predictive performance and clinical usability and has been externally validated in several heterogeneous cohorts. To improve clinical uptake, full integration of reliable models with sufficient face validity within the electronic health record is imperative.
Collapse
Affiliation(s)
- J E M Vernooij
- Department of Anaesthesia, Rijnstate Hospital, the Netherlands
| | - N J Koning
- Department of Anaesthesia, Rijnstate Hospital, the Netherlands
| | - J W Geurts
- Department of Anaesthesia, Rijnstate Hospital, the Netherlands
| | - S Holewijn
- Department of Vascular Surgery, Rijnstate Hospital, the Netherlands
| | - B Preckel
- Department of Anaesthesia, Amsterdam UMC, Amsterdam, the Netherlands
| | - C J Kalkman
- University Medical Centre, Utrecht, the Netherlands
| | - L M Vernooij
- Department of Anaesthesia, University Medical Centre Utrecht, the Netherlands
| |
Collapse
|
3
|
Does Adding a Measure of Social Vulnerability to a Surgical Risk Calculator Improve Its Performance? J Am Coll Surg 2022; 234:1137-1146. [PMID: 35703812 DOI: 10.1097/xcs.0000000000000187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Abstract
BACKGROUND Emerging literature suggests that measures of social vulnerability should be incorporated into surgical risk calculators. The Social Vulnerability Index (SVI) is a measure designed by the CDC that encompasses 15 socioeconomic and demographic variables at the census tract level. We examined whether adding the SVI into a parsimonious surgical risk calculator would improve model performance. STUDY DESIGN The eight-variable Surgical Risk Preoperative Assessment System (SURPAS), developed using the entire American College of Surgeons (ACS) NSQIP database, was applied to local ACS-NSQIP data from 2012 to 2018 to predict 12 postoperative outcomes. Patient addresses were geocoded and used to estimate the SVI, which was then added to the model as a ninth predictor variable. Brier scores and c-indices were compared for the models with and without the SVI. RESULTS The analysis included 31,222 patients from five hospitals. Brier scores were identical for eight outcomes and improved by only one to two points in the fourth decimal place for four outcomes with addition of the SVI. Similarly, c-indices were not significantly different (p values ranged from 0.15 to 0.96). Of note, the SVI was associated with most of the eight SURPAS predictor variables, suggesting that SURPAS may already indirectly capture this important risk factor. CONCLUSION The eight-variable SURPAS prediction model was not significantly improved by adding the SVI, showing that this parsimonious tool functions well without including a measure of social vulnerability.
Collapse
|
4
|
Pradhan N, Dyas AR, Bronsert MR, Lambert-Kerzner A, Henderson WG, Qiu H, Colborn KL, Mason NJ, Meguid RA. Attitudes about use of preoperative risk assessment tools: a survey of surgeons and surgical residents in an academic health system. Patient Saf Surg 2022; 16:13. [PMID: 35300719 PMCID: PMC8932286 DOI: 10.1186/s13037-022-00320-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 01/15/2022] [Indexed: 11/10/2022] Open
Abstract
Background Formal surgical risk assessment tools have been developed to predict risk of adverse postoperative patient outcomes. Such tools accurately predict common postoperative complications, inform patients and providers of likely perioperative outcomes, guide decision making, and improve patient care. However, these are underutilized. We studied the attitudes towards and techniques of how surgeons preoperatively assess risk. Methods Surgeons at a large academic tertiary referral hospital and affiliate community hospitals were emailed a 16-question survey via REDCap (Research Electronic Data Capture) between 8/2019-6/2020. Reminder emails were sent once weekly for three weeks. All completed surveys by surgical residents and attendings were included; incomplete surveys were excluded. Surveys were analyzed using descriptive statistics (frequency distributions and percentages for categorical variables, means, and standard deviations for continuous variables), and Fisher’s exact test and unpaired t-tests comparing responses by surgical attendings vs. residents. Results A total of 108 surgical faculty, 95 surgical residents, and 58 affiliate surgeons were emailed the survey. Overall response rates were 50.0% for faculty surgeons, 47.4% for residents, and 36.2% for affiliate surgeons. Only 20.8% of surgeons used risk calculators most or all of the time. Attending surgeons were more likely to use prior experience and current literature while residents used risk calculators more frequently. Risk assessment tools were more likely to be used when predicting major complications and death in older patients with significant risk factors. Greatest barriers for use of risk assessment tools included time, inaccessibility, and trust in accuracy. Conclusions A small percentage of surgeons use surgical risk calculators as part of their routine practice. Time, inaccessibility, and trust in accuracy were the most significant barriers to use. Supplementary Information The online version contains supplementary material available at 10.1186/s13037-022-00320-1.
Collapse
Affiliation(s)
- Nisha Pradhan
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Adam R Dyas
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.,Division of Cardiothoracic Surgery, Department of Surgery, Anschutz Medical Campus, University of Colorado Denver, 12631 E. 17th Avenue, C-310, Aurora, CO, 80045, USA
| | - Michael R Bronsert
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.,Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA
| | - Anne Lambert-Kerzner
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.,Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA
| | - William G Henderson
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.,Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA.,Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO, USA
| | - Howe Qiu
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Kathryn L Colborn
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA.,Division of Cardiothoracic Surgery, Department of Surgery, Anschutz Medical Campus, University of Colorado Denver, 12631 E. 17th Avenue, C-310, Aurora, CO, 80045, USA
| | - Nicholas J Mason
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Robert A Meguid
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA. .,Division of Cardiothoracic Surgery, Department of Surgery, Anschutz Medical Campus, University of Colorado Denver, 12631 E. 17th Avenue, C-310, Aurora, CO, 80045, USA. .,Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA.
| |
Collapse
|
5
|
Dyas AR, Colborn KL, Bronsert MR, Henderson WG, Mason NJ, Rozeboom PD, Pradhan N, Lambert-Kerzner A, Meguid RA. Comparison of Preoperative Surgical Risk Estimated by Thoracic Surgeons Versus a Standardized Surgical Risk Prediction Tool. Semin Thorac Cardiovasc Surg 2021; 34:1378-1385. [PMID: 34785355 DOI: 10.1053/j.semtcvs.2021.11.008] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Accepted: 11/10/2021] [Indexed: 11/11/2022]
Abstract
Considerable variability exists between surgeons' assessments of a patient's individual pre-operative surgical risk. Surgical risk calculators are not routinely used despite their validation. We sought to compare thoracic surgeons' prediction of patients' risk of postoperative adverse outcomes versus a surgical risk calculator, the Surgical Risk Preoperative Assessment System (SURPAS). We developed vignettes from 30 randomly selected patients who underwent thoracic surgery in the American College of Surgeons' National Surgical Quality Improvement Program (NSQIP) database. Twelve thoracic surgeons estimated patients' preoperative risks of postoperative morbidity and mortality. These were compared to SURPAS estimates of the same vignettes. C-indices and Brier scores were calculated for the surgeons' and SURPAS estimates. Agreement between surgeon estimates was examined using intraclass correlation coefficients (ICCs). Surgeons estimated higher morbidity risk compared to SURPAS for low-risk patients (ASA classes 1-2, 11.5% vs. 5.1%, p=<0.001) and lower morbidity risk compared to SURPAS for high-risk patients (ASA class 5, 37.6% vs. 69.8%, p<0.001). This trend also occurred in high-risk patients for mortality (ASA 5, 11.1% vs. 44.3%, p<0.001). C-indices for SURPAS vs. surgeons were 0.84 vs. 0.76 (p=0.3) for morbidity and 0.98 vs. 0.85 (p=0.001) for mortality. Brier scores for SURPAS vs. surgeons were 0.1579 vs. 0.1986 for morbidity (p=0.03) and 0.0409 vs. 0.0543 for mortality (p=0.006). ICCs showed that surgeons had moderate risk agreement for morbidity (ICC=0.654) and mortality (ICC=0.507). Thoracic surgeons and patients could benefit from using a surgical risk calculator to better estimate patients' surgical risks during the informed consent process.
Collapse
Affiliation(s)
- Adam R Dyas
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Kathryn L Colborn
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO, USA
| | - Michael R Bronsert
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA
| | - William G Henderson
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA; Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO, USA
| | - Nicholas J Mason
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Paul D Rozeboom
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Nisha Pradhan
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA
| | - Anne Lambert-Kerzner
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA
| | - Robert A Meguid
- Surgical Outcomes and Applied Research Program, Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA; Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA.
| |
Collapse
|