Correlation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes

Validity of task-specific metrics for assessment in perineal proctectomy

BACKGROUND

Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics.

METHODS

In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores.

RESULTS

In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ2 = 15.38; P < 0.001; df = 2) and task-specific (χ2 = 11.38; P = 0.003; df = 2) scores.

CONCLUSIONS

Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.

Laparoscopy experience in East, Central, and Southern Africa: insights from operative case volume analysis

BACKGROUND

With the primary objective of addressing the disparity in global surgical care access, the College of Surgeons of East, Central, and Southern Africa (COSECSA) trains surgeons. While sufficient operative experience is crucial for surgical training, the extent of utilization of minimally invasive techniques during COSECSA training remains understudied.

METHODS

We conducted an extensive review of COSECSA general surgery trainees' operative case logs from January 1, 2015, to December 31, 2020, focusing on the utilization of minimally invasive surgical procedures. Our primary objective was to determine the prevalence of laparoscopic procedures and compare this to open procedures. We analyzed the distribution of laparoscopic cases across common indications such as cholecystectomy, appendicitis, and hernia operations. Additionally, we examined the impact of trainee autonomy, country development index, and hospital type on laparoscopy utilization.

RESULTS

Among 68,659 total cases, only 616 (0.9%) were laparoscopic procedures. Notably, 34 cases were conducted during trainee external rotations in countries like the United Kingdom, Germany, and India. Gallbladder and appendix pathologies were most frequent among the 582 recorded laparoscopic cases performed in Africa. Laparoscopic cholecystectomy accounted for 29% (276 of 975 cases), laparoscopic appendectomy for 3% (76 of 2548 cases), and laparoscopic hernia repairs for 0.5% (26 of 5620 cases). Trainees self-reported lower autonomy for laparoscopic (22.5%) than open cases (61.5%). Laparoscopy usage was more prevalent in upper-middle-income (2.7%) and lower-middle-income countries (0.8%) compared with lower-income countries (0.5%) (p < 0.001). Private (1.6%) and faith-based hospitals (1.5%) showed greater laparoscopy utilization than public hospitals (0.5%) (p < 0.001).

CONCLUSIONS

The study highlights the relatively low utilization of minimally invasive techniques in surgical training within the ECSA region. Laparoscopic cases remain a minority, with variations observed based on specific diagnoses. The findings suggest a need to enhance exposure to minimally invasive procedures to ensure well-rounded training and proficiency in these techniques.

Operative Case Volumes and Variation for General Surgery Training in East, Central, and Southern Africa

BACKGROUND

Operative experience is a necessary part of surgical training. The College of Surgeons of East, Central, and Southern Africa (COSECSA), which oversees general surgery training programs in the region, has implemented guidelines for the minimum necessary case volumes upon completion of two (Membership) and five (Fellowship) years of surgical training. We aimed to review trainee experience to determine whether guidelines are being met and examine the variation of cases between countries.

METHODS

Operative procedures were categorized from a cohort of COSECSA general surgery trainees and compared to the guideline minimum case volumes for Membership and Fellowship levels. The primary and secondary outcomes were total observed case volumes and cases within defined categories. Variations by country and development indices were explored.

RESULTS

One hundred ninety-four trainees performed 69,283 unique procedures related to general surgery training. The review included 70 accredited hospitals and sixteen countries within Africa. Eighty percent of MCS trainees met the guideline minimum of 200 overall cases; however, numerous trainees did not meet the guideline minimum for each procedure. All FCS trainees met the volume target for total cases and orthopedics; however, many did not meet the guideline minimums for other categories, especially breast, head and neck, urology, and vascular surgery. The operative experience of trainees varied significantly by location and national income level.

CONCLUSIONS

Surgical trainees in East, Central, and Southern Africa have diverse operative training experience. Most trainees fulfill the overall case volume requirements; however, further exploration of how to meet the demands of specific categories and procedures is necessary.

COVID-19 Did Not Stop the Rising Tide: Trends in Case Volume Logged by Surgical Residents

INTRODUCTION

The coronavirus pandemic has profoundly impacted all facets of surgical care, including surgical residency training. The objective of this study was to assess the operative experience and overall case volume of surgery residents before and during the pandemic.

METHODS

Using data from the Accreditation Council for Graduate Medical Education annual operative log reports, operative volume for 2015 to 2021 graduates of Accreditation Council for Graduate Medical Education -accredited general, orthopedic, neuro- and plastic surgery residency programs was analyzed using nonparametric Kendall-tau correlation analysis. The period before the pandemic was defined as AY14-15 to AY18-19, and the pandemic period was defined as AY19-20 to AY20-21.

RESULTS

Operative data for 8556 general, 5113 orthopedic, 736 plastic, and 1278 neurosurgery residency graduates were included. Between 2015 and 2021, total case volume increased significantly for general surgery graduates (Kendall's tau-b: 0.905, p = 0.007), orthopedic surgery graduates (Kendall's tau-b: 1.000, p = 0.003), neurosurgery graduates (Kendall's tau-b: 0.905, p = 0.007), and plastic surgery graduates (Kendall's tau-b: 0.810, p = 0.016). Across all specialties, the mean total number of cases performed by residents graduating during the pandemic was higher than among residents graduating before the pandemic, though no formal significance testing was performed. Among general surgery residents, the number of cases performed as surgeon chief among residents graduating in AY19-20 decreased for the first time in 5 years, though the overall volume remained higher than the prior year, and returned to prepandemic trends in AY20-21.

CONCLUSIONS

Over the past 7 years, the case volume of surgical residents steadily increased. Surgical trainees who graduated during the coronavirus pandemic have equal or greater total operative experience compared to trainees who graduated prior to the pandemic.

Objective metrics for hand-sewn bowel anastomoses can differentiate novice from expert surgeons

BACKGROUND

Assessing performance automatically in a virtual reality trainer or from recorded videos is advantageous but needs validated objective metrics. The purpose of this study is to obtain expert consensus and validate task-specific metrics developed for assessing performance in double-layered end-to-end anastomosis.

MATERIALS AND METHODS

Subjects were recruited into expert (PGY 4-5, colorectal surgery residents, and attendings) and novice (PGY 1-3) groups. Weighted average scores of experts for each metric item, completion time, and the total scores computed using global and task-specific metrics were computed for assessment.

RESULTS

A total of 43 expert surgeons rated our task-specific metric items with weighted averages ranging from 3.33 to 4.5 on a 5-point Likert scale. A total of 20 subjects (10 novices and 10 experts) participated in validation study. The novice group completed the task significantly more slowly than the experienced group (37.67 ± 7.09 vs 25.47 ± 7.82 min, p = 0.001). In addition, both the global rating scale (23.47 ± 4.28 vs 28.3 ± 3.85, p = 0.016) and the task-specific metrics showed a significant difference in performance between the two groups (38.77 ± 2.83 vs 42.58 ± 4.56 p = 0.027) following partial least-squares (PLS) regression. Furthermore, PLS regression showed that only two metric items (Stay suture tension and Tool handling) could reliably differentiate the performance between the groups (20.41 ± 2.42 vs 24.28 ± 4.09 vs, p = 0.037).

CONCLUSIONS

Our study shows that our task-specific metrics have significant discriminant validity and can be used to evaluate the technical skills for this procedure.

Development and Validation of Task-Specific Metrics for the Assessment of Linear Stapler-Based Small Bowel Anastomosis

INTRODUCTION

Task-specific metrics facilitate the assessment of surgeon performance. This 3-phased study was designed to (1) develop task-specific metrics for stapled small bowel anastomosis, (2) obtain expert consensus on the appropriateness of the developed metrics, and (3) establish its discriminant validity.

METHODS

In Phase I, a hierarchical task analysis was used to develop the metrics. In Phase II, a survey of expert colorectal surgeons established the importance of the developed metrics. In Phase III, to establish discriminant validity, surgical trainees and surgeons, divided into novice and experienced groups, constructed a side-to-side anastomosis on porcine small bowel using a linear cutting stapler. The participants' performances were videotaped and rated by 2 independent observers. Partial least squares regression was used to compute the weights for the task-specific metrics to obtain weighted total score.

RESULTS

In Phase II, a total of 45 colorectal surgeons were surveyed: 28 with more than 15 years, 13 with 5 to 15 years, and 4 with less than 5 years of experience. The consensus was obtained on all the task-specific metrics in the more experienced groups. In Phase III, 20 subjects participated equally in both groups. The experienced group performed better than the novice group regardless of the rating scale used: global rating scale (p = 0.009) and the task-specific metrics (p = 0.012). After partial least squares regression, the weighted task-specific metric score continued to show that the experienced group performed better (p < 0.001).

CONCLUSION

Task-specific metric items were developed based on expert consensus and showed good discriminant validity compared with a global rating scale between experienced and novice operators. These items can be used for evaluating technical skills in a stapled small bowel anastomosis model.

Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?

BACKGROUND

Simulation using virtual reality (VR) simulators is an important tool in surgical training. VR laparoscopic simulators can provide immediate objective performance assessment without observer evaluation. This study aims to explore the correlation between subjective observer evaluation and VR laparoscopic simulator performance metrics in a laparoscopic cholecystectomy (LC) simulation module.

METHODS

A LC simulation module using a VR laparoscopic simulator was completed by PGY2-3 general surgery residents at a single institution. Simulation performance was recorded and evaluated by a trained evaluator using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) form, the Objective Structured Assessment of Technical Skills (OSATS) form, and a LC-specific simulation assessment form (LC-SIM). Objective performance metrics were also obtained from the simulator system. Performance before the curriculum (pre-test) and after the curriculum (post-test) were compared.

RESULTS

Fourteen residents were included in the study. There were significant improvements from pre-test to post-test on each component of GOALS, OSATS, and LC-SIM scores (all p values < 0.05). In terms of objective simulator metrics, significant improvements were noted in time to extract gallbladder (481 ± 221 vs 909 ± 366 min, p = 0.019), total number of movements (475 ± 264 vs 839 ± 324 min, p = 0.012), and total path length (955 ± 475 vs 1775 ± 632 cm, p = 0.012) from pre-test to post-test. While number of movements and total path lengths of both hands decreased, speed of right instrument also decreased from 4.1 + 2.7 to 3.0 ± 0.7 cm/sec (p = 0.007). Average speed of left instrument was associated with respect for tissue (r = 0.60, p < 0.05) and depth perception (r = 0.68, p < 0.05) on post-test evaluations.

CONCLUSION

Our study demonstrated significant improvement in technical skills based on subjective evaluator assessment as well as objective simulator metrics after simulation. The few correlations identified between the subjective evaluator and the objective simulator assessments suggest the two evaluation modalities were measuring different aspects of the technical skills and should both be considered in the evaluation process.

Assessment of Chief Resident Practice Readiness in a Porcine Lab: A 4-Year Experience

OBJECTIVE

General surgery training prepares residents for the autonomous practice of surgery; however, assessment for readiness for independent practice presents several challenges. The simulation lab offers a safe and standardized environment for assessing the technical skills of a resident in the absence of numerous confounders of the real operating room. We describe our experience with evaluation and remediation of chief resident assessments in a porcine simulation lab.

DESIGN

Operative skill assessment of surgical residents was conducted using anesthetized porcine models. Procedure's representative of basic and complex operative skill was chosen for the assessment. Faculty assessed the residents using a checklist for the completion of all critical operative steps. A "failing" score or "critical fail" on a given procedure determined mandatory remediation. For remediation, faculty provided immediate post-procedure feedback on all errors, and residents were offered supervised practice. Residents were then retested to demonstrate competency.

SETTING

Large animal research center at Indiana University School of Medicine, Indianapolis, IN PARTICIPANTS: From 2017 to 2020, thirty-seven PGY5 residents participated in the porcine lab over a 4-year period. These general surgery residents were assessed at the beginning of their chief year.

RESULTS

There were a total of 6 residents that failed 1 or more procedures. There were no failures in the cholecystectomy, 3 failures for Nissen, 4 failures for Hand sewn anastomosis, and 1 failure for stapled anastomosis. Two residents failed 2 procedures. All residents received remediation with a faculty member and were subsequently able to perform the procedure competently.

CONCLUSIONS

A formal simulation-based assessment of procedural competence can identify technical performance deficiencies even at the chief resident level. Combined with a formal remediation program, such deficiencies can be addressed well in advance of residency graduation. Determining the relationship of such simulation-based assessments with operative performance is currently underway.

How do Trainers and Trainees use Formative Workplace-based Assessments of Operative Competence to Deliver and Receive Structured Feedback?

OBJECTIVE

The aim of this study is to assess the quality of feedback provided to surgical trainees in the operating theatre, and to further investigate how trainees and trainers use workplace-based assessment in practice with regards to frequency and timing of assessments.

DESIGN

A retrospective study of all submitted Supervised Structured Assessments of Operative Performance (SSAOPs) from April 25, 2016 to February 2, 2021 was conducted.

SETTING

Surgical trainees in the Republic of Ireland across all national surgical training sites submitted SSAOPs through an online platform.

PARTICIPANTS

Assessments of operative competence (SSAOPs) from all Core Surgical Trainees (in their first two years of dedicated post-graduate surgical training) were included for analysis, regardless of surgical subspecialty. A total of 2294 assessments were submitted from April 25, 2016 to February 2, 2021 by 330 core surgical trainees and 379 surgeon assessors. Five hundred of these assessments were randomly selected and scored for quality of feedback using a modified "Task, Gap, Action (TGA)" framework.

RESULTS

Of all 2294 submitted assessments, 1905 (83.04%) were submitted in the latter 3 months of each rotation, and 803 (35%) were submitted in the last month. Only 51 of 270 (18.89%) of trainees in their first year and 33 of 236 trainees in their second year (13.98%) submitted more than the minimum required number of assessments (6 per year). Of 500 randomly selected assessments, 362 (72.4%) had documented written feedback. The mean 'Gap' and 'Action' scores were low, at 0.44/3 and 0.53/3 respectively.

CONCLUSIONS

Trainees do not submit more than the required number of operative workplace-based assessments. Assessments are submitted at the end of the trainee's rotation, limiting their formative value. The quality of written feedback is poor and could be improved significantly by encouraging a "Task," "Gap" and "Action" approach.

Prospective Factors that Predict American Board of Surgery In-Training Examination Performance: A Systematic Review

INTRODUCTION

American Board of Surgery In-Training Examination (ABSITE) performance has become an important factor when monitoring resident progress. Understanding which prospective factors predict performance can help identify residents at risk.

METHODS

A literature search was conducted searching PubMed, EMBASE, and JAMA Network from June 2011 to June 2021, in accordance with the PRISMA guidelines. Searches were performed for the terms "ABSITE" and "American Board of Surgery In-Training Examination." Prospective factors such as prior examination performance, clinical evaluations, and demographics were evaluated.

RESULTS

A final 35 studies were included. The prospective factor most consistently found to predict ABSITE performance is performance on prior knowledge-based examinations such as the USMLE step exams. The ACGME Medical Knowledge 1 milestone evaluation also appears to correlate to ABSITE performance, although clinical evaluations, in general, do not. Demographics have no significant correlation to ABSITE performance.

DISCUSSION

Using performance on prior knowledge-based examinations programs may be able to identify residents at risk for failing ABSITE. It may be possible to initiate early intervention before rather than only remediation after poor performance.

International Medical Graduates are Comparable to American Medical Graduates as General Surgery Interns

BACKGROUND

International medical graduates (IMGs) are often relegated to preliminary positions in general surgery (GS) owing to uncertainties about the candidate's performance in the American healthcare setting. We aimed to determine the comparative performance of IMGs and American medical graduates (AMGs) at baseline and assess these trends over the course of their GS internship.

METHODS

Evaluations of all IMG preliminary and AMG categorical interns from 2013 to 2017 at our GS residency program were obtained from three faculty members to score overall performance, technical skills, interpersonal communication, and medical knowledge on a 10-point Likert scale. Scores on the American Board of Surgery In-Training Exam, an in-house preparation test, United States Medical Licensing Exam, and performance during the biannual multistation objective assessments were compared between the two resident groups.

RESULTS

Seventy-two interns (28 [39%] AMG categorical and 44 [61%] IMG preliminary) met inclusion criteria. The AMG group had significantly higher median Step 1 and Step 2 scores compared with our IMG group (243 versus 231, P = 0.002, and 250 versus 246, P = 0.03, respectively).Although in-house preparation test scores were higher among IMGs (median [interquartile range] of 36 [33-40] among AMGs and 38 [34-45] among IMGs; P = 0.002), there were no statistically significant differences between the American Board of Surgery In-Training Exam scores of the two groups. The median scores for the four faculty evaluation components were similar between the AMG (7, 8, 7, 7) and IMG resident groups (7, 7, 7, 7; P = nonsignificant). IMGs scored significantly higher in both biannual multistation objective assessments than AMGs (median [interquartile range] July: 59 [47-91] versus 55 [37-62], P = 0.005; January: 103 [86-116] versus 91 [87-104], P = 0.03).

CONCLUSIONS

It is reassuring to confirm that no matter where they are from, great candidates can perform well as surgical interns in a GS training program.

The Experiential Benefit of an Orthopedic Trauma Fellowship: An Analysis of ACGME Case Log Data From 2006 to 2017

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) has published orthopedic case log data since the 2006/2007 academic year. Here, we use this data to analyze the variability in orthopedic trauma case experience reported by orthopedic trainees and to better understand the impact of an orthopedic trauma fellowship on orthopedic surgical training.

DESIGN, SETTING, AND PARTICIPANTS

Data were gathered from ACGME case log reports for orthopedic residents (reporting the cumulative case experience of graduating residents) and orthopedic trauma fellows (reporting the case experience of their fellowship year only) for all available years.

RESULTS

The average orthopedic trauma fellow reported significantly more trauma cases in multiple body regions ("Pelvis/Hip", "Femur/Knee", and "Foot/Toes") and "Open Complex" reductions (as defined by the ACGME) in their 1 year of fellowship than the average resident reported in their 5 years of residency.

CONCLUSION

On average, orthopedic trauma fellowships substantially increase the trauma case volumes of orthopedic trainees, especially with respect to lower extremity trauma.