Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy

Prediction of Basic Robotic Competence for Robotic Visceral Operations Using the O-Score within the "Robotic Curriculum for Young Surgeons" (RoCS)

INTRODUCTION

Surgical residency programs lack structured assessments of robotic surgery. The validated O-Score is an assessment tool for tracking robotic operative proficiency consisting of 9 items on a 5 point Likert scale. Surgical autonomy is one comprehensive binary item. This study aimed to establish a benchmark for the number of procedures and the O-Score sum score to achieve surgical autonomy in robotic visceral procedures.

MATERIAL AND METHODS

This single-center prospective pilot cohort study assessed robotic procedures between 2020 and 2023. Bedside and console assistance performances were analyzed separately based on the O-Score and the calculated total numerical sum of the individual item values of the O-Score. Bedside assistance was conducted for upper gastrointestinal, hepatopancreatobiliary, and colorectal procedures, whereas console assistance referred to either one of the three areas. The study participants included inexperienced robotic surgeons who were evaluated by 2 robotic experts.

RESULTS

In total, 273 procedures were included in this study. For 13 bedside assistants, 273 O-Score assessments were identified, and 62 O-Score assessments for six console assistants. Surgical autonomy was achieved in 50.9% for bedside assistance and in 11.3 % for assistance at the robotic console. Surgical autonomy was positively correlated with the O-Score sum for bedside (p = < 0.001) and console assistance (p = 0.004). The positive prediction of surgical autonomy for bedside (console) assistance ranged from 74% (60%) to 93% (100%), correlated with a range of the O-Score sum between 37 (37) and 40 (40) and a robotic caseload between 19 (17) and 33 (24) procedures.

CONCLUSIONS

A significant improvement in the basic robotic performance was observed. Benchmarks regarding number of cases and O-Score sum were established for bedside assistance regardless of the type of visceral robotic operation. Currently, data on console assistance are limited. Monitoring robotic operative skills and skill progression is feasible in daily routine using the O-Score and O-Score sum.

Using artificial intelligence to evaluate adherence to best practices in one anastomosis gastric bypass: first steps in a real-world setting

BACKGROUND

Safety in one anastomosis gastric bypass (OAGB) is judged by outcomes, but it seems reasonable to utilize best practices for safety, whose performance can be evaluated and therefore improved. We aimed to test an artificial intelligence-based model in real world for the evaluation of adherence to best practices in OAGB.Please check and confirm that the authors and their respective affiliations have been correctly identified and amend if necessary.

OK METHODS

A retrospective single-center study of 89 consecutive OAGB videos was captured and analyzed by an artificial intelligence platform (10/2020-12/2023). The platform currently provides assessment of four elements, including bougie insertion, full division of pouch, view of Treitz ligament, and leak test performed. Two bariatric surgeons viewed all videos, categorizing these elements into Yes/No adherence. Intra-rater and inter-rater agreements were computed. The estimates found in greatest consensus were used to determine the model's performance. Clinical data retrieval was performed.

RESULTS

Videos included primary (71.9%) and conversion (28.1%) OAGB. Patients' age was 41.5 ± 13.6y and body mass index 42.0 ± 5.7 kg/m2. Anastomosis width was 40 mm (IQR, 30-45), and biliopancreatic limb length was 200 cm (IQR, 180-200). Operative duration was 69.1 min (IQR 55.3-97.4), mainly spent on gastric transection (26%) and anastomosis (45%). Surgeons' intra-rater overall agreements ranged 93-100% (kappa 0.57-1). Inter-rater overall agreements increased to 99-100% (kappa 0.95-1) in the second review, set as reference point to the model. The model's overall accuracy ranged 82-98%, sensitivity 91-94%, and positive predictive value 88-99%. Specificity ranged 17-92% and negative predictive value 20-68%.

CONCLUSION

The model appears to have high accuracy, sensitivity, and positive predictive value for evaluating adherence to best practices for safety in OAGB. Considering the paucity of negative estimates in our study, more low-performance cases are needed to reliably define the model's specificity and negative predictive value. Adding more best practices, tested in multi-center studies will enable cross-border standardization of the procedure.

A prospective study of a training program for bronchial sleeve resection using operable 3-dimensional models

Objective

To develop a training program for bronchial sleeve reconstruction using our previously developed 3-dimensional (3D) operable airway model and evaluate its effectiveness in surgical trainees.

Methods

Eight trainees and 4 faculty surgeons were enrolled. Their right upper lobe sleeve reconstruction procedures were scored by 2 senior surgeons in a blinded fashion on a 5-point Likert scale on the following: airway wall tear, reapplied ligatures, reapplied needles, needle entry and exit, anastomotic bite, and caliber adjustment (full score: 30). The trainees were randomized into training and control groups (n = 4 in each group). The training group underwent 6 cycles of training guided by video-based instructions. The control group underwent regular clinical training. All trainees were reevaluated.

Results

Before training, the median score of faculty surgeons was better than that of trainees (27.0 [range, 21.0-28.0] vs 17.5 [range, 9.5, 26.5]; P = .05), suggesting the validity of the scoring method. The initial scores and anastomosis times were similar in the control and training groups. After training, the scores tended to be higher in the training than in the control group (median, 28.2 [range, 27.0-29.0] vs 20.8 [range, 15.0-28.0]; P = .11). The anastomosis time tended to be shorter in the training group (median, 20.0 [18.9, 21.6] minutes vs 24.6 [range 17.8-30.9] minutes; P = .69). The reduction in anastomosis time was significantly greater in the training group (median, -9.4 [range, -4.5 to -13.1] vs 0.0 [range, 5.3 to -6.0]; P = .05).

Conclusions

The training program for bronchial sleeve resection using 3D airway models with video-based instructions improved the trainees' skills.

Effect of video-based self-reflection on intraoperative skills: A pilot randomized controlled trial

BACKGROUND

The value of video-based self-assessment in enhancing surgical skills is uncertain. This study investigates the feasibility and estimates sample size for a full-scale randomized controlled trial to evaluate the effectiveness of video-based self-assessment to improve surgical performance of laparoscopic cholecystectomy in trainees.

METHODS

This parallel pilot randomized controlled trial included general surgery trainees performing supervised laparoscopic cholecystectomy randomized 1:1 to control (traditional intraoperative teaching) or intervention group (traditional teaching plus video-based self-assessment). Operative performance was measured by the attending surgeon blinded to group assignment at the time of surgery using standardized assessment tools (Global Operative Assessment of Laparoscopic Skills and Operative Performance Rating System). The intervention group had access to their video recordings on a web-based platform for review and self-assessment using the same instruments. The primary outcome for the estimation of sample size was the difference in faculty-assessed final operative performance (third submitted case). Feasibility criteria included >85% participation, >85% adherence to case submission and >85% completion of self-assessment.

RESULTS

Of 37 eligible trainees approached, 32 consented and were randomized (86%). There were 16 in the intervention group, 15 in the control group (55% male, 55% junior trainees), and 1 was excluded for protocol violation. Twenty-four (75%) of participants submitted 3 cases. Thirteen trainees (81%) accessed the platform and completed 26 (63.2%) case self-assessments. Fifty-five trainees per arm will be needed to power a full-scale laparoscopic cholecystectomy with Global Operative Assessment of Laparoscopic Skills and 130 trainees per arm with Operative Performance Rating System as the assessment tool.

CONCLUSION

This pilot study contributes important data to inform the design of an adequately powered randomized controlled trial of video-based self-assessment to improve trainee performance of laparoscopic cholecystectomy. Although a priori trial feasibility criteria were not achieved, automated video capture and storage could significantly improve adherence in future trials.

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety

BACKGROUND

Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric.

METHODS

The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2.

RESULTS

The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases.

CONCLUSIONS

A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

Using video-based assessment (VBA) to document fellow improvement in safely completing the jejunojejunostomy portion of laparoscopic Roux-en-Y gastric bypass (RYGB) surgery

BACKGROUND

Surgical assessment instruments are used for formative and summative trainee evaluations. To characterize the features of existing instruments and a novel 12-item objective, procedure-specific assessment tool for Roux-en-Y Gastric Bypass (RYGB-OPSA), we evaluated the progress of a single surgical fellow over 17 consecutive surgeries.

METHODS

Seventeen consecutive RYGB videos completed between 8/2021 and 1/2022 by an academic hospital surgical fellow were de-identified and assessed by four board-certified bariatric surgeons using Global Operative Assessment of Laparoscopic Skills (GOALS), General Assessment of Surgical Skill (GASS), and RYGB-OPSA which includes the reflection of transverse colon, identification of ligament of Treitz, biliopancreatic and Roux limbs orientation, jejunal division point selection, stapler use, mesentery division, bleeding control, jejunojejunostomy (JJ) anastomotic site selection, apposition of JJ anastomotic site, JJ creation, common enterotomy closure of JJ, and integrity of anastomosis. The GASS measured economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, and achievement of hemostasis. RYGB-OPSA and GASS items were scored "poor-unsafe," "acceptable-safe," or "good-safe." Change in performance was measured by linear trendline slope.

RESULTS

Over the course of 17 procedures, significant improvement was demonstrated by three GOALS items, GOALS overall score, GASS bimanual dexterity, and three RYGB-OPSA tasks: JJ creation, jejunal division point selection, and stapler use. Achievement of hemostasis declined but never rated "poor-unsafe." Overall RYGB-OPSA and GOALS trendlines documented significant increase across the 17 procedures.

CONCLUSION

This examination of a bariatric surgery fellow's operative training experience as measured by three surgical assessment instruments demonstrated anticipated improvements in general skills and safe completion of procedure-specific tasks. Effective surgical assessment instruments have enough sensitivity to show improvement to enable meaningful trainee feedback (low-stakes assessments) as well as the ability to determine safe surgical practice to enable promotion to greater autonomous practice.