Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg. 2013;257:476-482. [PMID: 23386240 DOI: 10.1097/sla.0b013e318275b72a]

Evaluation of a 3D printed training model with realistic spatial-anatomical conditions for head and neck microsurgery

OBJECTIVES

Although existing microsurgical models provide a high degree of realism in tissue properties, they often neglect the complex and constrained spatial-anatomical conditions typical of head and neck surgery. This study aims to evaluate the effectiveness of the Head and Neck Realistic Anatomical Condition Experience (RACE) model in enhancing microsurgical education.

METHODS

Using a microsurgical competency assessment tool and self-assessment questionnaires, the head and neck RACE model was evaluated through application in two student courses (10 participants) and one resident course (5 participants). In both groups, first the conventional chicken thigh model and then the RACE model were applied. Data were analyzed using a two-way repeated measures ANOVA with Welch's statistics to assess differences between the groups.

RESULTS

In pregraduate courses, the transition from the conventional chicken thigh model to the RACE model initially led to a decline across all eight microsurgical performance parameters (Q1.1-Q4.2). However, after an additional day of training with the RACE model, all parameters-except tissue-preserving technique (Q1.2) - returned to or significantly exceeded baseline levels (Q1.2 p = 0.373, Q1.3 p = 0.003, Q2.1 p < 0.001, Q2.2 p = 0.022, Q2.3 p = 0.008, Q3.1 = 0.014, Q4.1 p = 0.036, Q4.2 p = 0.002). Conversely, residents showed immediate improvement in all parameters, except for suture distance to the vessel's margin, upon switching to the RACE model.

CONCLUSIONS

Head and neck RACE models provide a challenging and practical addition to microsurgery teaching.

CLINICAL RELEVANCE

The positive impact on learning outcomes in this area supports the development of RACE models in other areas of microsurgical and general medical training, and therefore the education of students and clinical practitioners.

Essential components and validation of multi-specialty robotic surgical training curricula: a systematic review

INTRODUCTION

The rapid adoption of robotic surgical systems has overtook the development of standardized training and competency assessment for surgeons, resulting in an unmet educational need in this field. This systematic review aims to identify the essential components and evaluate the validity of current robotic training curricula across all surgical specialties.

METHODS

A systematic search of MEDLINE, EMBASE, Emcare, and CINAHL databases was conducted to identify the studies reporting on multi-specialty or specialty-specific surgical robotic training curricula, between January 2000 and January 2024. We extracted the data according to Kirkpatrick's curriculum evaluation model and Messick's concept of validity. The quality of studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

From the 3687 studies retrieved, 66 articles were included. The majority of studies were single-center ( n = 52, 78.8%) and observational ( n = 58, 87.9%) in nature. The most commonly reported curriculum components include didactic teaching ( n = 48, 72.7%), dry laboratory skills ( n = 46, 69.7%), and virtual reality (VR) simulation ( n = 44, 66.7%). Curriculum assessment methods varied, including direct observation ( n = 44, 66.7%), video assessment ( n = 26, 39.4%), and self-assessment (6.1%). Objective outcome measures were used in 44 studies (66.7%). None of the studies were fully evaluated according to Kirkpatrick's model, and five studies (7.6%) were fully evaluated according to Messick's framework. The studies were generally found to have moderate methodological quality with a median MERSQI of 11.

CONCLUSIONS

Essential components in robotic training curricula identified were didactic teaching, dry laboratory skills, and VR simulation. However, variability in assessment methods used and notable gaps in curricula validation remain evident. This highlights the need for standardized evidence-based development, evaluation, and reporting of robotic curricula to ensure the effective and safe adoption of robotic surgical systems.

Novel Use of Objective Sensor Technology: Creation of Individualized Education Plans to Develop Operative Mastery

BACKGROUND

Motion tracking has been shown to correlate with expert and novice performance but has not been used for skill development. For skill development, performance goals must be defined. We hypothesize that by using wearable sensor technology, motion-tracking outcomes can be identified in those deemed practice ready (PR) and used as benchmarks for precision learning.

STUDY DESIGN

At the American College of Surgeons Clinical Congress, surgeons and surgeons in training volunteered to wear motion-tracking sensors while performing intracorporeal suturing and knot tying, laparoscopic pattern cutting, and simulated laparoscopic cholecystectomy. Coach raters evaluated the participants using a modified SIMPL scale. Participants were divided into 2 groups based on coach ratings: PR and not PR. Motion results were compared. Benchmark performance for each motion-tracking outcome of the simulated laparoscopic cholecystectomy was set at the median (±median absolute deviation) of the PR cohort. For all participants, each motion-tracking outcome was compared with the benchmark.

RESULTS

A total of 94 participants were recruited. Fifty-three were rated PR. Differences between groups were identified in 2 of 10 metrics for intracorporeal suturing and knot tying, 4 of 10 metrics for pattern cutting, and 5 of 10 metrics for laparoscopic cholecystectomy. Set benchmarks for the metrics were compared with each participant's score for all individual metrics. A not PR individual was less likely to achieve the benchmarks (chi-square = 55.48, p < 0.00001).

CONCLUSIONS

Wearable sensor technology can identify differences between surgeons rated PR and not PR. More importantly, motion metric results can be used to develop benchmarks for training endpoints. This will allow for the development of an individualized report card and training protocol focused on areas in need of improvement instead of the current model of generalized training. Implementing such focused training may expedite competency and mastery of surgical skills.

Vascular pedicle dissection time in laparoscopic colectomies as a novel marker of surgical skill: a prospective cohort study

BACKGROUND

Outcomes after colorectal resections depend on patient, pathology and operative factors. Existing validated surgical skills scores (such as the competency assessment tool (CAT)) are directly correlated with outcomes but are time-consuming to administer, limiting their clinical utility. The vascular pedicle dissection time (VPDT) is a novel, simple surgical skill assessment measure with the potential for computer vision automation. This study aimed to assess the VPDT and benchmark it against the CAT score.

METHODS

A prospective multicentre study was performed in New Zealand, recording videos of laparoscopic colorectal resections. Patient, operation and histology characteristics were also collected. The VPDT was calculated from retraction of the vascular pedicle to completion of medial dissection, including vascular division. Each laparoscopic video was scored by two independent colorectal surgeons, and the median CAT score was grouped into tertiles.

RESULTS

In total, 154 patients were included between December 2020 and November 2023 (74 (48.1%) right-sided and 80 (51.9%) left-sided resections). Median VPDT was significantly different between the CAT score groups for the right-sided resections (lower, 15 min; middle, 13 min; higher, 10 min; p = 0.036) and the left-sided resections (lower, 46 min; middle, 40 min; higher, 26 min; p =  < 0.001). There was no significant difference in R1 resection, anastomotic leak rate, the occurrence of Clavien-Dindo > 3 complications or re-admission between the CAT groups.

CONCLUSIONS

This study showed that the VPDT was inversely correlated with the CAT score, indicating that it quantifies operative technical skill. A current study is evaluating the suitability of VPDT for real-time measurement using computer vision algorithms. This could allow for automated assessment of surgeons' learning curve and skills.

Validation of a simulator for oncoplastic breast conserving surgery

BACKGROUND

Therapeutic Mammoplasty (TM) is increasingly becoming the standard of care, especially for patients with large tumor-to-breast volume ratios. The wider dissemination of oncoplastic skills warrants systems for the acquisition and assessment of safe skills. To date, TM simulations have not been developed for the acquisition or assessment of oncoplastic skills. This study aimed to design, develop, and validate a synthetic TM simulator for specialist surgical training and assessment.

METHODS

A prospective, observational, and survey-based study. Breast surgeons collaborated with designers to construct a TM simulator. A modified Delphi approach was used to create a Competency Assessment Tool (CAT). Surgeons with varying operative experience performed simulated vertical scar TM. Procedures were videotaped (blinded, pseudo-anonymized), subsequently reviewed, and independently rated against CAT by three experts. Specimen radiographs and volumetric analysis were performed to assess specimen weight(g), volume(cm3), and adequacy of resection, derived as a percentage of deviation in uniformity around a 10 mm margin.

RESULTS

Thirty participants were recruited (10 consultants, 10 senior registrars (ST7-8), and 10 junior registrars (ST3-6)). Video-based rating scores (0-40) were significantly greater in consultants (median(IQR) = 34.0(30.5-38.0)) than in senior registrars (median(IQR) = 30.0(28.0-33.0)) and junior registrars (median(IQR) = 28.0(25.8-30.3)). The CAT scores varied significantly based on operator grade (p < 0.05). The inter-rater reliability showed fair agreement (κ = 0.379). Specimens resected by consultants had significantly greater weight and volume (p < 0.05). The consultants demonstrated the greatest uniformity in resection accuracy (p < 0.05).

CONCLUSION

A novel oncoplastic simulator was developed to practice and assess TM skills. Video-based ratings and end-product assessments differentiated experts from novice surgeons, suggesting construct validity.

Artificial intelligence assessment of tissue-dissection efficiency in laparoscopic colorectal surgery

PURPOSE

Several surgical-skill assessment tools emphasize the importance of efficient tissue-dissection, whose assessment relies on human judgment and is thus subject to bias. Automated assessment may help solve this problem. This study aimed to verify the feasibility of surgical-skill assessment using a deep learning-based recognition model.

METHODS

This retrospective study used multicenter intraoperative videos of laparoscopic colorectal surgery (sigmoidectomy or high anterior resection) for colorectal cancer obtained from 766 cases across Japan. Three groups with different skill levels were distinguished: high-, intermediate-, and low-skill. We developed a model to recognize tissue dissection by the monopolar device using deep learning-based computer-vision technology. Tissue-dissection time per monopolar device appearance time (efficient-dissection time ratio) was extracted as a quantitative parameter describing efficient dissection. We automatically measured the efficient-dissection time ratio using the recognition model of 8 surgical instruments and tissue-dissection on/off classification model. The efficient-dissection time ratio was compared among groups; the feasibility of distinguishing them was explored using the model. The model-calculated parameters were evaluated to determine whether they could differentiate high-, intermediate-, and low-skill groups.

RESULTS

The tissue-dissection recognition model had an overall accuracy of 0.91. There was a moderate correlation (0.542; 95% confidence interval, 0.288-0.724; P < 0.001) between manually and automatically measured efficient-dissection time ratios. Efficient-dissection time ratios by this model were significantly higher in the high-skill than in intermediate-skill (P = 0.0081) and low-skill (P = 0.0249) groups.

CONCLUSION

An automated efficient-dissection assessment model using a monopolar device was constructed with a feasible automated skill-assessment method.

Development and validation of a new tool for assessment of trainees' interventional musculoskeletal ultrasound skills

OBJECTIVES

Interventional musculoskeletal ultrasound (MSUS) procedures are routinely performed in rheumatology practice. However, the efficacy and safety of the procedures rely on the competence of the physician, and assessment of skills is crucial. Thus, this study aimed to develop and establish validity evidence for a tool assessing trainees' interventional MSUS skills.

METHODS

An expert panel of rheumatologists modified an existing tool for assessing competences in invasive abdominal and thoracic ultrasound procedures. The new tool (the Assessment of Interventional Musculoskeletal Ultrasound Skills [AIMUS] tool) reflects the essential steps in interventional MSUS. To establish validity evidence, physicians with different levels of interventional MSUS experience were enrolled and performed two procedures on a rubber phantom, simulating real patient cases. All performances were video-recorded, anonymized and assessed in random order by two blinded raters using the AIMUS tool.

RESULTS

65 physicians from 21 different countries were included and categorized into groups based on their experience, resulting in 130 videos for analysis. The internal consistency of the tool was excellent, with a Cronbach's α of 0.96. The inter-case reliability was good with a Pearson's correlation coefficient (PCC) of 0.74 and the inter-rater reliability was moderate to good (PCC 0.58). The ability to discriminate between different levels of experience was highly significant (P < 0.001).

CONCLUSION

We have developed and established validity evidence for a new interventional MSUS assessment tool. The tool can be applied in future competency-based educational programmes, provide structured feedback to trainees in daily clinical practice and ensure end-of-training competence.

TRIAL REGISTRATION

ClinicalTrials.gov, http://clinicaltrials.gov, NCT05303974.

Video-based surgical quality assessment of minimally invasive right hemicolectomy by medical students after specific training

BACKGROUND

Recently, a competency assessment tool has been developed within the RIGHT project, a national quality improvement program for minimally invasive right hemicolectomy in patients with colon cancer. This study aimed to evaluate whether trained medical students can reliably evaluate minimally invasive right hemicolectomy videos using a competency assessment tool.

METHODS

Nine expert colorectal surgeons, 13 trained medical students, and 17 untrained medical students assessed the surgical quality of 6 full-length minimally invasive right hemicolectomy videos with the competency assessment tool. The expert surgeons were trained using the competency assessment tool by the RIGHT project leaders, who were also involved in the development and validation of the competency assessment tool. Training for medical students included anatomy, step-by-step procedure explanation, and competency assessment tool review with 2 supervised video assessments. The untrained students were taught only anatomy and minimally invasive right hemicolectomy steps. The intraclass correlation coefficient was calculated to determine inter-rater reliability, and analysis of variance with the Bonferroni correction for multiple testing was used to assess potential differences between the groups per video.

RESULTS

The trained students demonstrated an overall excellent inter-rater reliability (intraclass correlation coefficient score of 0.885). When their scores were combined with those of the expert surgeons, a high inter-rater reliability was also demonstrated (intraclass correlation coefficient score of 0.945). Trained students consistently aligned with surgeons' mean total scores, also accurately identifying lower quality surgeries. Untrained students assigned statistically significantly higher scores to the 3 lower quality surgeries as compared with expert surgeons and trained students.

CONCLUSION

Among trained students, excellent inter-rater reliability and concordance with expert colorectal surgeons was found. The study highlights the potential to engage trained medical students for objective minimally invasive right hemicolectomy video assessment.

A Rectal Cancer Surgery Dataset: Use of artificial intelligence to aid automation of error identification

Minimally invasive surgery is complex and prone to variation not routinely objectively measured. We established an association between skills and patient outcomes. The evolving application of artificial intelligence techniques could assist intraoperative analysis. In this study, we analysed 77 rectal cancer operations' videos from a multicentre RCT that were recorded unedited and underwent blinded manual analysis using a validated, bespoke performance assessment tool (LapTMEpt) and the Objective Clinical Human Reliability Analysis (OCHRA). The OCHRA methodology involved segmentation of the 77 operations and manually annotating each case for the enacted errors and near misses. We provide a detailed description of the errors and near misses of over 380 hours of video analysis, containing 1377 errors. This dataset can inform machine learning to assist progress toward a fully automated, objective assessment of surgical skills.

How Accurate Are Surgeons at Assessing the Quality of Their Critical View of Safety During Laparoscopic Cholecystectomy?

INTRODUCTION

Obtaining the critical view of safety (CVS) is considered an important step to reduce bile duct injuries during laparoscopic cholecystectomy (LC). However, existing literature suggests that few surgeons obtain adequate CVS when LC videos are directly evaluated by experts. This discrepancy calls for effective, standardized CVS teaching methods. While self-assessment (SA) remains the principal tool utilized by practicing surgeons for performance improvement, its effectiveness is controversial. The aim of this study was to compare surgeon SAs of repeated LC performance and attainment of the CVS with that of expert raters.

METHODS

Multi-institutional study of surgeon members from the Society of American Gastrointestinal and Endoscopic Surgeons who volunteered to participate. All surgeons were asked to submit an LC video and complete a SA of the CVS quality using the Strasberg scale (0-6 score with ≥5 score indicating appropriate CVS). The same videos were reviewed by two blinded expert raters, members of the Society of American Gastrointestinal and Endoscopic Surgeons safe cholecystectomy task force, who had received prior rater training. Surgeon self-ratings and expert ratings were compared with a Wilcoxon signed-rank test.

RESULTS

Twenty-five surgeon-participants were recruited, 13 of whom submitted an LC video. Surgeons did not achieve adequate CVS in their first submitted video based on expert ratings. Surgeons in the SA group overestimated their performance across all four scales: Operative Performance Rating System (z = -0.36, P = 0.715), Global Operative Assessment of Laparoscopic Skills (z = -0.37, P = 0.712), Strasberg (z = -1.84, P = 0.066), and Competency Assessment Tool (z = -0.73, P = 0.465). Surgeons in the coaching group overestimated their performance on each scale as well: Operative Performance Rating System (z = -0.67, P = 0.500), Global Operative Assessment of Laparoscopic Skills (z = -1.48, P = 0.138), Strasberg (z = -1.07, P = 0.285), and Competency Assessment Tool (z = -1.21, P = 0.225).

CONCLUSIONS

Our study confirms that an adequate CVS is infrequently obtained during LC in a small but national sample of general surgeons. It further adds to the existing body of literature that suggests that SA alone may be inadequate for performance improvement. Effective teaching methods such as expert or artificial intelligence coaching are needed to improve the use of appropriate CVS by surgeons that may help decrease bile duct injury risk.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

IMPORTANCE

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

OBJECTIVE

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

DESIGN, SETTING, AND PARTICIPANTS

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

EXPOSURE

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

MAIN OUTCOMES AND MEASURES

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

RESULTS

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

CONCLUSIONS AND RELEVANCE

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Incentive policy for the comprehensive development of young medical talents: an evolutionary game study

Introduction

Currently in China, there is a lack of well-defined and viable incentive mechanisms at the governmental and hospital levels to support the development of young medical talents, thereby hindering their growth Existing studies primarily investigate the current state and trajectory of incentives, yet they inadequately address the distinctive characteristics of various stakeholders involved in medical talent incentive processes, particularly the lack of research on incentive mechanisms with Chinese attributes.

Methods

This study adopts evolutionary game theory to investigate the dynamics of replication and the strategies for achieving evolutionary stability in the comprehensive development of young medical talents, considering both scenarios with and without supportive policies.

Results

In the absence of any supportive policy measures, the evolutionarily stable strategy (ESS) point is O(0,0), the unstable equilibrium point is C(1,1), and the saddle points are A(0,1), B(1,0). The initial state of the system is at the unstable equilibrium point C(1,1), which means that the young medical talents and medical institutions adopt a combination of strategies (actively seeking comprehensive development and taking incentive measures). Under the scenario with supportive policies, the ESS point is C(1,1), the unstable equilibrium point is O(0,0), and the saddle points are A(0,1), B(1,0). The initial state of the system is at the unstable equilibrium point O(0,0), which means that young medical talents and medical institutions adopt (N,N) strategy combinations (inactively seeking comprehensive development, implementing no incentive measure).

Discussion

(1) Government incentives play a crucial role in motivating young medical talents to seek comprehensive development. (2) The level of government incentive support for young medical talents should exceed the cost increment of individual efforts. Additionally, the policy support provided by the government to medical institutions should surpass the incentive support offered by these institutions to young medical talents. This will enhance the motivation and encouragement efforts of medical institutions in actively promoting comprehensive development among young medical talents. (3) With the backing of certain government incentive policies, medical institutions implementing incentive measures and young medical talents actively seeking comprehensive development will establish a virtuous cycle of mutual promotion.

Nationwide standardization of minimally invasive right hemicolectomy for colon cancer and development and validation of a video-based competency assessment tool (the Right study)

BACKGROUND

Substantial variation exists when performing a minimally invasive right hemicolectomy (MIRH) due to disparities in training, expertise and differences in implementation of innovations. This study aimed to achieve national consensus on an optimal and standardized MIRH technique for colon cancer and to develop and validate a video-based competency assessment tool (CAT) for MIRH.

METHOD

Statements covering all elements of MIRH were formulated. Subsequently, the Delphi technique was used to reach consensus on a standardized MIRH among 76 colorectal surgeons from 43 different centres. A CAT was developed based on the Delphi results. Nine surgeons assessed the same 12 unedited full-length videos using the CAT, allowing evaluation of the intraclass correlation coefficient (ICC).

RESULTS

After three Delphi rounds, consensus (≥80% agreement) was achieved on 23 of the 24 statements. Consensus statements included the use of low intra-abdominal pressure, detailed anatomical outline how to perform complete mesocolic excision with central vascular ligation, the creation of an intracorporeal anastomosis, and specimen extraction through a Pfannenstiel incision using a wound protector. The CAT included seven consecutive steps to measure competency of the MIRH and showed high consistency among surgeons with an overall ICC of 0.923.

CONCLUSION

Nationwide consensus on a standardized and optimized technique of MIRH was reached. The CAT developed showed excellent interrater reliability. These achievements are crucial steps to an ongoing nationwide quality improvement project (the Right study).

Smart surgical glasses for orofacial cleft surgery training in high- and low-income countries: A proof of concept

INTRODUCTION

The coronavirus 2019 pandemic urged us to find alternatives for education through remote proctoring and international surgical collaborations among high-, middle-, and low-income countries. Smart surgical glasses are promising for remote surgical education and international surgical collaborations.

AIMS

This study aimed to assess the usability of smart surgical glasses during cleft surgery and explore their potential in remote surgical education and collaboration.

METHODS

Six plastic surgical cases were randomly selected and recorded using the RODS&CONES glasses in 4K (3840 × 2160p). A 23-point questionnaire was sent to one plastic surgeon, one plastic surgery resident, and eight doctors who were not trained to critically appraise the video and audio quality of the smart surgical glasses and their applicability for remote surgical education.

RESULTS

The participants indicated that the smart glasses had several significant advantages over conventional on-site education, such as facilitating a better view of the surgical field and providing possibilities for remote interaction. The audio quality was considered excellent. The main limitations were image stabilization issues and loss of video connection due to weak wireless fidelity.

CONCLUSIONS

All participants appreciated the use of smart glasses for remote education and considered them a promising tool for enhancing the quality of surgical education. The glasses can enable remote assistance and education of local surgical residents and may facilitate sustainable surgical collaborations among high-, middle-, and low-income countries.

Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos

BACKGROUND

Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model.

METHODS

Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score.

RESULTS

The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04).

CONCLUSION

The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.

Technical assessment in minimally invasive complete mesocolic excision: Is the complete mesocolic excision competency assessment tool valid and reliable?

AIM

The complete mesocolic excision competency assessment tool (CMECAT) is a novel tool designed to assess technical skills in minimally invasive complete mesocolic excision (CME) surgery. The aim of this study was to assess construct validity and reliability of CMECAT in a clinical context.

METHOD

Colorectal surgeons were asked to submit video recorded laparoscopic CME resections for independent assessment of their technical abilities. The videos were grouped by surgeons' training level, and four established CME experts were recruited as CMECAT assessors. Extended reliability analysis (G-theory) was applied to describe assessor agreement.

RESULTS

A total of 19 videos and 72 assessments were included in the analysis. Overall, technical skills assessed by CMECAT improved with increased training level: the experts scored significantly better than the untrained surgeons (3.3 vs. 2.5 points; p < 0.01). On right-sided resections, significantly higher scores were reported with increased training level for all categories and sections, while for left-sided resections, the variance across groups was smaller and significantly higher scores were only reported for oncological safety describing items. Overall, assessor agreement was high (G-coefficient: 0.81).

CONCLUSION

This study confirms that CMECAT can be applied to video recorded CME cases for technical skill assessment. Further, it can reliably assess technical performance in right sided CME surgery, where construct validity has now been established. More videos are required to evaluate its validity on left colonic CME. In the future, we hope CMECAT can improve feedback during CME training, serve as a tool in certification processes and contribute to distinguishing CME from conventional surgery in future research.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy

BACKGROUND

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.

METHODS

Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).

RESULTS

Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).

CONCLUSION

GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Procedure-based assessment implementation in colorectal surgery: a scoping review

BACKGROUND

Competency-based training (CBT) programs use procedure-based assessments (PBAs) to evaluate trainees' abilities to perform specific procedures in clinical settings, similar to Entrustable Professional Activities (EPAs). PBAs help determine trainees' readiness for advanced training levels. However, there is limited evidence on implementing colorectal-specific PBAs in surgical training schemes. This review aims to identify observed and perceived challenges to implementing PBAs in workplace settings.

METHODS

A scoping review following the Joanna Briggs Institute Protocol for Scoping Reviews (JBI-ScR) was conducted. Eligible studies provided evidence on the implementation, feasibility, and challenges of PBAs in colorectal surgery, including various study designs from retrospective to prospective.

RESULTS

Of the 80 screened studies, 75 were excluded based on exclusion criteria. Most of the included studies were conducted in national training institutions in the United Kingdom, assessing 778 colorectal procedures with specific PBAs. The main facilitators of implementing PBAs were structured assessments, focused assessors' training, and electronic forms usage.

CONCLUSION

This review offers insight into the practicality and feasibility of implementing PBAs in colorectal surgery. Identified challenges include the need for adequate assessor training and the time-consuming nature of the assessment. These findings could improve PBA implementation in colorectal surgery and enhance surgical education quality. However, the limited number of studies and existing literature heterogeneity call for more research to identify other gaps.

Automatic Surgical Skill Assessment System Based on Concordance of Standardized Surgical Field Development Using Artificial Intelligence

Importance

Automatic surgical skill assessment with artificial intelligence (AI) is more objective than manual video review-based skill assessment and can reduce human burden. Standardization of surgical field development is an important aspect of this skill assessment.

Objective

To develop a deep learning model that can recognize the standardized surgical fields in laparoscopic sigmoid colon resection and to evaluate the feasibility of automatic surgical skill assessment based on the concordance of the standardized surgical field development using the proposed deep learning model.

Design, Setting, and Participants

This retrospective diagnostic study used intraoperative videos of laparoscopic colorectal surgery submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017. Data were analyzed from April 2020 to September 2022.

Interventions

Videos of surgery performed by expert surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were used to construct a deep learning model able to recognize a standardized surgical field and output its similarity to standardized surgical field development as an AI confidence score (AICS). Other videos were extracted as the validation set.

Main Outcomes and Measures

Videos with scores less than or greater than 2 SDs from the mean were defined as the low- and high-score groups, respectively. The correlation between AICS and ESSQS score and the screening performance using AICS for low- and high-score groups were analyzed.

Results

The sample included 650 intraoperative videos, 60 of which were used for model construction and 60 for validation. The Spearman rank correlation coefficient between the AICS and ESSQS score was 0.81. The receiver operating characteristic (ROC) curves for the screening of the low- and high-score groups were plotted, and the areas under the ROC curve for the low- and high-score group screening were 0.93 and 0.94, respectively.

Conclusions and Relevance

The AICS from the developed model strongly correlated with the ESSQS score, demonstrating the model's feasibility for use as a method of automatic surgical skill assessment. The findings also suggest the feasibility of the proposed model for creating an automated screening system for surgical skills and its potential application to other types of endoscopic procedures.

Experience of practicing obstetrician-gynecologists in a surgical training program in total laparoscopic hysterectomy

BACKGROUND

Advanced surgical techniques, such as total laparoscopic hysterectomy, are often challenging to acquire beyond fellowship training programs for practicing obstetrician-gynecologists. A lack of formative data currently exist for continuing medical education programs, limiting our understanding of how improvement in surgical skills and training programs occur.

OBJECTIVE

This study aimed to investigate how practicing obstetrician-gynecologists acting as trainees experience a program that aims to teach them total laparoscopic hysterectomy, and to assess whether their surgical skills improve according to data from formative assessment tools and qualitative data from open-ended survey questions and in-depth interviews.

STUDY DESIGN

We report a process analysis of formative data collected during a pilot implementation trial of a surgical training program targeting practicing obstetrician-gynecologists. Eleven consultant obstetrician-gynecologists and 4 experienced surgical mentors participated in 4 hospitals in Queensland, Australia. Total laparoscopic hysterectomy was performed in 700 patients over the course of the study. A total laparoscopic hysterectomy surgical mentorship training program of 10 training days with up to 3 total laparoscopic hysterectomy procedures per day was performed. Both the obstetrician-gynecologists and the surgical mentor completed a formative assessment questionnaire analyzing the trainee's performance after each surgical procedure. Mentors were formatively assessed by the Structured Training Trainer Assessment Report (STTAR) and at the completion of the study by the mini-STTAR, a summative assessment of quality of mentorship. Obstetrician-gynecologists, mentors, hospital leaders, and surgical administrative staff participated in qualitative interviews about the training program.

RESULTS

Over time, there was a demonstrated improvement in trainee performance reported by both mentors and trainees in all competency assessment tool domains as the case number increased, with mentors consistently rating trainees' performance higher than the trainees themselves. Most trainees were satisfied with their mentor in all 31 areas during formative assessment, and at the end of the training, structure, attributes, and role modeling were all rated high (average score >4.5; range, 3.79-5.00), whereas training behavior was rated slightly lower at 4.1 (range, 3.79-4.45). Qualitative interviews demonstrated that the trainees found the training to be a beneficial, hands-on experience.

CONCLUSION

Formative assessment clearly documented improvement in surgical skills during a total laparoscopic hysterectomy training program for consultant obstetrician-gynecologists.

Preserving privacy in surgical video analysis using a deep learning classifier to identify out-of-body scenes in endoscopic videos

Surgical video analysis facilitates education and research. However, video recordings of endoscopic surgeries can contain privacy-sensitive information, especially if the endoscopic camera is moved out of the body of patients and out-of-body scenes are recorded. Therefore, identification of out-of-body scenes in endoscopic videos is of major importance to preserve the privacy of patients and operating room staff. This study developed and validated a deep learning model for the identification of out-of-body images in endoscopic videos. The model was trained and evaluated on an internal dataset of 12 different types of laparoscopic and robotic surgeries and was externally validated on two independent multicentric test datasets of laparoscopic gastric bypass and cholecystectomy surgeries. Model performance was evaluated compared to human ground truth annotations measuring the receiver operating characteristic area under the curve (ROC AUC). The internal dataset consisting of 356,267 images from 48 videos and the two multicentric test datasets consisting of 54,385 and 58,349 images from 10 and 20 videos, respectively, were annotated. The model identified out-of-body images with 99.97% ROC AUC on the internal test dataset. Mean ± standard deviation ROC AUC on the multicentric gastric bypass dataset was 99.94 ± 0.07% and 99.71 ± 0.40% on the multicentric cholecystectomy dataset, respectively. The model can reliably identify out-of-body images in endoscopic videos and is publicly shared. This facilitates privacy preservation in surgical video analysis.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

Assessment methods in laparoscopic colorectal surgery: a systematic review of available instruments

BACKGROUND

Laparoscopic surgery has become the golden standard for many procedures, requiring new skills and training methods. The aim of this review is to appraise literature on assessment methods for laparoscopic colorectal procedures and quantify these methods for implementation in surgical training.

MATERIALS AND METHODS

PubMed, Embase and Cochrane Central Register of Controlled Trials databases were searched in October 2022 for studies reporting learning and assessment methods for laparoscopic colorectal surgery. Quality was scored using the Downs and Black checklist. Included articles were categorized in procedure-based assessment methods and non-procedure-based assessment methods. A second distinction was made between capability for formative and/or summative assessment.

RESULTS

In this systematic review, nineteen studies were included. These studies showed large heterogeneity despite categorization. Median quality score was 15 (range 0-26). Fourteen studies were categorized as procedure-based assessment methods (PBA), and five as non-procedure-based assessment methods. Three studies were applicable for summative assessment.

CONCLUSIONS

The results show a considerable diversity in assessment methods with varying quality and suitability. To prevent a sprawl of assessment methods, we argue for selection and development of available high-quality assessment methods. A procedure-based structure combined with an objective assessment scale and possibility for summative assessment should be cornerstones.

An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery

OBJECTIVE

The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills.

BACKGROUND

Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery.

METHODS

Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS.

RESULTS

To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns.

CONCLUSIONS

VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.

AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

INTRODUCTION

Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers.

METHODS

A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection.

RESULTS

An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification.

CONCLUSIONS

Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Development of a procedure-specific tool for skill assessment in left- and right-sided laparoscopic complete mesocolic excision

AIM

To (1) develop an assessment tool for laparoscopic complete mesocolic excision (LCME) and (2) report evidence of its content validity.

METHOD

Assessment statements were revealed through (1) semi-structured expert interviews and (2) consensus by the Delphi method, both involving an expert panel of five LCME surgeons. All experts were interviewed and then asked to rate LCME describing statements from 1 (strongly disagree) to 5 (strongly agree). Responses were returned anonymously to the panel until consensus was reached. Statements were directly included as content in the assessment tool if ≥60% of the experts responded "agree" or "strongly agree" (ratings 4 and 5), with the remaining responses being "neither agree nor disagree" (rating 3). Interclass correlation coefficient (ICC) was calculated for expert agreement evaluation. All included statements were subsequently reformulated as tool items and approved by the experts.

RESULTS

Four Delphi rounds were performed to reach consensus. Disagreement was reported for statements describing instrument handling around pancreas; visualisation of landmarks before inferior mesenteric artery ligation; lymphadenectomy around the inferior mesenteric artery, and division of the terminal ileum and transverse colon. ICC in the last Delphi-round was 0.84. The final tool content included 73 statements, converted to 48 right- and 40 left-sided items for LCME assessment.

CONCLUSION

A procedure-specific, video-based tool, named complete mesocolic excision competency assessment tool (CMECAT), has been developed for LCME skill assessment. In the future, we hope it can facilitate assessment of LCME surgeons, resulting in improved patient outcome after colon cancer surgery.

Serbian National Training Programme for minimally invasive colorectal surgery (LapSerb): short-term clinical outcomes of over 1400 colorectal resections

INTRODUCTION

The Serbian National Training Programme for minimally invasive colorectal surgery (LapSerb) was introduced to implement laparoscopic colorectal surgery across Serbia. The programme aimed to accelerate training of established colorectal surgeons through a competency-based programme. This involved knowledge assessment, workshops, live operating, and competency-based assessment of unedited videos. The aim of this study is to report the outcomes of laparoscopic colorectal resection performed by LapSerb certified surgeons.

METHODS

LapSerb prospectively maintained multicentred database was analysed for laparoscopic colorectal resections from January 2015 to February 2021. Data collected included patient demographics, indications for surgery, perioperative data, and 30-day outcomes.

RESULTS

A total of 1456 laparoscopic colectomies by 24 certified surgeons were included in the final analysis. Mean age was 67 (± 12) years old and male to female ratio was 1:1.5. 83.1% of the colectomies were malignant, mainly due to adenocarcinoma. Anterior resection was the most common procedure with 699 (48%) cases, followed by right and left colectomies with 357 (24.5%) and 303 (21%) procedure respectively. 4.8% of patients required conversion to open surgery. Thirty-day readmission and reoperation rates were 2.3% and 4.7%, respectively. Overall mortality in all cases was 1.1% and R0 resections were achieved in 97.8% of malignant colectomies.

CONCLUSION

The LapSerb programme successfully and safely established laparoscopic colorectal surgery across the country with comparable and acceptable short-term clinical outcomes.

Validation of a Newly Developed Competency Assessment Tool for the Posterior Sagittal Anorectoplasty

INTRODUCTION

 The correction of an anorectal malformation (ARM) is complex and relatively infrequent. Simulation training and subsequent assessment may result in better clinical outcomes. Assessment can be done using a competency assessment tool (CAT). This study aims to develop and validate a CAT for the posterior sagittal anorectoplasty (PSARP) on a simulation model.

MATERIALS AND METHODS

 The CAT-PSARP was developed after consultation with experts in the field. The PSARP was divided into five steps, while tissue and instrument handling were scored separately. Participants of pediatric colorectal hands-on courses in 2019 and 2020 were asked to participate. They performed one PSARP procedure on an ARM simulation model, while being assessed by two objective observers using the CAT-PSARP.

RESULTS

 A total of 82 participants were enrolled. A fair interobserver agreement was found for general skills (intraclass correlation coefficient [ICC] = 0.524, p < 0.001), a good agreement for specific skills (ICC = 0.646, p < 0.001), and overall performance (ICC = 0.669, p < 0.001). The experienced group scored higher on all steps (p < 0.001), except for "anoplasty (p = 0.540)," compared with an inexperienced group.

CONCLUSION

 The CAT-PSARP is a suitable objective assessment tool for the overall performance of the included steps of the PSARP for repair of an ARM on a simulation model.

Standardization of ileoanal J-pouch surgery technique: Quality assessment of minimally invasive ileoanal J-pouch surgery videos

BACKGROUND

Ileal pouch anal anastomosis is a complex procedure associated with significant morbidity, with several complications after ileal pouch anal anastomosis surgery leading to pouch failure. The aim of the study is to evaluate the heterogeneity surrounding the technique of ileoanal J-pouch surgery by assessing the safety and quality of published online peer-reviewed surgical videos.

METHODS

Ileal pouch anal anastomosis videos published on peer-reviewed surgical journals and video channels were edited and anonymized to demonstrate specific steps of the surgical procedure: mobilization and division of the rectum, formation of the ileoanal J-pouch reservoir, anastomosis, and lengthening techniques. The anonymized videos were presented to a group of reviewers with expertise in ileal pouch anal anastomosis blinded to the names and affiliations of the surgeons performing the procedure. Primary outcome was the rate of interobserver variability in the assessment of specific technical steps of the ileal pouch anal anastomosis surgery procedure. Secondary outcome was the appropriateness of the use of surgical videos review as an assessment tool for ileal pouch anal anastomosis surgery, measured as rate of reviewers being unable to answer for poor video quality.

RESULTS

In total, 29 video fragments were distributed, and 13 assessors completed a 60-item survey, organized in 7 major domains. The survey completion rate was 93.4%. Out of a total 729 answers, in 23 (3.2%) the reviewers indicated they were unable to comment due to poor video image, and in 48 (6.5%) were unable to comment due to the particular step not being shown in the procedure. The proportion of assessors rating rectal mobilization technically appropriate ranged from 30.7% to 92.3% and from 7.7% to 69.2% for safety. The level of rectal division was considered appropriate in 0 to 53.8% of the videos, whereas the stapling technique used for rectal division was appropriate in 0 to 70% of the videos.

CONCLUSION

Our study assessed published peer-reviewed videos on ileal pouch anal anastomosis surgery and reported heterogeneity in the safety of the demonstrated techniques. Blind assessment of published peer-reviewed ileal pouch anal anastomosis videos reported a high rate of unsafe or inappropriate technique for rectal mobilization and transection in the reviewed videos, with fair interobserver agreement among reviewers. There is a need for consensus on what is considered safe and appropriate in ileal pouch anal anastomosis surgery. Peer review of ileal pouch anal anastomosis surgery videos could facilitate training and accreditation in this complex procedure.

Impact of Endoscopic Surgical Skill Qualification on Laparoscopic Resections for Rectal Cancer in Japan: The EnSSURE Study

Objective

This cohort study investigated short- and long-term postoperative outcomes of laparoscopic procedures for rectal cancer performed with versus without certified surgeons.

Background

In Japan, the Endoscopic Surgical Skill Qualification System (ESSQS) evaluates surgical skills deemed essential for laparoscopic surgery; however, it is unknown whether this certification contributes to procedural safety.

Methods

Outcomes of laparoscopic rectal resections for cStage II and III rectal cancer performed from 2014 to 2016 at 56 Japanese hospitals were retrospectively reviewed. The impact of having versus not having certified surgeons on postoperative complications and other short- and long-term outcomes were assessed. In cases with ESSQS-certified surgeons, surgeons attended surgery in the capacity of an operator, assistant, scope operator, or advisor.

Results

Overall, 3188 procedures were analyzed, with 2644 procedures performed with and 544 without ESSQS-certified surgeons. A multivariate logistic regression model showed that the adjusted odds ratio of postoperative complications after procedures performed with ESSQS-certified surgeons was 0.68 (95% confidence interval, 0.51-0.91; P = 0.009). The adjusted odds ratios for conversion and pathological R0 resection rates with ESSQS-certified surgeons were 0.20 (P < 0.001) and 2.10 (P = 0.04), respectively. Multiple linear regression analyses showed significantly shorter surgical duration and more harvested lymph nodes for operations performed with ESSQS-certified surgeons. Multivariate Cox regression showed that the adjusted hazard ratios for poor overall and recurrence-free survival after operations performed with ESSQS-certified surgeons were 0.88 (P = 0.35) and 1.04 (P = 0.71), respectively.

Conclusions

This study showed the superiority of the short-term postoperative results for laparoscopic rectal procedures performed with ESSQS-certified surgeons.

Artificial Intelligence-Based Total Mesorectal Excision Plane Navigation in Laparoscopic Colorectal Surgery

BACKGROUND

Total mesorectal excision is the standard surgical procedure for rectal cancer because it is associated with low local recurrence rates. To the best of our knowledge, this is the first study to use an image-guided navigation system with total mesorectal excision.

IMPACT OF INNOVATION

The impact of innovation is the development of a deep learning-based image-guided navigation system for areolar tissue in the total mesorectal excision plane. Such a system might be helpful to surgeons because areolar tissue can be used as a landmark for the appropriate dissection plane.

TECHNOLOGY, MATERIALS, AND METHODS

This was a single-center experimental feasibility study involving 32 randomly selected patients who had undergone laparoscopic left-sided colorectal resection between 2015 and 2019. Deep learning-based semantic segmentation of areolar tissue in the total mesorectal excision plane was performed. Intraoperative images capturing the total mesorectal excision scene extracted from left colorectal laparoscopic resection videos were used as training data for the deep learning model. Six hundred annotation images were created from 32 videos, with 528 images in the training and 72 images in the test data sets. The experimental feasibility study was conducted at the Department of Colorectal Surgery, National Cancer Center Hospital East, Chiba, Japan. Dice coefficient was used to evaluate semantic segmentation accuracy for areolar tissue.

PRELIMINARY RESULTS

The developed semantic segmentation model helped locate and highlight the areolar tissue area in the total mesorectal excision plane. The accuracy and generalization performance of deep learning models depend mainly on the quantity and quality of the training data. This study had only 600 images; thus, more images for training are necessary to improve the recognition accuracy.

CONCLUSION AND FUTURE DIRECTIONS

We successfully developed a total mesorectal excision plane image-guided navigation system based on an areolar tissue segmentation approach with high accuracy. This may aid surgeons in recognizing the total mesorectal excision plane for dissection.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

BACKGROUND

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

METHODS

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

RESULTS

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

CONCLUSION

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Development of a Reliable Surgical Quality Assurance System for 2-stage Esophagectomy in Randomized Controlled Trials

OBJECTIVE

The aim was to develop a reliable surgical quality assurance system for 2-stage esophagectomy. This development was conducted during the pilot phase of the multicenter ROMIO trial, collaborating with international experts.

SUMMARY OF BACKGROUND DATA

There is evidence that the quality of surgical performance in randomized controlled trials influences clinical outcomes, quality of lymphadenectomy and loco-regional recurrence.

METHODS

Standardization of 2-stage esophagectomy was based on structured observations, semi-structured interviews, hierarchical task analysis, and a Delphi consensus process. This standardization provided the structure for the operation manual and video and photographic assessment tools. Reliability was examined using generalizability theory.

RESULTS

Hierarchical task analysis for 2-stage esophagectomy comprised fifty-four steps. Consensus (75%) agreement was reached on thirty-nine steps, whereas fifteen steps had a majority decision. An operation manual and record were created. A thirty five-item video assessment tool was developed that assessed the process (safety and efficiency) and quality of the end product (anatomy exposed and lymphadenectomy performed) of the operation. The quality of the end product section was used as a twenty seven-item photographic assessment tool. Thirty-one videos and fifty-three photographic series were submitted from the ROMIO pilot phase for assessment. The overall G-coefficient for the video assessment tool was 0.744, and for the photographic assessment tool was 0.700.

CONCLUSIONS

A reliable surgical quality assurance system for 2-stage esophagectomy has been developed for surgical oncology randomized controlled trials.

ETHICAL APPROVAL

11/NW/0895 and confirmed locally as appropriate, 12/SW/0161, 16/SW/0098.Trial registration number: ISRCTN59036820, ISRCTN10386621.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Can complex surgical interventions be standardized? Reaching international consensus on posterior sagittal anorectoplasty using a modified-Delphi method

BACKGROUND/PURPOSE

In an effort to standardize educational experience, address future physician shortages, and improve quality of care to patients, many surgical specialties are discussing how to maximize exposure to index cases. One solution being explored is telementoring, which requires a well-developed educational curriculum with intraoperative objectives. The American College of Surgery Telementoring Task Force selected anorectal malformation and posterior sagittal anorectoplasty (PSARP) for the repair of imperforate anus as the initial educational focus for this pilot. The purpose of this study was to obtain international consensus on intraoperative learning objectives for a complex surgical procedure.

METHODS

A multidisciplinary team of medical educators and pediatric surgery experts created an outline of essential curricular content and intraoperative learning objectives for PSARP in three clinical scenarios. Twelve international subject matter experts were identified meeting strict inclusion criteria. Intraoperative checklists were revised using the modified-Delphi process.

RESULTS

After five rounds of modifications to the intraoperative checklists, international consensus was achieved for three different clinical scenarios requiring a PSARP: perineal or vestibular fistula, low prostatic fistula, and bladder neck fistula.

CONCLUSIONS

A modified-Delphi approach was successful in generating guidelines for surgical techniques that can be used to standardize intraoperative teaching and expectations for trainees.

TYPE OF STUDY

Diagnostic study LEVEL OF EVIDENCE: Level V (expert opinion).

Development and implementation of an assessment tool to evaluate technical skills in the insertion of implantable venous access devices, a Prospective Cohort Study

OBJECTIVE

Based on the Competency Assessment Tool, herein we developed an assessment instrument suitable to evaluate the implantation of central intravenous devices.

BACKGROUND

Surgical assessment is based mainly on the subjective impressions of the teacher. Based on the "Competency Assessment Tool" (CAT) developed for the evaluation of technical surgical skills in minimally invasive colorectal resection, we designed an assessment tool suitable to evaluate the implantation of central venous access devices performed by junior surgical trainees.

METHODS

Four major assessments during the different steps of the intervention were used in this evaluation. Each of these tasks was divided into four sub-domains according to surgical skill. In addition to the CAT score, the apprentices' skills were evaluated using a visual assessment that was quantified using an analogue scale (value from 1 to 10). The candidates were classified into junior and senior trainees depending on the number of procedures they had already performed and on their surgical experience.

RESULTS

71 procedures were evaluated during the study period. Seven senior trainees conducted 43 procedures and five junior trainees performed 28 interventions. The senior trainees had significantly higher CAT scores than junior candidates, and the scores fluctuated according to surgical experience, usually reaching their peak after 10 procedures.

CONCLUSIONS

The CAT model is well suited for the assessment of surgical trainees during central venous access device implantation. It enables a close assessment of the learning process and the technical skills of trainees, which helps them improving in a safe, standardized manner.

Development of a surgical competency assessment tool for sentinel lymph node dissection by minimally invasive surgery for endometrial cancer

INTRODUCTION

Sentinel lymph node dissection is widely used in the staging of endometrial cancer. Variation in surgical techniques potentially impacts diagnostic accuracy and oncologic outcomes, and poses barriers to the comparison of outcomes across institutions or clinical trial sites. Standardization of surgical technique and surgical quality assessment tools are critical to the conduct of clinical trials. By identifying mandatory and prohibited steps of sentinel lymph node (SLN) dissection in endometrial cancer, the purpose of this study was to develop and validate a competency assessment tool for use in surgical quality assurance.

METHODS

A Delphi methodology was applied, included 35 expert gynecological oncology surgeons from 16 countries. Interviews identified key steps and tasks which were rated mandatory, optional, or prohibited using questionnaires. Using the surgical steps for which consensus was achieved, a competency assessment tool was developed and subjected to assessments of validity and reliability.

RESULTS

Seventy percent consensus agreement standardized the specific mandatory, optional, and prohibited steps of SLN dissection for endometrial cancer and informed the development of a competency assessment tool. Consensus agreement identified 21 mandatory and three prohibited steps to complete a SLN dissection. The competency assessment tool was used to rate surgical quality in three preselected videos, demonstrating clear separation in the rating of the skill level displayed with mean skills summary scores differing significantly between the three videos (F score=89.4; P<0.001). Internal consistency of the items was high (Cronbach α=0.88).

CONCLUSION

Specific mandatory and prohibited steps of SLN dissection in endometrial cancer have been identified and validated based on consensus among a large number of international experts. A competency assessment tool is now available and can be used for surgeon selection in clinical trials and for ongoing, prospective quality assurance in routine clinical care.

Clinical Evaluation of Intraoperative Near Misses in Laparoscopic Rectal Cancer Surgery

OBJECTIVE

To investigate the frequency, nature, and severity of intraoperative adverse near miss events within advanced laparoscopic surgery and report any associated clinical impact.

BACKGROUND

Despite implementation of surgical safety initiatives, the intraoperative period is poorly documented with evidence of underreporting. Near miss analyses are undertaken in high-risk industries but not in surgical practice.

METHODS

Case video and data from 2 laparoscopic total mesorectal excision randomized controlled trials were analyzed (ALaCaRT ACTRN12609000663257, 2D3D ISRCTN59485808). Intraoperative adverse events were identified and categorized using the observational clinical human reliability analysis technique. The EAES classification was applied by 2 blinded assessors. EAES grade 1 events (nonconsequential error, no damage, or need for correction) were considered near misses. Associated clinical impact was assessed with early morbidity and histopathology outcomes.

RESULTS

One hundred seventy-five cases contained 1113 error events. Six hundred ninety-eight (62.7%) were near misses (median 3, IQR 2-5, range 0-15) with excellent inter-rater and test-retest reliability (κ=0.86, 95% CI 0.83-0.89, P < 0.001 and κ=0.88, 95% CI 0.85-0.9, P < 0.001 respectively). Significantly more near misses were seen in patients who developed early complications (4 (3-6) vs. 3 (2-4), P < 0.001). Higher numbers of near misses were seen in patients with more numerous (P = 0.002) and more serious early complications (P = 0.003). Cases containing major intraoperative adverse events contained significantly more near misses (5 (3-7) vs. 3 (2-5), P < 0.001) with a major event observed for every 19.4 near misses.

CONCLUSION

Intraoperative adverse events and near misses can be reliably and objectively captured in advanced laparoscopic surgery. Near misses are commonplace and closely associated with morbidity outcomes.

Development and validation of a recommended checklist for assessment of surgical videos quality: the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool

INTRODUCTION

There has been a constant increase in the number of published surgical videos with preference for open-access sources, but the proportion of videos undergoing peer-review prior to publication has markedly decreased, raising questions over quality of the educational content presented. The aim of this study was the development and validation of a standard framework for the appraisal of surgical videos submitted for presentation and publication, the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool.

METHODS

An international committee identified items for inclusion in the LAP-VEGaS video assessment tool and finalised the marking score utilising Delphi methodology. The tool was finally validated by anonymous evaluation of selected videos by a group of validators not involved in the tool development.

RESULTS

9 items were included in the LAP-VEGaS video assessment tool, with every item scoring from 0 (item not presented in the video) to 2 (item extensively presented in the video), with a total marking score ranging from 0 to 18. The LAP-VEGaS video assessment tool resulted highly accurate in identifying and selecting videos for acceptance for conference presentation and publication, with high level of internal consistency and generalisability.

CONCLUSIONS

We propose that peer review in adherence to the LAP-VEGaS video assessment tool could enhance the overall quality of published video outputs.

Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery

Importance

Complex surgical interventions are inherently prone to variation yet they are not objectively measured. The reasons for outcome differences following cancer surgery are unclear.

Objective

To quantify surgical skill within advanced laparoscopic procedures and its association with histopathological and clinical outcomes.

Design, Setting, and Participants

This analysis of data and video from the Australasian Laparoscopic Cancer of Rectum (ALaCaRT) and 2-dimensional/3-dimensional (2D3D) multicenter randomized laparoscopic total mesorectal excision trials, which were conducted at 28 centers in Australia, the United Kingdom, and New Zealand, was performed from 2018 to 2019 and included 176 patients with clinical T1 to T3 rectal adenocarcinoma 15 cm or less from the anal verge. Case videos underwent blinded objective analysis using a bespoke performance assessment tool developed with a 62-international expert Delphi exercise and workshop, interview, and pilot phases.

Interventions

Laparoscopic total mesorectal excision undertaken with curative intent by 34 credentialed surgeons.

Main Outcomes and Measures

Histopathological (plane of mesorectal dissection, ALaCaRT composite end point success [mesorectal fascial plane, circumferential margin, ≥1 mm; distal margin, ≥1 mm]) and 30-day morbidity. End points were analyzed using surgeon quartiles defined by tool scores.

Results

The laparoscopic total mesorectal excision performance tool was produced and shown to be reliable and valid for the specialist level (intraclass correlation coefficient, 0.889; 95% CI, 0.832-0.926; P < .001). A substantial variation in tool scores was recorded (range, 25-48). Scores were associated with the number of intraoperative errors, plane of mesorectal dissection, and short-term patient morbidity, including the number and severity of complications. Upper quartile-scoring surgeons obtained excellent results compared with the lower quartile (mesorectal fascial plane: 93% vs 59%; number needed to treat [NNT], 2.9, P = .002; ALaCaRT end point success, 83% vs 58%; NNT, 4; P = .03; 30-day morbidity, 23% vs 50%; NNT, 3.7; P = .03).

Conclusions and Relevance

Intraoperative surgical skill can be objectively and reliably measured in complex cancer interventions. Substantial variation in technical performance among credentialed surgeons is seen and significantly associated with clinical and pathological outcomes.

Patient-specific mental rehearsal with three-dimensional models before low anterior resection: randomized clinical trial

Background

It was hypothesized that preparing for a surgical procedure, taking into account individual patient characteristics, may facilitate the procedure and improve surgical quality. The aim of this study was to compare different case-specific, preoperative mental rehearsal methods before minimally invasive rectal cancer surgery.

Methods

In this RCT, patients were allocated in a 1 : 1 : 1 : 1 ratio to four groups: systematic mental rehearsal (SMR) using MRI scans; SMR and three-dimensional (3D) virtual models; SMR and synthetic 3D printed models; and routine practice (control group). Surgeons operating on all but the control group underwent mental rehearsal with the visual aids, including axial MRI scans of the pelvis, interactive 3D virtual models reconstructed from axial MRIs, and synthetic models, manufactured by 3D printing. Operations were video-recorded and assessed by two experts blinded to allocation using two validated scores, the Competency Assessment Tool (CAT) and Objective Clinical Human Reliability Analysis (OCHRA). The primary outcome of the study was surgical performance, measured by the CAT.

Results

Forty-nine patients were randomized and allocated to the four groups. There were 12 participants in each of the control, MRI and SMR, and virtual and SMR groups, whereas the SMR using physical models and simulation group included 13. No difference was observed between groups in median CAT scores (control 30.50, MRI 34.25, virtual 31.75, physical 34.00; P = 0.748, partial η² <0.001, where _pη² is indicative of effect size) or OCHRA scores (anterior, posterior, right and left lateral planes, transection P>0.200, _pη² =0.052–0.088). Time spent not performing dissection was significantly shorter for the SMR with MRI group than for the control (57.5 versus 42 respectively; P < 0.001, _pη² =0.212).

Conclusion

Mental rehearsal did not affect CAT and OCHRA scores of consultant surgeons. Reference number: ISRCTN 75603704 (https://www.isrctn.com).

Laparoscopic Colorectal Surgery Outcomes Improved After National Training Program (LAPCO) for Specialists in England

OBJECTIVE

To examine the impact of The National Training Program for Lapco on the rate of laparoscopic surgery and clinical outcomes of cases performed by Lapco surgeons after completion of training.

SUMMARY OF BACKGROUND DATA

Lapco provided competency-based supervised clinical training for specialist colorectal surgeons in England.

METHODS

We compared the rate of laparoscopic surgery, mortality, and morbidity for colorectal cancer resections by Lapco delegates and non-Lapco surgeons in 3-year periods preceding and following Lapco using difference in differences analysis. The changes in the rate of post-Lapco laparoscopic surgery with the Lapco sign-off competency assessment and in-training global assessment scores were examined using risk-adjusted cumulative sum to determine their predictive clinical validity with predefined competent scores of 3 and 5 respectively.

RESULTS

One hundred eight Lapco delegates performed 4586 elective colo-rectal resections pre-Lapco and 5115 post-Lapco while non-Lapco surgeons performed 72,930 matched cases. Lapco delegates had a 37.8% increase in laparoscopic surgery which was greater than non-Lapco surgeons by 20.9% [95% confidence interval (CI), 18.5-23.3, P < 0.001) with a relative decrease in 30-day mortality by -1.6% (95% CI, -3.4 to -0.2, P = 0.039) and 90-day mortality by -2.3% (95% CI, -4.3 to -0.4, P = 0.018). The change point of risk-adjusted cumulative sum was 3.12 for competency assessment tool and 4.74 for global assessment score whereas laparoscopic rate increased from 44% to 66% and 40% to 56%, respectively.

CONCLUSIONS

Lapco increased the rate of laparoscopic colorectal cancer surgery and reduced mortality and morbidity in England. In-training competency assessment tools predicted clinical performance after training.

A methodology for Dynamic Human Reliability Analysis in Robotic Surgery

Surgery has changed significantly in recent years due to the introduction of advanced technologies, resulting in increased system complexity at the technical, human and organisational levels, which may lead to higher variability of patient outcome due to new error pathways. Current approaches towards a safer surgery are largely based on ex-post analysis of events and process monitoring (e.g. root cause analysis, safety checklists, safety audits). However, adopting a proactive approach enables the prior identification of critical factors and the design of safer sociotechnical systems, thanks to a multi-level (or mesoergnomics) perspective. In this paper, a methodology for performing mesoergonomics analysis of surgical procedures is proposed. It is a methodology for Dynamic Human Reliability Analysis in Robotic Surgery based on a modified version of human error assessment and reduction technique (HEART) integrated with a method for incorporating uncertainties related to the influence of personal and organisational factors on the execution of a surgical procedure. The pilot application involves a robot-assisted radical prostatectomy procedure, and the results reveal that team-related factors have the greatest impact on patient outcome variability.

Accuracy and usefulness in assessing proficiency of the observational clinical human reliability assessment checklist of the open inguinal hernia repair procedure: A cross-sectional study

BACKGROUND

The Observational Clinical Human Reliability Assessment (OCHRA) can be used to score errors during surgical procedures. To construct an OCHRA-checklist, steps, substeps, and hazards of a surgical procedure need to be defined. A step-by-step framework was developed to segment surgical procedures into steps, substeps, and hazards. The first aim of this study was to investigate if the step-by-step framework could be used to construct an accurate Lichtenstein open inguinal hernia repair (LOIHR) stepwise description. The second aim was to investigate if the OCHRA-checklist based on this stepwise description was accurate and useful for surgical training and assessment.

MATERIALS AND METHODS

Ten expert surgeons rated statements regarding the accuracy of the LOIHR stepwise description, the accuracy, and the usefulness of the LOIHR OCHRA-checklist (eight, seven, and six statements, respectively) using a 5-point Likert scale. One-sample Wilcoxon signed-rank test was used to compare the outcomes to the neutral value of 3.

RESULTS

The accuracy of the stepwise description and the accuracy and usefulness of the OCHRA-checklist were rated statistically significantly higher than the neutral value of 3 (median 4.75 [5.00-4.00] with p = .009, median 5.00 [5.00-4.00] with p = .012, median 4.00 [5.00-4.00] with p = .047, respectively). The experts rated the OCHRA-checklist to be useful for the training (5.00 [5.00-4.00], p = .009), and assessment (4.50 [5.00-4.00], p = .010) of surgical residents.

CONCLUSION

This preliminary study showed that the stepwise LOIHR description constructed using the step-by-step framework was found to be accurate. The LOIHR OCHRA-checklist developed using the stepwise description was also accurate, and particularly useful for the training and assessment of proficiency of surgical residents.

The what? How? And Who? Of video based assessment

BACKGROUND

Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance.

METHODS

This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment.

RESULTS

Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video.

CONCLUSION

Validity, scalability and discoverability are current but removable barriers to video-based assessment.

Surgical Quality Assurance in COLOR III: Standardization and Competency Assessment in a Randomized Controlled Trial

OBJECTIVE

The aim of this study was to develop an objective and reliable surgical quality assurance system (SQA) for COLOR III, an international multicenter randomized controlled trial (RCT) comparing transanal total mesorectal excision (TaTME) with laparoscopic approach for rectal cancer.

BACKGROUND OF SUMMARY DATA

SQA influences outcome measures in RCTs such as lymph nodes harvest, in-hospital mortality, and locoregional cancer recurrence. However, levels of SQA are variable.

METHOD

Hierarchical task analysis of TaTME was performed. A 4-round Delphi methodology was applied for standardization of TaTME steps. Semistructured interviews were conducted in round 1 to identify key steps and tasks, which were rated as mandatory, optional, or prohibited in rounds 2 to 4 using questionnaires. Competency assessment tool (CAT) was developed and its content validity was examined by expert surgeons. Twenty unedited videos were assessed to test reliability using generalizability theory.

RESULTS

Eighty-three of 101 surgical tasks identified reached 70% agreement (26 mandatory, 56 optional, and 1 prohibited). An operative guide of standardized TaTME was created. CAT is matrix of 9 steps and 4 performance qualities: exposure, execution, adverse event, and end-product. The overall G-coefficient was 0.883. Inter-rater and interitem reliability were 0.883 and 0.986. To enter COLOR III, 2 unedited TaTME and 1 laparoscopic TME videos were submitted and assessed by 2 independent assessors using CAT.

CONCLUSION

We described an iterative approach to develop an objective SQA within multicenter RCT. This approach provided standardization, the development of reliable and valid CAT, and the criteria for trial entry and monitoring surgical performance during the trial.