Fundamentals of Laparoscopic Surgery manual test: is videotaped performance assessment an option?

Recent advances in benign gynecological laparoscopic surgery

Minimally invasive surgery continues to transform the field of gynecological surgery and is now the standard of care for the surgical treatment of many diseases in gynecology. Owing to minimally invasive surgery's clear advantages, new advances in technology are being employed rapidly and enabling even the most complicated procedures to be performed less invasively. We examine recent literature on minimally invasive surgical innovations, advances, and common practices in benign gynecology that, from our point of view, made an impact on the way laparoscopic surgery is performed and managed in the last decade.

Direct Observation vs. Video-Based Assessment in Flexible Cystoscopy

OBJECTIVE

Direct observation in assessment of clinical skills is prone to bias, demands the observer to be present at a certain location at a specific time, and is time-consuming. Video-based assessment could remove the risk of bias, increase flexibility, and reduce the time spent on assessment. This study investigated if video-based assessment was a reliable tool for cystoscopy and if direct observers were prone to bias compared with video-raters.

DESIGN

This study was a blinded observational trial. Twenty medical students and 9 urologists were recorded during 2 cystoscopies and rated by a direct observer and subsequently by 2 blinded video-raters on a global rating scale (GRS) for cystoscopy. Both intrarater and interrater reliability were explored. Furthermore, direct observer bias was explored by a paired samples t-test.

RESULTS

Intrarater reliability calculated by Pearson's r was 0.86. Interrater reliability was 0.74 for single measure and 0.85 for average measures. A hawk-dove effect was seen between the 2 raters. Direct observer bias was detected when comparing direct observer scores to the assessment by an independent video-rater (p < 0.001).

CONCLUSION

This study found that video-based assessment was a reliable tool for cystoscopy with 2 video-raters. There was a significant bias when comparing direct observation with blinded video-based assessment.

Ongoing evaluation of video-based assessment of proctors' scoring of the fundamentals of laparoscopic surgery manual skills examination

BACKGROUND

In continued efforts to maintain standards required for high-stakes examination administration, Society of American Gastrointestinal and Endoscopic Surgeons' Fundamentals of Laparoscopic Surgery (FLS) program requires all prospective test proctors to complete the New FLS Proctor Workshop. As the FLS proctor pool and training evolves, the influence of experience on performance is not clear. In the previous work, we proposed that asynchronous web-based performances might be used to train proctors. In this study, we assessed the accuracy of proctors' ratings from asynchronous web-based performances by comparing the sensitivity and the specificity of ratings from proctors having varied experience levels.

METHODS

A sample of 42 (26% of all registered proctors) FLS proctors (recently trained novice, n = 15; intermediate, n = 20; and expert, n = 7) viewed 2 videotaped, laparoscopic-view FLS performances via the web. The first performance (error) contained 1 predetermined critical performance error in each of the 5 tasks (5 total errors), whereas the second performance (no error) contained no critical errors. For both the performances, participants noted time to complete each of the 5 tasks and any critical errors they observed (dichotomously scored). Participants also completed a demographic section that captured years of experience as a proctor. Using the Kruskal-Wallis test, we compared new trainee, intermediate, and expert proctors' recorded task times for both the performances. The sensitivity and the specificity values were also independently calculated using ratings from the error and the no error performances and then compared using the same test.

RESULTS

There were no differences in recorded times across proctor groups for any of the tasks; p = 0.21 and 0.94. Rating sensitivity was 79% (novice), 75% (intermediate), and 83% (expert), with no significant differences across experience levels; p > 0.46. Following removal of the peg transfer task that had particularly low sensitivity, the sensitivity for the remaining 4 tasks increased [88% and 92%]. The specificity was 93% (novice), 86% (intermediate), and 100% (expert); p = 0.046. Comments from less-experienced proctors indicated that their focus on noncritical performance issues, such as "loop not on black line" and "air knot," were consistent with decreased specificity.

CONCLUSIONS

Favorable results of this pilot study suggest that web-based FLS performances may be used for assessing FLS proctors' rating quality with adequate sensitivity and specificity. Decreased rating specificity from intermediate proctors indicates that these proctors would benefit from increased testing frequencies, updated training on current FLS proctoring standards, and ongoing performance review.