Validation of the VBLaST: A Virtual Peg Transfer Task in Gynecologic Surgeons

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review

INTRODUCTION

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity.

MATERIAL AND METHODS

A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis.

RESULTS

A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes.

CONCLUSIONS

Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

A Systematic Review of Validity Evidence for the Fundamentals of Laparoscopic Surgery Examination in Gynecology

OBJECTIVE

The Fundamentals in Laparoscopic Surgery (FLS) examination is designed to test laparoscopic surgery skills. Our aim for this systematic review was to examine validity evidence supporting or refuting the FLS examination specifically as a high-stakes summative assessment tool in gynecology.

DATA SOURCES

The data sources were PubMed, MEDLINE, Embase, and Scopus.

METHODS OF STUDY SELECTION

The study eligibility criterion was the subject of the FLS examination as an assessment tool in gynecology. We developed a data extraction tool and assigned articles for screening and extraction to all authors, who then abstracted data independently. Conflicts that arose during the extraction process were resolved by consensus. We organized validity evidence for the cognitive and manual skills portions on the basis of the categories of current validation standards.

TABULATION, INTEGRATION, AND RESULTS

From 1971 citations identified, 9 studies were included, involving 319 participants. For the cognitive portion of the test, the results were mixed in 5 studies in relationships with the other variables category. For the manual portion of the test, most of the studies focused on the relationships with other variables evidence with mixed findings. The concerning findings in the manual skills portion included the lack of transferability of skills to the operating room, limited mixed evidence for improvement in operating room performance, and worse performance by obstetrics and gynecology surgeons compared with other specialties. We did not find supportive content-based, response process, or consequential evidence in either the cognitive or manual skills portion of the test.

CONCLUSION

Validity evidence for the FLS examination was either mixed, as it pertained to relationships with other variables, or lacking in other important evidence categories. Further evidence is required to justify the use of the FLS examination scores as a high-stakes summative assessment.

Characterizing the learning curve of a virtual intracorporeal suturing simulator VBLaST-SS©

BACKGROUND

The virtual basic laparoscopic skill trainer suturing simulator (VBLaST-SS©) was developed to simulate the intracorporeal suturing task in the FLS program. The purpose of this study was to evaluate the training effectiveness and participants' learning curves on the VBLaST-SS© and to assess whether the skills were retained after 2 weeks without training.

METHODS

Fourteen medical students participated in the study. Participants were randomly assigned to two training groups (7 per group): VBLaST-SS© or FLS, based on the modality of training. Participants practiced on their assigned system for one session (30 min or up to ten repetitions) a day, 5 days a week for three consecutive weeks. Their baseline, post-test, and retention (after 2 weeks) performance were also analyzed. Participants' performance scores were calculated based on the original FLS scoring system. The cumulative summation (CUSUM) method was used to evaluate learning. Two-way mixed factorial ANOVA was used to compare the effects of group, time point (baseline, post-test, and retention), and their interaction on performance.

RESULTS

Six out of seven participants in each group reached the predefined proficiency level after 7 days of training. Participants' performance improved significantly (p < 0.001) after training within their assigned group. The CUSUM learning curve shows that one participant in each group achieved 5% failure rate by the end of the training period. Twelve out of fourteen participants' CUSUM curves showed a negative trend toward achieving the 5% failure rate after further training.

CONCLUSION

The VBLaST-SS© is effective in training laparoscopic suturing skill. Participants' performance of intracorporeal suturing was significantly improved after training on both systems and was retained after 2 weeks of no training.