Validity evidence for a new portable, lower-cost platform for the fundamentals of endoscopic surgery skills test

Democratizing Flexible Endoscopy Training: Noninferiority Randomized Trial Comparing a Box-Trainer vs a Virtual Reality Simulator to Prepare for the Fundamental of Endoscopic Surgery Exam

BACKGROUND

A considerable number of surgical residents fail the mandated endoscopy exam despite having completed the required clinical cases. Low-cost endoscopy box trainers (BTs) could democratize training; however, their effectiveness has never been compared with higher-cost virtual reality simulators (VRSs).

STUDY DESIGN

In this randomized noninferiority trial, endoscopy novices trained either on the VRS used in the Fundamental of Endoscopic Surgery manual skills (FESms) exam or a validated BT-the Basic Endoscopic Skills Training (BEST) box. Trainees were tested at fixed timepoints on the FESms and on standardized ex vivo models. The primary endpoint was FESms improvement at 1 week. Secondary endpoints were FESms improvement at 2 weeks, FESms pass rates, ex vivo tests performance, and trainees' feedback.

RESULTS

Seventy-seven trainees completed the study. VRS and BT trainees showed comparable FESms improvements (25.16 ± 14.29 vs 25.58 ± 11.75 FESms points, respectively; p = 0.89), FESms pass rates (76.32% vs 61.54%, respectively; p = 0.16) and total ex vivo tasks completion times (365.76 ± 237.56 vs 322.68 ± 186.04 seconds, respectively; p = 0.55) after 1 week. Performances were comparable also after 2 weeks of training, but FESms pass rates increased significantly only in the first week. Trainees were significantly more satisfied with the BT platform (3.97 ± 1.20 vs 4.81 ± 0.40 points on a 5-point Likert scale for the VRS and the BT, respectively; p < 0.001).

CONCLUSIONS

Simulation-based training is an effective means to develop competency in endoscopy, especially at the beginning of the learning curve. Low-cost BTs like the BEST box compare well with high-tech VRSs and could help democratize endoscopy training.

The Fundamentals of Endoscopic Surgery (FES) skills test: factors associated with first-attempt scores and pass rate

BACKGROUND

The Fundamentals of Endoscopic Surgery (FES) program became required for American Board of Surgery certification as part of the Flexible Endoscopy Curriculum (FEC) for residents graduating in 2018. This study expands prior psychometric investigation of the FES skills test.

METHODS

We analyzed de-identified first-attempt skills test scores and self-reported demographic characteristics of 2023 general surgery residents who were required to pass FES.

RESULTS

The overall pass rate was 83%. "Loop Reduction" was the most difficult sub-task. Subtasks related to one another only modestly (Spearman's ρ ranging from 0.11 to 0.42; coefficient α = .55). Both upper and lower endoscopic procedural experience had modest positive association with scores (ρ = 0.14 and 0.15) and passing. Examinees who tested on the GI Mentor Express simulator had lower total scores and a lower pass rate than those tested on the GI Mentor II (pass rates = 73% vs. 85%). Removing an Express-specific scoring rule that had been applied eliminated these differences. Gender, glove size, and height were closely related. Women scored lower than men (408- vs. 489-point averages) and had a lower first-attempt pass rate (71% vs. 92%). Glove size correlated positively with score (ρ = 0.31) and pass rate. Finally, height correlated positively with score (r = 0.27) and pass rate. Statistically controlling for glove size and height did not eliminate gender differences, with men still having 3.2 times greater odds of passing.

CONCLUSIONS

FES skills test scores show both consistencies with the assessment's validity argument and several remarkable findings. Subtasks reflect distinct skills, so passing standards should perhaps be set for each subtask. The Express simulator-specific scoring penalty should be removed. Differences seen by gender are concerning. We argue those differences do not reflect measurement bias, but rather highlight equity concerns in surgical technology, training, and practice.

A pilot study to assess the utility of a freely downloadable mobile application simulator for undergraduate clinical skills training: a single-blinded, randomised controlled trial

BACKGROUND

Medical simulators offer an invaluable educational resource for medical trainees. However, owing to cost and portability restrictions, they have traditionally been limited to simulation centres. With the advent of sophisticated mobile technology, simulators have become cheaper and more accessible. Touch Surgery is one such freely downloadable mobile application simulator (MAS) used by over one million healthcare professionals worldwide. Nevertheless, to date, it has never been formally validated as an adjunct in undergraduate medical education.

METHODS

Medical students in the final 3 years of their programme were recruited and randomised to one of three revision interventions: 1) no formal revision resources, 2) traditional revision resources, or 3) MAS. Students completed pre-test questionnaires and were then assessed on their ability to complete an undisclosed male urinary catheterisation scenario. Following a one-hour quarantined revision period, all students repeated the scenario. Both attempts were scored by allocation-blinded examiners against an objective 46-point mark scheme.

RESULTS

A total of 27 medical students were randomised (n = 9 per group). Mean scores improved between baseline and post-revision attempts by 8.7% (p = 0.003), 19.8% (p = 0.0001), and 15.9% (p = 0.001) for no resources, traditional resources, and MAS, respectively. However, when comparing mean score improvements between groups there were no significant differences.

CONCLUSIONS

Mobile simulators offer an unconventional, yet potentially useful adjunct to enhance undergraduate clinical skills education. Our results indicate that MAS's perform comparably to current gold-standard revision resources; however, they may confer significant advantages in terms of cost-effectiveness and practice flexibility.

TRIAL REGISTRATION

Not applicable.