Self-directed learning by video as a means to improve technical skills in surgery residents: a randomized controlled trial

Generalizability of Artificial Intelligence Assessments in Laparoscopic Surgery Simulation

INTRODUCTION

The application of artificial intelligence (AI) in the assessment of procedural skills on a simulation platform using the global rating scale (GRS) has shown promise. Our team developed an open-source, low-cost simulation platform for the development of laparoscopic skills in low-resource settings, with skill assessment provided by video-based peer review and AI. The generalizability of AI trained on one procedure to evaluate general procedural skills within a single training system is unknown. This study examines the feasibility of generalizing AI-based assessments across procedures in a training system.

METHODS

AI was trained, with varied combinations of procedures, to score 111 laparoscopic performance videos of four procedures (57 salpingostomies, 20 appendectomies, 15 enterectomies, and 19 diaphragmatic repairs), using time and distance-based calculations. Predicted scores were generated using five-fold cross-validation and K-nearest neighbors, with both 5-class (scored 1-5) and 2-class (pass/fail) scoring systems. Videos were also scored in a conventional fashion using human video-based review, based on GRS competencies.

RESULTS

AI assessments achieved 42%-100% concordance with human reviews in the 5-class system and 68%-100% in the 2-class system, P = 0.005. Within the 5-class system, 100% accuracy was reached when AI trained on multiple procedures evaluated appendectomy. The 2-class system attained 100% accuracy in three procedures across the GRS competencies.

CONCLUSIONS

AI assessment trained on procedures using video-based review evaluated laparoscopic skills across different procedures within a simulation-based training system. Dichotomizing scoring to pass/fail improved accuracy, while supporting the potential to assess procedural competence.

Student access patterns for an online clinical skills procedural video library

CLINICAL RELEVANCE

Clinical skills training is essential in optometry curricula to develop core graduate entry competencies, including self-directed learning to facilitate life-long learning. Efficient and efficacious approaches are required to optimise student and educator time and resources.

BACKGROUND

A video library of optometric clinical skills was created in 2012 to support self-directed student learning and face-to-face training. Use of videos in higher education generally increased during the COVID-19 pandemic and has remained above pre-pandemic levels. This study aimed to capture and evaluate student access patterns in the library to determine which videos were accessed most, and if this changed with training stage and following the pandemic.

METHODS

Data on student viewing and critique submission were extracted for 71 videos from a clinical skills video library from 2018 to 2023. The number of videos viewed by students was analysed by year, cohort, video type (gold standard, scripted errors, summary, and student generated) and video category (history, screening, refraction, anterior segment, posterior segment and tonometry).

RESULTS

First-year students viewed the most videos, and this decreased significantly during and following the pandemic. Overall, the number of videos viewed decreased with increasing course stage. Video access, by category, aligned with the curriculum. Views were highest for gold standard and student videos. Viewing of scripted error videos and submission of critiques of procedural videos was low for all course stages and years.

CONCLUSION

A web-based video library of optometric clinical skills was used for self-directed learning, mostly by students early in their training. Similar resources developed in the future should align with curriculum and include exemplar and student-based videos. Exploration of student and educator perspectives regarding factors that impact engagement with the online library warrants investigation to facilitate optimal integration in post-pandemic times.

Exploring influencing factors and facilitating strategies for self-directed learning from the perspective of medical trainees: a multicentre qualitative study in China

OBJECTIVES

Exploration of influencing factors and promotion strategies for self-directed learning (SDL) is a current research hotspot. However, there is a dearth of relevant research among medical trainees. The objective of the present study was to explore the perceptions of SDL among medical trainees, while also identifying the multidimensional factors associated with SDL and potential facilitating strategies from the perspective of medical trainees.

DESIGN

This multicentre qualitative study used focus group discussions to gather insights into SDL, analysed through thematic analysis. NVivo V.12 was used for efficient data management and processing.

SETTING

Four focus group discussions were held at three large tertiary hospitals in mainland China from December 2022 to July 2023.

PARTICIPANTS

This study used purposive sampling to recruit eligible participants from the selected hospitals, primarily through instructor recommendations. Recruitment was further supported by snowball sampling, where trainees who completed the interviews referred additional participants. Medical trainees from three major tertiary hospitals in mainland China were ultimately invited to participate in the study.

RESULTS

A total of four focus group discussions were conducted involving 17 medical trainees. Thirteen themes were summarised and assigned into three categories: (1) perceptions of SDL among medical trainees, (2) multidimensional factors affecting SDL among medical trainees and (3) potential improvement strategies.

CONCLUSION

This qualitative study investigated the perspectives of medical trainees regarding SDL across various grades and hospitals in mainland China. The participants universally acknowledged the significance of SDL and expressed a strong aspiration to enhance their competence in this area. The study identified that the factors influencing SDL abilities are multifaceted, highlighting the need for tailored intervention strategies to address these challenges effectively.

Self-Directed Skills Laboratories Increase Emergency Medicine Physician Confidence in High-Acuity, Low-Opportunity Procedures

INTRODUCTION

The critical role of emergency physicians in military settings underscores the necessity for a broad and proficient skill set, especially in life-saving procedures such as thoracostomies, endotracheal intubations, and cricothyrotomies, to maintain combat readiness. The current peacetime phase, however, presents challenges in maintaining these skills because of decreased exposure to high-acuity medical scenarios. This decrease in exposure jeopardizes skills retention among military emergency medicine physicians, highlighted by studies showing a significant decline in performance over time because of reduced practice.

MATERIALS AND METHODS

This study was carried out at the Naval Medical Readiness Training Center Portsmouth under IRB approval, employing a prospective, observational, mixed-methods design. Participants included board-certified emergency medicine physicians engaged in a self-directed, small-group skills lab focusing on central venous catheterization, thoracostomy, intraosseous access, endotracheal intubation, cricothyrotomy, and resuscitative thoracotomy. Pre- and post-lab surveys on a 5-point confidence scale assessed the impact of the lab, with data analyzed via the Wilcoxon signed-rank test to evaluate significant changes.

RESULTS

Fourteen physicians reported pre-lab confidence levels, showing high confidence in central venous catheter access, thoracostomy tube placement, intraosseous catheter placement, and endotracheal intubation. The interventions of cricothyrotomy, resuscitative thoracotomy, and suprapubic catheterization started with notably lower confidence levels. Statistically significant improvements in confidence were observed post-lab for cricothyrotomy, resuscitative thoracotomy, and suprapubic catheterization suggesting the lab's effectiveness in addressing less frequently practiced procedures.

CONCLUSIONS

The Military Health System must find avenues to maintain the clinical skills of wartime procedures in the peacetime environment. Although there is no substitute for clinical encounters, alternative modalities are needed to augment skills retention in high-acuity, low-frequency procedures. Self-directed, small-group task trainers and cadaveric labs are a lower maintenance mechanism by which faculty can improve their confidence in certain procedural skills. Further studies should evaluate if this translates to changes in clinically oriented outcomes and how to optimize such training evolutions within the skills retention paradigm.

Can surgical skills be taught using technological advances online? A comparative study of online and face-to-face surgical skills training

INTRODUCTION

Online teaching has rapidly emerged as a viable alternative to traditional face-to-face education. How to teach surgical skills in the online environment, however, has not yet been fully established nor evaluated.

METHODS

An international 1-day online surgical skills course consisting of lectures, pre-recorded virtual workshops, live demonstrations and along with surgical skills teaching in breakout rooms was organised. Based on existing learning theories, new methods were developed to deliver skills teaching online. Simultaneously, traditional in-person surgical skills teaching was also conducted and used as a benchmark. Skills development was assessed by trained demonstrators and self-reported competency scores were compared between the online and face-to-face event.

RESULTS

553 delegates from 20 different countries attended the online course. Of these, 64 were trained in breakout rooms with a 1:5 demonstrator-to-delegate ratio whilst the remaining 489 delegates participated in didactic skills development sessions. In a separate face-to-face course, 20 delegates were trained with traditional methods. Demonstrators rated the competency of delegates for suturing, tendon repair and vascular anastomosis. There was no significant difference in the competency ratings of delegates receiving online teaching or face-to-face teaching (p = 0.253, p = 0.084, p = 1.00, respectively). The development of the same skills to "articulation" were not different between formats (p = 0.841, p = 0.792, p = 1.00, respectively). Post course self-rated competency scores improved for all technical skills (p < 0.001). Small group sessions, both online and face-to-face, received higher satisfaction ratings compared to large group sessions in terms of clarity of instructions, answers to questions and demonstrator feedback. Overall feedback on teaching quality, however, was equivalent across both groups.

DISCUSSION

Online teaching of surgical skills for early training years is an appropriate alternative to face-to-face teaching.

Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation

OBJECTIVE

This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation.

DESIGN

A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review.

SETTING/PARTICIPANTS

Original, peer-reviewed studies focused on VBC or assessment.

RESULTS

VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room.

CONCLUSIONS

VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.

Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation

In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.