Assessment of learning curves on a simulated neurosurgical task using metrics selected by artificial intelligence

First validated automated scoring system using the diagnostic arthroscopy skill score (DASS 2.0) for assessing proficiency in virtual reality arthroscopy

PURPOSE

Proficiency-based simulator training is a promising approach for learning the complex motor skills required for arthroscopy. However, its implementation requires an objective assessment tool to assess residents' arthroscopic skills. To address this need, an automated diagnostic arthroscopy skill score (DASS) was developed and validated as a replacement for manual scoring.

METHODS

An automated measurement system of arthroscopic skills was developed, replacing the manual assessment parameters of the DASS with objective measurement criteria. Data from arthroscopies performed by 20 experts were used to establish threshold values for scoring. To validate the new method, 125 videos of residents were recorded and evaluated by five raters. The results of the automated evaluation were then compared with those of established manual assessment. To assess the reliability of the manual evaluation, the intraclass correlation coefficient (ICC) and minimum detectable change (MDC) were calculated. Methodological agreement was evaluated using linear least squares regression and the Bland-Altman method.

RESULTS

A good to excellent level of reliability was found among the five raters (ICC for DASSpart1 = 0.89, 95% confidence interval [CI] = 0.84-0.92). The calculated MDC was 5.0 points. High methodological agreement was found between the manual and automated evaluations of the DASS. The 95% CI for the slope of the regression line included 1, and the 95% CI for the intercept included 0. According to the Bland-Altman method, the mean difference between manual and automated evaluations was 4.1 ± 5.4 points, and the scattering of the measurement differences was uniformly distributed, regardless of the total score.

CONCLUSIONS

Automated DASS measurement is a valid and reliable tool for assessing arthroscopic skills. Its advantages, such as rater independence, precise and objective measurements, and immediate evaluation, make it a powerful tool for evaluating arthroscopic performance during simulation training.

LEVEL OF EVIDENCE

Level II.

The impact of surgical simulation and training technologies on general surgery education

The landscape of general surgery education has undergone a significant transformation over the past few years, driven in large part by the advent of surgical simulation and training technologies. These innovative tools have revolutionized the way surgeons are trained, allowing for a more immersive, interactive, and effective learning experience. In this review, we will explore the impact of surgical simulation and training technologies on general surgery education, highlighting their benefits, challenges, and future directions. Enhancing the technical proficiency of surgical residents is one of the main benefits of surgical simulation and training technologies. By providing a realistic and controlled environment, With the use of simulations, residents may hone their surgical skills without compromising patient safety. Research has consistently demonstrated that training with simulations enhances surgical skills., reduces errors, and enhances overall performance. Furthermore, simulators can be programmed to mimic a wide range of surgical scenarios, enabling residents to cultivate the essential critical thinking and decision-making abilities required to manage intricate surgical cases. Another area of development is incorporating simulation-based training into the wider surgical curriculum. As simulation technologies become more widespread, they will need to be incorporated into the fabric of surgical education, rather than simply serving as an adjunct to traditional training methods. This will require a fundamental shift in the way surgical education is delivered, with a greater emphasis on simulation-based training and assessment.

The Role of Artificial Intelligence in Medical Education: A Systematic Review

BACKGROUND

To examine the artificial intelligence (AI) tools currently being studied in modern medical education, and critically evaluate the level of validation and the quality of evidence presented in each individual study.

METHODS

This review (PROSPERO ID: CRD42023410752) was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. A database search was conducted using PubMed, Embase, and Cochrane Library. Articles written in the English language between 2000 and March 2023 were reviewed retrospectively using the MeSH Terms "AI" and "medical education" A total of 4642 potentially relevant studies were found.

RESULTS

After a thorough screening process, 36 studies were included in the final analysis. These studies consisted of 26 quantitative studies and 10 studies investigated the development and validation of AI tools. When examining the results of studies in which Support vector machines (SVMs) were employed, it has demonstrated high accuracy in assessing students' experiences, diagnosing acute abdominal pain, classifying skilled and novice participants, and evaluating surgical training levels. Particularly in the comparison of surgical skill levels, it has achieved an accuracy rate of over 92%.

CONCLUSION

AI tools demonstrated effectiveness in improving practical skills, diagnosing diseases, and evaluating student performance. However, further research with rigorous validation is required to identify the most effective AI tools for medical education.

Real-Time multifaceted artificial intelligence vs In-Person instruction in teaching surgical technical skills: a randomized controlled trial

Trainees develop surgical technical skills by learning from experts who provide context for successful task completion, identify potential risks, and guide correct instrument handling. This expert-guided training faces significant limitations in objectively assessing skills in real-time and tracking learning. It is unknown whether AI systems can effectively replicate nuanced real-time feedback, risk identification, and guidance in mastering surgical technical skills that expert instructors offer. This randomized controlled trial compared real-time AI feedback to in-person expert instruction. Ninety-seven medical trainees completed a 90-min simulation training with five practice tumor resections followed by a realistic brain tumor resection. They were randomly assigned into 1-real-time AI feedback, 2-in-person expert instruction, and 3-no real-time feedback. Performance was assessed using a composite-score and Objective Structured Assessment of Technical Skills rating, rated by blinded experts. Training with real-time AI feedback (n = 33) resulted in significantly better performance outcomes compared to no real-time feedback (n = 32) and in-person instruction (n = 32), .266, [95% CI .107 .425], p < .001; .332, [95% CI .173 .491], p = .005, respectively. Learning from AI resulted in similar OSATS ratings (4.30 vs 4.11, p = 1) compared to in-person training with expert instruction. Intelligent systems may refine the way operating skills are taught, providing tailored, quantifiable feedback and actionable instructions in real-time.

Continuous Instrument Tracking in a Cerebral Corticectomy Ex Vivo Calf Brain Simulation Model: Face and Content Validation

BACKGROUND AND OBJECTIVES

Subpial corticectomy involving complete lesion resection while preserving pial membranes and avoiding injury to adjacent normal tissues is an essential bimanual task necessary for neurosurgical trainees to master. We sought to develop an ex vivo calf brain corticectomy simulation model with continuous assessment of surgical instrument movement during the simulation. A case series study of skilled participants was performed to assess face and content validity to gain insights into the utility of this training platform, along with determining if skilled and less skilled participants had statistical differences in validity assessment.

METHODS

An ex vivo calf brain simulation model was developed in which trainees performed a subpial corticectomy of three defined areas. A case series study assessed face and content validity of the model using 7-point Likert scale questionnaires.

RESULTS

Twelve skilled and 11 less skilled participants were included in this investigation. Overall median scores of 6.0 (range 4.0-6.0) for face validity and 6.0 (range 3.5-7.0) for content validity were determined on the 7-point Likert scale, with no statistical differences between skilled and less skilled groups identified.

CONCLUSION

A novel ex vivo calf brain simulator was developed to replicate the subpial resection procedure and demonstrated face and content validity.

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

The impact of artificial intelligence (AI) will be felt not only in the arena of patient care and deliverable therapies but will also be uniquely disruptive in medical education and healthcare simulation (HCS), in particular. As HCS is intertwined with computer technology, it offers opportunities for rapid scalability with AI and, therefore, will be the most practical place to test new AI applications. This will ensure the acquisition of AI literacy for graduates from the country's various healthcare professional schools. Artificial intelligence has proven to be a useful adjunct in developing interprofessional education and team and leadership skills assessments. Outcome-driven medical simulation has been extensively used to train students in image-centric disciplines such as radiology, ultrasound, echocardiography, and pathology. Allowing students and trainees in healthcare to first apply diagnostic decision support systems (DDSS) under simulated conditions leads to improved diagnostic accuracy, enhanced communication with patients, safer triage decisions, and improved outcomes from rapid response teams. However, the issue of bias, hallucinations, and the uncertainty of emergent properties may undermine the faith of healthcare professionals as they see AI systems deployed in the clinical setting and participating in diagnostic judgments. Also, the demands of ensuring AI literacy in our healthcare professional curricula will place burdens on simulation assets and faculty to adapt to a rapidly changing technological landscape. Nevertheless, the introduction of AI will place increased emphasis on virtual reality platforms, thereby improving the availability of self-directed learning and making it available 24/7, along with uniquely personalized evaluations and customized coaching. Yet, caution must be exercised concerning AI, especially as society's earlier, delayed, and muted responses to the inherent dangers of social media raise serious questions about whether the American government and its citizenry can anticipate the security and privacy guardrails that need to be in place to protect our healthcare practitioners, medical students, and patients.

A scoping review of artificial intelligence in medical education: BEME Guide No. 84

BACKGROUND

Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research.

METHODS

This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape.

RESULTS

The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education.

CONCLUSION

The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial

OBJECTIVE

To explore optimal feedback methodologies to enhance trainee skill acquisition in simulated surgical bimanual skills learning during brain tumor resections.

HYPOTHESES

(1) Providing feedback results in better learning outcomes in teaching surgical technical skill when compared to practice alone with no tailored performance feedback. (2) Providing more visual and visuospatial feedback results in better learning outcomes when compared to providing numerical feedback.

DESIGN

A prospective 4-parallel-arm randomized controlled trial.

SETTING

Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Canada.

PARTICIPANTS

Medical students (n = 120) from 4 Quebec medical schools.

RESULTS

Participants completed a virtually simulated tumor resection task 5 times while receiving 1 of 4 feedback based on their group allocation: (1) practice-alone without feedback, (2) numerical feedback, (3) visual feedback, and (4) visuospatial feedback. Outcome measures were participants' scores on 14-performance metrics and the number of expert benchmarks achieved during each task. There were no significant differences in the first task which determined baseline performance. A statistically significant interaction between feedback allocation and task repetition was found on the number of benchmarks achieved, F (10.558, 408.257)=3.220, p < 0.001. Participants in all feedback groups significantly improved their performance compared to baseline. The visual feedback group achieved significantly higher number of benchmarks than the practice-alone group by the third repetition of the task, p = 0.005, 95%CI [0.42 3.25]. Visual feedback and visuospatial feedback improved performance significantly by the second repetition of the task, p = 0.016, 95%CI [0.19 2.71] and p = 0.003, 95%CI [0.4 2.57], respectively.

CONCLUSION

Simulations with autonomous visual computer assistance may be effective pedagogical tools in teaching bimanual operative skills via visual and visuospatial feedback information delivery.

Advances in Pediatric Surgery Simulation-Based Training

Pediatric surgery is the diagnostic, operative, and postoperative surgical care of children with congenital and acquired anomalies and diseases. The early history of the specialty followed the classic "see one, do one, teach one" philosophy of training but has since evolved to modern methods including simulation-based training (SBT). Current trainees in pediatric surgery face numerous challenges, such as the decreasing incidence of congenital disease and reduced work hours. SBT consists of several modalities that together assist in the acquisition of technical skills and improve performance in the operating room. SBT has evolved to incorporate simulator models and video gaming technology, in parallel with the development of simulation in other surgical and non-surgical pediatric fields. SBT has advanced to a level of sophistication that means that it can improve the skills of not only pediatric surgery trainees but also practicing attending surgeons. In this review, we will discuss the history of pediatric surgery, simulation in pediatric surgery training, and the potential direction of pediatric surgical simulation training in the future.

Performance in a Simulated Virtual Reality Anterior Cervical Discectomy and Fusion Task: Disc Residual, Rate of Removal, and Efficiency Analyses

BACKGROUND AND OBJECTIVES

Anterior cervical discectomy and fusion (ACDF) is among the most common spine procedures. The Sim-Ortho virtual reality simulator platform contains a validated ACDF simulated task for performance assessment. This study aims to develop a methodology to extract three-dimensional data and reconstruct and quantitate specific simulated disc tissues to generate novel metrics to analyze performance metrics of skilled and less skilled participants.

METHODS

We used open-source platforms to develop a methodology to extract three-dimensional information from ACDF simulation data. Metrics generated included, efficiency index, disc volumes removed from defined regions, and rate of tissue removal from superficial, central, and deep disc regions. A pilot study was performed to assess the utility of this methodology to assess expertise during the ACDF simulated procedure.

RESULTS

The system outlined, extracts data allowing the development of a methodology which accurately reconstructs and quantitates 3-dimensional disc volumes. In the pilot study, data sets from 27 participants, divided into postresident, resident, and medical student groups, allowed assessment of multiple novel metrics, including efficiency index (surgical time spent in actively removing disc), where the postresident group spent 61.8% of their time compared with 53% and 30.2% for the resident and medical student groups, respectively ( P = .01). During the annulotomy component, the postresident group removed 47.4% more disc than the resident groups and 102% more than the medical student groups ( P = .03).

CONCLUSION

The methodology developed in this study generates novel surgical procedural metrics from 3-dimensional data generated by virtual reality simulators and can be used to assess surgical performance.

Neurosurgical skills analysis by machine learning models: systematic review

Machine learning (ML) models are being actively used in modern medicine, including neurosurgery. This study aimed to summarize the current applications of ML in the analysis and assessment of neurosurgical skills. We conducted this systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched the PubMed and Google Scholar databases for eligible studies published until November 15, 2022, and used the Medical Education Research Study Quality Instrument (MERSQI) to assess the quality of the included articles. Of the 261 studies identified, we included 17 in the final analysis. Studies were most commonly related to oncological, spinal, and vascular neurosurgery using microsurgical and endoscopic techniques. Machine learning-evaluated tasks included subpial brain tumor resection, anterior cervical discectomy and fusion, hemostasis of the lacerated internal carotid artery, brain vessel dissection and suturing, glove microsuturing, lumbar hemilaminectomy, and bone drilling. The data sources included files extracted from VR simulators and microscopic and endoscopic videos. The ML application was aimed at classifying participants into several expertise levels, analysis of differences between experts and novices, surgical instrument recognition, division of operation into phases, and prediction of blood loss. In two articles, ML models were compared with those of human experts. The machines outperformed humans in all tasks. The most popular algorithms used to classify surgeons by skill level were the support vector machine and k-nearest neighbors, and their accuracy exceeded 90%. The "you only look once" detector and RetinaNet usually solved the problem of detecting surgical instruments - their accuracy was approximately 70%. The experts differed by more confident contact with tissues, higher bimanuality, smaller distance between the instrument tips, and relaxed and focused state of the mind. The average MERSQI score was 13.9 (from 18). There is growing interest in the use of ML in neurosurgical training. Most studies have focused on the evaluation of microsurgical skills in oncological neurosurgery and on the use of virtual simulators; however, other subspecialties, skills, and simulators are being investigated. Machine learning models effectively solve different neurosurgical tasks related to skill classification, object detection, and outcome prediction. Properly trained ML models outperform human efficacy. Further research on ML application in neurosurgery is needed.

Extended Reality in Neurosurgical Education: A Systematic Review

Surgical simulation practices have witnessed a rapid expansion as an invaluable approach to resident training in recent years. One emerging way of implementing simulation is the adoption of extended reality (XR) technologies, which enable trainees to hone their skills by allowing interaction with virtual 3D objects placed in either real-world imagery or virtual environments. The goal of the present systematic review is to survey and broach the topic of XR in neurosurgery, with a focus on education. Five databases were investigated, leading to the inclusion of 31 studies after a thorough reviewing process. Focusing on user performance (UP) and user experience (UX), the body of evidence provided by these 31 studies showed that this technology has, in fact, the potential of enhancing neurosurgical education through the use of a wide array of both objective and subjective metrics. Recent research on the topic has so far produced solid results, particularly showing improvements in young residents, compared to other groups and over time. In conclusion, this review not only aids to a better understanding of the use of XR in neurosurgical education, but also highlights the areas where further research is entailed while also providing valuable insight into future applications.