Face, content, and construct validity of a novel VR/AR surgical simulator of a minimally invasive spine operation

Mixed-reality surgical simulators are seen more objective than conventional training. The simulators' utility in training must be established through validation studies. Establish face-, content-, and construct-validity of a novel mixed-reality surgical simulator developed by McGill University, CAE-Healthcare, and DePuy Synthes. This study, approved by a Research Ethics Board, examined a simulated L4-L5 oblique lateral lumbar interbody fusion (OLLIF) scenario. A 5-point Likert scale questionnaire was used. Chi-square test verified validity consensus. Construct validity investigated 276 surgical performance metrics across three groups, using ANOVA, Welch-ANOVA, or Kruskal-Wallis tests. A post-hoc Dunn's test with a Bonferroni correction was used for further analysis on significant metrics. Musculoskeletal Biomechanics Research Lab, McGill University, Montreal, Canada. DePuy Synthes, Johnson & Johnson Family of Companies, research lab. Thirty-four participants were recruited: spine surgeons, fellows, neurosurgical, and orthopedic residents. Only seven surgeons out of the 34 were recruited in a side-by-side cadaver trial, where participants completed an OLLIF surgery first on a cadaver and then immediately on the simulator. Participants were separated a priori into three groups: post-, senior-, and junior-residents. Post-residents rated validity, median > 3, for 13/20 face-validity and 9/25 content-validity statements. Seven face-validity and 12 content-validity statements were rated neutral. Chi-square test indicated agreeability between group responses. Construct validity found eight metrics with significant differences (p < 0.05) between the three groups. Validity was established. Most face-validity statements were positively rated, with few neutrally rated pertaining to the simulation's graphics. Although fewer content-validity statements were validated, most were rated neutral (only four were negatively rated). The findings underscored the importance of using realistic physics-based forces in surgical simulations. Construct validity demonstrated the simulator's capacity to differentiate surgical expertise.

The Development And Applications Of Augmented And Virtual Reality Technology In Spine Surgery Training: A Systematic Review

BACKGROUND

The COVID-19 pandemic has accelerated the growing global interest in the role of augmented and virtual reality in surgical training. While this technology grows at a rapid rate, its efficacy remains unclear. To that end, we offer a systematic review of the literature summarizing the role of virtual and augmented reality on spine surgery training.

METHODS

A systematic review of the literature was conducted on May 13th, 2022. PubMed, Web of Science, Medline, and Embase were reviewed for relevant studies. Studies from both orthopedic and neurosurgical spine programs were considered. There were no restrictions placed on the type of study, virtual/augmented reality modality, nor type of procedure. Qualitative data analysis was performed, and all studies were assigned a Medical Education Research Study Quality Instrument (MERSQI) score.

RESULTS

The initial review identified 6752 studies, of which 16 were deemed relevant and included in the final review, examining a total of nine unique augmented/virtual reality systems. These studies had a moderate methodological quality with a MERSQI score of 12.1 + 1.8; most studies were conducted at single-center institutions, and unclear response rates. Statistical pooling of the data was limited by the heterogeneity of the study designs.

CONCLUSION

This review examined the applications of augmented and virtual reality systems for training residents in various spine procedures. As this technology continues to advance, higher-quality, multi-center, and long-term studies are required to further the adaptation of VR/AR technologies in spine surgery training programs.

Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial

OBJECTIVE

To explore optimal feedback methodologies to enhance trainee skill acquisition in simulated surgical bimanual skills learning during brain tumor resections.

HYPOTHESES

(1) Providing feedback results in better learning outcomes in teaching surgical technical skill when compared to practice alone with no tailored performance feedback. (2) Providing more visual and visuospatial feedback results in better learning outcomes when compared to providing numerical feedback.

DESIGN

A prospective 4-parallel-arm randomized controlled trial.

SETTING

Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Canada.

PARTICIPANTS

Medical students (n = 120) from 4 Quebec medical schools.

RESULTS

Participants completed a virtually simulated tumor resection task 5 times while receiving 1 of 4 feedback based on their group allocation: (1) practice-alone without feedback, (2) numerical feedback, (3) visual feedback, and (4) visuospatial feedback. Outcome measures were participants' scores on 14-performance metrics and the number of expert benchmarks achieved during each task. There were no significant differences in the first task which determined baseline performance. A statistically significant interaction between feedback allocation and task repetition was found on the number of benchmarks achieved, F (10.558, 408.257)=3.220, p < 0.001. Participants in all feedback groups significantly improved their performance compared to baseline. The visual feedback group achieved significantly higher number of benchmarks than the practice-alone group by the third repetition of the task, p = 0.005, 95%CI [0.42 3.25]. Visual feedback and visuospatial feedback improved performance significantly by the second repetition of the task, p = 0.016, 95%CI [0.19 2.71] and p = 0.003, 95%CI [0.4 2.57], respectively.

CONCLUSION

Simulations with autonomous visual computer assistance may be effective pedagogical tools in teaching bimanual operative skills via visual and visuospatial feedback information delivery.

Current Trends in Artificial Intelligence-Assisted Spine Surgery: A Systematic Review

This systematic review summarizes existing evidence and outlines the benefits of artificial intelligence-assisted spine surgery. The popularity of artificial intelligence has grown significantly, demonstrating its benefits in computer-assisted surgery and advancements in spinal treatment. This study adhered to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), a set of reporting guidelines specifically designed for systematic reviews and meta-analyses. The search strategy used Medical Subject Headings (MeSH) terms, including "MeSH (Artificial intelligence)," "Spine" AND "Spinal" filters, in the last 10 years, and English- from January 1, 2013, to October 31, 2023. In total, 442 articles fulfilled the first screening criteria. A detailed analysis of those articles identified 220 that matched the criteria, of which 11 were considered appropriate for this analysis after applying the complete inclusion and exclusion criteria. In total, 11 studies met the eligibility criteria. Analysis of these studies revealed the types of artificial intelligence-assisted spine surgery. No evidence suggests the superiority of assisted spine surgery with or without artificial intelligence in terms of outcomes. In terms of feasibility, accuracy, safety, and facilitating lower patient radiation exposure compared with standard fluoroscopic guidance, artificial intelligence-assisted spine surgery produced satisfactory and superior outcomes. The incorporation of artificial intelligence with augmented and virtual reality appears promising, with the potential to enhance surgeon proficiency and overall surgical safety.

Continuous Instrument Tracking in a Cerebral Corticectomy Ex Vivo Calf Brain Simulation Model: Face and Content Validation

BACKGROUND AND OBJECTIVES

Subpial corticectomy involving complete lesion resection while preserving pial membranes and avoiding injury to adjacent normal tissues is an essential bimanual task necessary for neurosurgical trainees to master. We sought to develop an ex vivo calf brain corticectomy simulation model with continuous assessment of surgical instrument movement during the simulation. A case series study of skilled participants was performed to assess face and content validity to gain insights into the utility of this training platform, along with determining if skilled and less skilled participants had statistical differences in validity assessment.

METHODS

An ex vivo calf brain simulation model was developed in which trainees performed a subpial corticectomy of three defined areas. A case series study assessed face and content validity of the model using 7-point Likert scale questionnaires.

RESULTS

Twelve skilled and 11 less skilled participants were included in this investigation. Overall median scores of 6.0 (range 4.0-6.0) for face validity and 6.0 (range 3.5-7.0) for content validity were determined on the 7-point Likert scale, with no statistical differences between skilled and less skilled groups identified.

CONCLUSION

A novel ex vivo calf brain simulator was developed to replicate the subpial resection procedure and demonstrated face and content validity.

Performance in a Simulated Virtual Reality Anterior Cervical Discectomy and Fusion Task: Disc Residual, Rate of Removal, and Efficiency Analyses

BACKGROUND AND OBJECTIVES

Anterior cervical discectomy and fusion (ACDF) is among the most common spine procedures. The Sim-Ortho virtual reality simulator platform contains a validated ACDF simulated task for performance assessment. This study aims to develop a methodology to extract three-dimensional data and reconstruct and quantitate specific simulated disc tissues to generate novel metrics to analyze performance metrics of skilled and less skilled participants.

METHODS

We used open-source platforms to develop a methodology to extract three-dimensional information from ACDF simulation data. Metrics generated included, efficiency index, disc volumes removed from defined regions, and rate of tissue removal from superficial, central, and deep disc regions. A pilot study was performed to assess the utility of this methodology to assess expertise during the ACDF simulated procedure.

RESULTS

The system outlined, extracts data allowing the development of a methodology which accurately reconstructs and quantitates 3-dimensional disc volumes. In the pilot study, data sets from 27 participants, divided into postresident, resident, and medical student groups, allowed assessment of multiple novel metrics, including efficiency index (surgical time spent in actively removing disc), where the postresident group spent 61.8% of their time compared with 53% and 30.2% for the resident and medical student groups, respectively ( P = .01). During the annulotomy component, the postresident group removed 47.4% more disc than the resident groups and 102% more than the medical student groups ( P = .03).

CONCLUSION

The methodology developed in this study generates novel surgical procedural metrics from 3-dimensional data generated by virtual reality simulators and can be used to assess surgical performance.

AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training

Importance

To better elucidate the role of artificial intelligence (AI) in surgical skills training requires investigations in the potential existence of a hidden curriculum.

Objective

To assess the pedagogical value of AI-selected technical competencies and their extended effects in surgical simulation training.

Design, Setting, and Participants

This cohort study was a follow-up of a randomized clinical trial conducted at the Neurosurgical Simulation and Artificial Intelligence Learning Centre at the Montreal Neurological Institute, McGill University, Montreal, Canada. Surgical performance metrics of medical students exposed to an AI-enhanced training curriculum were compared with a control group of participants who received no feedback and with expert benchmarks. Cross-sectional data were collected from January to April 2021 from medical students and from March 2015 to May 2016 from experts. This follow-up secondary analysis was conducted from June to September 2022. Participants included medical students (undergraduate year 0-2) in the intervention cohorts and neurosurgeons to establish expertise benchmarks.

Exposure

Performance assessment and personalized feedback by an intelligent tutor on 4 AI-selected learning objectives during simulation training.

Main Outcomes and Measures

Outcomes of interest were unintended performance outcomes, measured by significant within-participant difference from baseline in 270 performance metrics in the intervention cohort that was not observed in the control cohort.

Results

A total of 46 medical students (median [range] age, 22 [18-27] years; 27 [59%] women) and 14 surgeons (median [range] age, 45 [35-59] years; 14 [100%] men) were included in this study, and no participant was lost to follow-up. Feedback on 4 AI-selected technical competencies was associated with additional performance change in 32 metrics over the entire procedure and 20 metrics during tumor removal that was not observed in the control group. Participants exposed to the AI-enhanced curriculum demonstrated significant improvement in safety metrics, such as reducing the rate of healthy tissue removal (mean difference, -7.05 × 10-5 [95% CI, -1.09 × 10-4 to -3.14 × 10-5] mm3 per 20 ms; P < .001) and maintaining a focused bimanual control of the operative field (mean difference in maximum instrument divergence, -4.99 [95% CI, -8.48 to -1.49] mm, P = .006) compared with the control group. However, negative unintended effects were also observed. These included a significantly lower velocity and acceleration in the dominant hand (velocity: mean difference, -0.13 [95% CI, -0.17 to -0.09] mm per 20 ms; P < .001; acceleration: mean difference, -2.25 × 10-2 [95% CI, -3.20 × 10-2 to -1.31 × 10-2] mm per 20 ms2; P < .001) and a significant reduction in the rate of tumor removal (mean difference, -4.85 × 10-5 [95% CI, -7.22 × 10-5 to -2.48 × 10-5] mm3 per 20 ms; P < .001) compared with control. These unintended outcomes diverged students' movement and efficiency performance metrics away from the expertise benchmarks.

Conclusions and Relevance

In this cohort study of medical students, an AI-enhanced curriculum for bimanual surgical skills resulted in unintended changes that improved performance in safety but negatively affected some efficiency metrics. Incorporating AI in course design requires ongoing assessment to maintain transparency and foster evidence-based learning objectives.

Neurosurgical skills analysis by machine learning models: systematic review

Machine learning (ML) models are being actively used in modern medicine, including neurosurgery. This study aimed to summarize the current applications of ML in the analysis and assessment of neurosurgical skills. We conducted this systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched the PubMed and Google Scholar databases for eligible studies published until November 15, 2022, and used the Medical Education Research Study Quality Instrument (MERSQI) to assess the quality of the included articles. Of the 261 studies identified, we included 17 in the final analysis. Studies were most commonly related to oncological, spinal, and vascular neurosurgery using microsurgical and endoscopic techniques. Machine learning-evaluated tasks included subpial brain tumor resection, anterior cervical discectomy and fusion, hemostasis of the lacerated internal carotid artery, brain vessel dissection and suturing, glove microsuturing, lumbar hemilaminectomy, and bone drilling. The data sources included files extracted from VR simulators and microscopic and endoscopic videos. The ML application was aimed at classifying participants into several expertise levels, analysis of differences between experts and novices, surgical instrument recognition, division of operation into phases, and prediction of blood loss. In two articles, ML models were compared with those of human experts. The machines outperformed humans in all tasks. The most popular algorithms used to classify surgeons by skill level were the support vector machine and k-nearest neighbors, and their accuracy exceeded 90%. The "you only look once" detector and RetinaNet usually solved the problem of detecting surgical instruments - their accuracy was approximately 70%. The experts differed by more confident contact with tissues, higher bimanuality, smaller distance between the instrument tips, and relaxed and focused state of the mind. The average MERSQI score was 13.9 (from 18). There is growing interest in the use of ML in neurosurgical training. Most studies have focused on the evaluation of microsurgical skills in oncological neurosurgery and on the use of virtual simulators; however, other subspecialties, skills, and simulators are being investigated. Machine learning models effectively solve different neurosurgical tasks related to skill classification, object detection, and outcome prediction. Properly trained ML models outperform human efficacy. Further research on ML application in neurosurgery is needed.

Current status of virtual reality simulation education for orthopedic residents: the need for a change in focus

Introduction

Advances in technology are changing surgical education. Simulation provides an important adjunct to operative experience. This pedagogy has arguably become more important in light of the COVID-19 pandemic, with resultant reduction in operative exposure for trainees. Virtual reality (VR) simulators may provide significant contribution to experiential learning; however, much of the investigative focus to date has, correctly, been on establishing validity evidence for these constructs. The aim of this work was to perform a scoping review to assess the current status of VR simulation education to determine curricular development efforts for orthopedic residents.

Methods

With a trained medical librarian, searches of PubMed, EMBASE, and Web of Science were conducted for all articles in the last 10 years (September 2011-September 2021). Controlled vocabulary Medical Subject Headings (MeSH) terms and natural language developed with subject matter experts describing virtual reality or VR simulation and orthopedic training were used. Two trained reviewers evaluated all abstracts for inclusion. Exclusion criteria were all articles that did not assess VR simulation education involving orthopedic residents. Data were extracted from the included full-text articles including: study design, type of participants, type of VR simulation, simulated orthopedic skill, type of educational event, learner assessment including Kirkpatrick's level, assessment of quality using the Medical Education Research Study Quality Instrument (MERSQI), and level of effectiveness (LoE).

Results

Initial search identified 1,394 articles, of which 61 were included in the final qualitative synthesis. The majority (54%) were published in 2019- 2021, 49% in Europe. The commonest VR simulator was ArthroS (23%) and the commonest simulated skill was knee arthroscopy (33%). The majority of studies (70%) focused on simulator validation. Twenty-three studies described an educational module or curriculum, and of the 21 (34%) educational modules, 43% were one-off events. Most modules (18/21, 86%) assessed learners at Kirkpatrick level 2. With regard to methodological quality, 44% of studies had MERSQI 11.5-15 and 89% of studies had LoE of 2. Two studies had LoE of 3.

Conclusion

Current literature pertaining to VR training for orthopedic residents is focused on establishing validity and rarely forms part of a curriculum. Where the focus is education, the majority are discrete educational modules and do not teach a comprehensive amalgam of orthopedic skills. This suggests focus is needed to embed VR simulation training within formal curricula efforts guided by the work of Kern, and assess the efficacy of these against patient outcomes.

Utilizing artificial intelligence and electroencephalography to assess expertise on a simulated neurosurgical task

Virtual reality surgical simulators have facilitated surgical education by providing a safe training environment. Electroencephalography (EEG) has been employed to assess neuroelectric activity during surgical performance. Machine learning (ML) has been applied to analyze EEG data split into frequency bands. Although EEG is widely used in fields requiring expert performance, it has yet been used to classify surgical expertise. Thus, the goals of this study were to (a) develop an ML model to accurately differentiate skilled and less-skilled performance using EEG data recorded during a simulated surgery, (b) explore the relative importance of each EEG bandwidth to expertise, and (c) analyze differences in EEG band powers between skilled and less-skilled individuals. We hypothesized that EEG recordings during a virtual reality surgery task would accurately predict the expertise level of the participant. Twenty-one participants performed three simulated brain tumor resection procedures on the NeuroVR™ platform (CAE Healthcare, Montreal, Canada) while EEG data was recorded. Participants were divided into 2 groups. The skilled group was composed of five neurosurgeons and five senior neurosurgical residents (PGY4-6), and the less-skilled group was composed of six junior residents (PGY1-3) and five medical students. A total of 13 metrics from EEG frequency bands and ratios (e.g., alpha, theta/beta ratio) were generated. Seven ML model types were trained using EEG activity to differentiate between skilled and less-skilled groups. The artificial neural network achieved the highest testing accuracy of 100% (AUROC = 1.0). Model interpretation via Shapley analysis identified low alpha (8-10 Hz) as the most important metric for classifying expertise. Skilled surgeons displayed higher (p = 0.044) low-alpha than the less-skilled group. Furthermore, skilled surgeons displayed significantly lower TBR (p = 0.048) and significantly higher beta (13-30 Hz, p = 0.049), beta 1 (15-18 Hz, p = 0.014), and beta 2 (19-22 Hz, p = 0.015), thus establishing these metrics as important markers of expertise. ACGME CORE COMPETENCIES: Practice-Based Learning and Improvement.

Advancing Surgical Education: The Use of Artificial Intelligence in Surgical Training

The technology of artificial intelligence (AI) has made significant in-roads into the field of medicine over the last decade. With surgery being a discipline where repetition is the key to mastery, the scope of AI presents enormous potential for resident education through the analysis of technique and delivery of structured feedback for performance improvement. In an era marred by a raging pandemic that has decreased exposure and opportunity, AI offers an attractive solution towards improving operating room efficiency, safe patient care in the hands of supervised residents and can ultimately culminate in reduced health care costs. Through this article, we elucidate the current adoption of the artificial intelligence technology and its prospects for advancing surgical education.

Current Applications of Machine Learning in Spine: From Clinical View

STUDY DESIGN

Narrative review.

OBJECTIVES

This review aims to present current applications of machine learning (ML) in spine domain to clinicians.

METHODS

We conducted a comprehensive PubMed search of peer-reviewed articles that were published between 2006 and 2020 using terms (spine, spinal, lumbar, cervical, thoracic, machine learning) to examine ML in spine. Then exclude research of other domain, case report, review or meta-analysis, and which without available abstract or full text.

RESULTS

Total 1738 articles were retrieved from database, and 292 studies were finally included. Key findings of current applications were compiled and summarized in this review. Main clinical applications of those techniques including image processing, diagnosis, decision supporting, operative assistance, rehabilitation, surgery outcomes, complications, hospitalization and cost.

CONCLUSIONS

ML had achieved excellent performance and hold immense potential in spine. ML could help clinical staff to improve medical level, enhance work efficiency, and reduce adverse events. However more randomized controlled trials and improvement of interpretability are essential to clinicians accepting models' assistance in real work.

Assessment of learning curves on a simulated neurosurgical task using metrics selected by artificial intelligence

OBJECTIVE

Understanding the variation of learning curves of experts and trainees for a given surgical procedure is important in implementing formative learning paradigms to accelerate mastery. The study objectives were to use artificial intelligence (AI)-derived metrics to determine the learning curves of participants in 4 groups with different expertise levels who performed a series of identical virtual reality (VR) subpial resection tasks and to identify learning curve differences among the 4 groups.

METHODS

A total of 50 individuals participated, 14 neurosurgeons, 4 neurosurgical fellows and 10 senior residents (seniors), 10 junior residents (juniors), and 12 medical students. All participants performed 5 repetitions of a subpial tumor resection on the NeuroVR (CAE Healthcare) platform, and 6 a priori-derived metrics selected using the K-nearest neighbors machine learning algorithm were used to assess participant learning curves. Group learning curves were plotted over the 5 trials for each metric. A mixed, repeated-measures ANOVA was performed between the first and fifth trial. For significant interactions (p < 0.05), post hoc Tukey's HSD analysis was conducted to determine the location of the significance.

RESULTS

Overall, 5 of the 6 metrics assessed had a significant interaction (p < 0.05). The 4 groups, neurosurgeons, seniors, juniors, and medical students, showed an improvement between the first and fifth trial on at least one of the 6 metrics evaluated.

CONCLUSIONS

Learning curves generated using AI-derived metrics provided novel insights into technical skill acquisition, based on expertise level, during repeated VR-simulated subpial tumor resections, which will allow educators to develop more focused formative educational paradigms for neurosurgical trainees.

Use of surgical video-based automated performance metrics to predict blood loss and success of simulated vascular injury control in neurosurgery: a pilot study

OBJECTIVE

Experts can assess surgeon skill using surgical video, but a limited number of expert surgeons are available. Automated performance metrics (APMs) are a promising alternative but have not been created from operative videos in neurosurgery to date. The authors aimed to evaluate whether video-based APMs can predict task success and blood loss during endonasal endoscopic surgery in a validated cadaveric simulator of vascular injury of the internal carotid artery.

METHODS

Videos of cadaveric simulation trials by 73 neurosurgeons and otorhinolaryngologists were analyzed and manually annotated with bounding boxes to identify the surgical instruments in the frame. APMs in five domains were defined-instrument usage, time-to-phase, instrument disappearance, instrument movement, and instrument interactions-on the basis of expert analysis and task-specific surgical progressions. Bounding-box data of instrument position were then used to generate APMs for each trial. Multivariate linear regression was used to test for the associations between APMs and blood loss and task success (hemorrhage control in less than 5 minutes). The APMs of 93 successful trials were compared with the APMs of 49 unsuccessful trials.

RESULTS

In total, 29,151 frames of surgical video were annotated. Successful simulation trials had superior APMs in each domain, including proportionately more time spent with the key instruments in view (p < 0.001) and less time without hemorrhage control (p = 0.002). APMs in all domains improved in subsequent trials after the participants received personalized expert instruction. Attending surgeons had superior instrument usage, time-to-phase, and instrument disappearance metrics compared with resident surgeons (p < 0.01). APMs predicted surgeon performance better than surgeon training level or prior experience. A regression model that included APMs predicted blood loss with an R2 value of 0.87 (p < 0.001).

CONCLUSIONS

Video-based APMs were superior predictors of simulation trial success and blood loss than surgeon characteristics such as case volume and attending status. Surgeon educators can use APMs to assess competency, quantify performance, and provide actionable, structured feedback in order to improve patient outcomes. Validation of APMs provides a benchmark for further development of fully automated video assessment pipelines that utilize machine learning and computer vision.

Artificial Neural Network Approach to Competency-Based Training Using a Virtual Reality Neurosurgical Simulation

BACKGROUND

The methodology of assessment and training of surgical skills is evolving to deal with the emergence of competency-based training. Artificial neural networks (ANNs), a branch of artificial intelligence, can use newly generated metrics not only for assessment performance but also to quantitate individual metric importance and provide new insights into surgical expertise.

OBJECTIVE

To outline the educational utility of using an ANN in the assessment and quantitation of surgical expertise. A virtual reality vertebral osteophyte removal during a simulated surgical spine procedure is used as a model to outline this methodology.

METHODS

Twenty-one participants performed a simulated anterior cervical diskectomy and fusion on the Sim-Ortho virtual reality simulator. Participants were divided into 3 groups, including 9 postresidents, 5 senior residents, and 7 junior residents. Data were retrieved from the osteophyte removal component of the scenario, which involved using a simulated burr. The data were manipulated to initially generate 83 performance metrics spanning 3 categories (safety, efficiency, and motion) of which only the most relevant metrics were used to train and test the ANN.

RESULTS

The ANN model was trained on 6 safety metrics to a testing accuracy of 83.3%. The contributions of these performance metrics to expertise were revealed through connection weight products and outlined 2 identifiable learning patterns of technical skills.

CONCLUSION

This study outlines the potential utility of ANNs which allows a deeper understanding of the composites of surgical expertise and may contribute to the paradigm shift toward competency-based surgical training.

Nondominant Hand Skills Spatial and Psychomotor Analysis During a Complex Virtual Reality Neurosurgical Task-A Case Series Study

BACKGROUND

Virtual reality surgical simulators provide detailed psychomotor performance data, allowing qualitative and quantitative assessment of hand function. The nondominant hand plays an essential role in neurosurgery in exposing the operative area, assisting the dominant hand to optimize task execution, and hemostasis. Outlining expert-level nondominant hand skills may be critical to understand surgical expertise and aid learner training.

OBJECTIVE

To (1) provide validity for the simulated bimanual subpial tumor resection task and (2) to use this simulation in qualitative and quantitative evaluation of nondominant hand skills for bipolar forceps utilization.

METHODS

In this case series study, 45 right-handed participants performed a simulated subpial tumor resection using simulated bipolar forceps in the nondominant hand for assisting the surgery and hemostasis. A 10-item questionnaire was used to assess task validity. The nondominant hand skills across 4 expertise levels (neurosurgeons, senior trainees, junior trainees, and medical students) were analyzed by 2 visual models and performance metrics.

RESULTS

Neurosurgeon median (range) overall satisfaction with the simulated scenario was 4.0/5.0 (2.0-5.0). The visual models demonstrated a decrease in high force application areas on pial surface with increased expertise level. Bipolar-pia mater interactions were more focused around the tumoral region for neurosurgeons and senior trainees. These groups spent more time using the bipolar while interacting with pia. All groups spent significantly higher time in the left upper pial quadrant than other quadrants.

CONCLUSION

This work introduces new approaches for the evaluation of nondominant hand skills which may help surgical trainees by providing both qualitative and quantitative feedback.

Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation

In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.

Machine learning for technical skill assessment in surgery: a systematic review

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.

PROSPERO: CRD42020226071

Artificial Intelligence and Surgical Education: A Systematic Scoping Review of Interventions

OBJECTIVE

To synthesize peer-reviewed evidence related to the use of artificial intelligence (AI) in surgical education DESIGN: We conducted and reported a scoping review according to the standards outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis with extension for Scoping Reviews guideline and the fourth edition of the Joanna Briggs Institute Reviewer's Manual. We systematically searched eight interdisciplinary databases including MEDLINE-Ovid, ERIC, EMBASE, CINAHL, Web of Science: Core Collection, Compendex, Scopus, and IEEE Xplore. Databases were searched from inception until the date of search on April 13, 2021.

SETTING/PARTICIPANTS

We only examined original, peer-reviewed interventional studies that self-described as AI interventions, focused on medical education, and were relevant to surgical trainees (defined as medical or dental students, postgraduate residents, or surgical fellows) within the title and abstract (see Table 2). Animal, cadaveric, and in vivo studies were not eligible for inclusion.

RESULTS

After systematically searching eight databases and 4255 citations, our scoping review identified 49 studies relevant to artificial intelligence in surgical education. We found diverse interventions related to the evaluation of surgical competency, personalization of surgical education, and improvement of surgical education materials across surgical specialties. Many studies used existing surgical education materials, such as the Objective Structured Assessment of Technical Skills framework or the JHU-ISI Gesture and Skill Assessment Working Set database. Though most studies did not provide outcomes related to the implementation in medical schools (such as cost-effective analyses or trainee feedback), there are numerous promising interventions. In particular, many studies noted high accuracy in the objective characterization of surgical skill sets. These interventions could be further used to identify at-risk surgical trainees or evaluate teaching methods.

CONCLUSIONS

There are promising applications for AI in surgical education, particularly for the assessment of surgical competencies, though further evidence is needed regarding implementation and applicability.

Virtual Reality in the Neurosciences: Current Practice and Future Directions

Virtual reality has made numerous advancements in recent years and is used with increasing frequency for education, diversion, and distraction. Beginning several years ago as a device that produced an image with only a few pixels, virtual reality is now able to generate detailed, three-dimensional, and interactive images. Furthermore, these images can be used to provide quantitative data when acting as a simulator or a rehabilitation device. In this article, we aim to draw attention to these areas, as well as highlight the current settings in which virtual reality (VR) is being actively studied and implemented within the field of neurosurgery and the neurosciences. Additionally, we discuss the current limitations of the applications of virtual reality within various settings. This article includes areas in which virtual reality has been used in applications both inside and outside of the operating room, such as pain control, patient education and counseling, and rehabilitation. Virtual reality's utility in neurosurgery and the neurosciences is widely growing, and its use is quickly becoming an integral part of patient care, surgical training, operative planning, navigation, and rehabilitation.

Exploring the roles of artificial intelligence in surgical education: A scoping review

BACKGROUND

Technology-enhanced teaching and learning, including Artificial Intelligence (AI) applications, has started to evolve in surgical education. Hence, the purpose of this scoping review is to explore the current and future roles of AI in surgical education.

METHODS

Nine bibliographic databases were searched from January 2010 to January 2021. Full-text articles were included if they focused on AI in surgical education.

RESULTS

Out of 14,008 unique sources of evidence, 93 were included. Out of 93, 84 were conducted in the simulation setting, and 89 targeted technical skills. Fifty-six studies focused on skills assessment/classification, and 36 used multiple AI techniques. Also, increasing sample size, having balanced data, and using AI to provide feedback were major future directions mentioned by authors.

CONCLUSIONS

AI can help optimize the education of trainees and our results can help educators and researchers identify areas that need further investigation.

Utilizing a multilayer perceptron artificial neural network to assess a virtual reality surgical procedure

BACKGROUND

Virtual reality surgical simulators are a safe and efficient technology for the assessment and training of surgical skills. Simulators allow trainees to improve specific surgical techniques in risk-free environments. Recently, machine learning has been coupled to simulators to classify performance. However, most studies fail to extract meaningful observations behind the classifications and the impact of specific surgical metrics on the performance. One benefit from integrating machine learning algorithms, such as Artificial Neural Networks, to simulators is the ability to extract novel insights into the composites of the surgical performance that differentiate levels of expertise.

OBJECTIVE

This study aims to demonstrate the benefits of artificial neural network algorithms in assessing and analyzing virtual surgical performances. This study applies the algorithm on a virtual reality simulated annulus incision task during an anterior cervical discectomy and fusion scenario.

DESIGN

An artificial neural network algorithm was developed and integrated. Participants performed the simulated surgical procedure on the Sim-Ortho simulator. Data extracted from the annulus incision task were extracted to generate 157 surgical performance metrics that spanned three categories (motion, safety, and efficiency).

SETTING

Musculoskeletal Biomechanics Research Lab; Neurosurgical Simulation and Artificial Intelligence Learning Center, McGill University, Montreal, Canada.

PARTICIPANTS

Twenty-three participants were recruited and divided into 3 groups: 11 post-residents, 5 senior and 7 junior residents.

RESULTS

An artificial neural network model was trained on nine selected surgical metrics, spanning all three categories and achieved 80% testing accuracy.

CONCLUSIONS

This study outlines the benefits of integrating artificial neural networks to virtual reality surgical simulators in understanding composites of expertise performance.

Best Practices Using Ex Vivo Animal Brain Models in Neurosurgical Education to Assess Surgical Expertise

BACKGROUND

Ex vivo animal brain simulation models are being increasingly used for neurosurgical training because these models can replicate human brain conditions. The goal of the present report is to provide the neurosurgical community interested in using ex vivo animal brain simulation models with guidelines for comprehensively and rigorously conducting, documenting, and assessing this type of research.

METHODS

In consultation with an interdisciplinary group of physicians and researchers involved in ex vivo models and a review of the literature on the best practices guidelines for simulation research, we developed the "ex vivo brain model to assess surgical expertise" (EVBMASE) checklist. The EVBMASE checklist provides a comprehensive quantitative framework for analyzing and reporting studies involving these models. We applied The EVBMASE checklist to the studies reported of ex vivo animal brain models to document how current ex vivo brain simulation models are used to train surgical expertise.

RESULTS

The EVBMASE checklist includes defined subsections and a total score of 20, which can help investigators improve studies and provide readers with techniques to better assess the quality and any deficiencies of the research. We classified 18 published ex vivo brain models into modified (group 1) and nonmodified (group 2) models. The mean total EVBMASE score was 11 (55%) for group 1 and 4.8 (24.2%) for group 2, a statistically significant difference (P = 0.006) mainly attributed to differences in the simulation study design section (P = 0.003).

CONCLUSIONS

The present findings should help contribute to more rigorous application, documentation, and assessment of ex vivo brain simulation research.

A systematic review of virtual reality for the assessment of technical skills in neurosurgery

OBJECTIVE

Virtual reality (VR) and augmented reality (AR) systems are increasingly available to neurosurgeons. These systems may provide opportunities for technical rehearsal and assessments of surgeon performance. The assessment of neurosurgeon skill in VR and AR environments and the validity of VR and AR feedback has not been systematically reviewed.

METHODS

A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was conducted through MEDLINE and PubMed. Studies published in English between January 1990 and February 2021 describing the use of VR or AR to quantify surgical technical performance of neurosurgeons without the use of human raters were included. The types and categories of automated performance metrics (APMs) from each of these studies were recorded.

RESULTS

Thirty-three VR studies were included in the review; no AR studies met inclusion criteria. VR APMs were categorized as either distance to target, force, kinematics, time, blood loss, or volume of resection. Distance and time were the most well-studied APM domains, although all domains were effective at differentiating surgeon experience levels. Distance was successfully used to track improvements with practice. Examining volume of resection demonstrated that attending surgeons removed less simulated tumor but preserved more normal tissue than trainees. More recently, APMs have been used in machine learning algorithms to predict level of training with a high degree of accuracy. Key limitations to enhanced-reality systems include limited AR usage for automated surgical assessment and lack of external and longitudinal validation of VR systems.

CONCLUSIONS

VR has been used to assess surgeon performance across a wide spectrum of domains. The VR environment can be used to quantify surgeon performance, assess surgeon proficiency, and track training progression. AR systems have not yet been used to provide metrics for surgeon performance assessment despite potential for intraoperative integration. VR-based APMs may be especially useful for metrics that are difficult to assess intraoperatively, including blood loss and extent of resection.

Quantitation of Tissue Resection Using a Brain Tumor Model and 7-T Magnetic Resonance Imaging Technology

BACKGROUND

Animal brain tumor models can be useful educational tools for the training of neurosurgical residents in risk-free environments. Magnetic resonance imaging (MRI) technologies have not used these models to quantitate tumor, normal gray and white matter, and total tissue removal during complex neurosurgical procedures. This pilot study was carried out as a proof of concept to show the feasibility of using brain tumor models combined with 7-T MRI technology to quantitatively assess tissue removal during subpial tumor resection.

METHODS

Seven ex vivo calf brain hemispheres were used to develop the 7-T MRI segmentation methodology. Three brains were used to quantitate brain tissue removal using 7-T MRI segmentation methodology. Alginate artificial brain tumor was created in 4 calf brains to assess the ability of 7-T MRI segmentation methodology to quantitate tumor and gray and white matter along with total tissue volumes removal during a subpial tumor resection procedure.

RESULTS

Quantitative studies showed a correlation between removed brain tissue weights and volumes determined from segmented 7-T MRIs. Analysis of baseline and postresection alginate brain tumor segmented 7-T MRIs allowed quantification of tumor and gray and white matter along with total tissue volumes removed and detection of alterations in surrounding gray and white matter.

CONCLUSIONS

This pilot study showed that the use of animal tumor models in combination with 7-T MRI technology provides an opportunity to increase the granularity of data obtained from operative procedures and to improve the assessment and training of learners.

Application of Artificial Intelligence in Medicine: An Overview

Artificial intelligence (AI) is a new technical discipline that uses computer technology to research and develop the theory, method, technique, and application system for the simulation, extension, and expansion of human intelligence. With the assistance of new AI technology, the traditional medical environment has changed a lot. For example, a patient's diagnosis based on radiological, pathological, endoscopic, ultrasonographic, and biochemical examinations has been effectively promoted with a higher accuracy and a lower human workload. The medical treatments during the perioperative period, including the preoperative preparation, surgical period, and postoperative recovery period, have been significantly enhanced with better surgical effects. In addition, AI technology has also played a crucial role in medical drug production, medical management, and medical education, taking them into a new direction. The purpose of this review is to introduce the application of AI in medicine and to provide an outlook of future trends.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine

Simulation-based training is increasingly being used for assessment and training of psychomotor skills involved in medicine. The application of artificial intelligence and machine learning technologies has provided new methodologies to utilize large amounts of data for educational purposes. A significant criticism of the use of artificial intelligence in education has been a lack of transparency in the algorithms' decision-making processes. This study aims to 1) introduce a new framework using explainable artificial intelligence for simulation-based training in surgery, and 2) validate the framework by creating the Virtual Operative Assistant, an automated educational feedback platform. Twenty-eight skilled participants (14 staff neurosurgeons, 4 fellows, 10 PGY 4-6 residents) and 22 novice participants (10 PGY 1-3 residents, 12 medical students) took part in this study. Participants performed a virtual reality subpial brain tumor resection task on the NeuroVR simulator using a simulated ultrasonic aspirator and bipolar. Metrics of performance were developed, and leave-one-out cross validation was employed to train and validate a support vector machine in Matlab. The classifier was combined with a unique educational system to build the Virtual Operative Assistant which provides users with automated feedback on their metric performance with regards to expert proficiency performance benchmarks. The Virtual Operative Assistant successfully classified skilled and novice participants using 4 metrics with an accuracy, specificity and sensitivity of 92, 82 and 100%, respectively. A 2-step feedback system was developed to provide participants with an immediate visual representation of their standing related to expert proficiency performance benchmarks. The educational system outlined establishes a basis for the potential role of integrating artificial intelligence and virtual reality simulation into surgical educational teaching. The potential of linking expertise classification, objective feedback based on proficiency benchmarks, and instructor input creates a novel educational tool by integrating these three components into a formative educational paradigm.