Construct Validity and Reliability of Structured Assessment of endoVascular Expertise in a Simulated Setting

Fundamental Technical Skills of Endovascular Surgery: A Preliminary Study on Its Impact on Skills and Stress during Procedures

BACKGROUND

Our objective is twofold: determining if simulation allows residents to reach proficient surgeons' performance concerning fundamental technical skills of endovascular surgery (FEVS) while investigating effects of the program on surgeons' stress.

METHODS

Using a FEVS training simulator, 8 endovascular FEVS were performed by vascular surgery residents (simulator-naive or simulator-experienced residents [SER]) and seniors. Total time needed to complete the 8 tasks, called total completion time (TCT), was the main evaluation criterion. Analgesia Nociception Index (ANI) was monitored during simulation. Likert scale questionnaire was filled out after each simulation.

RESULTS

For each task, TCT was significantly lower for SER and seniors than simulator-naive residents (P = 0.0163). After only 5 simulations, SER were able to reach and even exceed the seniors' level in terms of TCT, with a median time of 10.8 min for SER and 11.9 min for seniors, and wire's movements with a median distance during cannulation of 4.44 m for SER and 4.17 m for seniors. Seniors remained better than SER in terms of precise wire manipulation (wire movement after cannulation), 4.17 m against 4.44 m (3.72-5.96), respectively. Based on the Likert scale stress analysis, seniors felt less stressed than both residents' groups (P = 0.0618). Seniors' initial ANI and mean ANI over the session were significantly lower than those of the residents, P = 0.0358 and P = 0.0250, respectively.

CONCLUSIONS

We showed that 5 simulation sessions allowed residents to reach experienced surgeons' capacities on FEVS concerning TCT. Subjectively, seniors felt less stressed than residents, contrary to the results of our objective measures of stress.

The role of big data, risk prediction, simulation, and centralization for emergency vascular problems: Lessons learned and future directions

Vascular specialists remain in high demand in current practice and commonly oversee care delivery for a variety of clinical emergencies. Accordingly, the contemporary vascular surgeon must be facile with treating a spectrum of problems, including a complex, heterogeneous group of acute arteriovenous thromboembolic and bleeding diatheses. It has been documented previously that there are substantial current workforce limitations placing constraints on vascular surgical care provision. Moreover, with the aging at-risk population, there remains a considerable national urgency to improve timely diagnoses, specialty consultation, and appropriate transfer of patients to centers of excellence capable of providing a comprehensive compendium of emergency vascular services. Clinical decision aids, simulation training, and regionalization of nonelective vascular problems are all strategies that have been increasingly recognized to address these service gaps. Notably, clinical research in vascular surgery has traditionally focused on identification of patient- and procedure-related factors that influence outcomes by using resource-intensive causal inference methodology. By comparison, large data sets have only more recently been recognized to be a valuable tool that can provide heuristic algorithms to address more complex health care problems. Such data can be manipulated to generate clinical risk scores and decision aids, as well as robust outcome descriptions, which stand to inform stakeholders regarding best practice. The purpose of this review was to provide a robust overview of the lessons derived from the application of big data, risk prediction, and simulation in the management of vascular emergencies.

Focus on Radiation Protection Improves Both Correct Behavior and Procedural Performance During Simulation-Based Training - A Randomized Comparison

BACKGROUND

To explore whether simulation-based endovascular training with focus on radiation safety could improve correct behavior without jeopardizing the learning of procedural skills.

METHODS

Twenty-four residents without previous endovascular experience completed 10 clinical scenarios on a virtual-reality endovascular simulator with software for peripheral endovascular interventions. Participants were randomized to receive feedback (n = 12) or not (n = 12) on radiation protection (RP) performance after each case. Expert assessments were done at the first, second, fourth, seventh, and 10th case on RP and endovascular skills (ES). Automatic simulator metrics on procedure time, contrast dose, handling errors, and estimated radiation exposure to patient and operator were registered. Outcome metrics were analyzed by two-way mixed analysis of variance pairwise comparisons with independent t-tests. Correlations were explored using Pearson's r for internal consistency reliability.

RESULTS

The RP performance was similar in both groups at their first attempt (P = 0.61), but the feedback group significantly outperformed the control group over time (P < 0.001 for all comparisons). The feedback group was however slower to learn the ES at start (P = 0.047 at second performance), but after 7 attempts no difference was shown (P = 0.59). The feedback group used more time (19.5 vs. 15.3 min; P = 0.007) but less contrast (60 vs. 100 mL; P < 0.001). The number of errors was the same in both groups, but all metrics regarding radiation exposure favored the feedback group (P-values from 0.001 to 0.008).

CONCLUSIONS

Simulation-based training (SBT) is effective to acquire basic endovascular intervention skills and concurrently learn RP behavior when feedback on radiation culture is provided.

A Systematic Review of Simulation-Based Training in Vascular Surgery

Introduction

Recent advancements in surgical technology, reduced working hours, and training opportunities exacerbated by the COVID-19 pandemic have led to an increase in simulation-based training. Furthermore, a rise in endovascular procedures has led to a requirement for high-fidelity simulators that offer comprehensive feedback. This review aims to identify vascular surgery simulation models and assess their validity and levels of effectiveness (LoE) for each model in order to successfully implement them into current training curricula.

Methods

PubMed and EMBASE were searched on January 1, 2021, for full-text English studies on vascular surgery simulators. Eligible articles were given validity ratings based on Messick’s modern concept of validity alongside an LoE score according to McGaghie’s translational outcomes.

Results

Overall 76 eligible articles validated 34 vascular surgery simulators and training courses for open and endovascular procedures. High validity ratings were achieved across studies for: content (35), response processes (12), the internal structure (5), relations to other variables (57), and consequences (2). Only seven studies achieved an LoE greater than 3/5. Overall, ANGIO Mentor was the most highly validated and effective simulator and was the only simulator to achieve an LoE of 5/5.

Conclusions

Simulation-based training in vascular surgery is a continuously developing field with exciting future prospects, demonstrated by the vast number of models and training courses. To effectively integrate simulation models into current vascular surgery curricula and assessments, there is a need for studies to look at trainee skill retention over a longer period of time. A more detailed discussion on cost-effectiveness is also needed.

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

PURPOSE

Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD

In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS

Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS

G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

PURPOSE

Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory.

METHOD

Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability.

RESULTS

Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations).

CONCLUSIONS

Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Evaluation of a modular in vitro neurovascular procedure simulation for intracranial aneurysm embolization

BACKGROUND

Rapid development in endovascular aneurysm therapy continuously drives demand for suitable neurointerventional training opportunities.

OBJECTIVE

To investigate the value of an integrated modular neurovascular training environment for aneurysm embolization using additively manufactured vascular models.

METHODS

A large portfolio of 30 patient-specific aneurysm models derived from different treatment settings (eg, coiling, flow diversion, flow disruption) was fabricated using additive manufacturing. Models were integrated into a customizable neurointerventional simulator with interchangeable intracranial and cervical vessel segments and physiological circuit conditions ('HANNES'; Hamburg ANatomic Neurointerventional Endovascular Simulator). Multiple training courses were performed and participant feedback was obtained using a questionnaire.

RESULTS

Training for aneurysm embolization could be reliably performed using HANNES. Case-specific clinical difficulties, such as difficult aneurysm access or coil dislocation, could be reproduced. During a training session, models could be easily exchanged owing to standardized connectors in order to switch to a different treatment situation or to change from 'treated' back to 'untreated' condition. Among 23 participants evaluating hands-on courses using a five-point scale from 1 (strongly agree) to 5 (strongly disagree), HANNES was mostly rated as 'highly suitable for practicing aneurysm coil embolization' (1.78±0.79).

CONCLUSION

HANNES offers a wide variability and flexibility for case-specific hands-on training of intracranial aneurysm treatment, providing equal training conditions for each situation. The high degree of standardization offered may be valuable for analysis of device behavior or assessment of physician skills. Moreover, it has the ability to reduce the need for animal experiments.

The model for Fundamentals of Endovascular Surgery (FEVS) successfully defines the competent endovascular surgeon

OBJECTIVE

Fundamental skills testing is now required for certification in general surgery. No model for assessing fundamental endovascular skills exists. Our objective was to develop a model that tests the fundamental endovascular skills and differentiates competent from noncompetent performance.

METHODS

The Fundamentals of Endovascular Surgery model was developed in silicon and virtual-reality versions. Twenty individuals (with a range of experience) performed four tasks on each model in three separate sessions. Tasks on the silicon model were performed under fluoroscopic guidance, and electromagnetic tracking captured motion metrics for catheter tip position. Image processing captured tool tip position and motion on the virtual model. Performance was evaluated using a global rating scale, blinded video assessment of error metrics, and catheter tip movement and position. Motion analysis was based on derivations of speed and position that define proficiency of movement (spectral arc length, duration of submovement, and number of submovements).

RESULTS

Performance was significantly different between competent and noncompetent interventionalists for the three performance measures of motion metrics, error metrics, and global rating scale. The mean error metric score was 6.83 for noncompetent individuals and 2.51 for the competent group (P < .0001). Median global rating scores were 2.25 for the noncompetent group and 4.75 for the competent users (P < .0001).

CONCLUSIONS

The Fundamentals of Endovascular Surgery model successfully differentiates competent and noncompetent performance of fundamental endovascular skills based on a series of objective performance measures. This model could serve as a platform for skills testing for all trainees.

The role of simulation in neurosurgery

PURPOSE

In an era of residency duty-hour restrictions, there has been a recent effort to implement simulation-based training methods in neurosurgery teaching institutions. Several surgical simulators have been developed, ranging from physical models to sophisticated virtual reality systems. To date, there is a paucity of information describing the clinical benefits of existing simulators and the assessment strategies to help implement them into neurosurgical curricula. Here, we present a systematic review of the current models of simulation and discuss the state-of-the-art and future directions for simulation in neurosurgery.

METHODS

Retrospective literature review.

RESULTS

Multiple simulators have been developed for neurosurgical training, including those for minimally invasive procedures, vascular, skull base, pediatric, tumor resection, functional neurosurgery, and spine surgery. The pros and cons of existing systems are reviewed.

CONCLUSION

Advances in imaging and computer technology have led to the development of different simulation models to complement traditional surgical training. Sophisticated virtual reality (VR) simulators with haptic feedback and impressive imaging technology have provided novel options for training in neurosurgery. Breakthrough training simulation using 3D printing technology holds promise for future simulation practice, proving high-fidelity patient-specific models to complement residency surgical learning.

Peak and ceiling effects in final-product analysis of mastoidectomy performance

BACKGROUND

Virtual reality surgical simulation of mastoidectomy is a promising training tool for novices. Final-product analysis for assessing novice mastoidectomy performance could be limited by a peak or ceiling effect. These may be countered by simulator-integrated tutoring.

METHODS

Twenty-two participants completed a single session of self-directed practice of the mastoidectomy procedure in a virtual reality simulator. Participants were randomised for additional simulator-integrated tutoring. Performances were assessed at 10-minute intervals using final-product analysis.

RESULTS

In all, 45.5 per cent of participants peaked before the 60-minute time limit. None of the participants achieved the maximum score, suggesting a ceiling effect. The tutored group performed better than the non-tutored group but tutoring did not eliminate the peak or ceiling effects.

CONCLUSION

Timing and adequate instruction is important when using final-product analysis to assess novice mastoidectomy performance. Improved real-time feedback and tutoring could address the limitations of final product based assessment.

National Simulation-Based Training of Fellows: The Vascular Surgery Example

Vascular surgery has evolved dramatically as a specialty, with new training paradigms and an ever-developing, technically demanding field. The ability to evaluate trainees on their fundamental skills is an important step in ensuring some uniformity in trainees' basic technical abilities. This article describes the development and implementation of the fundamentals of vascular and endovascular surgery, including lessons applied from the Fundamentals of Laparoscopic Surgery and Fundamentals of Endoscopic Surgery programs.

A pilot study to assess the construct and face validity of the Northwestern Objective Microanastomosis Assessment Tool

OBJECT

Microsurgical skills remain an integral component of neurosurgical education. There is a need for an objective scale to assess microsurgical skills. The objective of this study was to assess the face and construct validity of a bench training microanastomosis module and an objective assessment scale, i.e., the Northwestern Objective Microanastomosis Assessment Tool (NOMAT).

METHODS

Medical students, neurosurgical residents, and postdoctoral research fellows at Northwestern University were enrolled in the study. Trainees were divided into 3 groups based on microsurgical experience: 1) experienced, 2) exposed, and 3) novices. Each trainee completed two end-to-end microanastomoses using a 1-mm and a 3-mm synthetic vessel. Two cameras were installed to capture procedural footage. One neurosurgeon blindly graded the performance of trainees using both objective and subjective methods to assess construct validity. Two neurosurgeons reviewed the contents of the simulation module to assess face validity.

RESULTS

Twenty-one trainees participated in the study, including 6 experienced, 6 exposed, and 9 novices. The mean NOMAT score for experienced trainees on the 1-mm module was 47.3/70 compared with 26.0/70 and 25.8/70 for exposed and novice trainees, respectively (p = 0.02). Using subjective grading, experienced trainees performed significantly better on the 1-mm module (64.2/100) compared with exposed or novice trainees (23.3/100 and 25.0/100, respectively; p = 0.02). No statistical difference between groups was noted for the 3-mm module with both NOMAT and subjective grading. Experienced trainees took less time to perform both tasks compared with the others.

CONCLUSIONS

Face and construct validities of the microanastomosis module were established. The scale and the microanastomosis module could help assess the microsurgical skills of neurosurgical trainees and serve as a basis for the creation of a microsurgical curriculum.

Technical skills assessment in a coronary angiography simulator for construct validation

INTRODUCTION

The aim of this study was to evaluate technical skills in a coronary angiography (CA) simulator to establish the performance level of trainees and experts in virtual CA.The traditional master-apprentice way of learning CA is by practicing on patients despite a known risk for complications during training. Safe CA training is warranted, and simulators might be one possibility. Simulators used must be validated regarding their ability to separate trainees from experts. Construct validation of a CA simulator, to our knowledge, has not yet been published.

METHODS

Ten cardiology residents without experience in CA, 4 intermediate, and 10 CA experts performed 5 CAs in the Mentice VIST (Vascular Intervention Simulation Trainer). Metrics reflecting proficiency skills such as total procedure time, fluoroscopy time, and contrast volume were extracted from the simulator computer and compared between the groups. All examinations were videotaped, and the number of handling errors was examined. The videos were evaluated by 2 experts blinded to the test object's performance level.

RESULTS

Experts outperformed trainees in all metrics measured by the simulator. Improvement was demonstrated in all metrics through all 5 CAs. Furthermore, beginners had more handling errors compared with experts.

CONCLUSIONS

Mentice VIST simulator can distinguish between trainees and experts in CA in the metrics extracted from the computer and therefore prove the concept of construct validity.

A systematic review of assessment of skill acquisition and operative competency in vascular surgical training

OBJECTIVE

The aim of this systematic review is to describe the literature and assessment tools evaluating vascular surgical operative performance that could potentially be used for the assessment of educational outcomes applicable to the Milestone Project and the Next Accreditation System.

METHODS

A systematic review of PubMed/MEDLINE, EMBASE, PsycINFO, and key journals from 1985 to 2013 was performed to identify English-language articles describing assessment of vascular surgical skills and competence. Qualifying studies were abstracted for data concerning study aims, study and assessment setting, skills measured, and metrics used to determine competency. Strengths, weaknesses, and psychometric robustness of the assessment tools were determined.

RESULTS

The literature search identified 617 citations. After title and abstract review, 65 articles were retrieved for full-text assessment and 48 articles were included in the final review. Twenty-nine articles assessed open vascular skills; 19, endovascular skills; six, nontechnical skills; and one, teamwork skills. The majority (84%) of studies were performed in a simulated environment, four (8%) were performed in the operating room, and the remaining three were performed in both a simulated environment and an operating room. Strengths and weaknesses of assessment tools were study and assessor dependent, with none applicable to all study scenarios or procedures.

CONCLUSIONS

The literature describing assessment tools pertinent to vascular surgery is diverse. Existing assessment tools may be relevant to individual technical skill acquisition assessment; however, an operative assessment tool relevant to vascular/endovascular surgery and generalizable to the wide spectrum of technical and nontechnical skills pertinent to vascular surgery needs to be developed, validated, and implemented to allow the practical assessment of resident readiness to operate in an unsupervised setting.

Fine-motor skills testing and prediction of endovascular performance

BACKGROUND

Performing endovascular procedures requires good control of fine-motor digital movements and hand-eye coordination. Objective assessment of such skills is difficult. Trainees acquire control of catheter/wire movements at various paces. However, little is known to what extent talent plays for novice candidates at entry to practice.

PURPOSE

To study the association between performance in a novel aptitude test of fine-motor skills and performance in simulated procedures.

MATERIAL AND METHODS

The test was based on manual course-tracking using a proprietary hand-operated roller-bar device coupled to a personal computer with monitor view rotation. A total of 40 test repetitions were conducted separately with each hand. Test scores were correlated with simulator performance. Group A (n = 14), clinicians with various levels of endovascular experience, performed a simulated procedure of contralateral iliac artery stenting. Group B (n = 19), medical students, performed 10 repetitions of crossing a challenging aortic bifurcation in a simulator.

RESULTS

The test score differed markedly between the individuals in both groups, in particular with the non-dominant hand. Group A: the test score with the non-dominant hand correlated significantly with simulator performance assessed with the global rating scale SAVE (R = -0.69, P = 0.007). There was no association observed from performances with the dominant hand. Group B: there was no significant association between the test score and endovascular skills acquisition neither with the dominant nor with the non-dominant hand.

CONCLUSION

Clinicians with increasing levels of endovascular technical experience had developed good fine-motor control of the non-dominant hand, in particular, that was associated with good procedural performance in the simulator. The aptitude test did not predict endovascular skills acquisition among medical students, thus, cannot be suggested for selection of novice candidates. Procedural experience and practice probably supplant the influence of innate abilities (talent) over time.

Experience With a Simulator-Based Angiography Course for Neurosurgical Residents

BACKGROUND:

Simulation is an increasingly useful means of teaching in the era of duty hour restrictions. Since the completion of our diagnostic cerebral angiography simulator curriculum pilot program, we have performed this resident course at 2 Congress of Neurological Surgeons (CNS) annual meetings with larger participant numbers.

OBJECTIVE:

To report the ongoing results of these courses.

METHODS:

A 120-minute simulator-based training course was performed at 2 CNS annual meetings. Precourse written and simulator skills assessments were performed, followed by instructor-guided training on an endovascular simulator. Postcourse written and simulator practical assessments were then performed and compared with precourse scores.

RESULTS:

Thirty-seven neurosurgery resident participants completed the course module: 16 completed the first course provided and 21 completed the second. Posttest written scores were significantly higher than pretest scores (mean ± SEM, 8.5 ± 0.40.3 vs 4.9 ± 0.3; P &lt; .001). Instructor assessments of practical posttest scores of participants were significantly higher than pretest practical scores for both the CNS 2011 and CNS 2012 groups (P &lt; .001).

CONCLUSION:

The expansion of a curriculum-based, cerebral angiography simulator pilot program to trainees through courses at national neurosurgical meetings demonstrated excellent results with significant improvements in written test scores and instructor assessments of participant technical skills. With ever-expanding improvements in simulation technology and realism, simulator training for cerebral angiography may become an integral component of resident training in the future.