Performance Assessment of Arthroscopic Rotator Cuff Repair and Labral Repair in a Dry Shoulder Simulator

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review

INTRODUCTION

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery.

METHODS

A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality.

RESULTS

Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools.

DISCUSSION

The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

The frequency of assessment tools in arthroscopic training: a systematic review

BACKGROUND

Multiple assessment tools are used in arthroscopic training and play an important role in feedback. However, it is not fully recognized as to the standard way to apply these tools. Our study aimed to investigate the use of assessment tools in arthroscopic training and determine whether there is an optimal way to apply various assessment tools in arthroscopic training.

METHODS

A search was performed using PubMed, Embase and Cochrane Library electronic databases for articles published in English from January 2000 to July 2021. Eligible for inclusion were primary research articles related to using assessment tools for the evaluation of arthroscopic skills and training environments. Studies that focussed only on therapeutic cases, did not report outcome measures of technical skills, or did not mention arthroscopic skills training were excluded.

RESULTS

A total of 28 studies were included for review. Multiple assessment tools were used in arthroscopic training. The most common objective metric was completion time, reported in 21 studies. Technical parameters based on simulator or external equipment, such as instrument path length, hand movement, visual parameters and injury, were also widely used. Subjective assessment tools included checklists and global rating scales (GRS). Among these, the most commonly used GRS was the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Most of the studies combined objective metrics and subjective assessment scales in the evaluation of arthroscopic skill training.

CONCLUSIONS

Overall, both subjective and objective assessment tools can be used as feedback for basic arthroscopic skill training, but there are still differences in the frequency of application in different contexts. Despite this, combined use of subjective and objective assessment tools can be applied to more situations and skills and can be the optimal way for assessment.

LEVEL OF EVIDENCE

Level III, systematic review of level I to III studies. Key messagesBoth subjective and objective assessment tools can be used as feedback for basic arthroscopic skill training.Combined use of subjective and objective assessment tools can be applied to more situations and skills and can be the optimal way for assessment.

The FAST Workstation Shows Construct Validity and Participant Endorsement

Purpose

To determine in what way the proposed simulation-based intervention (SBI) is an effective intervention for use in basic arthroscopic skills training.

Methods

Twenty candidates were recruited and grouped according to experience. Performance metrics included the time to activity completion, errors made, and Global Rating Scale score. Qualitative data were collected using a structured questionnaire.

Results

Performance on the SBI differed depending on previous arthroscopic training received. Performance on the simulator differed between groups to a statistically significant level regarding time to completion. A difference was also present between participants with no previous training and those with previous training when assessed using the Global Rating Scale. The SBI was deemed acceptable, user-friendly, and realistic. Participants practicing at the expert level believe that such an SBI would be beneficial in developing basic arthroscopic skills.

Conclusions

The results of this study provide evidence that the use of an SBI consisting of a benchtop workstation, laptop viewing platform, 30° arthroscope, and defined performance metrics can detect differences in the level of arthroscopic experience. This format of SBI has been deemed acceptable and useful to the intended user, increasing the feasibility of introducing it into surgical training.

Clinical Relevance

This study adds to the existing body of evidence supporting the potential benefits of benchtop SBIs in arthroscopic skills training. Improved performance on such an SBI may be beneficial for the purpose of basic arthroscopic skills training, and we would support the inclusion of this system in surgical training programs such as those developed by the Arthroscopy Association of North America and American Board of Orthopaedic Surgery.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Does Sawbone-Based Arthroscopy Module (SBAM) Can Help Elbow Surgeons?

INTRODUCTION

The use of elbow arthroscopy is becoming increasingly common in orthopaedic practice; nevertheless, it is still considered a difficult procedure with a long learning curve. The aim of the study is to evaluate the role of a new elbow Sawbone-Based Arthroscopy Module (e-SBAM) in the training of elbow surgeons.

METHODS

Fourteen surgeons were classified as "Expert" (n: 7; more than 10 years of experience in arthroscopic surgery) and "Not-expert" surgeons (n: 7; less than 10 years of experience). During a dedicated arthroscopic session, using the Sawbones Elbow model (Sawbones Europe AB®), all participants were asked to perform an arthroscopic round and to touch three specific landmarks. An independent observer measured the time that each participant needed to perform this task (Performance 1). The same measurement was repeated after two weeks of eSBAM training (Performance 2).

RESULTS

"Not-expert" surgeons needed significantly more time (41 s; range 26-120) than "Expert" ones (13 s; range 8-36) to complete Performance 1. One "Not-expert" surgeon did not complete Performance 1 and needed more than 120 s for Performance 2. The whole study group required a median of 5 s less to complete Performance 2. A tendency towards an improvement was observed in the group of the non-experienced surgeons as compared with the experienced ones.

CONCLUSIONS

The simulation training can be advantageous in the learning curve of young elbow surgeons and helpful for experienced surgeons. E-SBAM can be used as an effective tool for the current stepwise arthroscopic elbow training programs with the aim of improving arthroscopic elbow skills.

LEVEL OF EVIDENCE

Basic Science Study.

Arthroscopic Simulation: The Future of Surgical Training: A Systematic Review

BACKGROUND

Arthroscopic simulation has rapidly evolved recently with the introduction of higher-fidelity simulation models, such as virtual reality simulators, which provide trainees an environment to practice skills without causing undue harm to patients. Simulation training also offers a uniform approach to learn surgical skills with immediate feedback. The aim of this article is to review the recent research investigating the use of arthroscopy simulators in training and the teaching of surgical skills.

METHODS

A systematic review of the Embase, MEDLINE, and Cochrane Library databases for English-language articles published before December 2019 was conducted. The search terms included arthroscopy or arthroscopic in combination with simulation or simulator.

RESULTS

We identified a total of 44 relevant studies involving benchtop or virtually simulated ankle, knee, shoulder, and hip arthroscopy environments. The majority of these studies demonstrated construct and transfer validity; considerably fewer studies demonstrated content and face validity.

CONCLUSIONS

Our review indicates that there is a considerable evidence base regarding the use of arthroscopy simulators for training purposes. Further work should focus on the development of a more uniform simulator training course that can be compared with current intraoperative training in large-scale trials with long-term follow-up at tertiary centers.

Innovation in Orthopaedic Surgery Education: Novel Tools for Modern Times

Orthopaedic residency training has and will continue to evolve with a wide variety of changes. Hands-on surgical simulation "boot camps," computerized simulation of surgical process, and even virtual reality simulators, all can help trainees acquire surgical experience without compromising patient care. Low-cost training modules help remedy the rising costs associated with teaching complex orthopaedic surgery skills. Motion tracking and checklists help refine standardization of assessment. As technology and healthcare systems continue to grow, we encourage training programs to keep pace by considering engagement of these tools.

Skills in Open Surgery do not Predict Arthroscopic Skills in Medical Students

OBJECTIVE

Teaching of basic open surgical skills like skin closure has been well established in medical schools' surgical curricula. The same cannot be said for basic arthroscopic skills even though the importance of arthroscopic surgery in Orthopaedics has been well documented. The goal of this study was to evaluate the relationship between basic skills in open surgery and basic arthroscopic skills in a cohort of medical students. Our hypothesis was that performance in open surgical skills does not correlate with skills in arthroscopic surgery.

DESIGN

The performance of medical students in basic tasks in open surgery (porcine model: subcutaneous and skin suturing) and arthroscopic surgery (bench top model: arthroscopic triangulation and object transfer) was assessed. For both surgical techniques an introductory course was given followed by a standardized assessment. After 1 week both open and arthroscopic tasks were reassessed. All procedures were recorded and scored by 2 independent observers in a blinded fashion.

SETTING

The study was performed at the skills lab of the Clinic for Orthopaedics and Tumororthopaedics at University Hospital Muenster in Muenster Germany.

PARTICIPANTS

A cohort of 21 medical students (average age 22.2 years) participated in this study. The cohort consisted of 17 female (81%) and 4 male (19%) students. All students were in the clinical part of their study and had not received any prior surgical education.

RESULTS

Interobserver reliability was very high for the arthroscopic tasks and high for the open surgery tasks. No correlation was found between open and arthroscopic skills. For the first assessment the correlation coefficient was r = 0.197 (p = 0.391). For the second assessment the correlation coefficient was r = 0.212 (p = 0.356). Significant improvement from first to second assessment was only found for the arthroscopic tasks (p < 0.0001) while improvement in the open surgery performance failed to reach statistical significance (p = 0.184).

CONCLUSIONS

The results of this study suggest that performance in open surgical skills does not correlate with performance in arthroscopic skills and should be taught independent from each other. Arthroscopic skills can effectively be taught with bench top training systems in a time- and resource efficient manner, with measurable results even in a 1-day-course setup. Arthroscopic skills training may be offered in undergraduate curricula in addition to open surgical skills training to students with an interest in Orthopaedics.

Global Rating Scales for the Assessment of Arthroscopic Surgical Skills: A Systematic Review

PURPOSE

To evaluate whether sufficient validity and reliability evidence exists to support the use of global rating scales (GRS) as evaluation tools in both formative assessment and competency assessment of arthroscopic procedures.

METHODS

A search of PubMed, Embase, and Scopus was conducted for articles published between 1990 and 2018. Studies reporting measures of validity and reliability of GRS relating to arthroscopic skills were included. Procedural checklists and other assessment tools were excluded.

RESULTS

A total of 39 articles met the inclusion criteria. In total, 7 de novo GRS specific for arthroscopic education and 3 pre-existing GRS repurposed 4 times for arthroscopic education were identified in the literature. The 11 GRS were used to assess 1175 surgeons 3890 times. Three GRS tools explicitly defined an arbitrary minimum competency threshold, 6 of 11 tools demonstrated construct validity-the ability to significantly discriminate between groups of differing experience-and 5 of 11 tools assessed inter-rater reliability, but only the Arthroscopic Surgical Skills Evaluation Tool demonstrated excellent inter-rater reliability. The Arthroscopic Surgical Skills Evaluation Tool was validated by 16 articles for a total of 537 surgeons for hip, knee, shoulder, and ankle arthroscopy in both simulated and clinical environments but was found to be invalid in wrist arthroscopy. The Basic Arthroscopic Knee Skill Scoring System was validated by 15 articles for a total of 497 surgeons for knee, hip, and shoulder in both clinical and simulated environments. The remaining 9 GRS were validated by 2 or fewer studies.

CONCLUSIONS

Overall, GRS have contributed to training, feedback, and formative assessment practices. The GRS reviewed demonstrate both construct and concurrent validity as well as reliability in multiple arthroscopic procedures in multiple joints. Currently, there is sufficient evidence to use GRS as a feedback tool. However, there is insufficient evidence for its use in high-stakes examinations or as a minimum competency assessment.

LEVEL OF EVIDENCE

Level III, systematic review of level I to III studies.

Development of a certification examination for orthopedic sports medicine fellows

Background

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship.

Methods

Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale.

Results

Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair.

Conclusion

There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

Improvement of arthroscopic surgical performance using a new wide-angle arthroscope in the surgical training

Background

We have developed a new arthroscope with a field of view of 150°. This arthroscope requires less motion to maneuver and exhibits reduced optical error. It also improves how novices learn arthroscopy. We hypothesized that the surgical performance with this arthroscope is superior to that with a conventional arthroscope. This study tested the hypothesis by using motion analysis and a new validated parameter, “dimensionless squared jerk” (DSJ).

Methods

We compared the surgical performance between the use of the wide-angle arthroscope and that of the conventional arthroscope among 14 novice orthopedic residents who performed 3 standardized tasks 3 times with each arthroscope. The tasks simulated the surgical skills in arthroscopic rotator cuff repair. The arthroscope motion was analyzed using an optical tracking system. The differences in performance parameters, such as the time taken to complete the tasks, average acceleration of the hands (m/s²), number of movements, and total path length (m) including DSJ between the 2 arthroscopes were investigated using paired t-tests.

Results

All estimated values for the tasks using the 150° arthroscope were lower than those for the tasks using the 105° arthroscope. Statistically significant differences in performance between the 2 arthroscopes were observed only for DSJ (p = 0.014) and average acceleration (p = 0.039).

Conclusions

DSJ and average acceleration are reliable parameters for representing hand-eye coordination. The surgical performance of novice arthroscopists was better with the new wide-angle arthroscope than with the conventional arthroscope.

A Simple and Low-cost Drilling Simulator for Training Plunging Distance Among Orthopedic Surgery Residents

OBJECTIVE

Drilling through bone is a complex action that requires precise motor skills of an orthopedic surgeon. In order to minimize plunging and soft tissue damage, the surgeon must halt drill progression precisely following penetration of the far cortex. The purpose of this study was to create a low-cost and easy-to-use drilling simulator to train orthopedic residents in reducing the drill plunging depth.

DESIGN, SETTING, PARTICIPANTS

This prospective observational study was performed in the division of orthopedic surgery of a single tertiary medical center. The participants included 13 residents and 7 orthopedic specialists. The simulator consisted of a synthetic femur bone model and ordinary modeling clay, and the training unit consisted of a disposable plastic tube (∼US$14), clamps (∼US$58), and a power drill + drill bit (standard hospital equipment). Plunging depths were measured by the simulator and compared between orthopedic specialists, the 6 "senior residents" (3+ years) and the 7 "junior residents" during a training session. Measurements were taken again 2 weeks following the training session.

RESULTS

Initially, the plunging depths of the junior residents were significantly greater compared to those of the orthopedic specialists (7.00 mm vs. 5.28 mm, respectively, p < 0.038). There was no similarly significant difference between the senior residents and the orthopedic experts ([6.33 mm vs. 5.28 mm, respectively; p = 0.18). The senior residents achieved plunging depths of 5.17 mm at the end of the training session and 4.7 mm 2 weeks later compared to 7.14 mm at the end of the training session and 6 mm 2 weeks later for the junior residents.

CONCLUSIONS

This study demonstrated the capability of a low-cost drilling simulator as a training model for reducing the plunging depth during the drilling of bone and soft tissue among junior and senior residents.

Utility of Modern Arthroscopic Simulator Training Models: A Meta-analysis and Updated Systematic Review

PURPOSE

To determine the utility of modern arthroscopic simulators in transferring skills learned on the model to the operating room.

METHODS

A meta-analysis and systematic review of all English-language studies relevant to validated arthroscopic simulation models using PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines from 1999 to 2016 was performed. Data collected included the specific simulator model, the joint used, participant demographic characteristics, participant level of training, training session information, type and number of tasks, pre- and post-training assessments, and overall outcomes of simulator performance. Three independent reviewers analyzed all studies.

RESULTS

Fifty-seven studies with 1,698 participants met the study criteria and were included. Of the studies, 25 (44%) incorporated an arthroscopic training program into the study methods whereas 32 (56%) did not. In 46 studies (81%), the studies' respective simulator models were used to assess arthroscopic performance, whereas 9 studies (16%) used Sawbones models, 8 (14%) used cadaveric models, and 4 (7%) evaluated subject performance on a live patient in the operating room. In 21 studies (37%), simulator performance was compared with experience level, with 20 of these (95%) showing that clinical experience correlated with simulator performance. In 25 studies (44%), task performance was evaluated before and after simulator training, with 24 of these (96%) showing improvement after training. All 4 studies that included live-patient arthroscopy reported improved operating room performance after simulator training compared with the performance of subjects not participating in a training program.

CONCLUSIONS

This review suggests that (1) training on arthroscopic simulators improves performance on arthroscopic simulators and (2) performance on simulators for basic diagnostic arthroscopy correlates with experience level. Limited data suggest that simulator training can improve basic diagnostic arthroscopy skills in vivo.

LEVEL OF EVIDENCE

Level IV, systematic review of Level I through IV studies.

Editorial Commentary: Dry Shoulder Models Are Excellent for Training Shoulder Arthroscopic Skills

Recent research supports the use of arthroscopic simulation using dry models in the training of residents and fellows learning arthroscopic shoulder surgery.