Cognitive and Psychomotor Entrustable Professional Activities: Can Simulators Help Assess Competency in Trainees?

Procedural simulation training in orthopaedics and traumatology: Nationwide survey among surgeon educators and residents in France

BACKGROUND

Simulation is among the tools used in France to train residents specialising in orthopaedic and trauma surgery (OTS). However, implementing simulation-based training (SBT) is complex and poorly reported. The objective of this study was to describe the use of simulation for OTS training in France.

HYPOTHESIS

Nationwide, SBT is not used to its full capacity for teaching OTS in France, and differences in opinions about SBT may exist between surgeon educators and residents.

STUDY DESIGN

Nationwide questionnaire survey in France.

MATERIALS AND METHODS

We built two specific self-questionnaires then e-mailed them between December 2020 and February 2021 to the surgeon educators who were members of the national university council and to the residents specialising in OTS during the current academic year. The questions were about the 2018-2019 academic year, before the COVID-19 pandemic. Two classes of residents who were still medical students during this period were not included, leaving three classes for the analysis.

RESULTS

The participation rates were 57% (67/117) for the educators and 24% (87/369) for the three classes of residents. Of the 67 educators, 47 (70%) reported being involved in SBT and identified the university (70%) and industry (53%) as the main funders of this teaching modality. The educators indicated that the mean number of SBT laboratories in their region was 1.4±0.9 (range, 0-4). The main types of simulators were saw bones (77%); cadavers (85%); and commercial simulators (74%), notably for the knee (87%) and shoulder (78%). The educators estimated that they had achieved a mean of 33%±23% (range, 0%-100%) of the teaching objectives set out in the OTS curriculum and that the main obstacles were insufficient funding (81%) and lack of time (67%). Only 21% of educators reported conducting SBT research. The residents reported that they accessed SBT via the OTS teaching module (28/87, 32%), local university degrees (23/87, 26%), their hospital department (17/87, 18%), or the industry (15/87, 17%); 25/87 (29%) had never received SBT. On a 0-10 scale (0, completely disagrees; 10, completely agrees), the mean score for SBT effectiveness was 8.6±2.1 for residents and 7.1±3.0 for educators (p<0.001); the corresponding values for the quality of SBT integration in the region were 1.5±1.8 and 3.8±2.6, respectively (p<0.001).

CONCLUSION

SBT is not yet used to its full potential for teaching OTS in France. Insufficient funding and lack of time were identified by the educators as the main obstacles to greater use of SBT. Both the residents and the educators felt that SBT mightbe beneficial for training.

LEVEL OF EVIDENCE

IV, nationwide survey.

Comparison of a dichotomous versus trichotomous checklist for neonatal intubation

BACKGROUND

To compare validity evidence for dichotomous and trichotomous versions of a neonatal intubation (NI) procedural skills checklist.

METHODS

NI skills checklists were developed utilizing an existing framework. Experts were trained on scoring using dichotomous and trichotomous checklists, and rated recordings of 23 providers performing simulated NI. Videolaryngoscope recordings of glottic exposure were evaluated using Cormack-Lehane (CL) and Percent of Glottic Opening scales. Internal consistency and reliability of both checklists were analyzed, and correlations between checklist scores, airway visualization, entrustable professional activities (EPA), and global skills assessment (GSA) were calculated.

RESULTS

During rater training, raters gave significantly higher scores on better provider performance in standardized videos (both p < 0.001). When utilized to evaluate study participants' simulated NI attempts, both dichotomous and trichotomous checklist scores demonstrated very good internal consistency (Cronbach's alpha 0.868 and 0.840, respectively). Inter-rater reliability was higher for dichotomous than trichotomous checklists [Fleiss kappa of 0.642 and 0.576, respectively (p < 0.001)]. Sum checklist scores were significantly different among providers in different disciplines (p < 0.001, dichotomous and trichotomous). Sum dichotomous checklist scores correlated more strongly than trichotomous scores with GSA and CL grades. Sum dichotomous and trichotomous checklist scores correlated similarly well with EPA.

CONCLUSIONS

Neither dichotomous or trichotomous checklist was superior in discriminating provider NI skill when compared to GSA, EPA, or airway visualization assessment. Sum scores from dichotomous checklists may provide sufficient information to assess procedural competence, but trichotomous checklists may permit more granular feedback to learners and educators. The checklist selected may vary with assessment needs.

Simulation for assessment of Entrustable Professional Activities in an emergency medicine residency program

In 2018, Canadian post-graduate Emergency Medicine (EM) programs transitioned to Competence-by-Design. Residents are now assessed using Entrustable Professional Activities (EPAs). We developed and implemented simulation for assessment to mitigate anticipated challenges with residents completing the required number of observations of resuscitation-based EPAs. Our survey of trainees who participated in these sessions suggests that it may be a feasible and acceptable method for EPA assessment.

A review of participant recruitment transparency for sound validation of hip surgery simulators: a novel umbrella approach

Malposition of implants is associated with complications, higher wear and increased revision rates in total hip replacement (THR) along with surgeon inexperience. Training THR residents to reach expert proficiency is affected by the high cost and resource limitations of traditional training techniques. Research in extended reality (XR) technologies can overcome such barriers. These offer a platform for learning, objective skill-monitoring and, potentially, for automated certification. Prior to their incorporation into curricula however, thorough validation must be undertaken. As validity is heavily dependent on the participants recruited, there is a need to review, scrutinise and define recruitment criteria in the absence of pre-defined standards, for sound simulator validation. A systematic review on PubMed and IEEE databases was conducted. Training simulator validation research in fracture, arthroscopy and arthroplasty relating to the hip was included. 46 validation studies were reviewed. It was observed that there was no uniformity in reporting or recruitment criteria, rendering cross-comparison challenging. This work developed Umbrella categories to help prioritise recruitment, and has formulated a detailed template of fields and guidelines for reporting criteria so that, in future, research may come to a consensus as to recruitment criteria for a hip "expert" or "novice".

Entrustable Professional Activities in Orthopaedics

Background

An entrustable professional activity (EPA) is defined as a core task of a specialty that is entrusted to a trainee once sufficient competence has been reached. A group of EPAs reflects the activities that clinicians commonly do on a day-to-day basis. Lists of EPAs have been created for most medical subspecialties, but not orthopaedic surgery. The aim of this study was to create a peer-reviewed list of essential EPAs that a resident must perform independently before completing orthopaedic residency training.

Methods

A focus group of 7 orthopaedic surgeons from the University of Toronto developed a comprehensive list of 285 EPAs. For each subspecialty group, the opinions of at least 15 academic and nonacademic surgeons, as well as subspecialty-trained and non-subspecialty-trained surgeons, were used. The modified Delphi method was used to rank EPAs on a five-point scale from not important to mandatory for a resident to competent before exiting training. Two Delphi rounds were used, using a threshold of >50% of surgeons considering the EPA as mandatory before being considered for the next round. A final list of EPAs was ratified using the focus group of academic surgeons involved in the study.

Results

Seventy-five (75) of 107 (70%) surgeons invited responded to the survey. Nearly half (129) of the 285 EPAs were discarded after the first round of Delphi. A further 118 EPAs were discarded after the second Delphi round, leaving 49 final EPAs, across 9 subspecialties in orthopaedic surgery.

Conclusions

Expert consensus was used to create a list of EPAs considered mandatory for completion of resident training in orthopaedics in our province. The final 49 peer-reviewed EPAs will be a valuable benchmark in curriculum design and assessment in orthopaedic surgery in the competency-based era for other programs.

Surgical Games: A Simulation-Based Structured Assessment of Orthopedic Surgery Resident Technical Skill

BACKGROUND

Simulation-based education and objective evaluation of surgical skill have been incorporated into many surgical training programs. We describe the development and implementation of a timed, multitask, station-based Surgical Games to evaluate orthopedic resident surgical skills.

METHODS

Participants in the study were postgraduate-year 2 to 5 orthopedic surgery residents from a single institution. Residents completed 4-timed simulated tasks: cadaveric carpal tunnel release (CTR), Sawbones model of total knee arthroplasty (TKA), Sawbones model of ankle fracture open reduction internal fixation (ORIF), and knee arthroscopy simulator (KAS) of removal of loose body. Evaluations were performed using standardized score sheets by attending surgeons. Resident performance was analyzed by postgraduate-year and number of weeks of prior task-related residency training.

RESULTS

A total of 32 residents were assessed at the 4 stations. Total scores were significantly different for CTR (p = 0.006), TKA (p = 0.05), and the KAS (p = 0.004) by year of training, but not for the ankle ORIF task. Residents with more task-specific experience performed significantly better on the KAS (p < 0.001), TKA (p = 0.002), and CTR (p = 0.02) tasks but not on the ankle ORIF task (p = 0.1). Overall, residents rated the exercise valuable to their education with mean scores of 3.9 ± 0.54 on a 5-point Likert Scale.

CONCLUSIONS

This Surgical Games exercise provided an objective evaluation of surgical skill that correlated with year-in-training and prior experience in skill-specific rotations for the KAS, TKA, and CTR tasks. This surgical skills assessment provided an opportunity for effective structured feedback and identification of areas for improvement.

Establishing an Orthopedic Program-Specific, Comprehensive Competency-Based Education Program

BACKGROUND

Competency-based education (CBE) seeks to determine resident proficiency in the knowledge, skills, and behaviors required for independent patient care. Multiple assessment instruments evaluate technical skills or direct patient care in the clinic setting, but there are few reports incorporating both within an orthopedic specialty rotation. This study reports a residency program's comprehensive CBE initiative using formative assessments in the clinic and operating room during a sports medicine rotation.

MATERIALS AND METHODS

The sports medicine rotation used validated formative assessments to evaluate resident performance during clinic encounters and program-defined surgical entrustable professional activities (EPAs). Junior resident (postgraduate year [PGY] 1-2) EPAs included basic knee/shoulder arthroscopic procedures. Senior resident (PYG 5) EPAs comprised anterior cruciate ligament reconstruction, biceps tenodesis, shoulder stabilization, and rotator cuff repair. Assessment scores were compared between individuals and PGY groups.

RESULTS

Sixty-six clinical skills (CS) and 106 surgical skills assessments were conducted for 22 residents in one academic year. Surgical skills assessments demonstrated significant differences between each PGY group (P < 0.01). All PGY2 and PGY5 residents achieved independence on the evaluated EPAs. PGY5s earned higher scores in CS assessments than the other classes (P < 0.01). PGY2 residents scored higher than PGY1s in 7 of 9 CS domains. CS independence was achieved by 21 of 22 residents by the end of the rotation.

CONCLUSIONS

The CBE program effectively quantified expected differences in resident performance by PGY for clinic and surgical assessments on a sports medicine rotation. Assessments built an environment where feedback was more structured and standardized, creating a culture to improve resident education.

Development of a certification examination for orthopedic sports medicine fellows

Background

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship.

Methods

Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale.

Results

Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair.

Conclusion

There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

Technical Obstacles in Total Knee Arthroplasty Learning: A Steps Breakdown Evaluation

Introduction

Total knee arthroplasty (TKA) is a common procedure practiced in both the community and academic setting and one that all orthopaedic surgery residents are expected to become competent in. The aim of this study is to determine the most common technical obstacles encountered during TKA learning.

Methods

This is a prospective, cohort observational study performed from September 2017 to April 2018. After routine primary TKA, faculty completed a survey of the trainees in the case through a series of 10 questions. The questions were scored on a 0 to 5 scale based on performance proficiency. Exclusion criteria included revision TKA and complex primary TKA. Participants were divided into two groups based on year in training multiplied by the number of cases performed: group 1 (junior-n = 44) was <20, whereas group 2 (senior-n = 59) was >20.

Results

The senior experience group scored higher for all questions (P < 0.05). Skills competency and technique were related to each other, independent of experience. When evaluating the relationships between the steps, the scores on every step were linked to the previous and following step at all experience levels (P < 0.05), with some dictating the success of the rest of the case with high significance (P < 0.01).

Conclusion

We have shown that most senior-level residents cannot necessarily perform all steps of a TKA proficiently, potentially leading to issues in independent practice. We have also demonstrated that residents have the most difficulty with conceptual tasks, rather than technical ones. Teaching has traditionally focused on technical skills, but this implies conceptual tasks may require more teaching focus.

Virtual Reality Simulation Facilitates Resident Training in Total Hip Arthroplasty: A Randomized Controlled Trial

BACKGROUND

No study has yet assessed the efficacy of virtual reality (VR) simulation for teaching orthopedic surgery residents. In this blinded, randomized, and controlled trial, we asked if the use of VR simulation improved postgraduate year (PGY)-1 orthopedic residents' performance in cadaver total hip arthroplasty and if the use of VR simulation had a preferentially beneficial effect on specific aspects of surgical skills or knowledge.

METHODS

Fourteen PGY-1 orthopedic residents completed a written pretest and a single cadaver total hip arthroplasty (THA) to establish baseline levels of knowledge and surgical ability before 7 were randomized to VR-THA simulation. All participants then completed a second cadaver THA and retook the test to assess for score improvements. The primary outcomes were improvement in test and cadaver THA scores.

RESULTS

There was no significant difference in the improvement in test scores between the VR and control groups (P = .078). In multivariate regression analysis, the VR cohort demonstrated a significant improvement in overall cadaver THA scores (P = .048). The VR cohort demonstrated greater improvement in each specific score category compared with the control group, but this trend was only statistically significant for technical performance (P = .009).

CONCLUSIONS

VR-simulation improves PGY-1 resident surgical skills but has no significant effect on medical knowledge. The most significant improvement was seen in technical skills. We anticipate that VR simulation will become an indispensable part of orthopedic surgical education, but further study is needed to determine how best to use VR simulation within a comprehensive curriculum.

LEVEL OF EVIDENCE

Level 1.

Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools

BACKGROUND

Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.

METHODS

Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.

RESULTS

There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.

CONCLUSIONS

This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.

CLINICAL RELEVANCE

This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

A systematic review of 7 years of research on entrustable professional activities in graduate medical education, 2011-2018

PURPOSE

This review aimed to synthesise some of the extant work on the use of entrustable professional activities (EPAs) for postgraduate physicians, to assess the quality of the work and provide direction for future research and practice.

METHOD

Systematic searches were conducted within five electronic databases (Medline, Scopus, Web of Science, PsycINFO and CINAHL) in September 2018. Reference lists, Google Scholar and Google were also searched. Methodological quality was assessed using the Quality Assessment Tool for Studies with Diverse Designs (QATSDD).

RESULTS

In total, 49 studies were included, classified as Development of EPAs (n = 37; 76% of total included), Implementation and/or assessment of EPAs (n = 10; 20%), or both (n = 2; 4%). EPAs were described for numerous specialties, including internal medicine (n = 14; 36%), paediatrics (n = 8; 21%) and psychiatry (n = 4; 10%). Of the development studies, 92% utilised more than one method to generate EPAs. The two most commonly used methods were developing initial EPAs in a working group, (n = 27; 69%) and revising through deliberation (n = 21; 54%). Development papers were of variable quality (mean QATSDD score = 20, range 6-41). Implementation and assessment studies utilised methods that included observing trainee performance (n = 6; 50%) and enrolling trainees in competency-based curricula, which included EPAs (n = 4; 33%). The methodological quality of these implementation studies varied (mean QATSDD score = 19.5, range = 6-32).

CONCLUSIONS

This review highlighted a need for: (i) consideration of best practice guidelines for EPA development; (ii) focus on the methodological quality of research on EPA development and of EPAs, and (iii) further work investigating the implementation of EPAs in the curriculum.

Assessment Tools for Feedback and Entrustment Decisions in the Clinical Workplace: A Systematic Review

BACKGROUND: Entrustable Professional Activities (EPAs) combine feedback and evaluation with a permission to act under a specified level of supervision and the possibility to schedule learners for clinical service. This literature review aims to identify workplace-based assessment tools that indicate progression toward unsupervised practice, suitable for entrustment decisions and feedback to learners. METHODS: A systematic search was performed in the PubMed, Embase, ERIC, and PsycINFO databases. Based on title/abstract and full text, articles were selected using predetermined inclusion and exclusion criteria. Information on workplace-based assessment tools was extracted using data coding sheets. The methodological quality of studies was assessed using the medical education research study quality instrument (MERSQI). RESULTS: The search yielded 6,371 articles (180 were evaluated in full text). In total, 80 articles were included, identifying 67 assessment tools. Only a few studies explicitly mentioned assessment tools used as a resource for entrustment decisions. Validity evidence was frequently reported, and the MERSQI score was 10.0 on average. CONCLUSIONS: Many workplace-based assessment tools were identified that potentially support learners with feedback on their development and support supervisors with providing feedback. As expected, only few articles referred to entrustment decisions. Nevertheless, the existing tools or the principals could be used for entrustment decisions, supervision level, or autonomy.

Ear Disease Knowledge and Otoscopy Skills Transfer to Real Patients: A Randomized Controlled Trial

OBJECTIVE

To determine which teaching method-otoscopy simulation (OS), web-based module (WM), or standard classroom instruction (SI)-produced greater translation of knowledge and otoscopy examination skills to real patients.

DESIGN

In a prospective randomized controlled nonclinical trial, medical students were randomized to 1 of 3 interventional arms: (1) OS, (2) WM, or (3) SI. Students were assessed at baseline for diagnostic accuracy and otoscopy skills on 5 volunteer patients (total of 10 ears), followed by the intervention. Testing was repeated immediately after intervention on the same patients. Student reported confidence in diagnostic accuracy and otoscopy examination were also captured. Assessors were blinded to the intervention group, and whether students were pre- or post-intervention.

SETTING

Clinical Teaching Centre, Queen's University.

PARTICIPANTS

Twenty-nine participants were initially randomized. Two students were unable to attend their specific intervention sessions and withdrew. Final group sizes were: OS-10, WM-9, SI-8. Five patients with external/middle ear pathologies were voluntarily recruited to participate as testing subjects.

RESULTS

Baseline diagnostic accuracy and otoscopy clinical skills did not differ across the groups. Post-intervention, there were improvements in diagnostic accuracy from all groups: OS (127.78%, 2.30 ± 1.42, p = 0.0006), WM (76.40%, 1.44 ± 1.88, p = 0.0499), and SI (100.00%, 1.50 ± 1.20, p = 0.0093). For otoscopy skills, post-intervention improvements were noted from OS (77.00%, 3.85 ± 2.55, p < 0.0001) and SI (22.20%, 1.25 ± 1.20, p = 0.0011), with no significant improvement from WM (13.46%, 0.78 ± 1.92, p = 0.1050). Students across all groups reported significantly improved confidence in diagnostic accuracy (p < 0.0001) and otoscopy skill (p < 0.0001) after the intervention.

CONCLUSION

All 3 teaching modalities showed an improvement in diagnostic accuracy immediately post-intervention. Otoscopy clinical skills were found to have increased only in OS and SI, with the OS group demonstrating the largest improvement. Simulation-based medical education in Otolaryngology may provide the greatest transfer of medical knowledge and technical skills when evaluated with real patients.

Prior Generic Arthroscopic Volume Correlates with Hip Arthroscopic Proficiency: A Simulator Study

BACKGROUND

Changing trends in surgical education and patient expectation are leading to proficiency models of progression and the use of simulators. Hip arthroscopy is increasingly performed and has a steep learning curve mainly addressed during fellowship training. The aim of this study was to assess the impact of previous generic arthroscopic experience on performance at a simulated hip arthroscopy task to both estimate the minimum case numbers that correlate with expert proficiency levels and help to guide selection for hip arthroscopy fellowships.

METHODS

Fifty-two participants were recruited to a cross-sectional study. Four consultants (expert hip arthroscopists), 28 trainees (residents and fellows), and 20 novices (interns and medical students) performed a standardized bench-top simulated hip arthroscopy task. A validated global rating scale (GRS) score and motion analysis were used to assess surgical performance. Prior arthroscopic experience was recorded from surgical electronic logbooks. Receiver operating characteristic (ROC) curve analyses were conducted to identify optimum cut-points for task proficiency at both expert and competent GRS levels.

RESULTS

There were significant differences (p < 0.05) between the arthroscopic ability of all experience groups based on GRS assessment and for all motion analysis metrics. There was a significant positive correlation between logbook numbers and GRS scores (p < 0.0001). ROC curve analysis demonstrated that a minimum of 610 prior arthroscopic procedures were necessary to achieve an expert GRS score, and 78 prior arthroscopic procedures were necessary for a competent score.

CONCLUSIONS

Performing a basic hip arthroscopy task competently requires substantial previous generic arthroscopic experience. The numbers identified in this study provide targets for residents. Program directors appointing to hip arthroscopy fellowship training posts may find these results useful as a guide during the selection process.

Performance Assessment of Arthroscopic Rotator Cuff Repair and Labral Repair in a Dry Shoulder Simulator

PURPOSE

To evaluate the use of dry models to assess performance of arthroscopic rotator cuff repair (RCR) and labral repair (LR).

METHODS

Residents, fellows, and sports medicine staff performed an arthroscopic RCR and LR on a dry model. Any prior RCR and LR experience was noted. Staff surgeons assessed participants by use of task-specific checklists, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), and a final overall global rating. All procedures were video recorded and were scored by a fellow blinded to the year of training of each participant.

RESULTS

A total of 51 participants and 46 participants performed arthroscopic RCR and LR, respectively, on dry models. The internal consistency or reliability (Cronbach α) using the total ASSET score for the RCR and LR was high (>0.9). One-way analysis of variance for the total ASSET score showed a difference between participants based on year of training (P < .001) for both procedures. The inter-rater reliability for the ASSET score was excellent (>0.9) for both procedures. A good correlation was seen between the ASSET score and the year of training, as well as the previous number of sports rotations.

CONCLUSIONS

The results of this study show evidence of construct validity when using dry models to assess performance of arthroscopic RCR and LR by residents.

CLINICAL RELEVANCE

The results of this study support the use of arthroscopic simulation in the training of residents and fellows learning arthroscopic shoulder surgery.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

PURPOSE

The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model.

METHODS

Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review.

RESULTS

The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05).

CONCLUSIONS

The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.

CLINICAL RELEVANCE

As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.