Learning curves in health professions education

Evaluation of dental students' learning curve in intraligamentary anesthesia using different syringe systems: A prospective crossover study

PURPOSE/OBJECTIVES

This prospective crossover preclinical trial aimed to evaluate the learning curve of dental students in successfully administering intraligamentary anesthesia (ILA) using three different syringe systems.

METHODS

Dental students performed ILA using three devices in two separate sessions, each targeting mandibular and/or maxillary premolars. The devices included two manual systems (pistol-type and lever-based) and one computer-controlled local anesthetic delivery system (CCLAD). The primary research parameter was the success rate of anesthesia, defined as the percentage of successful ILA administrations confirmed by a negative response to a cold test. Secondary parameters included pain experienced during needle penetration and injection, students' self-reported levels of mental tension and handling of the syringes, and any potential side effects.

RESULTS

A total of 110 students performed ILA on 599 teeth during the study period. When comparing the CCLAD system to the manual syringes, the CCLAD system exhibited a significantly higher overall success rate in the first session (92.5% vs. 77.4%; p < 0.001), potentially due to its precise control of anesthetic flow and pressure, which likely facilitated more effective anesthetic delivery. However, when examining the individual manual techniques, no significant difference was found between the pistol-type manual and the CCLAD system (p = 0.66). All techniques' success rate increased from the first to the second session (80.4% vs. 86.9%; p = 0.0357). Additionally, penetration pain demonstrated a significant decrease across all techniques (p < 0.01). Notably, students' anxiety levels decreased, and self-assurance increased significantly over the sessions. Undesired reversible side effects were documented in 10.9% of cases.

CONCLUSION

These findings suggest that repeated practice of ILA, particularly with different syringe systems, enhances anesthetic success and psychological readiness for patient interaction. Additional training sessions may further improve proficiency.

Evaluating Learning Curves in Virtual Reality Cortical Mastoidectomy Training Across Expertise Levels

HYPOTHESIS

Virtual reality (VR) simulation has been established as an effective method of supplementing traditional surgical training. Learning curves can analyze skill acquisition over time in VR settings. Although previous studies explored learning curves of mastoidectomy performances on single specimen, this study analyzed learning curves on anatomically different virtual temporal bones across three expertise levels (novice, intermediate, and expert).

METHODS

Thirty participants were divided into three groups: 10 medical students (novice), 10 ear, nose and throat (ENT) registrars (intermediate), and 10 senior ear surgeons (expert). They performed mastoidectomy on eight anatomically different temporal bones on the University of Melbourne Temporal Bone Surgery Simulator. A blinded senior ENT surgeon assessed the final products of the dissections using the Melbourne Mastoidectomy Scale (MMS). Learning curves of MMS scores, number of strokes, average force, drilling time, and total time were compared between groups using Friedman tests. Within-group analyses were conducted with Wilcoxon signed-rank tests.

RESULTS

All performance metrics showed significant differences across all groups. Only comparison between intermediate and expert groups for average force was not significant. Within-group analyses showed significant differences in expert group for total and drilling time, intermediate group for average force, and novice group for total time. Individual learning curves of intermediate group demonstrated varied learning behavior.

CONCLUSIONS

Trainee performance was not seen to reach the level of experts after eight repetitions on anatomically different specimens. Enhancing training with individualized feedback and increased repetitions may optimize skills acquisition.

A prospective study of the acquisition of vaginal examination skills using simulation

OBJECTIVE

To determine the number of sessions required using procedural simulation to acquire the skill of vaginal examination, which is an essential part of obstetrics, but a difficult learned skill.

METHODS

Using a high-fidelity simulator, we conducted a prospective, single-center, single-blind study, at the Angers School of Midwifery. A class of students completed a theory course, and took part in three simulation sessions. During the simulation sessions, each student was asked to describe five different cervixes, under five criteria: position, length, consistency, dilation, and head station. Each participant received individual feedback as part of a debrief session, after completing their description. A pass rate of 80% was set for the entire class.

RESULTS

Twenty-six students participated. The class achieved a mean score of 70.77 ± 10.23% in the first session, 81.85 ± 9.91% in the second session, and 81.23 ± 8.63% in the third session. There was a significant improvement only between the first and second sessions (P < 0.001). Of the 26 participants, 6 (23%) scored over 80% in the first session, 17 participants (65%) scored above 80% in the first two sessions, and 21 participants (80%) scored above 80% over the three sessions.

CONCLUSION

Learning vaginal examination by procedural simulation with the aid of a high-fidelity simulator, and receiving individual feedback and debrief, resulted in an 80% pass rate in two practical sessions, working to describe 10 cervixes.

Learning Curve of Microsurgical Anastomosis: Training for Resident Education

BACKGROUND

Acquiring microsurgical anastomosis skills requires considerable time and effort. Moreover, appropriate and systematic training for acquiring microsurgical anastomosis skills is lacking. Therefore, this study investigated the learning curve for acquiring microsurgical anastomosis techniques among orthopaedic surgery residents.

METHODS

The study involved 12 orthopaedic surgery residents without experience in microsurgical anastomosis. The residents were divided into two groups: the 'Experienced group' with more than 6 months of suturing experience and the 'Inexperienced group' with no suturing experience. Each participant underwent 30 practice sessions, suturing a 3.5-mm diameter silastic tube. The time taken for each anastomosis and its quality were evaluated. Individual learning curves were derived, and the number of trials required to reach the time plateau was determined.

RESULTS

The Experienced group reached the time plateau after an average of 16.3 ± 1.4 attempts while the Inexperienced group reached it after an average of 24.2 ± 2.5 attempts. The time required for the first two attempts was 40.4 ± 6.2 min for the Experienced group and 61.2 ± 8.6 min for the Inexperienced group (P < 0.001). The time required for the last two attempts was 11.4 ± 0.7 min for the Experienced group and 12.8 ± 0.8 min for the Inexperienced group. Comparing the quality scores of the first two attempts, the Experienced group scored 4.3 ± 0.5 points and the Inexperienced group scored 3.1 ± 0.5 points (P < 0.001). The Experienced group scored 7.8 ± 0.5 points for the last two attempts while the Inexperienced group scored 6.9 ± 0.3 points (P < 0.001).

CONCLUSIONS

Individuals new to suturing improved anastomosis time and quality by approximately 30 times. This finding suggests that practitioners can optimize their training while educators can refine the curriculum by predicting learning curves and providing timely feedback to enhance skill development.

Longitudinal Improvement in Public Speaking Skills Through Participation in a Resident Public Speaking Curriculum

OBJECTIVE

To promote public speaking skills, a pediatrics residency program developed a longitudinal public speaking curriculum grounded in deliberate practice and reflective practice.

METHODS

Residents delivered annual presentations and received formal feedback. Audience evaluation forms from 2005 to 2017 were included for analysis. The form used five-point scales (5 =best) for specific presentation elements (clarity, eye contact/body language, pace, succinct text, minimally distracting delivery, clear conclusion, appropriate learning objectives, achieving learning objectives, and answering questions) and for overall quality. Longitudinal changes in scores were analyzed with paired t tests.

RESULTS

Overall, 5771 evaluations of 276 presentations given by 97 residents were analyzed. Between post-graduate year (PGY)-1 and PGY-3 presentations, mean overall rating increased from 4.38 to 4.59 (P < .001, d=0.51). The median percentage of five-point scores increased from 50.0% (IQR, 24.3%-65.4%) to 72.5% (IQR, 53.3%-81.2%). Eight of nine specific elements showed significant increases (median effect size 0.55). Residents whose initial presentations ranked in the bottom quartile had larger improvements than residents initially ranked in the top quartile.

CONCLUSIONS

After pediatric residents participated in a public speaking curriculum with targeted objectives, formal feedback, and repeated practice, their public speaking skills improved. Public speaking curricula can and should be adopted more broadly in graduate medical education.

Five Domains of a Conceptual Framework of Continuing Professional Development

ABSTRACT

Continuing professional development (CPD) for health professionals involves efforts at improving health of individuals and the population through educational activities of health professionals who previously attained a recognized level of acceptable proficiency (licensure). However, those educational activities have inconsistently improved health care outcomes of patients. We suggest a conceptual change of emphasis in designing CPD to better align it with the goals of improving health care value for patients through the dynamic incorporation of five distinct domains to be included in learning activities. We identify these domains as: (1) identifying, appraising, and learning new information [New Knowledge]; (2) ongoing practicing of newly or previously acquired skills to maintain expertise [New Skills and Maintenance]; (3) sharing and transfer of new learning for the health care team which changes their practice [Teams]; (4) analyzing data to identify problems and drive change resulting in improvements in the health care system and patient outcomes [Quality Improvement]; and (5) promoting population health and prevention of disease [Prevention]. We describe how these five domains can be integrated into a comprehensive conceptual framework of CPD, supported by appropriate learning theories that align with the goals of the health care delivery system. Drawing on these distinct but interrelated areas of CPD will help organizers and directors of learning events to develop their activities to meet the goals of learners and the health care system.

Examining the learning curves in robotic cardiac surgery wet lab simulation training

BACKGROUND

Simulation-based training has gained distinction in cardiothoracic surgery as robotic-assisted cardiac procedures evolve. Despite the increasing use of wet lab simulators, the effectiveness of these training methods and skill acquisition rates remain poorly understood.

OBJECTIVES

This study aimed to compare learning curves and assess the robotic cardiac surgical skill acquisition rate for cardiac and noncardiac surgeons who had no robotic experience in a wet lab simulation setting.

METHODS

In this prospective cohort study, participants practiced 3 robotic tasks in a porcine model: left atriotomy closure, internal thoracic artery harvesting and mitral annular suturing. Participants were novice robotic cardiac and noncardiac surgeons alongside experienced robotic cardiac surgeons who established performance benchmarks. Performance was evaluated using the time-based score and modified global evaluative assessment of robotic skills (mGEARS).

RESULTS

The participants were 15 novice surgeons (7 cardiac; 8 noncardiac) and 4 experienced robotic surgeons. Most novices reached mastery in 52 (±22) min for atrial closure, 32 (±18) for internal thoracic artery harvesting and 34 (±12) for mitral stitches, with no significant differences between the cardiac and noncardiac surgeons. However, for mGEARS, noncardiac novices faced more challenges in internal thoracic artery harvesting. The Thurstone learning curve model indicated no significant difference in the learning rates between the groups.

CONCLUSIONS

Wet lab simulation facilitates the rapid acquisition of robotic cardiac surgical skills to expert levels, irrespective of surgeons' experience in open cardiac surgery. These findings support the use of wet lab simulators for standardized, competency-based training in robotic cardiac surgery.

Delineating Neurosurgery Resident Development: Insights from a 10-Year National Accreditation Council for Graduate Medical Education Milestones and Learning Trajectories

BACKGROUND AND OBJECTIVES

The Accreditation Council for Graduate Medical Education Milestones assessment system provides a structured framework for evaluating the developmental progression of neurosurgery residents. Understanding the variability in learning trajectories and identifying critical intervention points are essential for refining educational strategies and enhancing training outcomes. In this article, we aim to identify learning trajectories of neurosurgery residents and pinpoint pivotal junctures for potential interventions to aid residents struggling to meet expected competency levels.

METHODS

We analyzed Accreditation Council for Graduate Medical Education Milestones data from 3541 neurosurgery residents across 124 programs from 2013-2014 to 2022-2023. We examined the reliability of Milestone assessments across programs and addressed potential biases because of program-level differences. Through cluster analysis, we identified distinct learning trajectories, focusing on competency-specific developments and crucial intervention points based on divergences in trajectory.

RESULTS

Our analysis revealed 5 distinct learning trajectories among neurosurgery residents, highlighting diverse developmental pathways. Cluster 1 (n = 1,719, 48.5%) and Cluster 2 (n = 341, 9.6%) demonstrated initial foundational competencies but varied in progression, with Cluster 2 failing to meet the threshold for independent practice by postgraduate year 7 consistently. Cluster 3 (n = 459, 13%), starting with the lowest initial competency levels, surpassed independence thresholds, illustrating the potential for significant growth regardless of initial competency levels. Cluster 4 (n = 520, 14.7%) and Cluster 5 (n = 502, 14.2%) exhibited accelerated competency development, achieving early readiness for independent practice. We found no significant association between cluster membership and program characteristics, suggesting that the trajectories reflect individual variations rather than programmatic differences.

CONCLUSION

Our findings highlight the need for educational strategies tailored to residents' unique development paths to support the achievement of competency thresholds. While the data suggest potential benefits of adaptive learning, further exploration is required to confirm its impact on educational outcomes.

New test for systematic skills enhancement and improvement in maxillofacial surgery training: multicentre pilot study

We know of few studies in the international scientific literature that specifically address the evaluation of surgical and clinical progress among physicians undergoing specialist training in maxillofacial surgery. Identifying a reliable tool to accurately assess both theoretical knowledge and surgical skills of trainees is essential. The primary aim of this study therefore was to design a comprehensive assessment tool that is capable of evaluating both the theoretical and practical skills of physicians undergoing specialist training in maxillofacial surgery. The methodology employed aims to ensure fairness and effectiveness in skills development, thereby optimising training activities. To meet this need, an evaluation and self-assessment test was developed for maxillofacial surgery trainees at the Ospedali Riuniti of Ancona. Data collection involved digitally administered evaluations and self-assessment tests focused on maxillofacial traumatology, based on AO trauma surgery references. Data were processed into graphs which revealed a progressive learning trend following an initial adjustment phase, leading to optimal outcomes in both clinical and surgical domains. The evaluation and self-assessment test proved to be a valuable learning tool with which to gauge advancements in clinical and surgical skills among maxillofacial surgery residents.

Entrustable professional activities (EPAs) in surgical pathology: implementation experience and longitudinal observations of resident development

Entrustable professional activities (EPAs) have been implemented in various medical specialties, and the Pathology National EPA Working Group has piloted the implementation of four pathology EPAs. We recently published the development of EPAs within our surgical pathology rotation. Following a six-month pilot, a survey demonstrated that faculty and residents found the forms helpful and easy to use and easy to understand, and EPAs have been fully incorporated into our surgical pathology rotation. Here, we discuss our experience, challenges, and results of resident EPA performance for intraoperative consultations (IOC) and sign-out (SO) after 21 months of implementation. Between June 2022 and March 2024, 24 residents were evaluated by 13 faculty members, resulting in 136 IOC and 298 SO EPA forms. Paper forms were predominantly used, with only five electronic forms submitted. EPA performance scores for SO increased from 2.4 ± 0.8 in Block 1 to 4.6 ± 0.2 in Block 14 (p < 0.0001), whereas performance scores for IOC increased from 2.7 ± 1.0 in Block 1 to 4.8 ± 0.2 in Block 14 (p < 0.0001). The progressive decrease in the standard deviation throughout residency denotes higher competence homogeneity as residency graduation approaches. Overall, our EPA evaluation method showed ease of use, provided valuable tracking tools, and long-term feasibility. EPAs are robust tools for tracking resident progression toward independent practice in surgical pathology, offering valuable insights for program and rotation directors to assess and track individual EPA skills, identify intervention points, and provide an opportunity for immediate, actionable feedback based on current performance.

Learning curves of novice residents on cataract surgery simulator: the E3CAPS pedagogic study

INTRODUCTION

Training novice ophthalmology residents on the EyeSi® simulator increases cataract surgery safety. However, there is no consensus regarding how much training residents should perform before their first time on patients. We evaluated the French national training program through the analysis of the learning curves of novice residents.

METHODS

This prospective multicentric pedagogic study was conducted with French novice residents. Each resident completed the recommended four two-hour training sessions and performed a standardized assessment simulating standard cataract surgery before the first session (A0), at the end of the first (A1), second (A2), third (A3) and fourth (A4) sessions. For each surgical step of each attempt, the following data were collected: score, odometer, completion time, posterior capsular rupture and cumulative energy delivered (ultrasounds) during phacoemulsification. A performance threshold was set at a score of 80/100 for each surgical step, 400/500 for the overall procedure. Only descriptive statistics were employed.

RESULTS

Sixteen newly nominated ophthalmology residents were included. Median score progressively increased from 95 [IQR 53; 147]) at A0 to 425 [IQR 411; 451] at A4. Despite a significant progression, the "emulsification" step had the lowest A4 scores 86 [IQR 60; 94] without reduction in completion time, odometer or ultrasounds delivered. The rate of posterior capsular rupture decreased linearly from 75% at A0 to 13% at A4 during "emulsification" and from 69 to 0% during "irrigation and aspiration". At A4, only 25% [8; 53] of residents had > 80 at each step and only 75% [47; 92] had > 400/500 overall.

CONCLUSION

A training program consisting of four two-hour sessions on the EyeSi simulator over four consecutive days effectively enhances the surgical skills of novice ophthalmology residents. Undergoing more training sessions may improve scores and decrease the incidence of surgical complications, particularly at the emulsification step of cataract surgery. The learning curves presented here can reassure residents who are progressing normally and help identify those who need a further personalized training program.  TRIAL REGISTRATION: ClinicalTrials registration number: NCT05722080 (first submitted 22/12/2022, first posted 10/02/2023).

The Effectiveness of Instruction on Lumbar Spine Thrust Joint Manipulation Task Performance

INTRODUCTION

Uncertainty exists regarding the best method for teaching thrust joint manipulation (TJM) to student physical therapists. The purpose of this study was to compare the effectiveness of Peyton's 4-step (P4) approach with the "see one, do one" (S1D1) approach for teaching students to perform a lumbar spine TJM task in an academic setting. Secondary objectives were to compare the effects of each instructional approach on students' attitudes and beliefs toward spinal TJM and on their motivation to learn to perform lumbar spine TJM.

REVIEW OF LITERATURE

The S1D1 approach is used in the health care professions for teaching clinical tasks to students. It is unclear whether the P4 approach may better prepare students to practice TJM.

SUBJECTS

Student physical therapists.

METHODS

Using a factorial quasi-experimental design, an equal number of students were assigned to a P4 or S1D1 instruction group for the TJM task. Students' performance accuracy, time, and outcome performing TJM in an academic setting were measured. Paper surveys were used to collect data about students' attitudes and beliefs toward spinal TJM and their motivation to learn TJM. A generalized estimating equations approach was used for data analysis.

RESULTS

Fifty-eight students (29 per group) completed the study. There was an interaction between the instruction group and time on task performance accuracy favoring the P4 approach (P = .03). There was no interaction between the instruction group and task performance time, task performance outcome, attitudes and beliefs toward spinal TJM, or motivation to learn TJM (all P > .19).

DISCUSSION AND CONCLUSION

The P4 approach more effectively improved student accuracy when performing the TJM task in an academic setting than the S1D1 approach. However, no differences between instruction were found for performance time or outcome. Students reported a favorable perception of learning lumbar spine TJM. These preliminary results suggest that instructors may use the P4 approach to improve students' TJM procedural knowledge before task practice. However, limitations of the study may affect the internal validity and generalizability of results.

From admissions to licensure: education data associations from a multi-centre undergraduate medical education collaboration

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements.

Point-of-Care Ultrasound Psychomotor Learning Curves: A Systematic Review of the Literature

OBJECTIVES

Use of point-of-care ultrasound (POCUS) in clinical medicine and inclusion in medical training is increasing. Some professional societies recommend that 25-50 POCUS examinations be completed for each application learned; however, the amount of practice required is not well studied. As such, a better understanding of the learning curves of POCUS psychomotor skills is needed. This systematic review characterizes the learning curves for POCUS psychomotor skill acquisition.

METHODS

With the assistance of a research librarian, the available literature through August 28, 2023, was identified. The titles and abstracts, and then the full text were reviewed by two reviewers to screen for inclusion. All studies included after full-text review then underwent data extraction and analysis.

RESULTS

The search identified 893 unique studies. Forty-five studies underwent full-text review, with 17 meeting full inclusion criteria. Substantial heterogeneity was noted in study design, duration of education, number and type of learners, and methods for statistical analysis. Clear and validated definitions for learning endpoints, such as plateau points or competency, are lacking. Learning curves and endpoints differ for different applications of POCUS.

CONCLUSION

The results are overall supportive of the recommendations to complete 25-50 examinations per application of POCUS learned. However, specific applications require more practice than others. Certain applications, such as cardiac and the Focused Assessment with Sonography in Trauma (FAST) exams, are closer to 50; while others, such as soft tissue, airway, and eye require no more than 25.

Using latent class growth analysis to detect group developmental trajectories in preclinical medical education

Medical educators and programs are deeply interested in understanding and projecting the longitudinal developmental trajectories of medical students after these students are matriculated into medical schools so appropriate resources and interventions can be provided to support students' learning and progression during the process. As students have different characteristics and they do not learn and progress at the same pace, it is important to identify student subgroups and address their academic needs to create more equitable learning opportunities. Using latent class growth analysis, this study explored students' developmental trajectories and detected group differences based on their coursework performance in Anatomy within the two years of preclinical education in one medical school. Four subgroups were identified with various intercepts and slopes. There were significant group differences between these subgroups and their standardized scores in MCAT and UCMLE Step 1. The study provides evidence about the heterogeneity of the student population and points out future research directions.

Assessing Facial Palsy: Does Feedback Improve Assessment Using the eFACE and Sunnybrook Facial Grading System?

OBJECTIVE(S)

To explore learning effects when applying the clinician-graded electronic facial function scale (eFACE) and the Sunnybrook Facial Grading System (Sunnybrook).

METHODS

Surgeons, facial rehabilitation therapists, and medical students were randomly allocated to the eFACE (n = 7) or Sunnybrook (n = 6) and graded 60 videos (Massachusetts Eye and Ear Infirmary open-source standard set); 10 persons with normal facial function and 50 patients with a wide variation of facial palsy severity. Participants received an introduction and individual feedback after each set of 10 videos. Scores were compared to the reference score provided with the set. Multilevel analysis was performed to analyze learning effect.

RESULTS

A learning effect was only found for the eFACE, with significant difference scores in set 1 and 2 compared to set 6, and no significant difference scores in the following sets. The difference score was associated with the reference score (severity of facial palsy) for eFACE (β = -0.19; SE = 0.04; p < 0.001) and Sunnybrook (β = -0.15; SE = 0.04; p < 0.001). Age of participants was also associated with the difference score in the eFACE group (β = 0.18; SE = 0.03; p < 0.001). No differences in scores were found between groups of participants.

CONCLUSION

The eFACE showed a learning effect of feedback while the Sunnybrook did not.

LEVEL OF EVIDENCE

NA Laryngoscope, 134:3105-3111, 2024.

Competency based medical education in nuclear cardiology: A tale of two axes

BACKGROUND

Across medical specialties, including nuclear cardiology, competency based medical education (CBME) changes the emphasis of learning from a time or experiential emphasis to a proficiency focused approached. Plotted on a learning-curve graph the emphasis on learning has shifted from the duration/ volume-based x-axis to the performance-based y-axis.

CURRENT STATUS

It has proven difficult to establish y-axis-based standards within nuclear cardiology to assess learning. As such there is a paucity of data to verify current experiential training targets and only recently is data emerging that seeks to find CBME targets by which proficiency (y-axis units) can be evaluated. Initial reports from such CBME-oriented studies indicate that in current nuclear cardiology practice, the number of studies required to achieve competency is dependent upon the chosen measure of competency that is assessed (summed stress score versus % LV ischemia), the case mix, and the modality being learnt (PET versus SPECT). Recent findings have also suggested that prior levels of experiential training may be an underestimation of the number of supervised studies learners need to interpret before they achieve competency.

SUMMARY

Nuclear cardiology training has adopted the concept of CBME and is progressing toward a more modern approach to trainee assessment. This brief review provides the background, current requirements and insights into new developments in nuclear cardiology training.

A comprehensive value-based method for new nuclear medical service pricing: with case study of radium [223Ra] bone metastases treatment

IMPORTANCE

Innovative nuclear medicine services offer substantial clinical value to patients. However, these advancements often come with high costs. Traditional payment strategies do not incentivize medical institutes to provide new services nor determine the fair price for payers. A shift towards a value-based pricing strategy is imperative to address these challenges. Such a strategy would reconcile the cost of innovation with incentives, foster transparent allocation of healthcare resources, and expedite the accessibility of essential medical services.

OBJECTIVE

This study aims to develop and present a comprehensive, value-based pricing model for new nuclear medicine services, illustrated explicitly through a case study of the radium [223Ra] treatment for bone metastases. In constructing the pricing model, we have considered three primary value determinants: the cost of the new service, associated service risk, and the difficulty of the service provision. Our research can help healthcare leaders design an evidence-based Fee-For-Service (FFS) payment reference pricing with nuclear medicine services and price adjustments.

DESIGN, SETTING AND PARTICIPANTS

This multi-center study was conducted from March 2021 to February 2022 (including consultation meetings) and employed both qualitative and quantitative methodologies. We organized focus group consultations with physicians from nuclear medicine departments in Beijing, Chongqing, Guangzhou, and Shanghai to standardize the treatment process for radium [223Ra] bone metastases. We used a specially designed 'Radium Nuclide [223Ra] Bone Metastasis Data Collection Form' to gather nationwide resource consumption data to extract information from local databases. Four interviews with groups of experts were conducted to determine the add-up ratio, based on service risk and difficulty. The study organized consultation meeting with key stakeholders, including policymakers, service providers, clinical researchers, and health economists, to finalize the pricing equation and the pricing result of radium [223Ra] bone metastases service.

MAIN OUTCOMES AND MEASURES

We developed and detailed a pricing equation tailored for innovative services in the nuclear medicine department, illustrating its application through a step-by-step guide. A standardized service process was established to ensure consistency and accuracy. Adhering to best practice guidelines for health cost data analysis, we emphasized the importance of cross-validation of data, where validated data demonstrated less variation. However, it required a more advanced health information system to manage and analyze the data inputs effectively.

RESULTS

The standardized service of radium [223Ra] bone metastases includes: pre-injection assessment, treatment plan, administration, post-administration monitoring, waste disposal and monitoring. The average duration for each stage is 104 min, 39 min, 25 min, 72 min and 56 min. A standardized monetary value for medical consumables is 54.94 yuan ($7.6), and the standardised monetary value (medical consumables cost plus human input) is 763.68 yuan ($109.9). Applying an agreed value add-up ratio of 1.065, the standardized value is 810.19 yuan ($116.9). Feedback from a consultation meeting with policymakers and health economics researchers indicates a consensus that the pricing equation developed was reasonable and well-grounded.

CONCLUSION

This research is the first study in the field of nuclear medicine department pricing methodology. We introduce a comprehensive value-based nuclear medical service pricing method and use radium[223Ra] bone metastases treatment pricing in China as a case study. This study establishes a novel pricing framework and provides practical instructions on its implementation in a real-world healthcare setting.

Using ACGME milestones as a formative assessment for the internal medicine clerkship: a consecutive two-year outcome and follow-up after graduation

BACKGROUND

This study evaluated the utility of using Accreditation Council for Graduate Medical Education (ACGME) Milestones as a formative assessment tool for the fifth- and sixth-grade medical students' performance in their internal medicine (IM) clerkship and the same students' performance in their post-graduate year (PGY) IM training.

METHODS

Retrospective data were collected from 65 medical students completing the two-year IM clerkship in the academic years 2019 and 2020 and 26 of the above students completing their PGY-1 training at the same university hospital in the academic year 2021. Data included the assessment results of 7 of the ACGME IM Milestones, information on admitted patients assigned to the students, and surveys of the students' satisfaction.

RESULTS

The analysis included 390 assessment results during the IM clerkship and 78 assessment results during the PGY-1 training. Clinical teachers commonly rated level 3 to medical students in the IM clerkship, with PC-2 subcompetency receiving the lowest rating among seven subcompetencies. The levels of most subcompetencies showed stationary in the two-year IM clerkship. Significant improvement was observed in all subcompetencies during the PGY-1 training. The medical students in the second-year IM clerkship expressed higher satisfaction with implementing Milestones than in their first-year IM clerkship and perceived Milestones assessments' usefulness as learning feedback.

CONCLUSIONS

Using ACGME Milestones as a formative assessment tool in the IM clerkship yielded promising outcomes. Longitudinal follow-up of subcompetencies facilitated tracking students' development and providing constructive feedback.

Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment

ABSTRACT

In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

Researching models of innovation and adoption in health professions education

BACKGROUND

Despite the constant presence of change and innovation in health professions education (HPE), there has been relatively little theoretical modelling of such change, the experiences of change, the ideology associated with change or the unexpected consequences of change. In this paper, the authors explore theoretical approaches to the adoption of innovations in HPE as a way of mapping a broader theoretical landscape of change.

METHOD

The authors, HPE researchers with an interest in technology adoption and systemic change, present a narrative review of the literature based on a series of thought experiments regarding how communities and individuals respond to the introduction of new ideas or methods. This research investigates the stages of innovation adoption, from the emergence and hype around new ideas to the concrete experiences of early adopters.

RESULTS

When an innovation first emerges, there is often little concrete information available to inform potential adopters, leaving it susceptible to hype, both positive and negative. This can be described using the Gartner Hype Cycle model, albeit with important caveats. Once the adoption of an innovation gets underway, early adopter user experiences can inform those that follow. This can be described using Rogers' diffusion of innovation model, again with caveats. Notably, neither model goes beyond the point of single point-in-time, yes/no, individual adoption. Other approaches, such as learning curve theory, are needed to track uptake and maintenance by individuals over time.

SIGNIFICANCE

This expanded theoretical base, while still somewhat instrumentalist, combined with complementary theoretical perspectives can afford opportunities to better explore reasons for variance, volunteerism and resistance to change. In summary, change is complicated and nuanced, and better models and theories are needed to understand and work meaningfully with change in HPE. To that end, the authors seek to encourage richer and more thoughtful research and scholarly thinking about change and a more nuanced approach to the pursuit of change in HPE as a whole.

[Analysis of learning curves for mini-gastric bypass in 341 patients with obesity]

OBJECTIVE

To analyze learning curves and appropriate experience on the features of mini-bypass surgery in 341 obese patients.

MATERIAL AND METHODS

A total of 341 laparoscopic mini-gastric bypass surgeries performed by one surgeon were studied. The median age of patients was 40.5 [34; 48.3] years. There were 284 (83.2%) women and 57 (16.8%) men. The median BMI was 45 [40;52] kg/m2.

RESULTS

The period of MGB development consisted of 138 interventions. Surgery time was 120 [100; 130] min and 90 [82.5; 100] mins after development of this technique (p=0.001). Complications occurred in 5 (1.5%) patients (1 patient with Clavien Dindo grade IIIA and 4 ones with grade IIIB). Of these, there were 3 patients with stapler suture defects. There were no complications only in the 4th quartile of surgeries. Surgical experience significantly affects postoperative outcomes. Surgery time was more influenced by surgical skill rather technique of anastomosis imposing.

CONCLUSION

Polynomial regression objectively characterizes development of surgical skills lasting 138 interventions. MGB is safe for morbid obesity with a complication rate of 1.5% and no mortality.

Tapping into Efficient Learning: An Exploration of the Impact of Sequential Learning on Skill Gains and Learning Curves in Central Venous Catheterization Simulator Training

OBJECTIVE

Medical residents learn how to perform many complex procedures in a short amount of time. Sequential learning, or learning in stages, is a method applied to complex motor skills to increase skill acquisition and retention but has not been widely applied in simulation-based training (SBT). Central venous catheterization (CVC) training could benefit from the implementation of sequential learning. CVC is typically taught with task trainers such as the dynamic haptic robotic trainer (DHRT). This study aims to determine the impact of sequential learning on skill gains and learning curves in CVC SBT by implementing a sequential learning walkthrough into the DHRT.

METHODS

103 medical residents participated in CVC training in 2021 and 2022. One group (N = 44) received training on the original DHRT system while the other group (N = 59) received training on the DHRTsequential with interactive videos and assessment activities. All residents were quantitatively assessed on (e.g. first trial success rate, distance to vein center, overall score) the DHRT or DHRTsequential systems.

RESULTS

Residents in the DHRTsequential group exhibited a 3.58 times higher likelihood of successfully completing needle insertion on their first trial than those in the DHRT only group and required significantly fewer trials to reach a pre-defined mastery level of performance. The DHRTsequential group also had fewer significant learning curves compared to the DHRT only group.

CONCLUSION

Implementing sequential learning into the DHRT system significantly benefitted CVC training by increasing the efficiency of initial skill gain, reducing the number of trials needed to complete training, and flattening the slope of the subsequent learning curve.

Twelve Tips for using Learning Curves in Health Professions Education Research

Learning curves can be used to design, implement, and evaluate educational interventions. Attention to key aspects of the method can improve the fidelity of this representation of learning as well as its suitability for education and research purposes. This paper addresses when to use a learning curve, which graphical properties to consider, how to use learning curves quantitatively, and how to use observed thresholds to communicate meaning. We also address the associated ethics and policy considerations. We conclude with a best practices checklist for both educators and researchers seeking to use learning curves in their work.

Assessing the utility of a novel entrustment-supervision assessment tool

BACKGROUND

Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools.

METHODS

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor.

RESULTS

A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8.

CONCLUSION

The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.

Gathering validity evidence for a 3D-printed simulator for training of myringotomy and ventilation tube insertion

Objectives

This study aimed to gather validity evidence according to Messick's framework for a novel 3D-printed simulator for myringotomy with ventilation tube insertion for use in technical skills training of otorhinolaryngology (ORL) residents.

Methods

The study included 15 junior ORL residents (trainees) and 13 experienced teaching otolaryngologists (experts). Experts and trainees first received an identically structured introduction to the procedure, simulator, and simulation setup. Five procedures performed by each participant were video-recorded and ordered randomly for blinded rating by two independent raters. The rating tools used were a global rating scale (GBRS) and a task-specific checklist. Validity evidence was collected according to Messick's framework. Differences in time consumption and performance scores were analyzed. Finally, a pass/fail standard was established using the contrasting groups' method.

Results

Trainees used significantly more time per procedure (109 s, 95% CI: 99-120) than experts (82 s, 95% CI: 71-93; p < .001). Adjusted for repetition and rater leniency, experts achieved an average GBRS score of 18.8 (95% CI: 18.3-19.2) out of 20 points, whereas trainees achieved an average of 17.1 points (95% CI: 16.6-17.5; p < .001). In contrast to the task-specific checklist, the GBRS score discriminated between repetition number and participant experience. The pass/fail standard for the GBRS was established at 18.4 points.

Conclusion

We established educational validity evidence for a novel 3D-printed model for simulation-based training of ventilation tube insertion and established a reliable pass/fail standard.

Level of Evidence

1b.

Fundamentals of Microsurgery: A Novel Simulation Curriculum Based on Validated Laparoscopic Education Approaches

BACKGROUND

 Microsurgical techniques have a steep learning curve. We adapted validated surgical approaches to develop a novel, competency-based microsurgical simulation curriculum called Fundamentals of Microsurgery (FMS). The purpose of this study is to present our experience with FMS and quantify the effect of the curriculum on resident performance in the operating room.

METHODS

 Trainees underwent the FMS curriculum requiring task progression: (1) rubber band transfer, (2) coupler tine grasping, (3) glove laceration repair, (4) synthetic vessel anastomosis, and (5) vessel anastomosis in a deep cavity. Resident anastomoses were also evaluated in the operative room with the Stanford Microsurgery and Resident Training (SMaRT) tool to evaluate technical performance. The National Aeronautics and Space Administration Task Load Index (NASA-TLX) and Short-Form Spielberger State-Trait Anxiety Inventory (STAI-6) quantified learner anxiety and workload.

RESULTS

 A total of 62 anastomoses were performed by residents in the operating room during patient care. Higher FMS task completion showed an increased mean SMaRT score (p = 0.05), and a lower mean STAI-6 score (performance anxiety) (p = 0.03). Regression analysis demonstrated residents with higher SMaRT score had lower NASA-TLX score (mental workload) (p < 0.01) and STAI-6 scores (p < 0.01).

CONCLUSION

 A novel microsurgical simulation program FMS was implemented. We found progression of trainees through the program translated to better technique (higher SMaRT scores) in the operating room and lower performance anxiety on STAI-6 surveys. This suggests that the FMS curriculum improves proficiency in basic microsurgical skills, reduces trainee mental workload, anxiety, and improves intraoperative clinical proficiency.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.

Learning curves for point-of-care ultrasound image acquisition for novice learners in a longitudinal curriculum

BACKGROUND

A learning curve is graphical representation of the relationship between effort, such as repetitive practice or time spent, and the resultant learning based on specific outcomes. Group learning curves provide information for designing educational interventions or assessments. Little is known regarding the learning curves for Point-of-Care Ultrasound (POCUS) psychomotor skill acquisition of novice learners. As POCUS inclusion in education increases, a more thorough understanding of this topic is needed to allow educators to make informed decisions regarding curriculum design. The purpose of this research study is to: (A) define the psychomotor skill acquisition learning curves of novice Physician Assistant students, and (B) analyze the learning curves for the individual image quality components of depth, gain and tomographic axis.

RESULTS

A total of 2695 examinations were completed and reviewed. On group-level learning curves, plateau points were noted to be similar for abdominal, lung, and renal systems around 17 examinations. Bladder scores were consistently good across all exam components from the start of the curriculum. For cardiac exams, students improved even after 25 exams. Learning curves for tomographic axis (angle of intersection of the ultrasound with the structure of interest) were longer than those for depth and gain. Learning curves for axis were longer than those for depth and gain.

CONCLUSION

Bladder POCUS skills can be rapidly acquired and have the shortest learning curve. Abdominal aorta, kidney, and lung POCUS have similar learning curves, while cardiac POCUS has the longest learning curve. Analysis of learning curves for depth, axis, and gain demonstrates that axis has the longest learner curve of the three components of image quality. This finding has previously not been reported and provides a more nuanced understanding of psychomotor skill learning for novices. Learners might benefit from educators paying particular attention to optimizing the unique tomographic axis for each organ system.

Prehospital Surgical Airway Management Skills in a Rural Emergency Medical Service System

BACKGROUND

The objective of this study is to describe the education, training, and use of prehospital surgical airways in a rural Emergency Medical Service (EMS) system.

MATERIALS AND METHODS

We conducted an internet-based survey instrument of all advanced life support (ALS) EMS agencies in a seven-county rural EMS system in Pennsylvania. ALS agencies were queried regarding basic demographic information as well as the number of surgical airways performed in the previous 10 years as well as the education and training of EMS providers in surgical airways.

RESULTS

The survey was completed by 11 of 20 ALS EMS agencies in our region (55% rate of return). The content and frequency of training varied considerably among EMS agencies. Only four prehospital surgical airways were performed during the study period. One patient survived to hospital discharge to home.

CONCLUSION

Surgical airways are an infrequently performed procedure in the rural prehospital setting. There is no universally accepted standard for teaching or evaluating the competency of this potentially life-saving procedure. Further efforts to establish a core educational curriculum appear warranted.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery

Importance

Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.

Objective

To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.

Design, Setting, and Participants

This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities.

Exposures

Longitudinal SIMPL ratings.

Main Outcomes and Measures

Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year.

Results

Using 63 248 SIMPL ratings, the association between prior and future performance was positive (β, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated.

Conclusions and Relevance

In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

Coaching the coach: A randomized controlled study of a novel curriculum for procedural coaching during intubation

Background

Videolaryngoscopy allows real-time procedural coaching during intubation. This study sought to develop and assess an online curriculum to train pediatric emergency medicine attending physicians to deliver procedural coaching during intubation.

Methods

Curriculum development consisted of semistructured interviews with 12 pediatric emergency medicine attendings with varying levels of airway expertise analyzed using a constructivist grounded theory approach. Following development, the curriculum was implemented and assessed through a multicenter randomized controlled trial enrolling participants in one of three cohorts: the coaching module, unnarrated video recordings of intubations, and a module on ventilator management. Participants completed identical pre and post assessments asking them to select the correct coaching feedback and provided reactions for qualitative thematic analysis.

Results

Content from interviews was synthesized into a video-enhanced 15-min online coaching module illustrating proper technique for intubation and strategies for procedural coaching. Eighty-seven of 104 randomized physicians enrolled in the curriculum; 83 completed the pre and post assessments (80%). The total percentage correct did not differ between pre and post assessments for any cohort. Participants receiving the coaching module demonstrated improved performance on patient preparation, made more suggestions for improvement, and experienced a greater increase in confidence in procedural coaching. Qualitative analysis identified multiple benefits of the module, revealed that exposure to video recordings without narration is insufficient, and identified feedback on suggestions for improvement as an opportunity for deliberate practice.

Conclusions

This study leveraged clinical and educational digital technology to develop a curriculum dedicated to the content expertise and coaching skills needed to provide feedback during intubations performed with videolaryngoscopy. This brief curriculum changed behavior in simulated coaching scenarios but would benefit from additional support for deliberate practice.

Learning insertion of a Resuscitative Endovascular Balloon Occlusion of the Aorta (REBOA) catheter: Is clinical experience necessary? A prospective trial

BACKGROUND

Resuscitative endovascular balloon occlusion of the aorta (REBOA) is an emerging and potentially life-saving procedure, necessitating qualified operators in an increasing number of centres. The procedure shares technical elements with other vascular access procedures using the Seldinger technique, which is mastered by doctors not only in endovascular specialties but also in trauma surgery, emergency medicine, and anaesthesiology. We hypothesised that doctors mastering the Seldinger technique (experienced anaesthesiologist) would learn the technical aspects of REBOA with limited training and remain technically superior to doctors unfamiliar with the Seldinger technique (novice residents) given similar training.

METHODS

This was a prospective trial of an educational intervention. Three groups of doctors were enroled: novice residents, experienced anaesthesiologists, and endovascular experts. The novices and the anaesthesiologists completed 2.5 h of simulation-based REBOA training. Their skills were tested before and 8-12 weeks after training using a standardised simulated scenario. The endovascular experts, constituting a reference group, were equivalently tested. All performances were video recorded and rated by three blinded experts using a validated assessment tool for REBOA (REBOA-RATE). Performances were compared between groups and with a previously published pass/fail cutoff.

RESULTS

Sixteen novices, 13 board-certified specialists in anaesthesiology, and 13 endovascular experts participated. Before training, the anaesthesiologists outperformed the novices by 30 percentage points of the maximum REBOA-RATE score (56% (SD 14.0) vs 26% (SD 17%), p<0.01). After training, there was no difference in skills between the two groups (78% (SD 11%) vs 78 (SD 14%), p = 0.93). Neither group reached the endovascular experts' skill level (89% (SD 7%), p<0.05).

CONCLUSION

For doctors mastering the Seldinger technique, there was an initial inter-procedural transfer of skills advantage when performing REBOA. However, after identical simulation-based training, novices performed equally well to anaesthesiologists, indicating that vascular access experience is not a prerequisite to learning the technical aspects of REBOA. Both groups would need more training to reach technical proficiency.

Influence of practical and clinical experience on dexterity performance measured using haptic virtual reality simulator

INTRODUCTION

Development of dexterity, hand-eye coordination and self-assessment are essential during the preclinical training of dental students. To meet this requirement, dental simulators have been developed combining virtual reality with a force feedback haptic interface. The aim of this study was to assess the capability of the VirTeaSy^© haptic simulator to discriminate between users with different levels of practical and clinical experience.

MATERIALS AND METHODS

Fifty-six volunteers divided into five groups (non-dentists, 1st/3rd/final-year dental students, recent graduates) had three attempts to prepare an occlusal amalgam cavity using the simulator. Percentages of volumes prepared inside (%IV) and outside (%OV) the required cavity, skill index and progression rate, referring to the evolution of skill index between trials 1 and 3, were assessed. The dental students and recent graduates completed a questionnaire to gather their opinions about their first hands-on experience with a haptic simulator.

RESULTS

The results showed no significant difference between the groups at the first attempt. Following the third attempt, the skill index was improved significantly. Analysis of progression rates, characterised by large standard deviations, did not reveal significant differences between groups. The third attempt showed significant differences in skill index and %IV between 1st-year undergraduate dental students and both non-dentists and recent dental graduates. The questionnaire indicated a tendency for dental operators to consider the simulator as a complement to their learning and not a substitute for traditional methods.

CONCLUSION

This study did not show the ability of a basic aptitude test on VirTeaSy^© haptic simulator to discriminate between users of different levels of expertise. Optimisations must be considered in order to make simulation-based assessment clinically relevant.

Longitudinal assessment of competency development at The Ohio State University using the competency-based veterinary education (CBVE) model

With the development of the American Association of Veterinary Medical Colleges' Competency-Based Veterinary Education (CBVE) model, veterinary schools are reorganizing curricula and assessment guidelines, especially within the clinical rotation training elements. Specifically, programs are utilizing both competencies and entrustable professional activities (EPAs) as opportunities for gathering information about student development within and across clinical rotations. However, what evidence exists that use of the central tenets of the CBVE model (competency framework, milestones and EPAs) improves our assessment practices and captures reliable and valid data to track competency development of students as they progress through their clinical year? Here, we report on validity evidence to support the use of scores from in-training evaluation report forms (ITERs) and workplace-based assessments of EPAs to evaluate competency progression within and across domains described in the CBVE, during the final year clinical training period of The Ohio State University's College of Veterinary Medicine (OSU-CVM) program. The ITER, used at the conclusion of each rotation, was modified to include the CBVE competencies that were assessed by identifying the stage of student development on a series of descriptive milestones (from pre-novice to competent). Workplace based assessments containing entrustment scales were used to assess EPAs from the CBVE model within each clinical rotation. Competency progression and entrustment scores were evaluated on each of the 31 rotations offered and high-stakes decisions regarding student performance were determined by a collective review of all the ITERs and EPAs recorded for each learner across each semester and the entire year. Results from the class of 2021, collected on approximately 190 students from 31 rotations, are reported with more than 55 299 total competency assessments combined with milestone placement and 2799 complete EPAs. Approximately 10% of the class was identified for remediation and received additional coaching support. Data collected longitudinally through the ITER on milestones provides initial validity evidence to support using the scores in higher stakes contexts such as identifying students for remediation and for determining whether students have met the necessary requirements to successfully complete the program. Data collected on entrustment scores did not, however, support such decision making. Implications are discussed.

Evaluation of an advanced critical care echocardiography program: a mixed methods study

PURPOSE

Limited data exist on advanced critical care echocardiography (CCE) training programs for intensivists. We sought to describe a longitudinal echocardiography program and investigate the effect of distributed conditional supervision vs predefined en-bloc supervision, as well as the effect of an optional echocardiography laboratory rotation, on learners' engagement.

METHODS

In this mixed methods study, we enrolled critical care fellows and faculty from five University of Toronto-affiliated intensive care units (ICU) between July 2015 and July 2018 in an advanced training program, comprising theoretical lectures and practical sessions. After the first year, the program was modified with changes to supervision model and inclusion of a rotation in the echo laboratory. We conducted semistructured interviews and investigated the effects of curricular changes on progress toward portfolio completion (150 transthoracic echocardiograms) using a Bayesian framework.

RESULTS

Sixty-five learners were enrolled and 18 were interviewed. Four (9%) learners completed the portfolio. Learners reported lack of time and supervision, and skill complexity as the main barriers to practicing independently. Conditional supervision was associated with a higher rate of submitting unsupervised echocardiograms than unconditional supervision (rate ratio, 1.11, 95% credible interval, 1.08 to 1.14). After rotation in the echocardiography laboratory, submission of unsupervised echocardiograms decreased.

CONCLUSION

Trainees perceived lack of time and limited access to supervision as major barriers to course completion. Nevertheless, successful portfolio completion was related to factors other than protected time in the echocardiography laboratory or unconditional direct supervision in ICU. Further research is needed to better understand the factors promoting success of CCE training programs.

The Case Against Stapedectomy as a Core Competency for General Practice: A Study of Care Patterns in the Department of Defense

OBJECTIVE

Stapedectomy remains a key indicator case reportable to the Accreditation Council of Graduate Medical Education despite the decline in the incidence of otosclerosis over the last half century. This study compared the rates of stapedectomy performed by otolaryngologists at academic and nonacademic centers.

STUDY DESIGN

Retrospective review.

SETTING

Tertiary referral academic centers, nonacademic centers, and civilian purchased-care across the Department of Defense between 2015 and 2020.

PATIENTS

Department of Defense beneficiaries with otosclerosis near a military treatment facility with an otolaryngologist.

INTERVENTIONS

Stapedectomy (Current Procedural Terminology codes 69,660, 69,661, and 69,662).

MAIN OUTCOME MEASURES

Number of stapedectomies performed by setting.

RESULTS

From 2015 to 2020, 426 stapedectomies were performed at or near a military treatment facility with an otolaryngologist (274 directly by military otolaryngologists, 152 by community providers). Military providers performed 94% of stapedectomies at or near military academic centers, versus only 30% at or near nonacademic centers ( p < 0.0001). Among the 60 stapedectomies performed at nonacademic centers, only 30 were performed by general otolaryngologists (7% of all stapedectomies performed; 11% of procedures by military providers) while the rest were performed by fellowship-trained otologist or neurotologist.

CONCLUSIONS

Low stapes surgical volume by military general otolaryngologists reinforces recent epidemiologic trends and suggests that few general otolaryngologists graduate residency with sufficient competency to pursue independently performing stapedectomy or have difficulty maintaining competency after graduation.

Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education

Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.

Determining the learning curve for a novel microsurgical procedure using histopathology

PURPOSE

Wet laboratories are becoming an increasingly important training tool as part of a push to a proficiency-based training model. We created a microsurgical wet laboratory to investigate the utility of histopathology use in assessing surgical outcomes and determine the learning curve of a novel microsurgical procedure.

METHODS

A microsurgical wet laboratory was established using pig eyes to simulate the human cornea. Three novice surgeons and an experienced surgeon performed an anterior cornea lamellar dissection and the duration of the procedure was recorded. With the aid of histological analysis, the thickness and characteristics of the dissected graft was recorded. The number of attempts to complete the experiment, defined as three successful dissections with mean thickness below 100 μm, was documented.

RESULTS

The use of histopathology was highly successful allowing in-depth analysis of the dissected graft for each attempt. Trainees reached the endpoint of the study in 21, 26 and 36 attempts (mean: 28 attempts) whilst the corneal surgeon completed the experiment in 12 attempts (p = 0.07). Mean dissection thickness decreased over time for all participants. The mean dissection time for trainees was 10.6 ± 4.2 min compared to the corneal surgeon with a mean of 8.2 ± 3.1 min (p = 0.03).

CONCLUSION

We propose a corneal wet laboratory model that allows for simple, efficient, and flexible microsurgical training. The use of histopathological analysis allows for careful graft analysis, providing objective feedback throughout the training exercise. Trainees demonstrated improvements in the three key aspects of the procedure: accuracy as evidenced by decreasing histological thickness, confidence by self-report and fluidity by decreasing duration of the procedure.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

IMPORTANCE

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

OBJECTIVE

To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

DESIGN, SETTING, AND PARTICIPANTS

This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

INTERVENTIONS

Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

MAIN OUTCOMES AND MEASURES

To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

RESULTS

This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

CONCLUSIONS AND RELEVANCE

These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Percutaneous transhepatic drainage is safe and effective in biliary obstruction-A single-center experience of 599 patients

BACKGROUND

Historically, surgical bilioenteric bypass was the only treatment option for extrahepatic bile duct obstruction, but with technological advancements, percutaneous transhepatic drainage (PTD) and endoscopic solutions were introduced as a less invasive alternative. Endoscopic methods may lead to a decreasing indication of PTD in the future, but today it is still the standard treatment method, especially in hilar obstructions.

METHODS

In our retrospective data analysis, we assessed technical success rate, reintervention rate, morbidity, mortality, and the learning curve of patients treated with PTD over 12 years in a tertiary referral center.

RESULTS

599 patients were treated with 615 percutaneous interventions. 94.5% (566/599) technical success rate; 2.7% (16/599) reintervention rate were achieved. 111 minor and 22 major complications occurred including 1 case of death. In perihilar obstruction, cholangitis were significantly more frequent in cases where endoscopic retrograde cholangiopancreatography had also been performed prior to PTD compared to PTD alone, with 39 (18.2%) and 15 (10.5%) occurrences, respectively.

DISCUSSION

The results and especially the excellent success rates demonstrate that PTD is safe and effective, and it is appropriate for first choice in the treatment algorithm of perihilar stenosis. Ultimately, we concluded that PTD should be performed in experienced centers to achieve low mortality, morbidity, and high success rates.

The Equivalency of the Binocular Indirect Ophthalmoscope Simulator to Peer Practice: A Pilot Study

SIGNIFICANCE

Optometric educators are constantly looking for learning and teaching approaches to improve clinical skills training. In addition, the COVID-19 pandemic has made educators scrutinize the time allocated to face-to-face teaching and practice. Simulation learning is an option, but its use must first be evaluated against traditional learning methods.

PURPOSE

The purpose of this study was to compare the training of binocular indirect ophthalmoscopy skills achieved by students and optometrists through deliberate practice on the Eyesi Indirect Ophthalmoscope simulator with deliberate practice using a peer.

METHODS

Students and optometrists were randomly allocated to practice on either the simulator or a peer. Binocular indirect ophthalmoscopy performance was assessed using a peer and the simulator as the patients at different time points. Knowledge and confidence were examined before and following all practice sessions using a quiz and survey.

RESULTS

Significant improvements in binocular indirect ophthalmoscopy performance using either a peer or the simulator as the patient for assessment were seen after 8 hours of student practice (P < .001) and after a half-hour practice time for optometrists (P < .001). There was no significant difference in performance overall between those practicing on a simulator and those practicing on a peer (P > .05). Confidence in ability to perform the technique was lower for students who had practiced on the simulator.

CONCLUSIONS

The simulator has similar efficacy to peer practice for teaching binocular indirect ophthalmoscopy to students and maintenance of this clinical skill for optometrists. Simulation does not replace the need for peer practice but may be a useful adjunct reducing the face-to-face hours required. These findings present a need for further research regarding diverse applications of the Eyesi Indirect Ophthalmoscope simulator in the curriculum for training optometry students and as a continuing professional development offering for optometrists, given the short exposure duration required to observe a significant improvement in skill.

Using Individual Residents' Learning Trajectories to Better Understand the Impact of Gaps in Practice

PURPOSE

To optimize learning, health professional training programs need to achieve the right balance between depth of practice (gaining more experience with particular skills) and breadth of practice (spreading experience across an array of activities). Better understanding how training for a particular skill set is impacted by periods of focus on a different skill set would allow improved curriculum and assessment design, thereby enhancing the efficiency of training and effectiveness of care. To this end, learning curves were used to compare performance in surgery after prolonged periods of practice to performance after gaps in surgical training.

METHOD

Daily operative assessments from the Dalhousie obstetrics and gynecology program were analyzed retrospectively and learning curves were generated. In addition to examining the variability in learning trajectories, the impact of gaps was systematically assessed by comparing resident scores after 2 successive months in which they were not assessed operatively to those collected after 2 successive months in which they were assessed at least once.

RESULTS

Four thousand four hundred sixteen scores for 33 residents over a 10-year period were analyzed. Trajectories and peak performances were identified. Residents performed better during their third sequential month of being assessed (mean = 4.40, 95% CI = 4.33-4.46) relative to during months following a period of being away from the operating room for at least 2 months (mean = 4.21, 95% CI = 4.13-4.29; P < .01; d = 0.7). However, maximum performance achieved was more strongly related to the number of times residents experienced a gap in training (r = 0.50) than to the number of times residents experienced 3 consecutive months of training (r = 0.25).

CONCLUSIONS

Distinct patterns of development exist for individual residents. Time away from surgical practice and assessment negatively impacted short-term performance, but may improve long-term learning trajectories. This speaks to the value of spaced education and is important for the design of longitudinal skills-based training programs.

Cost-Utility Analysis of a Medication Adherence Management Service Alongside a Cluster Randomized Control Trial in Community Pharmacy

BACKGROUND

It is necessary to determine the cost utility of adherence interventions in chronic diseases due to humanistic and economic burden of non-adherence.

PURPOSE

To evaluate, alongside a cluster-randomized controlled trial, the cost-utility of a pharmacist-led medication adherence management service (MAMS) compared with usual care in community pharmacies.

MATERIALS AND METHODS

The trial was conducted over six months. Patients with treatments for hypertension, asthma or chronic obstructive pulmonary disease (COPD) were included. Patients in the intervention group (IG) received a MAMS based on a brief complex intervention, whilst patients in the control group (CG) received usual care. The cost-utility analysis adopted a health system perspective. Costs related to medications, healthcare resources and adherence intervention were included. The effectiveness was estimated as quality-adjusted life years (QALYs), using a multiple imputation missing data model. The incremental cost-utility ratio (ICUR) was calculated on the total sample of patients.

RESULTS

A total of 1186 patients were enrolled (IG: 633; CG: 553). The total intervention cost was estimated to be €27.33 ± 0.43 per patient for six months. There was no statistically significant difference in total cost of medications and healthcare resources per patient between IG and CG. The values of EQ-5D-5L at 6 months were significantly higher in the IG [IG: 0.881 ± 0.005 vs CG: 0.833 ± 0.006; p = 0.000]. In the base case, the service was more expensive and more effective than usual care, resulting in an ICUR of €1,494.82/QALY. In the complete case, the service resulted in an ICUR of €2,086.30/QALY, positioned between the north-east and south-east quadrants of the cost-utility plane. Using a threshold value of €20,000/QALY gained, there is a 99% probability that the intervention is cost-effective.

CONCLUSION

The medication adherence management service resulted in an improvement in the quality of life of the population with chronic disease, with similar costs compared to usual care. The service is cost-effective.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.