Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education

Inter-school variations in the standard of examiners' graduation-level OSCE judgements

INTRODUCTION

Ensuring equivalence in high-stakes performance exams is important for patient safety and candidate fairness. We compared inter-school examiner differences within a shared OSCE and resulting impact on students' pass/fail categorisation.

METHODS

The same 6 station formative OSCE ran asynchronously in 4 medical schools, with 2 parallel circuits/school. We compared examiners' judgements using Video-based Examiner Score Comparison and Adjustment (VESCA): examiners scored station-specific comparator videos in addition to 'live' student performances, enabling 1/controlled score comparisons by a/examiner-cohorts and b/schools and 2/data linkage to adjust for the influence of examiner-cohorts. We calculated score impact and change in pass/fail categorisation by school.

RESULTS

On controlled video-based comparisons, inter-school variations in examiners' scoring (16.3%) were nearly double within-school variations (8.8%). Students' scores received a median adjustment of 5.26% (IQR 2.87-7.17%). The impact of adjusting for examiner differences on students' pass/fail categorisation varied by school, with adjustment reducing failure rate from 39.13% to 8.70% (school 2) whilst increasing failure from 0.00% to 21.74% (school 4).

DISCUSSION

Whilst the formative context may partly account for differences, these findings query whether variations may exist between medical schools in examiners' judgements. This may benefit from systematic appraisal to safeguard equivalence. VESCA provided a viable method for comparisons.

The nature of a specialty

Myers unpacks the defining features of a specialty and their implications for Clinical Competency Committees.

Current approaches in development and implementation of medical education strategies among pediatric hematology/oncology fellowship programs

Application of evidenced-based practices is often lacking in medical education. Increased recognition that methods of knowledge delivery based in relevant learning theories are more effective has led to improved ability to meet the needs of adult learners. We designed a study to assess approaches to education for Pediatric Hematology/Oncology (PHO) trainees and identify opportunities for incorporation of evidence-based practices. A national survey was shared with program directors (PDs) of PHO programs to investigate current trends in education design. Respondents were contacted via email, the survey was distributed by REDCap, and data were collected from April-July 2023. Quantitative data were analyzed descriptively and free-text responses using directed content analysis. Of the 77 eligible participants, 46 completed the survey (59.74% response rate), representing various geographic regions and class sizes. Respondents reported a wide range of familiarity with adult learning theory (range 1-5, median 3) and self-regulated learning (range 1-5, median 2). While programs employed active learning strategies to varying degrees, the lecture format was the most used method of education delivery. PDs recognized the need for development in several domains, including incorporation of educational frameworks, support for structure and content of didactics, and implementation of feedback related to education sessions. Differences in approaches were described, revealing a wide range of familiarity with and application of specific educational theories, as well as inconsistent efforts to consider principles of adult learning theory and self-regulated learning. These data demonstrate opportunities for targeted education and curriculum development.

Conceptualizing and Developing Competence in Newborn Medicine Among Military Pediatricians

INTRODUCTION

Competence in neonatal care is especially important for military pediatricians because military pediatricians can be asked to serve in remote duty locations with limited resources. We sought to understand how this competence is defined, developed, and assessed by military pediatric training programs.

MATERIALS AND METHODS

After Institutional Review Board approval was obtained, we interviewed educators and recent graduates from every pediatric military training program to construct a shared definition of competence. We then used Kern's Six Steps for curriculum development to understand how competence is taught and assessed.

RESULTS

Participants felt that competence for military pediatricians in the neonatal setting meant that learners should be able to provide a full spectrum of newborn care in any military setting. Participants confirmed that this competence was particularly important for military pediatricians because of the possibility of remote duty locations. Participants felt that specific knowledge, skills, and attitudes supported competence. Knowledge domains include distinguishing normal newborns from abnormal newborns, managing normal newborn care, managing common newborn abnormalities, and creating a safe escalation plan for complicated or uncommon newborn abnormalities. Specific skills that support competence are newborn resuscitation, delivery of effective ventilation, and neonatal circumcision. Specific attitudes that support competence are, understanding the personal limits of knowledge and understanding the resources for escalation of care. Educators use a variety of modalities to teach toward competence, including the structured curricula, bedside teaching, and simulation. According to participants, the assessment of learners occurs primarily through narrative assessment and feedback but would ideally occur through direct observation.

CONCLUSIONS

Competence in the neonatal setting is particularly important for military pediatricians. Essential skills undergo differential assessment and current assessment methods differ from ideal assessment methods. Future work should focus on how these facets can support a unified curriculum in newborn medicine.

Supporting Patient Involvement in U.S. Medical Education Through Changes in Accreditation

For over half of a century, there have been calls for greater patient and community involvement in U.S. medical education. Accrediting agencies, as the regulatory authorities for medical education, develop policies that impact every program in the U.S.; they have the ability to support patient involvement across the medical education system. In this article, we first review the requirements of U.S. accrediting agencies for undergraduate and graduate medical education to involve patients in educational programs. While agencies have patient members on their committees, they do little to encourage patient involvement through their standards or procedures. We then describe opportunities for accreditation to support patient involvement across teaching and learning activities, curriculum design and evaluation, policymaking and governance, and scholarly endeavors. We link these opportunities to specific standards that could be revised or have their data reporting requirements adjusted. U.S. agencies could also follow the examples of their counterparts outside the U.S., which have created new standards to encourage patient involvement. Ensuring patient representation on educational programs' governing and policymaking bodies is one among many immediate actions that could be taken by accrediting authorities to encourage system-level reforms. As medical school and residency training represent the beginnings of decades of practice for physicians, properly involving patients would maximize benefits for learners, educators, and society.

Variation in Case Exposure During Internal Medicine Residency

Importance

Variation in residency case exposure affects resident learning and readiness for future practice. Accurate reporting of case exposure for internal medicine (IM) residents is challenging because feasible and reliable methods for linking patient care to residents are lacking.

Objective

To develop an integrated education-clinical database to characterize and measure case exposure variability among IM residents.

Design, Setting, and Participants

In this cohort study, an integrated educational-clinical database was developed by linking patients admitted during overnight IM in-hospital call shifts at 5 teaching hospitals to senior on-call residents. The senior resident, who directly cares for all overnight IM admissions, was linked to their patients by the admission date, time, and hospital. The database included IM residents enrolled between July 1, 2010, and December 31, 2019, in 1 Canadian IM residency. Analysis occurred between August 1, 2023, and June 30, 2024.

Main Outcomes and Measures

Case exposure was defined by patient demographic characteristics, discharge diagnoses, volumes, acuity (eg, critical care transfer), medical complexity (eg, Charlson Comorbidity Index), and social determinants of health (eg, from long-term care). Residents were grouped into quartiles for each exposure measure, and the top and bottom quartiles were compared using standardized mean difference (SMD). Variation between hospitals was evaluated by calculating the SMD between the hospitals with the highest and lowest proportions for each measure. Variation over time was assessed using linear and logistic regression.

Results

The integrated educational-clinical database included 143 632 admissions (median [IQR] age, 71 [55-83] years; 71 340 [49.7%] female) linked to 793 residents (median [IQR] admissions per shift, 8 [6-12]). At the resident level, there was substantial variation in case exposure for demographic characteristics, diagnoses, volumes, acuity, complexity, and social determinants. For example, residents in the highest quartile had nearly 4 times more admissions requiring critical care transfer compared with the lowest quartile (3071 of 30 228 [10.2%] vs 684 of 25 578 [2.7%]; SMD, 0.31). Hospital-level variation was also significant, particularly in patient volumes (busier hospital vs less busy hospital: median [IQR] admissions per shift, 10 [8-12] vs 7 [5-9]; SMD, 0.96). Over time, residents saw more median (IQR) admissions per shift (2010 vs 2019: 7.6 [6.6-8.4] vs 9.0 [7.6-10.0]; P = .04) and more complex patients (2010 vs 2019: Charlson Comorbidity Index ≥2, 3851 of 13 762 [28.0%] vs 2862 of 8188 [35.0%]; P = .03), while working similar shifts per year (median [IQR], 11 [8-14]).

Conclusions

In this cohort study of IM residents in a Canadian residency program, significant variation in case exposure was found between residents, across sites, and over time.

When words are your scalpel, what and how information is exchanged may be differently salient to assessors

PURPOSE

Variable assessments of learner performances can occur when different assessors determine different elements to be differently important or salient. How assessors determine the importance of performance elements has historically been thought to occur idiosyncratically and thus be amenable to assessor training interventions. More recently, a main source of variation found among assessors was two underlying factors that were differently emphasised: medical expertise and interpersonal skills. This gave legitimacy to the theory that different interpretations of the same performance may represent multiple truths. A faculty development activity introducing assessors to entrustable professional activities in which they estimated a learner's level of readiness for entrustment provided an opportunity to qualitatively explore assessor variation in the context of an interaction and in a setting in which interpersonal skills are highly valued.

METHODS

Using a constructivist grounded theory approach, we explored variation in assessment processes among a group of palliative medicine assessors who completed a simulated direct observation and assessment of the same learner interaction.

RESULTS

Despite identifying similar learner strengths and areas for improvement, the estimated level of readiness for entrustment varied substantially among assessors. Those who estimated the learner as not yet ready for entrustment seemed to prioritise what information was exchanged and viewed missed information as performance gaps. Those who estimated the learner as ready for entrustment seemed to prioritise how information was exchanged and viewed the same missed information as personal style differences or appropriate clinical judgement. When presented with a summary, assessors expressed surprise and concern about the variation.

CONCLUSION

A main source of variation among our assessors was the differential salience of performance elements that align with medical expertise and interpersonal skills. These data support the theory that when assessing an interaction, differential salience for these two factors may be an important and perhaps inevitable source of assessor variation.

Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients

Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

A multispecialty ethnographic study of clinical competency committees (CCCs)

PURPOSE

Clinical competency committees (CCCs) assess residents' performance on their specialty specific milestones, however there is no 'one-size fits all' blueprint for accomplishing this. Thus, CCCs have had to develop their own procedures. The goal of this study was to examine these efforts to assist new programs embarking on this venture and established programs looking to improve their CCC practices and processes.

METHODS

We purposefully sampled CCCs across multiple specialties and institutions. Data from three sources were triangulated: (1) online demographic survey, (2) ethnographic observations of CCC meetings and (3) post-observation semi-structured interviews with the program director and/or CCC chairperson. Template analysis was used to build the coding structure.

RESULTS

Sixteen observations were completed with 15 different CCCs at 9 institutions. Three main thematic categories that impact the operations of CCCs emerged: (1) Membership structure and members roles, (2) Roles of the CCC in residency and 3) CCC processes, including trainee presentation to the committee and decision-making. While effective practices were observed, substantial variation existed in all three thematic areas.

CONCLUSIONS

While CCCs used some known effective practices, substantial variation in structure and processes was notable across CCCs. Future work should explore the impact of this variation on educational outcomes.

Some Things Change, Some Things Stay the Same: Trends in Canadian Education in Paediatric Cardiology and the Cardiac Sciences

Education in paediatric cardiology has evolved along with clinical care. The availability and application of new technologies in education, in particular, have had a significant impact. Artificial intelligence; virtual, augmented, and mixed reality learning tools; and gamification of learning have all resulted in new opportunities for today's trainees compared with those of the past. A new training model is also being used. Though currently focused on residency education, competency-based medical education is also being applied to undergraduate education in some Canadian medical schools. Competency-based medical education offers a more transparent relationship between education and physicians' social contract with society. It provides greater accountability for programmes and learners to teach and learn the skills required to function as competent specialists. However, it has not come without challenges. Coincident with the application of this model for learners, there has been increased educational accountability for physicians in practice and for the institutions training them. Despite these changes, some things have remained the same. On the positive side, the importance of good clinical teachers to effective learning remains constant. Unfortunately, the mistreatment of learners within our education system also remains and is perhaps the most important challenge facing medical education in Canada today. Learning to be better teachers and learner advocates is an important goal for all of those involved in educating Canadian medical learners.