Competency-Based Frameworks in Medical School Education Programs: A Thematic Analysis of the Academic Medicine Snapshots, 2020

How Internal Medicine Clerkship Directors Are Using Entrustable Professional Activities: A National Survey Study

BACKGROUND

In the present assessment environment in undergraduate medical education at U.S. medical schools, the prevalence and implementation of Entrustable Professional Activities (EPAs) in internal medicine (IM) clerkships are not well understood.

OBJECTIVE

To describe the prevalence and approach to EPA use in U.S. IM clerkships.

DESIGN

Cross-sectional, nationally representative survey of core IM clerkship directors.

PARTICIPANTS

One-hundred forty IM clerkship directors at Liaison Committee on Medical Education-accredited U.S./U.S. territory-based allopathic medical schools with membership in the Clerkship Directors in Internal Medicine (CDIM) as of December 2022.

MAIN MEASURES

Use of EPAs in IM core clerkships, including use for grading, types of EPAs, use of supportive measures for assessment, and current validity frameworks.

KEY RESULTS

The survey response was 80% (112/140); two additional respondents completed the section on EPA use (n = 114). Approximately half of respondents (47%) reported their IM clerkship used EPAs. Among schools accredited after 1977, a higher percentage was associated with having incorporated EPAs (p = 0.03). The Association of American Medical Colleges Core EPAs for Entering Residency (CEPAER) was the most common framework used by Clerkship Directors (CDs) for developing EPAs (55%). Most CDs (56%) used EPAs for both formative and summative assessments, and approximately half of CDs (48%) used EPAs for a portion of the final grade determination. CDs who used EPAs were no more likely to report efforts to ensure the validity of assessment, the use of faculty development, or that written assessments were a valid measure of students' performance compared to those who did not use EPAs.

CONCLUSIONS

Although EPAs have experienced substantial uptake in the IM clerkship and contribute to formative and summative assessment of learners, their use does not appear to be associated with enhanced efforts to obtain validity information.

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study

BACKGROUND

Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs.

METHODS

The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.

RESULTS

Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.

CONCLUSION

The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

Core sepsis-related competencies for medical students: an international consensus by Delphi technique

BACKGROUND

Sepsis is a life-threatening condition which may arise from infection in any organ system and requires early recognition and management. Healthcare professionals working in any specialty may need to manage patients with sepsis. Educating medical students about this condition may be an effective way to ensure all future doctors have sufficient ability to diagnose and treat septic patients. However, there is currently no consensus on what competencies medical students should achieve regarding sepsis recognition and treatment. This study aims to outline what sepsis-related competencies medical students should achieve by the end of their medical student training in both high or upper-middle incomes countries/regions and in low or lower-middle income countries/regions.

METHODS

Two separate panels from high or upper-middle income and low or lower-middle income countries/regions participated in a Delphi method to suggest and rank sepsis competencies for medical students. Each panel consisted of 13-18 key stakeholders of medical education and doctors in specialties where sepsis is a common problem (both specialists and trainees). Panelists came from all continents, except Antarctica.

RESULTS

The panels reached consensus on 38 essential sepsis competencies in low or lower-middle income countries/regions and 33 in high or upper-middle incomes countries/regions. These include competencies such as definition of sepsis and septic shock and urgency of antibiotic treatment. In the low or lower-middle income countries/regions group, consensus was also achieved for competencies ranked as very important, and was achieved in 4/5 competencies rated as moderately important. In the high or upper-middle incomes countries/regions group, consensus was achieved in 41/57 competencies rated as very important but only 6/11 competencies rated as moderately important.

CONCLUSION

Medical schools should consider developing curricula to address essential competencies, as a minimum, but also consider addressing competencies rated as very or moderately important.

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

PURPOSE

The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM).

METHOD

Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings.

RESULTS

During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS).

CONCLUSIONS

The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Residents need competence not confidence: A retrospective evaluation of the new competency education program for Korean neurology residents

The objective of our study was to scrutinize the learning experiences of Korean neurology residents, with an emphasis on the implications of the novel competency-based curriculum implemented in 2021. We hypothesized that this revised curriculum could modulate residents' cognitive conduct, primarily the manifestation of overconfidence, in distinctive ways across different stages of training. Our investigative framework was three-fold. Initially, we began with a qualitative inquiry involving in-depth interviews with a purposively selected cohort of eight residents from four training sites. This approach facilitated comprehensive insight into their perceptions of their competence and confidence across the continuum of a four-year residency program. Subsequently, we incorporated the K-NEPA13 assessment instrument, administered to the residents and their overseeing supervisors. This stage aimed to dissect potential cognitive biases, particularly overconfidence and consistency, within the resident population. The final study involved a comprehensive survey administered to a group of 97 Korean neurology residents, allowing us to consolidate and validate our preceding findings. Our findings revealed that junior residents portrayed heightened confidence in their clinical capabilities compared to their senior peers. Intriguingly, junior residents also displayed a stronger inclination towards reevaluating their clinical judgments, a behavior we hypothesize is stimulated by the recently introduced competency-based curriculum. We identified cognitive divergence between junior and senior residents, with the latter group favoring more consistent and linear cause-and-effect reasoning, while the former demonstrated receptiveness to introspection and reconsideration. We speculate this adaptability might be engendered by the supervisor assignment protocol intrinsic to the new curriculum. Our study highlights the essentiality of incorporating cognitive behaviors when devising medical education strategies. Acknowledging and addressing these diverse cognitive biases, and instilling a spirit of adaptability, can nurture a culture that persists in continuous learning and self-reflection among trainee doctors.