Shared Mental Models Among Clinical Competency Committees in the Context of Time-Variable, Competency-Based Advancement to Residency

EMYWAY Workplace-Based Entrustable Professional Activities Assessments in Otolaryngology Residency Training: A Nationwide Experience

OBJECTIVE

To present workplace-based entrustable professional activities (EPAs) assessment data from the first 2 years of the EMYWAY platform in otolaryngology residency training in Taiwan.

STUDY DESIGN

Two-year cross-sectional study.

SETTING

Otolaryngology training programs.

METHODS

In 2020, the Taiwan Society of Otorhinolaryngology-Head and Neck Surgery (TSO-HNS) developed a workplace-based assessment (WBA) framework with 11 EPAs, integrating milestones to evaluate resident competency. In 2021, TSO-HNS piloted the EMYWAY platform for WBAs, which includes an EPA-based assessment workflow, coaching feedback, and a dashboard displaying residents' entrustment-supervision levels. Data are analyzed annually for accreditation and curriculum enhancement. This study reports on the pilot year and the first full-scale year of implementation.

RESULTS

Eleven programs participated in the pilot year. Subsequently, 362 faculty members and 274 resident physicians from 34 programs nationwide engaged with EMYWAY. In the full-scale year from August 2022 to July 2023, 9805 responses were recorded, primarily from surgical theaters (45.9%; 4502/9805) and third-year residents (23.8%; 2331/9805). The most frequently evaluated EPAs were "head and neck" (17.5%; 1716/9805), "sinonasal" (13.5%; 1324/9805), and "ear" (12.2%; 1193/9805), with task complexity increasing with resident seniority (P < .0001). A positive correlation was found between residents' self-assessments and faculty members' ratings (r = 0.531; P < .001). Over 98.2% of residents and 88.4% of faculty members provided substantial feedback (>10 words). Analysis of WBAs reported by training programs identified faculty development targets and teaching-intensive tasks.

CONCLUSION

EMYWAY effectively documents workplace learning and tracks resident competency progression. Continuous improvement of WBA quality is essential for advancing the competency-based medical education ecosystem.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

IMPORTANCE

Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency.

OBJECTIVE

To assess progress in developing an entrustment process in the Core EPAs framework.

DESIGN, SETTING, AND PARTICIPANTS

In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database.

INTERVENTIONS

Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations.

MAIN OUTCOMES AND MEASURES

On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded.

RESULTS

Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs.

CONCLUSIONS AND RELEVANCE

These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

Bridging the Gap: Using Consensus to Explore Entrustment Decisions and Feedback Receptivity in Competency-Based Emergency Medicine Residency Programs Through the Construction of a Q-Sample Incorporating a Delphi Technique

Background

Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively.

Methods

In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study.

Results

A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning.

Conclusion

Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.

Learner Levels of Supervision Across the Continuum of Pediatrics Training

PURPOSE

To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship.

METHOD

The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings.

RESULTS

Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency.

CONCLUSIONS

Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.