Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended

Fixing disconnects: Exploring the emergence of principled adaptations in a competency-based curriculum

PURPOSE

Competency-based medical education (CBME) promises to improve medical education through curricular reforms to support learner development. This intention may be at risk in the case of a Canadian approach to CBME called Competence by Design (CBD), since there have been negative impacts on residents. According to Joseph Schwab, teachers, learners and milieu must be included in the process of curriculum-making to prevent misalignments between intended values and practice. This study considered what can be learned from the process of designing, enacting and adapting CBD to better support learners.

METHODS

This qualitative study explored the making of CBD through the perspectives of implementation leads (N = 18) at national, institutional and programme levels. A sociomaterial orientation to agency in curriculum-making guided the inductive approach to interviewing and analysis in phase one. A deductive analysis in phase two applied Schwab's theory to further understand sources of misalignments and the purpose of adaptive responses.

RESULTS

Misalignments occurred when the needs of teachers, learners and milieu were initially underestimated in the process of curriculum-making, disconnecting assessment practices from experiences of teaching, learning and entrustment. While technical and structural issues posed significant constraints on agency, some implementation leads were able to make changes to the curriculum or context to fix the disconnects. We identified six purposes for principled adaptations to align with CBME values of responsive teaching, individualised learning and meaningful entrustment.

CONCLUSION

Collectively, the adaptations we characterise demonstrate constructive alignment, a foundational principle of CBME in which assessment and teaching work together to support learning. This study proposes a model for making context-shaped, values-based adaptations to CBME to achieve its promise.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities

Importance

A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known.

Objective

To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions.

Design, Setting, and Participants

This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023.

Exposure

Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle.

Main Outcomes and Measures

The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence.

Results

Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association.

Conclusions and Relevance

In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.

Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process

BACKGROUND AND OBJECTIVES

Entrustable professional activities (EPAs) will be used for initial certification by the American Board of Pediatrics by 2028. Less than half of pediatric fellowships currently use EPAs for assessment, yet all will need to adopt them. Our objectives were to identify facilitators and barriers to the implementation of EPAs to assess pediatric fellows and to determine fellowship program directors' (FPD) perceptions of EPAs and Milestones.

METHODS

We conducted a survey of FPDs from 15 pediatric subspecialties. EPA users were asked about their implementation of EPAs, barriers encountered, and perceptions of EPAs. Nonusers were queried about deterrents to using EPAs. Both groups were asked about potential facilitators of implementation and their perceptions of Milestones.

RESULTS

The response rate was 65% (575/883). Of these, 344 (59.8%) were EPA users and 231 (40.2%) were nonusers. Both groups indicated work burden as a barrier to implementation. Nonusers reported more barriers than users (mean [SD]: 7 [3.8] vs 5.8 [3.4], P < .001). Both groups identified training materials and premade assessment forms as facilitators to implementation. Users felt that EPAs were easier to understand than Milestones (89%) and better reflected what it meant to be a practicing subspecialty physician (90%). In contrast, nonusers felt that Milestones were easy to understand (57%) and reflected what it meant to be a practicing subspecialist (58%).

CONCLUSIONS

Implementing EPA-based assessment will require a substantial investment by FPDs, facilitated by guidance and easily accessible resources provided by multiple organizations. Perceived barriers to be addressed include FPD time constraints, a need for additional assessment tools, and outcomes data.

Co-designing Entrustable Professional Activities in General Practitioner's training: a participatory research study

BACKGROUND

In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented.

METHODS

We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly.

RESULTS

During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes.

CONCLUSION

Accommodating competing stakeholders' needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.

The Importance of Adhering to Terminology When Implementing Competency-Based Veterinary Education (CBVE)

The American Association of Veterinary Medical Colleges (AAVMC) Competency-Based Veterinary Education (CBVE) Model was developed with consideration of the lessons learned over the past 20 years by other health care professions regarding the implementation of outcomes-based education. As veterinary education programs consider the benefits of outcomes-based training, and more programs begin adopting this model of education, it is more critical than ever to ensure fidelity of the model for successful implementation. Fidelity, or the accuracy with which something is reproduced, was identified as an important factor in successfully implementing competency-based training in medical education (CBME). Without fidelity of the core components of CBME as defined in the medical education literature, programs were challenged to evaluate the effectiveness of the new educational model, and in some cases, this led to premature notions of failure when all components of CBME had not been successfully implemented. Consistency in terminology related to competency-based education is critical for successful implementation of the CBVE Model. The terminology used in higher education, and in other disciplines, describes concepts that are underpinned by research, just as they are in competency-based education. Without shared understanding and accurate use of terminology to describe the tools and strategies used in CBVE, there is a considerable risk of failure or even perceived failure in transitioning to CBVE. The authors of this commentary, the AAVMC Council on Outcomes-based Veterinary Education, continue to encourage veterinary programs across the world to recognize the value of the AAVMC CBVE Model in transforming veterinary education. Through use of shared terminology and consistent application of the components of the model, we envision the expansion of CBVE as an opportunity to advance veterinary education and to promote new graduate success in the veterinary profession.

Exploring Factors for Implementation of EPAs in Pediatric Subspecialty Fellowships: A Qualitative Study of Program Directors

OBJECTIVE

To understand fellowship program directors' (FPDs) perspectives on facilitators and barriers to using entrustable professional activities (EPAs) in pediatric subspecialty training.

METHODS

We performed a qualitative study of FPDs, balancing subspecialty, program size, geographic region and current uses of EPAs. A study coordinator conducted 1-on-1 interviews using a semistructured approach to explore EPA use or nonuse and factors supporting or preventing their use. Investigators independently coded transcribed interviews using an inductive approach and the constant comparative method. Group discussion informed code structure development and refinement. Iterative data collection and analysis continued until theoretical sufficiency was achieved, yielding a thematic analysis.

RESULTS

Twenty-eight FPDs representing 11 pediatric subspecialties were interviewed, of whom 16 (57%) reported current EPA use. Five major themes emerged: (1) facilitators including the intuitive nature and simple wording of EPAs; (2) barriers such as workload burden and lack of a regulatory requirement; (2) variable knowledge and training surrounding EPAs, leading to differing levels of understanding; (3) limited current use of EPAs, even among self-reported users; and (4) complementary nature of EPAs and milestones. FPDs acknowledged the differing strengths of both EPAs and milestones but sought additional knowledge about the value added by EPAs for assessing trainees, including the impact on outcomes.

CONCLUSIONS

Identified themes can inform effective and meaningful EPA implementation strategies: Supporting and educating FPDs, ongoing assessment of the value of EPAs in training, and practical integration with current workflow. Generating additional data and engaging stakeholders is critical for successful implementation for the pediatric subspecialties.

Evaluating Feedback Comments in Entrustable Professional Activities: A Cross-Sectional Study

INTRODUCTION

Competency-based medical education (CBME) has transformed postgraduate medical training, prioritizing competency acquisition over traditional time-based curricula. Integral to CBME are Entrustable Professional Activities (EPAs), that aim to provide high-quality feedback for trainee development. Despite its importance, the quality of feedback within EPAs remains underexplored.

METHODS

We employed a cross-sectional study to explore feedback quality within EPAs, and to examine factors influencing length of written comments and their relationship to quality. We collected and analyzed 1163 written feedback comments using the Quality of Assessment for Learning (QuAL) score. The QuAL aims to evaluate written feedback from low-stakes workplace assessments, based on 3 quality criteria (evidence, suggestion, connection). Afterwards, we performed correlation and regression analyses to examine factors influencing feedback length and quality.

RESULTS

EPAs facilitated high-quality written feedback, with a significant proportion of comments meeting quality criteria. Task-oriented and actionable feedback was prevalent, enhancing value of low-stakes workplace assessments. From the statistical analyses, the type of assessment tool significantly influenced feedback length and quality, implicating that direct and video observations can yield superior feedback in comparison to case-based discussions. However, no correlation between entrustment scores and feedback quality was found, suggesting potential discrepancies between the feedback and the score on the entrustability scale.

CONCLUSION

This study indicates the role of the EPAs to foster high-quality feedback within CBME. It also highlights the multifaceted feedback dynamics, suggesting the influence of factors such as feedback length and assessment tool on feedback quality. Future research should further explore contextual factors for enhancing medical education practices.

Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program

BACKGROUND

Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design.

METHODS

We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs.

RESULTS

Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning.

CONCLUSION

Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

Entrustable professional activities in dental education: a scoping review

Background Competency-based education has led to the introduction of entrustable professional activities (EPAs) in health sciences education. EPAs are assessment tools that serve to certify a trainee's preparedness to conduct a given clinical activity. Objective Given its modest introduction into dental education, we sought to examine the current situation of EPAs in our field and identify possible barriers and facilitators. This review aimed to summarise evidence about EPA introduction in dental education programmes.Data sources A review of articles published between January 2005 and December 2021 was conducted using the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews methodology in three databases (PubMed, Cochrane and Embase).Data selection Keywords were 'entrustable professional activity' OR 'entrustable professional activities' AND 'dentistry' OR 'dental education'.Data extraction Eligibility criteria targeted papers published in English describing dental education EPA.Data synthesis Ten publications were selected: six reported on new EPAs, three discussed their relevance for dental education and one article described a process for developing validated EPAs. EPAs focused on clinical examination, health promotion, treatment planning, pain and/or anxiety control and topics related to a specific discipline or care procedures.Conclusion Topics covered by EPAs described in our study adequately reflect the specificities of today's dental professional. The benefits of EPAs will ensure tomorrow's dentists consistently meet societal expectations.

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students' Self-Assessed Skills by Specialty

BACKGROUND

The Association of American Medical Colleges described 13 Core Entrustable Professional Activities (EPAs) that graduating students should be prepared to perform under indirect supervision on day one of residency. Surgery program directors recently recommended entrustability in these Core EPAs for incoming surgery interns. We sought to determine if graduating students intending to enter surgery agreed they had the skills to perform these Core EPAs.

STUDY DESIGN

Using de-identified, individual-level data collected from and about 2019 Association of American Medical Colleges Graduation Questionnaire respondents, latent profile analysis was used to group respondents based on their self-assessed Core EPAs skills' response patterns. Associations between intended specialty, among other variables, and latent profile analysis group were assessed using independent sample t -tests and chi-square tests and multivariable logistic regression methods.

RESULTS

Among 12,308 Graduation Questionnaire respondents, latent profile analysis identified 2 respondent groups: 7,863 (63.9%) in a high skill acquisition agreement (SAA) group and 4,445 (36.1%) in a moderate SAA group. Specialty was associated with SAA group membership (p < 0.001), with general surgery, orthopaedic surgery, and emergency medicine respondents (among others) overrepresented in the high SAA group. In the multivariable logistic regression models, each of anesthesiology, ophthalmology, pediatrics, psychiatry, and radiology (vs general surgery) specialty intention was associated with a lower odds of high SAA group membership.

CONCLUSION

Graduating students' self-assessed Core EPAs skills were higher for those intending general surgery than for those intending some other specialties. Our findings can inform collaborative efforts to ensure graduates' acquisition of the skills expected of them at the start of residency.

Fidelity to Best Practices in EPA Implementation: Outcomes Supporting Use of the Core Components Framework From the University of Virginia Entrustable Professional Activity Program

PROBLEM

The rapid expansion of entrustable professional activity (EPA) assessment programs has led to calls to ensure fidelity in implementation and integrity in meeting the goals of competency-based medical education. Initiated in July 2017, in advance of the articulated core components of EPA implementation, this article describes the structure and outcomes of the University of Virginia (UVA) EPA Program and provides support for the identified essential components.

APPROACH

The UVA EPA Program includes workplace assessments by residents/fellows, attending faculty, and master assessors (MAs), experienced clinicians who assess students across disciplines and clinical settings. All assessors participate in formal professional development and provide verbal and written comments to support their supervision ratings. The Entrustment Committee, composed of 12 MAs, uses a shared mental model and aggregates all assessor data to make a high-stakes summative entrustment decision about students' readiness to assume the role of an acting intern.

OUTCOMES

Since 2017, over 2,000 assessors have completed 56,969 EPA assessments for 1,479 students. Ninety-four percent of assessments have been done during the clerkship phase. Residents/fellows have completed a mean of 18 assessments, attending faculty a mean of 27, and MAs a mean of 882. Seventy-four percent of observed encounters involved patients with acute concerns with or without a co-morbid condition. Fifty percent of assessments occurred in inpatient and 32% in ambulatory settings. Eighty-seven percent of assessments contained narrative comments with more than 100 characters.

NEXT STEPS

Planned next steps will include earlier identification of students who require individualized learning to promote the development of skills related to EPAs, expansion of the remediation program to enable more students to engage in a clinical performance mastery elective, and creation of targeted professional development for assessors to reinforce the tenets of the EPA program.

Developing a dashboard for program evaluation in competency-based training programs: a design-based research project

BACKGROUND

Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires the assessment of entrustable professional activities (EPAs). Dashboards could be used to track the completion of EPAs to support program evaluation.

METHODS

Using a design-based research process, we identified program evaluation needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. We interviewed leaders from the emergency medicine program and postgraduate medical education office at the University of Saskatchewan. Two investigators thematically analyzed interview transcripts to identify program evaluation needs that were audited by two additional investigators. Identified needs were described using quotes, analytics, and visualizations.

RESULTS

Between July 1, 2019 and April 6, 2021 we conducted 17 interviews with six participants (two program leaders and four institutional leaders). Four needs emerged as themes: tracking changes in overall assessment metrics, comparing metrics to the assessment plan, evaluating rotation performance, and engagement with the assessment metrics. We addressed these needs by presenting analytics and visualizations within a dashboard.

CONCLUSIONS

We identified program evaluation needs related to EPA assessments and designed dashboard elements to meet them. This work will inform the development of other CBME assessment dashboards designed to support program evaluation.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Core competencies for a biomedical laboratory scientist - a Delphi study

BACKGROUND

After completing university education, biomedical laboratory scientists work in clinical laboratories, in biomedical research laboratories, in biotech, and in pharmaceutical companies. Laboratory diagnostics have undergone rapid development over the recent years, with the pace showing no signs of abatement. This rapid development challenges the competence of the staff and will most certainly influence the education of future staff. This study aimed to examine what was considered the necessary competencies needed to pursue a career as a biomedical laboratory scientist.

METHODS

A modified Delphi technique was used, with the panel of experts expressing their views in a series of three questionnaire. Consensus was defined as the point which 75 % or more of the panel participants agreed that a particular competency was necessary.

RESULTS

The study highlights the perceived importance of mostly generic competencies that relate to quality, quality assurance, and accuracy, as well as different aspects of safety, respect, trustworthiness (towards patients/clients and colleagues), and communication skills. The results also stress the significance of self-awareness and professionality.

CONCLUSIONS

We identified important competencies for biomedical laboratory scientists. Together with complementary information from other sources, i.e., guidelines, laws, and scientific publications, the competencies identified can be used as learning outcomes in a competency-based education to provide students with all the competencies needed to work as professional biomedical laboratory scientists.