Entrustable Professional Activities in General Surgery: Trends in Resident Self-Assessment

A Modified Delphi Consensus Approach to Define Entrustable Professional Activities for Neurocritical Care Advanced Practice Providers

OBJECTIVES

To define consensus entrustable professional activities (EPAs) for neurocritical care (NCC) advanced practice providers (APPs), establish validity evidence for the EPAs, and evaluate factors that inform entrustment expectations of NCC APP supervisors.

DESIGN

A three-round modified Delphi consensus process followed by application of the EQual rubric and assessment of generalizability by clinicians not affiliated with academic medical centers.

SETTING

Electronic surveys.

SUBJECTS

NCC APPs ( n = 18) and physicians ( n = 12) in the United States with experience in education scholarship or APP program leadership.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The steering committee generated an initial list of 61 possible EPAs. The panel proposed 30 additional EPAs. A total of 47 unique nested EPAs were retained by consensus opinion. The steering committee defined six core EPAs addressing medical knowledge, procedural competencies, and communication proficiency which encompassed the nested EPAs. All core EPAs were retained and subsequently met the previously described cut score for quality and structure using the EQual rubric. Most clinicians who were not affiliated with academic medical centers rated each of the six core EPAs as very important or mandatory. Entrustment expectations did not vary by prespecified groups.

CONCLUSIONS

Expert consensus was used to create EPAs for NCC APPs that reached a predefined quality standard and were important to most clinicians in different practice settings. We did not identify variables that significantly predicted entrustment expectations. These EPAs may aid in curricular design for an EPA-based assessment of new NCC APPs and may inform the development of EPAs for APPs in other critical care subspecialties.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

National pilot of entrustable professional activities in pathology residency training

Entrustable professional activities (EPAs) are observable clinical skills and/or procedures that have been introduced into medical education at the student and resident levels in most specialties to determine readiness to advance into residency or independent practice, respectively. This publication describes the process and outcomes of a pilot study looking at the feasibility of using two anatomic pathology and two clinical pathology EPAs in pathology residency in 6 pathology residency programs that volunteered for the study. Faculty development on EPAs and their assessment was provided to pilot program faculty, and EPA assessment tools were developed and used by the pilot programs. Pre- and post-study surveys were given to participating residents, faculty, and program directors to gauge baseline practices and to gather feedback on the EPA implementation experience. Results demonstrated overall good feasibility in implementing EPAs. Faculty acceptance of EPAs varied and was less than that of program directors. Residents reported a significant increase in the frequency with which faculty provided formative assessments that included specific examples of performance and specific ways to improve, as well as increased frequency with which faculty provided summative assessments that included specific ways to improve. EPAs offered the most benefit in setting clear expectations for performance of each task, for providing more specific feedback to residents, and in increasing Program director's understanding of resident strengths abilities and weaknesses.

Differences in Resident Self-Evaluation and Clinical Competency Committee Evaluation Using ACGME Milestone Versions 1.0 and 2.0

OBJECTIVE

Intentionally self-driven professional development of surgical resident physicians is a hallmark of surgical training and is expected to gain further traction as Entrustable Professional Activities (EPAs) become the new paradigm for surgical education. We aimed to analyze how surgical residents rate themselves as compared to the evaluation of the Clinical Competency Committee using ACGME Milestones Version 1 (M1.0) and Version 2 (M2.0).

DESIGN

We asked 22 general surgical trainees for self-evaluation of Milestones (both M1.0 and M2.0) from 2017 semiannually to 2022. ACGME-required Milestone evaluations by the Clinical Competency Committee (CCC) were independently performed after the time window for resident self-evaluation. Neither trainees nor CCC were aware of the other party's evaluations. There were 1552 paired data available for evaluating individual competencies by both trainees and CCC. Paired Wilcoxon signed-rank tests were then performed among the corresponding pairs.

SETTING

MercyOne Des Moines Medical Center, Des Moines, IA; Teaching tertiary referral center.

PARTICIPANTS

Twenty-two general surgical trainees at this hospital and 28 faculty surgeons participated in this study.

RESULTS

The average self-evaluation of surgical residents was lower in the M1.0 cohort compared to the corresponding CCC evaluation (1.96 ± 0.72 vs. 2.11 ± 0.67; p < 0.001). M1.0 self-assessments and CCC-assessments were statistically similar for ICS (p = 0.548) and PROF (p = 0.554) competencies and differed for MK (p < 0.001), PBLI (p < 0.001), PC (p < 0.001), SBP (p = 0.008). On the contrary, the M2.0 cohort demonstrated higher average self-evaluation of surgical residents compared to the corresponding CCC evaluation (2.75 ± 0.87 vs. 2.12 ± 0.97; p < 0.001). Significant differences were observed for all 6 ACGME competencies using M2.0 self-assessments and CCC-assessments (all p < 0.001). Multivariate regression modeling (p < 0.001, R2 = 0.255) predicted the degree of discordance between self-assessment and CCC-assessed achievement of competencies with a significant effect of gender (baseline male: coef = -0.232, p < 0.001), PGY level (-0.083 per year, p < 0.001) and Milestone version (0.831, p < 0.001). A significant interaction exists for all gender/Milestone combinations except for the female trainees with M1.0.

CONCLUSIONS

The difference between self-evaluated Milestone achievement and faculty-driven CCC evaluation of surgical resident physician performance is more evident in Milestones 2.0 than in Milestones 1.0. Residents self-evaluate higher compared to faculty using Milestones 2.0. This discrepancy is seen among both genders and is more pronounced among male residents overestimating core competencies with M2.0 self-evaluation than formal CCC assessment.

Reimagining the Clinical Competency Committee to Enhance Education and Prepare for Competency-Based Time-Variable Advancement

Assessing residents and clinical fellows is a high-stakes activity. Effective assessment is important throughout training so that identified areas of strength and weakness can guide educational planning to optimize outcomes. Assessment has historically been underemphasized although medical education oversight organizations have strengthened requirements in recent years. Growing acceptance of competency-based medical education and its logical extension to competency-based time-variable (CB-TV) graduate medical education (GME) further highlights the importance of implementing effective evidence-based approaches to assessment. The Clinical Competency Committee (CCC) has emerged as a key programmatic structure in graduate medical education. In the context of launching a multi-specialty pilot of CB-TV GME in our health system, we have examined several program's CCC processes and reviewed the relevant literature to propose enhancements to CCCs. We recommend that all CCCs fulfill three core goals, regularly applied to every GME trainee: (1) discern and describe the resident's developmental status to individualize education, (2) determine readiness for unsupervised practice, and (3) foster self-assessment ability. We integrate the literature and observations from GME program CCCs in our institutions to evaluate how current CCC processes support or undermine these goals. Obstacles and key enablers are identified. Finally, we recommend ways to achieve the stated goals, including the following: (1) assess and promote the development of competency in all trainees, not just outliers, through a shared model of assessment and competency-based advancement; (2) strengthen CCC assessment processes to determine trainee readiness for independent practice; and (3) promote trainee reflection and informed self-assessment. The importance of coaching for competency, robust workplace-based assessments, feedback, and co-production of individualized learning plans are emphasized. Individual programs and their CCCs must strengthen assessment tools and frameworks to realize the potential of competency-oriented education.

An update on current EPAs in graduate medical education: A scoping review

The purpose of this scoping review is to update the recent progress of EPAs research in GME, focusing on the topical concern of EPAs effectiveness, and to provide a reference for medical researchers in countries/regions interested in introducing EPAs. Guided by Arksey and O'Malley's framework regarding scoping reviews, the researchers, in January 2021, conducted a search in five databases to ensure the comprehensiveness of the literature. After the predetermined process, 29 articles in total were included in this study. The most common areas for the implementation and evaluation of EPAs were Surgery (n = 7,24.1%), Pediatric (n = 5,17.2%) and Internal medicine (n = 4,13.8%), a result that shows a relatively large change in the research trend of EPAs in the last two years. Prior to 2018, EPAs research focused on internal medicine, psychiatry, family medicine, and primary care. The articles in the category of EPAs implementation and evaluation had four main themes: (1) validation of EPAs (n = 16,55.2%); (2) describing the experience of implementing EPAs (n = 11,37.9%); (3) examining the factors and barriers that influence the implementation and evaluation of EPAs (n = 6,20.6%); and (4) researching the experiences of faculty, interns, and other relevant personnel in using EPAs. Training programs were the most common EPAs implementation setting (n = 26,89.6%); direct observation and evaluation (n = 12,41.4%), and evaluation by scoring reports (n = 5,17.2%) were the two most common means of assessing physicians' EPA levels; 19 papers (65.5%) used faculty evaluation, and nine of these papers also used self-assessment (31.0%); the most frequently used tools in the evaluation of EPAs were mainly researcher-made instruments (n = 37.9%), assessment form (n = 7,24.1%), and mobile application (n = 6,20.7%). Although EPAs occupy an increasingly important place in international medical education, this study concludes that the implementation and diffusion of EPAs on a larger scale is still difficult.

Temporal changes in medical student perceptions of their clinical skills and needs using a repeated self-assessment instrument

BACKGROUND

Medical student needs in clinical skill training may change over time, but data on this topic are limited. This study uses repeated self-assessments on clinical rotations during medical school to evaluate students' perceptions of their clinical skill growth.

METHODS

A self-assessment rating was completed by students during each clinical rotation as they progressed through their core clinical rotation year. The instrument consisted of questions on 5 clinical skill categories where students rated their performance as "below", "at" or "above" expected, and open-ended questions on strengths and challenges. We evaluated changes in self-ratings between the first (n=136) and third (n=118) quarters by matched-pair analysis of the shift in responses between time points using a Sign Test. We also identified the main themes from the students' responses to open-ended questions.

RESULTS

We found 22.4 % and 13.3 % of students increased their self-assessment ratings on "Oral Presentation Skills" and on "Differential Diagnosis", respectively. In contrast, perceived ability to communicate with patients saw the largest negative shifts. "Patient Interaction" was the most commonly identified area of strength and "Knowledge and Organization" was most frequently cited as a barrier.

CONCLUSIONS

Students demonstrated a positive shift in perceived competence in some core clinical skills that are not strongly emphasized in the preclinical curriculum, likely reflecting increased exposure over time. However, their perceived competence in communication skills declined. This may reflect initial over-estimation or true decline due to competing needs/interests. These patterns of change can inform the design of longitudinal curricula that anticipate and address students' needs during clinical rotations, such as placing increased emphasis on presentation skills and differential diagnosis earlier in the curriculum, and adding more emphasis to communication skills in later phases.

A Phased Approach: The General Surgery Experience Adopting Entrustable Professional Activities in the United States

Entrustable professional activities (EPAs) have been increasingly used as an assessment framework to formally capture the myriad ad hoc entrustment decisions that occur on a daily basis in clinical settings with learners present. Following the definition of Core EPAs for Entering Residency by the Association of American Medical Colleges in 2014, the American Board of Surgery (ABS) began to explore the utility of EPAs as a framework to support competency-based resident education within general surgery in 2016. As the complement of EPAs drafted for a specialty serve to define the core tasks of a professional within that discipline, initial efforts to define the entire scope of general surgery were fraught with difficulty as no commonly accepted definition of a general surgeon currently exists. Opting to prioritize a pilot of the EPA conceptual framework within surgical training rather than defining the entirety of the specialty, ABS leaders identified 5 EPAs that represent a core of general surgery with which to begin. This article details the process of choosing the initial set of EPAs and provides a roadmap for other disciplines interested in testing the feasibility of this assessment framework while garnering buy-in among the community of educators. Future steps, including implementation of the existing 5 EPAs beyond the initial pilot sites and drafting and implementation of the additional complement of EPAs, are also described.

Fundamentos para un currículo nacional en Cirugía General basado en competencia: Consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía

Frente a los desafíos de la Educación Basada en Competencia, este consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía informa sobre las mínimas competencias profesionales esperadas del cirujano egresado de los veinte programas de Especialización en Cirugía General en Colombia. Un total de 105 profesores de los programas de especialización evaluaron tres áreas de competencia profesional: 1) atributos profesionales generales del residente durante su formación, 2) competencias prácticas (procedimientos quirúrgicos) que los residentes deben realizar al final de su entrenamiento y 3) Actividades Profesionales Confiables (APC) que los residentes deben ejecutar sin supervisión al final de su entrenamiento. Los resultados informan un alto nivel de consenso en el 100 % los atributos profesionales generales y APC, y del 75 % en diferentes procedimientos quirúrgicos. El consenso abre la puerta para el desarrollo de un currículo nacional de la especialidad y tiene implicaciones para la práctica educativa e investigación futura.