1
|
Kitto S, Fantaye AW, Zevin B, Fowler A, Sachdeva AK, Raiche I. A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs. JOURNAL OF SURGICAL EDUCATION 2024; 81:823-840. [PMID: 38679495 DOI: 10.1016/j.jsurg.2024.02.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 02/20/2024] [Indexed: 05/01/2024]
Abstract
OBJECTIVE Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties. DESIGN Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace. PARTICIPANTS A total of 42 empirical and nonempirical articles were included. RESULTS Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education. CONCLUSIONS This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.
Collapse
Affiliation(s)
- Simon Kitto
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore
| | - Arone W Fantaye
- Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa, Ottawa, Canada
| | - Boris Zevin
- Department of Surgery, Queen's University, Kingston, Canada
| | - Amanda Fowler
- Department of Surgery, Memorial University of Newfoundland, St. John's, Newfoundland, Canada
| | - Ajit K Sachdeva
- Division of Education, American College of Surgeons, Chicago, Illinios
| | - Isabelle Raiche
- Department of Surgery, University of Ottawa, Ottawa, Canada.
| |
Collapse
|
2
|
Montgomery KB, Mellinger JD, McLeod MC, Jones A, Zmijewski P, Sarosi GA, Brasel KJ, Klingensmith ME, Minter RM, Buyske J, Lindeman B. Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities. JAMA Surg 2024:2818486. [PMID: 38717759 PMCID: PMC11079788 DOI: 10.1001/jamasurg.2024.0809] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Accepted: 02/02/2024] [Indexed: 05/12/2024]
Abstract
Importance A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known. Objective To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions. Design, Setting, and Participants This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023. Exposure Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle. Main Outcomes and Measures The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence. Results Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association. Conclusions and Relevance In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.
Collapse
Affiliation(s)
| | - John D. Mellinger
- American Board of Surgery, Philadelphia, Pennsylvania
- Department of Surgery, Southern Illinois University, Springfield
| | | | - Andrew Jones
- American Board of Surgery, Philadelphia, Pennsylvania
| | | | | | - Karen J. Brasel
- Department of Surgery, Oregon Health & Science University, Portland
| | - Mary E. Klingensmith
- American Board of Surgery, Philadelphia, Pennsylvania
- Accreditation Council for Graduate Medical Education, Chicago, Illinois
- Department of Surgery, Washington University in St Louis, St Louis, Missouri
| | | | - Jo Buyske
- American Board of Surgery, Philadelphia, Pennsylvania
- Department of Surgery, University of Pennsylvania, Philadelphia
| | | |
Collapse
|
3
|
Montgomery KB, Mellinger JD, Lindeman B. Entrustable Professional Activities in Surgery: A Review. JAMA Surg 2024; 159:571-577. [PMID: 38477902 DOI: 10.1001/jamasurg.2023.8107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/14/2024]
Abstract
Importance Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees. Observations An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods. Conclusions and Relevance EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.
Collapse
|
4
|
Filiberto AC, Cochran A. Standardizing Resident Assessment and Competency-Milestones Matter. JAMA Surg 2024; 159:553. [PMID: 38477925 DOI: 10.1001/jamasurg.2024.0045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/14/2024]
Affiliation(s)
- Amanda C Filiberto
- Division of Vascular Surgery, Department of Surgery, University of Alabama, Birmingham
| | - Amalia Cochran
- Department of Surgery, University of Florida Health, Gainesville
- Web and Social Media Editor, JAMA Surgery
| |
Collapse
|
5
|
Montgomery KB, Mellinger JD, Jones A, McLeod MC, Zmijewski P, Sarosi GA, Brasel KJ, Klingensmith ME, Minter RM, Buyske J, Lindeman B. Validity of Entrustable Professional Activities in a National Sample of General Surgery Residency Programs. J Am Coll Surg 2024; 238:376-384. [PMID: 38224150 PMCID: PMC10942265 DOI: 10.1097/xcs.0000000000000967] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2024]
Abstract
BACKGROUND The American Board of Surgery has endorsed competency-based education as vital to the assessment of surgical training. From 2018 to 2020, a national pilot study was conducted at 28 general surgery programs to evaluate feasibility of implementing entrustable professional activities (EPAs) for 5 common general surgical conditions. ACGME core competency Milestones were also rated for each resident by program clinical competency committees. This study aimed to evaluate the validity of general surgery EPAs compared with Milestones. STUDY DESIGN Prospectively collected, de-identified EPA Pilot Study data were analyzed. EPAs studied were EPA-1 (gallbladder), EPA-2 (inguinal hernia), EPA-3 (right lower quadrant pain), EPA-4 (trauma), and EPA-5 (consult). Variables abstracted included levels of EPA entrustment (1 to 5) and corresponding ACGME Milestone subcompetency ratings (1 to 5) for the same study cycle. Spearman's correlations were used to evaluate the relationship between summative EPA scores and corresponding Milestone ratings. RESULTS A total of 493 unique residents received a summative entrustment decision. EPA summative entrustment scores had moderate-to-strong positive correlation with mapped Milestone subcompetencies, with median rho value of 0.703. Among operation-focused EPAs, median rho values were similar between EPA-1 (0.688) and EPA-2 (0.661), but higher for EPA-3 (0.833). EPA-4 showed a strong positive correlation with diagnosis and communication competencies (0.724), whereas EPA-5, mapped to the most Milestone subcompetencies, had the lowest median rho value (0.455). CONCLUSIONS Moderate-to-strong positive correlation was noted between EPAs and patient care, medical knowledge, and communication Milestones. These findings support the validity of EPAs in general surgery and suggest that EPA assessments can be used to inform Milestone ratings by clinical competency committees.
Collapse
Affiliation(s)
| | - John D Mellinger
- American Board of Surgery, Philadelphia, PA
- Department of Surgery, Southern Illinois University, Springfield, IL
| | | | - M Chandler McLeod
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL
| | - Polina Zmijewski
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL
| | - George A Sarosi
- Department of Surgery, University of Florida, Gainesville, FL
| | - Karen J Brasel
- Department of Surgery, Oregon Health & Science University, Portland, OR
| | - Mary E Klingensmith
- Accreditation Council for Graduate Medical Education, Chicago, IL
- Department of Surgery, Washington University in St. Louis, St. Louis, MO
| | | | - Jo Buyske
- American Board of Surgery, Philadelphia, PA
- Department of Surgery, University of Pennsylvania, Philadelphia, PA
| | - Brenessa Lindeman
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL
| |
Collapse
|
6
|
Schmidbauer ML, Pinilla S, Kunst S, Biesalski AS, Bösel J, Niesen WD, Schramm P, Wartenberg K, Dimitriadis K. Fit for Service: Preparing Residents for Neurointensive Care with Entrustable Professional Activities: A Delphi Study. Neurocrit Care 2024; 40:645-653. [PMID: 37498455 PMCID: PMC10959831 DOI: 10.1007/s12028-023-01799-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 06/30/2023] [Indexed: 07/28/2023]
Abstract
BACKGROUND Although the relevance of neurointensive medicine and high-quality training of corresponding physicians is increasingly recognized, there is high heterogeneity in the nature, duration, and quality of neurointensive care curricula around the world. Thus, we aimed to identify, define, and establish validity evidence for entrustable professional activities (EPAs) for postgraduate training in neurointensive care to determine trainees' readiness for being on-call. METHODS After defining EPAs through an iterative process by an expert group, we used a modified Delphi approach with a single-center development process followed by a national consensus and a single-center validation step. EPAs were evaluated by using the EQual rubric (Queen's EPA Quality Rubric). Interrater reliability was measured with Krippendorff's α. RESULTS The expert group defined seven preliminary EPAs for neurointensive care. In two consecutive Delphi rounds, EPAs were adapted, and consensus was reached for level of entrustment and time of expiration. Ultimately, EPAs reached a high EQual score of 4.5 of 5 and above. Interrater reliability for the EQual scoring was 0.8. CONCLUSIONS Using a multistep Delphi process, we defined and established validity evidence for seven EPAs for neurointensive medicine with a high degree of consensus to objectively describe readiness for on-call duty in neurointensive care. This operationalization of pivotal clinical tasks may help to better train clinical residents in neurointensive care across sites and health care systems and has the potential to serve as a blueprint for training in general intensive care medicine. It also represents a starting point for further research and development of medical curricula.
Collapse
Affiliation(s)
| | - Severin Pinilla
- University Hospital for Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
- Institute for Medical Education (IML), University of Bern, Bern, Switzerland
| | - Stefan Kunst
- Department of Neurology, LMU University Hospital, LMU Munich, Munich, Germany
| | - Anne-Sophie Biesalski
- Department of Neurology, Ruhr-Universität Bochum, St. Josef Hospital, Bochum, Germany
| | - Julian Bösel
- Department of Neurology, Heidelberg University Hospital, Heidelberg, Germany
| | - Wolf-Dirk Niesen
- Department of Neurology and Neurophysiology, University Medical Center Freiburg, Freiburg, Germany
| | - Patrick Schramm
- Department of Neurology, Universitätsklinikum Giessen und Marburg, Standort Giessen, Justus-Liebig-University, Giessen, Germany
| | - Katja Wartenberg
- Department of Neurology, University of Leipzig, Leipzig, Germany
| | - Konstantinos Dimitriadis
- Department of Neurology, LMU University Hospital, LMU Munich, Munich, Germany.
- Institute for Stroke and Dementia Research (ISD), LMU University Hospital, LMU Munich, Munich, Germany.
| |
Collapse
|
7
|
Cifra N, Pitts S, Mink R, Schwartz A, Herman B, Turner DA, Yussman S. Analysis of fellowship program director opinions of entrustable professional activities in adolescent medicine fellowship. Int J Adolesc Med Health 2024; 0:ijamh-2023-0154. [PMID: 38522004 DOI: 10.1515/ijamh-2023-0154] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Accepted: 03/08/2024] [Indexed: 03/25/2024]
Abstract
OBJECTIVES This study aimed to explore the minimum entrustable professional activity (EPA) supervision levels at which pediatric fellowship program directors (FPDs) would be willing to graduate fellows and the levels deemed necessary for safe and effective practice for each of the common pediatric subspecialty and the four adolescent medicine-specific EPAs. METHODS This cross-sectional study utilized survey data from pediatric FPDs in 2017. FPDs indicated the minimum level of supervision (LOS) for fellows at graduation and for safe and effective practice. RESULTS 82 percent (23/28) of adolescent medicine FPDs completed the survey. For each EPA, there were differences (p<0.05) between LOS expected for graduation and for safe and effective practice. There was also variability in the level at which FPDs would graduate fellows. CONCLUSIONS This study summarizes pediatric FPD opinions regarding the minimum levels of supervision required for fellows at the time of graduation as well as the levels deemed necessary for safe and effective practice. The difference between the minimum LOS at which FPDs would graduate a fellow and that deemed appropriate for safe and effective practice, along with variability in minimum LOS for graduation, highlight the need for clearer standards for fellowship graduation as well as more structured early career support for ongoing learning. These data highlight variability in FPD opinion regarding such expectations and both the need to better define desired training outcomes and potential need for post-graduation supervision in clinical practice.
Collapse
Affiliation(s)
- Nicole Cifra
- Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, USA
| | - Sarah Pitts
- Department of Pediatrics, Boston Children's Hospital, Boston, USA
| | | | - Alan Schwartz
- Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, USA
| | | | | | - Susan Yussman
- Department of Pediatrics, University of Rochester School of Medicine and Dentistry, Rochester, USA
| |
Collapse
|
8
|
Weaver ML. Ready or not, here EPAs come. Am J Surg 2023; 226:586-587. [PMID: 37640637 DOI: 10.1016/j.amjsurg.2023.08.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 08/14/2023] [Accepted: 08/14/2023] [Indexed: 08/31/2023]
Affiliation(s)
- M Libby Weaver
- Division of Vascular and Endovascular Surgery, University of Virginia, Charlottesville, VA, USA.
| |
Collapse
|
9
|
Farmer DL, O'Connell PR, Pugh CM, Lang H, Greenberg CC, Borel-Rinkes IH, Mellinger JD, Pinto-Marques H. Surgical Education. Ann Surg 2023; 278:642-646. [PMID: 37796749 DOI: 10.1097/sla.0000000000006078] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/07/2023]
Abstract
This paper summarizes the proceedings of the joint European Surgical Association ESA/American Surgical Association symposium on Surgical Education that took place in Bordeaux, France, as part of the celebrations for 30 years of ESA scientific meetings. Three presentations on the use of quantitative metrics to understand technical decisions, coaching during training and beyond, and entrustable professional activities were presented by American Surgical Association members and discussed by ESA members in a symposium attended by members of both associations.
Collapse
Affiliation(s)
- Diana L Farmer
- Department of Surgery, University of California Davis Health, Davis, CA
| | | | - Carla M Pugh
- Department of Surgery, Stanford Medicine, Stanford, CA
| | - Hauke Lang
- University Medical Centre of the Johannes Gutenberg University Mainz, Germany
| | | | | | | | | |
Collapse
|
10
|
Huynh A, Nguyen A, Beyer RS, Harris MH, Hatter MJ, Brown NJ, de Virgilio C, Nahmias J. Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2023; 98:769-774. [PMID: 36780667 DOI: 10.1097/acm.0000000000005168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.
Collapse
Affiliation(s)
- Ashley Huynh
- A. Huynh is a first-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-4413-6829
| | - Andrew Nguyen
- A. Nguyen is a first-year medical student, University of Florida College of Medicine, Gainesville, Florida; ORCID: https://orcid.org/0000-0002-8131-150X
| | - Ryan S Beyer
- R.S. Beyer is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-0283-3749
| | - Mark H Harris
- M.H. Harris is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-1598-225X
| | - Matthew J Hatter
- M.J. Hatter is a second-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0003-2922-6196
| | - Nolan J Brown
- N.J. Brown is a fourth-year medical student, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0002-6025-346X
| | - Christian de Virgilio
- C. de Virgilio is professor of surgery, Harbor-UCLA Medical Center, Torrance, California
| | - Jeffry Nahmias
- J. Nahmias is professor of trauma, burns, surgical critical care, and acute care surgery, University of California, Irvine, School of Medicine, Irvine, California; ORCID: https://orcid.org/0000-0003-0094-571X
| |
Collapse
|
11
|
Chang YC, Khwepeya M, Nkambule NS, Chuang RS, Chaou CH. Emergency residents' self-perceived readiness for practice: the association of milestones, entrustable professional activities, and professional identities-a multi-institutional survey. Front Med (Lausanne) 2023; 10:1032516. [PMID: 37250629 PMCID: PMC10213224 DOI: 10.3389/fmed.2023.1032516] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Accepted: 04/17/2023] [Indexed: 05/31/2023] Open
Abstract
Background As a successful innovation, competency-based medical education and its assessment tools continue to be a key strategy in training future doctors and tracking their performance trajectories. Linked to professional identity, evidence suggests that clinical competence is related to thinking, acting and feeling like a physician. Thus, incorporating the values and attitudes of healthcare professions as part of their professional identity in the clinical workplace improves professional performance. Methods Through a cross-sectional study, we examined the association of milestone, entrustable professional activities (EPA) and professional identity using self-reported tools among emergency medicine residents from 12 teaching hospitals across Taiwan. Milestone, EPA and professional identity were assessed using the Emergency Medicine Milestone Scale, Entrustable Professional Activity Scale and Emergency Physician Professional Identity and Value Scale, respectively. Results The results of a Pearson correlation indicated a significant positive correlation between milestone-based core competencies and EPAs (r = 0.40 ~ 0.74, p < 0.01). The professional identity domain of skills acquisition, capabilities and practical wisdom was positively correlated with milestone-based core competencies of patient care, medical knowledge, practice-based learning and improvement, and system-based practice (r = 0.18 ~ 0.21, p ≤ 0.05), and six items of EPA (r = 0.16 ~ 0.22, p < 0.05). Additionally, the professional identity domain of professional recognition and self-esteem was positively correlated with practice-based learning and improvement, and system-based practice milestone competencies (r = 0.16 ~ 0.19, p < 0.05). Conclusion This study demonstrates milestone and EPA assessment tools are highly linked and therefore, can be synergistically used by supervisors and clinical educators to evaluate clinical performance during residency training. Emergency physicians' professional identity is partly influenced by the advancement of skills and a resident's ability to learn, effectively perform tasks and make appropriate medical decisions at the system level in their clinical practice. Further research is warranted to understand the importance of residents' competency in relation to their professional identity development trajectory during clinical training.
Collapse
Affiliation(s)
- Yu-Che Chang
- Chang Gung Medical Education Research Centre, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- College of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Madalitso Khwepeya
- Chang Gung Medical Education Research Centre, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- College of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Nothando S. Nkambule
- International Graduate Program of Education and Human Development (IGPEHD), National Sun Yat-Sen University, Kaohsiung, Taiwan
| | - Renee S. Chuang
- Health Policy and Leadership Program, School of Public Health, Loma Linda University, Loma Linda, CA, United States
| | - Chung-Hsien Chaou
- Chang Gung Medical Education Research Centre, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- College of Medicine, Chang Gung University, Taoyuan, Taiwan
| |
Collapse
|
12
|
Porter S, Prendiville E, Allen BFS, Booth G, Boublik J, Burnett GW, Elkassabany N, Hausman J, Klesius L, Le-Wendling L, Machi AT, Maniker R, Parra M, Rosenquist R, Spofford CM, Suresh S, Tedore T, Wilson EH, Zhou JY, Woodworth G. Development of entrustable professional activities for regional anesthesia and pain medicine fellowship training. Reg Anesth Pain Med 2022; 47:rapm-2022-103854. [PMID: 35878963 DOI: 10.1136/rapm-2022-103854] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 07/14/2022] [Indexed: 11/04/2022]
Abstract
INTRODUCTION The Accreditation Council for Graduate Medical Education (ACGME) offers descriptions of competencies and milestones but does not provide standardized assessments to track trainee competency. Entrustable professional activities (EPAs) and special assessments (SAs) are emerging methods to assess the level of competency obtained by regional anesthesiology and acute pain medicine (RAAPM) fellows. METHODS A panel of RAAPM physicians with experience in education and competency assessment and one medical student were recruited to participate in a modified Delphi method with iterative rounds to reach consensus on: a list of EPAs, SAs, and procedural skills; detailed definitions for each EPA and SA; a mapping of the EPAs and SAs to the ACGME milestones; and a target level of entrustment for graduating US RAAPM fellows for each EPA and procedural skill. A gap analysis was performed and a heat map was created to cross-check the EPAs and SAs to the ACGME milestones. RESULTS Participants in EPA and SA development included 19 physicians and 1 medical student from 18 different programs. The Delphi rounds yielded a final list of 23 EPAs, a defined entrustment scale, mapping of the EPAs to ACGME milestones, and graduation targets. A list of 73 procedural skills and 7 SAs were similarly developed. DISCUSSION A list of 23 RAAPM EPAs, 73 procedural skills, and 7 SAs were created using a rigorous methodology to reach consensus. This framework can be utilized to help assess RAAPM fellows in the USA for competency and allow for meaningful performance feedback.
Collapse
Affiliation(s)
- Steven Porter
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic Florida, Jacksonville, Florida, USA
| | - Elaine Prendiville
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| | | | - Gregory Booth
- Department of Anesthesiology and Pain Medicine, Naval Medical Center Portsmouth Department of Anesthesiology and Pain Medicine, Portsmouth, Virginia, USA
| | - Jan Boublik
- Anesthesiology, Stanford Hospital and Clinics, Stanford, California, USA
| | - Garrett W Burnett
- Anesthesiology, Perioperative & Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Nabil Elkassabany
- Department of Anesthesiology and Critical Care, University of Pennsylvania Health System, Philadelphia, Pennsylvania, USA
| | - Jonathan Hausman
- Department of Anesthesiology, Cedars-Sinai Medical Center, Los Angeles, California, USA
| | - Lisa Klesius
- Department of Anesthesiology, University of Wisconsin System, Madison, Wisconsin, USA
| | | | - Anthony T Machi
- Anesthesiology and Pain Management, University of Texas Southwestern Medical Center at Dallas, Dallas, Texas, USA
| | - Robert Maniker
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, California, USA
- Department of Anesthesiology, Columbia University Medical Center, New York, New York, USA
| | | | | | - Christina M Spofford
- Department of Anesthesiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Santhanam Suresh
- Pediatric Anesthesiology, Northwestern Medicine, Chicago, Illinois, USA
| | - Tiffany Tedore
- Anesthesiology, NewYork-Presbyterian Hospital/Weill Cornell Medical Center, New York, New York, USA
| | - Elizabeth H Wilson
- Department of Anesthesiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, USA
| | - Jon Yan Zhou
- Anesthesiology and Pain Medicine, University of California Davis Health System, Sacramento, California, USA
| | - Glenn Woodworth
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| |
Collapse
|
13
|
Brazelle M, Zmijewski P, McLeod C, Corey B, Porterfield JR, Lindeman B. Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents. J Am Coll Surg 2022; 234:938-946. [PMID: 35426408 DOI: 10.1097/xcs.0000000000000168] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency. STUDY DESIGN This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings. RESULTS From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05). CONCLUSIONS The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.
Collapse
Affiliation(s)
- Morgan Brazelle
- From the Department of Surgery, Advent Health Central Florida, Orlando, FL (Brazelle)
| | - Polina Zmijewski
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL (Zmijewski, McLeod, Corey, Porterfield, Lindeman)
| | - Chandler McLeod
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL (Zmijewski, McLeod, Corey, Porterfield, Lindeman)
| | - Britney Corey
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL (Zmijewski, McLeod, Corey, Porterfield, Lindeman)
| | - John R Porterfield
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL (Zmijewski, McLeod, Corey, Porterfield, Lindeman)
| | - Brenessa Lindeman
- Department of Surgery, University of Alabama at Birmingham, Birmingham, AL (Zmijewski, McLeod, Corey, Porterfield, Lindeman)
| |
Collapse
|
14
|
Liu L, Jiang Z, Qi X, Xie A, Wu H, Cheng H, Wang W, Li H. An update on current EPAs in graduate medical education: A scoping review. MEDICAL EDUCATION ONLINE 2021; 26:1981198. [PMID: 34569433 PMCID: PMC8477952 DOI: 10.1080/10872981.2021.1981198] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/16/2021] [Revised: 08/23/2021] [Accepted: 09/11/2021] [Indexed: 05/31/2023]
Abstract
The purpose of this scoping review is to update the recent progress of EPAs research in GME, focusing on the topical concern of EPAs effectiveness, and to provide a reference for medical researchers in countries/regions interested in introducing EPAs. Guided by Arksey and O'Malley's framework regarding scoping reviews, the researchers, in January 2021, conducted a search in five databases to ensure the comprehensiveness of the literature. After the predetermined process, 29 articles in total were included in this study. The most common areas for the implementation and evaluation of EPAs were Surgery (n = 7,24.1%), Pediatric (n = 5,17.2%) and Internal medicine (n = 4,13.8%), a result that shows a relatively large change in the research trend of EPAs in the last two years. Prior to 2018, EPAs research focused on internal medicine, psychiatry, family medicine, and primary care. The articles in the category of EPAs implementation and evaluation had four main themes: (1) validation of EPAs (n = 16,55.2%); (2) describing the experience of implementing EPAs (n = 11,37.9%); (3) examining the factors and barriers that influence the implementation and evaluation of EPAs (n = 6,20.6%); and (4) researching the experiences of faculty, interns, and other relevant personnel in using EPAs. Training programs were the most common EPAs implementation setting (n = 26,89.6%); direct observation and evaluation (n = 12,41.4%), and evaluation by scoring reports (n = 5,17.2%) were the two most common means of assessing physicians' EPA levels; 19 papers (65.5%) used faculty evaluation, and nine of these papers also used self-assessment (31.0%); the most frequently used tools in the evaluation of EPAs were mainly researcher-made instruments (n = 37.9%), assessment form (n = 7,24.1%), and mobile application (n = 6,20.7%). Although EPAs occupy an increasingly important place in international medical education, this study concludes that the implementation and diffusion of EPAs on a larger scale is still difficult.
Collapse
Affiliation(s)
- Lu Liu
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Zhehan Jiang
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Xin Qi
- Peking University First Hospital, Beijing, China
| | - A’Na Xie
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Hongbin Wu
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Huaqin Cheng
- Institute of Medical Education, Peking University, Beijing, China
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Weimin Wang
- National Center for Health Professions Education Development, Peking University, Beijing, China
| | - Haichao Li
- Peking University First Hospital, Beijing, China
| |
Collapse
|
15
|
Kearse LE, Schmiederer IS, Anderson TN, Dent DL, Payne DH, Korndorffer JR. American Board of Surgery Entrustable Professional Activities (EPAs): Assessing Graduating Residents' Perception of Preoperative Entrustment. JOURNAL OF SURGICAL EDUCATION 2021; 78:e183-e188. [PMID: 34602378 DOI: 10.1016/j.jsurg.2021.09.004] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/25/2021] [Revised: 07/21/2021] [Accepted: 09/09/2021] [Indexed: 06/13/2023]
Abstract
OBJECTIVE To determine if graduating surgical residents are achieving entrustment of surgical entrustable professional activities (EPAs). We hypothesize that postgraduate year 5 (PGY5) residents are achieving evaluation and management entrustment in the selected EPAs. DESIGN In January 2020, surgical residents completed a survey following the American Board of Surgery In-Training Examination (ABSITE) to measure their levels of entrustment in 4 of the 5 ABS-selected EPAs. A Resident Entrustability Index (REI) score was developed to ascertain PGY5 residents' levels of entrustment (range 1-5). Residents indicated how often their assessments and operative plans were modified in the prior 6 months for each EPA (1=Always, 2=Frequently, 3=Occasionally, 4=Rarely, 5=Never). An independent clinical decision-making score (ICDM) was developed with subsequent evaluation of its relationships to intrinsic, resident-related and extrinsic, program-dependent factors. SETTING A national post-ABSITE survey. PARTICIPANTS All general surgery residents participating in ABSITE were invited to participate. Of the 1367 PGY5 residents that completed the survey, 1049 residents (76.7%) responded to the surgical EPA items. RESULTS Residents achieved an average REI of 4, indicating rare modification of assessments and operative plans for the 4 EPAs assessed. Complete entrustment was reported for inguinal hernias and penetrating abdominal trauma (Median REI = 5, IQR 4, 5) indicating assessments and operative plans were never modified. Lack of entrustment (REI ≤3) was reported by a minority of residents (ranging from 8.6% for operative plan of right lower quadrant pathology to 12.8% for operative plan of blunt abdominal trauma). Significant resident-related and program-dependent factors associated with achievement of expected ICDM was socializing with a co-resident (p = 0.001), while training in one's hometown (p < 0.001) and policies that mandate attendings be scrubbed in (p = 0.022) were associated with decreased achievement of expected ICDM. Overall, 89.2% and 90.3% of PGY5 residents are attaining appropriate levels of entrustment and ICDM abilities, respectively, within 6 months of graduating. CONCLUSIONS Of the EPAs evaluated, PGY5 residents are achieving appropriate levels of entrustment in evaluation and management. Although this is the case for a vast majority of PGY5 residents, there is still work to be done to ensure that all PGY5 residents are attaining entrustment prior to graduation. Our study also provides content validity for the surgical EPAs in assessing levels of entrustment in PGY5 residents.
Collapse
Affiliation(s)
- LaDonna E Kearse
- Department of Surgery, Stanford University, Stanford, California
| | | | | | - Daniel L Dent
- Department of Surgery, University of Texas Health at San Antonio, San Antonio, Texas
| | - Davis H Payne
- Department of Surgery, University of Texas Health at San Antonio, San Antonio, Texas
| | | |
Collapse
|
16
|
Schwartz A, Borman-Shoap E, Carraccio C, Herman B, Hobday PM, Kaul P, Long M, O'Connor M, Mink R, Schumacher DJ, Turner DA, West DC. Learner Levels of Supervision Across the Continuum of Pediatrics Training. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S42-S49. [PMID: 34183601 DOI: 10.1097/acm.0000000000004095] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship. METHOD The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings. RESULTS Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency. CONCLUSIONS Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.
Collapse
Affiliation(s)
- Alan Schwartz
- A. Schwartz is Michael Reese Endowed Professor of Medical Education and research professor, pediatrics, University of Illinois College of Medicine, Chicago, Illinois, and network director, Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN), McLean, Virginia; ORCID: http://orcid.org/0000-0003-3809-6637
| | - Emily Borman-Shoap
- E. Borman-Shoap is associate professor, residency program director, and vice chair of education, Department of Pediatrics, University of Minnesota, Minneapolis, Minnesota; ORCID: https://orcid.org/0000-0001-7514-7793
| | - Carol Carraccio
- C. Carraccio was vice president for competency-based medical education, American Board of Pediatrics, Chapel Hill, North Carolina, at the time the work was completed; ORCID: https://orcid.org/0000-0001-5473-8914
| | - Bruce Herman
- B. Herman is professor and vice chair of education, Department of Pediatrics, University of Utah, Salt Lake City, Utah
| | - Patricia M Hobday
- P.M. Hobday is assistant professor and director, Education in Pediatrics Across the Continuum (EPAC), Department of Pediatrics, University of Minnesota, Minneapolis, Minnesota
| | - Paritosh Kaul
- P. Kaul is professor and associate director, EPAC, Department of Pediatrics, University of Colorado, School of Medicine, Denver, Colorado; ORCID: https://orcid.org/0000-0003-4155-7406
| | - Michele Long
- M. Long is professor and director, EPAC, Department of Pediatrics, University of California, San Francisco, San Francisco, California; ORCID: https://orcid.org/0000-0002-8399-5589
| | - Meghan O'Connor
- M. O'Connor is assistant professor and director, EPAC, Department of Pediatrics, University of Utah, Salt Lake City, Utah; ORCID: https://orcid.org/0000-0002-5403-923X
| | - Richard Mink
- R. Mink is professor of pediatrics, David Geffen School of Medicine at the University of California, Los Angeles (UCLA), Los Angeles, California, pediatric critical care fellowship program director, Harbor-UCLA Medical Center, Torrance, California, and director, APPD Subspecialty Pediatrics Investigator Network, McLean, Virginia; ORCID: http://orcid.org/0000-0002-7998-4713
| | - Daniel J Schumacher
- D.J. Schumacher is associate professor, Department of Pediatrics, Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0001-5507-8452
| | - David A Turner
- D.A. Turner was associate professor, Division of Pediatric Critical Care, Department of Pediatrics, and associate director, Graduate Medical Education, Duke University Hospital and Health System, Durham, North Carolina, at the time the work was completed
| | - Daniel C West
- D.C. West is professor and senior director of medical education, Department of Pediatrics, Children's Hospital of Philadelphia and The Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-0909-4213
| |
Collapse
|
17
|
Taylor N, Defenbaugh N, Mira AE, Bendas E. What's new in academic medicine? Focus on evolving models of competence in Graduate Medical Education. INTERNATIONAL JOURNAL OF ACADEMIC MEDICINE 2021. [DOI: 10.4103/ijam.ijam_30_21] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
|