1
|
Ryan MS, Gielissen KA, Shin D, Perera RA, Gusic M, Ferenchick G, Ownby A, Cutrer WB, Obeso V, Santen SA. How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study. MEDICAL EDUCATION 2024; 58:825-837. [PMID: 38167833 DOI: 10.1111/medu.15291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 10/27/2023] [Accepted: 11/16/2023] [Indexed: 01/05/2024]
Abstract
BACKGROUND Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs. METHODS The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability. RESULTS Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability. CONCLUSION The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.
Collapse
Affiliation(s)
- Michael S Ryan
- Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA
- School of Health Professions Education, Maastricht University, Maastricht, The Netherlands
| | - Katherine A Gielissen
- Departments of Medicine and Pediatrics, Emory University School of Medicine, Atlanta, Georgia, USA
| | - Dongho Shin
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Robert A Perera
- Department of Biostatistics, Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
| | - Maryellen Gusic
- Departments of Pediatrics, Biomedical Education and Data Science, Lewis Katz School of Medicine, Philadelphia, Pennsylvania, USA
| | - Gary Ferenchick
- Department of Medicine, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
| | - Allison Ownby
- McGovern Medical School at UTHealth Houston, Houston, Texas, USA
| | - William B Cutrer
- Department of Pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee, USA
| | - Vivian Obeso
- Department of Medical Education, University of Miami Miller School of Medicine, Miami, Florida, USA
| | - Sally A Santen
- Virginia Commonwealth University School of Medicine, Richmond, Virginia, USA
- Emergency Medicine and Medical Education at University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
2
|
Schumacher DJ, Michelson C, Winn AS, Turner DA, Martini A, Kinnear B. A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees. MEDICAL EDUCATION 2024; 58:812-824. [PMID: 38088227 DOI: 10.1111/medu.15296] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 11/03/2023] [Accepted: 11/21/2023] [Indexed: 06/04/2024]
Abstract
INTRODUCTION The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why? METHODS Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs. RESULTS PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes. DISCUSSION PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.
Collapse
Affiliation(s)
- Daniel J Schumacher
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| | - Catherine Michelson
- Ann and Robert H. Lurie Children's Hospital of Chicago/Northwestern University, Chicago, Illinois, USA
| | - Ariel S Winn
- Boston Children's Hospital/Harvard Medical School, Boston, Massachusetts, USA
| | - David A Turner
- American Board of Pediatrics, Chapel Hill, North Carolina, USA
| | - Abigail Martini
- Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
| | - Benjamin Kinnear
- Cincinnati Children's Hospital Medical Center/University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
| |
Collapse
|
3
|
Fuller SL, Ambardekar AP, Diachun CAB, Kearney MD, Long TR, Miller Juve AK, Mitchell JD, Woodworth GE. Competency-Based Time-Variable Anesthesiology Residency Training: Identification of Problems and Solutions. Anesth Analg 2024; 138:848-855. [PMID: 37450642 DOI: 10.1213/ane.0000000000006625] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/18/2023]
Abstract
BACKGROUND Global medical education is gradually moving toward more comprehensive implementations of a competency-based education (CBE) model. Elimination of standard time-based training and adoption of time-variable training (competency-based time-variable training [CB-TVT]) is one of the final stages of implementation of CBE. While CB-TVT has been implemented in some programs outside the United States, residency programs in the United States are still exploring this approach to training. The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) are encouraging member boards and residency review committees to consider innovative ways programs could implement CB-TVT. The goals of this study were to (1) identify potential problems with the implementation of CB-TVT in anesthesiology residency training, (2) rank the importance of the problems and the perceived difficulty of solving them, and (3) develop proposed solutions to the identified problems. METHODS Study participants were recruited from key stakeholder groups in anesthesiology education, including current or former program directors, department chairs, residents, fellows, American Board of Anesthesiology (ABA) board members, ACGME residency review committee members or ACGME leaders, designated institutional officials, residency program coordinators, clinical operations directors, and leaders of large anesthesiology community practice groups. This study was conducted in 2 phases. In phase 1, survey questionnaires were iteratively distributed to participants to identify problems with the implementation of CB-TVT. Participants were also asked to rank the perceived importance and difficulty of each problem and to identify relevant stakeholder groups that would be responsible for solving each problem. In phase 2, surveys focused on identifying potential solutions for problems identified in phase 1. RESULTS A total of 36 stakeholders identified 39 potential problems, grouped into 7 major categories, with the implementation of CB-TVT in anesthesiology residency training. Of the 39 problems, 19 (48.7%) were marked as important or very important on a 5-point scale and 12 of 19 (63.2%) of the important problems were marked as difficult or very difficult to solve on a 5-point scale. Stakeholders proposed 165 total solutions to the identified problems. CONCLUSIONS CB-TVT is a promising educational model for anesthesiology residency, which potentially results in learner flexibility, individualization of curricula, and utilization of competencies to determine learner advancement. Because of the potential problems with the implementation of CB-TVT, it is important for future pilot implementations of CB-TVT to document realized problems, efficacy of solutions, and effects on educational outcomes to justify the burden of implementing CB-TVT.
Collapse
Affiliation(s)
- Skylar L Fuller
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - Aditee P Ambardekar
- Department of Anesthesiology and Pain Management, University of Texas, Southwestern Medical Center, Dallas, Texas
| | - Carol Ann B Diachun
- Department of Anesthesiology, University of Florida-Jacksonville, Jacksonville, Florida
| | - Matthew D Kearney
- Department of Family Medicine and Community Health, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania
| | - Timothy R Long
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, Minnesota
| | - Amy K Miller Juve
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| | - John D Mitchell
- Department of Anesthesiology, Critical Care, and Perioperative Medicine, Henry Ford Health, Detroit, Michigan
| | - Glenn E Woodworth
- From the Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon
| |
Collapse
|
4
|
Caretta-Weyer HA, Schumacher DJ, Kinnear B. Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:243-246. [PMID: 38011041 DOI: 10.1097/acm.0000000000005578] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
ABSTRACT In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.
Collapse
|
5
|
Gummesson C, Alm S, Cederborg A, Ekstedt M, Hellman J, Hjelmqvist H, Hultin M, Jood K, Leanderson C, Lindahl B, Möller R, Rosengren B, Själander A, Svensson PJ, Särnblad S, Tejera A. Entrustable professional activities (EPAs) for undergraduate medical education - development and exploration of social validity. BMC MEDICAL EDUCATION 2023; 23:635. [PMID: 37667366 PMCID: PMC10478490 DOI: 10.1186/s12909-023-04621-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 08/28/2023] [Indexed: 09/06/2023]
Abstract
BACKGROUND The development of entrustable professional activities (EPAs) as a framework for work-based training and assessment in undergraduate medical education has become popular. EPAs are defined as units of a professional activity requiring adequate knowledge, skills, and attitudes, with a recognized output of professional labor, independently executable within a time frame, observable and measurable in its process and outcome, and reflecting one or more competencies. Before a new framework is implemented in a specific context, it is valuable to explore social validity, that is, the acceptability by relevant stakeholders. AIM The aim of our work was to define Core EPAs for undergraduate medical education and further explore the social validity of the constructs. METHOD AND MATERIAL In a nationwide collaboration, EPAs were developed using a modified Delphi procedure and validated according to EQual by a group consisting of teachers nominated from each of the seven Swedish medical schools, two student representatives, and an educational developer (n = 16). In the next step, social validity was explored in a nationwide survey. The survey introduced the suggested EPAs. For each EPA, the importance of the EPA was rated, as was the rater's perception of the present graduates' required level of supervision when performing the activity. Free-text comments were also included and analyzed. RESULTS Ten Core EPAs were defined and validated. The validation scores for EQual ranged from 4.1 to 4.9. The nationwide survey had 473 responders. All activities were rated as "important" by most responders, ranging from 54 to 96%. When asked how independent current graduates were in performing the ten activities, 6 to 35% reported "independent". The three themes of the free text comments were: 'relevant target areas and content'; 'definition of the activities'; and 'clinical practice and learning'. CONCLUSION Ten Core EPAs were defined and assessed as relevant for Swedish undergraduate medical education. There was a consistent gap between the perceived importance and the certainty that the students could perform these professional activities independently at the time of graduation. These results indicate that the ten EPAs may have a role in undergraduate education by creating clarity for all stakeholders.
Collapse
Affiliation(s)
- Christina Gummesson
- Faculty of Medicine, Faculty of Odontology, Lund University, Malmö University, Malmö, Sweden.
| | - Stina Alm
- Department of Clinical Sciences, Futurum - the Academy for Health and Care, Region Jönköping County, Umeå University, Paediatrics, Umeå, Sweden
| | - Anna Cederborg
- Institute of Medicine, Sahlgrenska Academy, Department of Medicine, University of Gothenburg, Sahlgrenska University Hospital, Gothenburg, Sweden
| | - Mattias Ekstedt
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, 581 83, Sweden
| | - Jarl Hellman
- Department of Medical Sciences, Uppsala University, Uppsala, Sweden
| | - Hans Hjelmqvist
- Department of Anesthesiology and Intensive Care, Faculty of Medicine and Health, Örebro University, Örebro, Sweden
| | - Magnus Hultin
- Department of Surgical and Perioperative Sciences, Anesthesiology and Intensive Care Medicine, Umeå University, Umeå, Sweden
| | - Katarina Jood
- Department of Clinical Neuroscience, Institute of Neuroscience and Physiology, Sahlgrenska Academy, Department of Neurology, University of Gothenburg, Sahlgrenska University Hospital, Gothenburg, Sweden
| | - Charlotte Leanderson
- Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden
| | - Bertil Lindahl
- Department of Medical Sciences, Uppsala University, Uppsala, Sweden
| | - Riitta Möller
- Department of Clinical Science, Intervention and Technology, Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Karolinska Institutet, Stockholm, Sweden
| | - Björn Rosengren
- Clinical and Molecular Osteoporosis Research Unit, Department of Orthopedics and Clinical Sciences, Skåne University Hospital Malmö, Lund University, Malmö, Sweden
| | - Anders Själander
- Department of Public Health and Clinical Medicine, Umeå University, Umeå, Sweden
| | - Peter J Svensson
- Department of Clinical Sciences Lund University, Skåne University Hospital, Malmö, Sweden
| | - Stefan Särnblad
- Department of Pediatrics, Faculty of Medicine and Health, Örebro University, Örebro, Sweden
| | - Alexander Tejera
- Division of Translational Cancer Research, Department of Laboratory Medicine, Skåne University Hospital, Lund University, Lund University, Lund, Sweden
| |
Collapse
|
6
|
Eliasz KL, Nick MW, Zabar S, Buckvar-Keltz L, Ng GM, Riles TS, Kalet AL. Viewing Readiness-for-Residency through Binoculars: Mapping Competency-Based Assessments to the AAMC's 13 Core Entrustable Professional Activities (EPAs). TEACHING AND LEARNING IN MEDICINE 2023; 35:436-441. [PMID: 35668557 DOI: 10.1080/10401334.2022.2082432] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/04/2020] [Accepted: 05/04/2022] [Indexed: 06/15/2023]
Abstract
Construct: The construct being assessed is readiness-for-residency of graduating medical students, as measured through two assessment frameworks. Background: Readiness-for-residency of near-graduate medical students should be but is not consistently assessed. To address this, the Association of American Medical Colleges (AAMC), in 2014, identified and described 13 core Entrustable Professional Activities (EPAs), which are tasks that all residents should be able to perform unsupervised upon entering residency. However, the AAMC did not initially provide measurement guidelines or propose standardized assessments. We designed Night-onCall (NOC), an immersive simulation for our near-graduating medical students to assess and address their readiness-for-residency, framed around tasks suggested by the AAMC's core EPAs. In adopting this EPA assessment framework, we began by building upon an established program of competency-based clinical skills assessments, repurposing competency-based checklists to measure components of the EPAs where possible, and designing new checklists, when necessary. This resulted in a blended suite of 14 checklists, which theoretically provide substantive assessment of all 13 core EPAs. In this paper, we describe the consensus-based mapping process conducted to ensure we understood the relationship between competency and EPA-based assessment lenses and could therefore report meaningful feedback on both to transitioning students in the NOC exercise. Approach: Between January-November 2017, five clinician and two non-clinician health professions educators at NYU Grossman School of Medicine conducted a rigorous consensus-based mapping process, which included each rater mapping each of the 310 NOC competency-based checklist items to lists of entrustable behaviors expected of learners according to the AAMC 13 core EPAs. Findings: All EPAs were captured to varying degrees by the 14 NOC checklists (overall Intraclass Correlation Coefficient (ICC) = 0.77). Consensus meetings resolved discrepancies and improved ICC values for three (EPA-9, EPA-10, EPA-12) of the four EPAs that initially showed poor reliability. Conclusions: Findings suggest that with some limitations (e.g., EPA-7 "form clinical questions/retrieve evidence") established competency-based assessments can be repurposed to measure readiness-for-residency through an EPA lens and both can be reported to learners and faculty.
Collapse
Affiliation(s)
- Kinga L Eliasz
- Division of General Internal Medicine and Clinical Innovation, New York University Grossman School of Medicine, New York, New York, USA
- Department of Medicine, New York University Grossman School of Medicine, New York, New York, USA
| | - Michael W Nick
- Program on Medical Education and Technology, New York University Grossman School of Medicine, New York, New York, USA
| | - Sondra Zabar
- Division of General Internal Medicine and Clinical Innovation, New York University Grossman School of Medicine, New York, New York, USA
- Department of Medicine, New York University Grossman School of Medicine, New York, New York, USA
| | - Lynn Buckvar-Keltz
- Department of Medicine, New York University Grossman School of Medicine, New York, New York, USA
| | - Grace M Ng
- New York Simulation Center for the Health Sciences, A Partnership of the City University of New York and New York University Grossman School of Medicine, New York, New York, USA
| | - Thomas S Riles
- Departments of Surgery and Medical Education and Technology, New York University Grossman School of Medicine, New York, New York, USA
| | - Adina L Kalet
- Robert D. and Patricia E. Kern Institute for the Transformation of Medical Education at Medical College of Wisconsin, Wauwatosa, Wisconsin, USA
| |
Collapse
|
7
|
Park YS, Sachdeva AK, Liscum K, Alseidi A, Gesbeck M, Blair PG, Salcedo E, Sullivan M, Bordage G. The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study. Ann Surg 2023; 277:704-711. [PMID: 34954752 DOI: 10.1097/sla.0000000000005358] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
OBJECTIVE To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program. SUMMARY AND BACKGROUND DATA ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors. METHODS Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences. RESULTS Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%). CONCLUSIONS ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.
Collapse
Affiliation(s)
- Yoon Soo Park
- Massachusetts General Hospital, Harvard Medical School, Boston, MA
- Department of Medical Education, University of Illinois at Chicago, Chicago, IL
| | - Ajit K Sachdeva
- Division of Education, American College of Surgeons, Chicago, IL
| | - Kathy Liscum
- Division of Education, American College of Surgeons, Chicago, IL
| | - Adnan Alseidi
- Department of Surgery, University of California San Francisco, San Francisco, CA
| | - Melissa Gesbeck
- Division of Education, American College of Surgeons, Chicago, IL
| | | | - Edgardo Salcedo
- Department of Surgery, University of California Davis, Sacramento, CA
| | - Maura Sullivan
- Department of Surgery, University of Southern California, Los Angeles, CA
| | - Georges Bordage
- Department of Medical Education, University of Illinois at Chicago, Chicago, IL
| |
Collapse
|
8
|
Bremer AE, van de Pol MHJ, Laan RFJM, Fluit CRMG. An Innovative Undergraduate Medical Curriculum Using Entrustable Professional Activities. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2023; 10:23821205231164894. [PMID: 37123076 PMCID: PMC10134152 DOI: 10.1177/23821205231164894] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/07/2022] [Accepted: 03/02/2023] [Indexed: 05/03/2023]
Abstract
The need to educate medical professionals in changing medical organizations has led to a revision of the Radboudumc's undergraduate medical curriculum. Entrustable professional activities (EPAs) were used as a learning tool to support participation and encourage feedback-seeking behavior, in order to offer students the best opportunities for growth. This paper describes the development of the Radboudumc's EPA-based Master's curriculum and how EPAs can facilitate continuity in learning in the clerkships. Four guiding principles were used to create a curriculum that offers possibilities for the students' development: (1) working with EPAs, (2) establishing entrustment, (3) providing continuity in learning, and (4) organizing smooth transitions. The new curriculum was designed with the implementation of EPAs and an e-portfolio, based on these 4 principles. The authors found that the revised curriculum corresponds to daily practice in clerkships. Students used their e-portfolios throughout all clerkships, which stimulates feedback-seeking behavior. Moreover, EPAs promote continuity in learning while rotating clerkships every 1 to 2 months. This might encourage curriculum developers to use EPAs when aiming for greater continuity in the development of students. Future research needs to focus on the effect of EPAs on transitions across clerkships in order to further improve the undergraduate medical curriculum.
Collapse
Affiliation(s)
- Anne E Bremer
- Radboud Institute for Health Sciences,
Department of Radboudumc Health Academy, Radboud University Medical Center, Nijmegen, the Netherlands
- Anne E Bremer, Radboudumc Health Academy,
Postbus 9101, 6500 HB, Nijmegen, The Netherlands.
| | - Marjolein H J van de Pol
- Department of Primary and Community
Care, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Roland F J M Laan
- Department of Radboudumc Health Academy, Radboud University Medical Center, Nijmegen, the
Netherlands
| | - Cornelia R M G Fluit
- Department of Radboudumc Health Academy, Radboud University Medical Center, Nijmegen, the
Netherlands
| |
Collapse
|
9
|
Kuehl SE, Spicer JO. Using entrustable professional activities to better prepare students for their postgraduate medical training: A medical student's perspective. PERSPECTIVES ON MEDICAL EDUCATION 2022; 11:359-364. [PMID: 36441351 PMCID: PMC9743878 DOI: 10.1007/s40037-022-00731-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 10/12/2022] [Accepted: 10/14/2022] [Indexed: 06/16/2023]
Abstract
THE PROBLEM Medical students graduate underprepared for postgraduate medical training despite years of classroom and clinical training. In this article, a medical student shares her personal perspectives on three factors contributing to this problem in undergraduate medical education: students' peripheral roles in the clinical environment impede learning, students receive inadequate feedback, and assessments do not measure desired learning outcomes. A SOLUTION The authors describe how using entrustable professional activities (EPAs) could address these issues and promote students' clinical engagement by clarifying their roles, providing them with frequent and actionable feedback, and aligning their assessments with authentic work. These factors combined with grading schemes rewarding improvement could contribute to a growth mindset that reprioritizes clinical skill acquisition. The authors explore how medical schools have begun implementing the EPA framework, highlight insights from these efforts, and describe barriers that must be addressed. THE FUTURE Incorporating EPAs into medical school curricula could better prepare students for postgraduate training while also alleviating issues that contribute to student burnout by defining students' roles, improving feedback, and aligning assessments with desired learning outcomes.
Collapse
Affiliation(s)
- Sarah E Kuehl
- Emory University School of Medicine and Goizueta Business School, Atlanta, GA, USA.
| | - Jennifer O Spicer
- J. Willis Hurst Internal Medicine Residency Program, Division of Infectious Diseases, Department of Medicine, Emory University School of Medicine, Atlanta, GA, USA
| |
Collapse
|
10
|
Brown DR, Moeller JJ, Grbic D, Andriole DA, Cutrer WB, Obeso VT, Hormann MD, Amiel JM. Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020. JAMA Netw Open 2022; 5:e2233342. [PMID: 36156144 PMCID: PMC9513644 DOI: 10.1001/jamanetworkopen.2022.33342] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
IMPORTANCE Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency. OBJECTIVE To assess progress in developing an entrustment process in the Core EPAs framework. DESIGN, SETTING, AND PARTICIPANTS In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database. INTERVENTIONS Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations. MAIN OUTCOMES AND MEASURES On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded. RESULTS Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs. CONCLUSIONS AND RELEVANCE These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.
Collapse
Affiliation(s)
- David R. Brown
- Division of Family and Community Medicine, Department of Humanities, Health, and Society, Florida International University Herbert Wertheim College of Medicine, Miami
| | - Jeremy J. Moeller
- Department of Neurology, Yale University School of Medicine, New Haven, Connecticut
| | - Douglas Grbic
- Medical Education Research, Association of American Medical Colleges, Washington, District of Columbia
| | - Dorothy A. Andriole
- Medical Education Research, Association of American Medical Colleges, Washington, District of Columbia
| | - William B. Cutrer
- Department of Pediatrics, Division of Critical Care Medicine at Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Vivian T. Obeso
- Division of Internal Medicine, Department of Translational Medicine, Florida International University Herbert Wertheim College of Medicine, Miami
| | - Mark D. Hormann
- Division of Community and General Pediatrics, Department of Pediatrics, McGovern Medical School at UTHealth, Houston, Texas
| | - Jonathan M. Amiel
- Dean’s Office, Columbia University Vagelos College of Physicians and Surgeons, New York, New York
- Department of Psychiatry, Columbia University Vagelos College of Physicians and Surgeons, New York, New York
| |
Collapse
|
11
|
Zelewski SK, Basson MD. Design and implementation of a core EPA-based acting internship curriculum. MEDICAL TEACHER 2022; 44:922-927. [PMID: 35358009 DOI: 10.1080/0142159x.2022.2049732] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE The Association of American Medical Colleges published Core Entrustable Professional Activities (Core EPAs), expected independent clinical skills for first-day interns. We sought to determine whether a required acting internship (AI) in the fourth-year curriculum could be used for summative assessment of students' mastery of Core EPAs to a predefined level that would readily generalize across disciplines and campuses. METHODS The University of North Dakota School of Medicine and Health Sciences MD Program created a standardized, required Core EPA-based AI curriculum for multiple specialties at multiple geographic sites providing a final entrustability assessment for 10 EPAs in a single course. RESULTS The course was successfully designed and launched for all students in a single class. During the AI, students functioned at the level of an acting intern, rated the courses as superior, and performed at satisfactory exit-level competence for 10 Core EPAs. CONCLUSIONS A standardized, EPA-based AI curriculum can provide an opportunity for exit level EPA assessment in the medical curriculum. This model functions well within multiple specialties and at diverse community-based, volunteer faculty teaching sites.
Collapse
Affiliation(s)
- Susan K Zelewski
- Department of Pediatrics, Assistant Dean Northeast Campus and Director of the Year 3 and 4 MD Curriculum, School of Medicine and Health Sciences, University of North Dakota, Grand Forks, ND, USA
| | - Marc D Basson
- Department of Surgery, Biomedical Sciences, and Pathology, and Senior Associate Dean for Medicine and Research, School of Medicine and the Health Sciences, University of North Dakota, Grand Forks, ND, USA
| |
Collapse
|
12
|
Porter S, Prendiville E, Allen BFS, Booth G, Boublik J, Burnett GW, Elkassabany N, Hausman J, Klesius L, Le-Wendling L, Machi AT, Maniker R, Parra M, Rosenquist R, Spofford CM, Suresh S, Tedore T, Wilson EH, Zhou JY, Woodworth G. Development of entrustable professional activities for regional anesthesia and pain medicine fellowship training. Reg Anesth Pain Med 2022; 47:rapm-2022-103854. [PMID: 35878963 DOI: 10.1136/rapm-2022-103854] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 07/14/2022] [Indexed: 11/04/2022]
Abstract
INTRODUCTION The Accreditation Council for Graduate Medical Education (ACGME) offers descriptions of competencies and milestones but does not provide standardized assessments to track trainee competency. Entrustable professional activities (EPAs) and special assessments (SAs) are emerging methods to assess the level of competency obtained by regional anesthesiology and acute pain medicine (RAAPM) fellows. METHODS A panel of RAAPM physicians with experience in education and competency assessment and one medical student were recruited to participate in a modified Delphi method with iterative rounds to reach consensus on: a list of EPAs, SAs, and procedural skills; detailed definitions for each EPA and SA; a mapping of the EPAs and SAs to the ACGME milestones; and a target level of entrustment for graduating US RAAPM fellows for each EPA and procedural skill. A gap analysis was performed and a heat map was created to cross-check the EPAs and SAs to the ACGME milestones. RESULTS Participants in EPA and SA development included 19 physicians and 1 medical student from 18 different programs. The Delphi rounds yielded a final list of 23 EPAs, a defined entrustment scale, mapping of the EPAs to ACGME milestones, and graduation targets. A list of 73 procedural skills and 7 SAs were similarly developed. DISCUSSION A list of 23 RAAPM EPAs, 73 procedural skills, and 7 SAs were created using a rigorous methodology to reach consensus. This framework can be utilized to help assess RAAPM fellows in the USA for competency and allow for meaningful performance feedback.
Collapse
Affiliation(s)
- Steven Porter
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic Florida, Jacksonville, Florida, USA
| | - Elaine Prendiville
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| | | | - Gregory Booth
- Department of Anesthesiology and Pain Medicine, Naval Medical Center Portsmouth Department of Anesthesiology and Pain Medicine, Portsmouth, Virginia, USA
| | - Jan Boublik
- Anesthesiology, Stanford Hospital and Clinics, Stanford, California, USA
| | - Garrett W Burnett
- Anesthesiology, Perioperative & Pain Medicine, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Nabil Elkassabany
- Department of Anesthesiology and Critical Care, University of Pennsylvania Health System, Philadelphia, Pennsylvania, USA
| | - Jonathan Hausman
- Department of Anesthesiology, Cedars-Sinai Medical Center, Los Angeles, California, USA
| | - Lisa Klesius
- Department of Anesthesiology, University of Wisconsin System, Madison, Wisconsin, USA
| | | | - Anthony T Machi
- Anesthesiology and Pain Management, University of Texas Southwestern Medical Center at Dallas, Dallas, Texas, USA
| | - Robert Maniker
- Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, California, USA
- Department of Anesthesiology, Columbia University Medical Center, New York, New York, USA
| | | | | | - Christina M Spofford
- Department of Anesthesiology, Medical College of Wisconsin, Milwaukee, Wisconsin, USA
| | - Santhanam Suresh
- Pediatric Anesthesiology, Northwestern Medicine, Chicago, Illinois, USA
| | - Tiffany Tedore
- Anesthesiology, NewYork-Presbyterian Hospital/Weill Cornell Medical Center, New York, New York, USA
| | - Elizabeth H Wilson
- Department of Anesthesiology, University of Wisconsin-Madison School of Medicine and Public Health, Madison, Wisconsin, USA
| | - Jon Yan Zhou
- Anesthesiology and Pain Medicine, University of California Davis Health System, Sacramento, California, USA
| | - Glenn Woodworth
- Department of Anesthesiology and Perioperative Medicine, Oregon Health & Science University, Portland, Oregon, USA
| |
Collapse
|
13
|
Chang YC, Chuang RS, Hsiao CT, Khwepeya M, Nkambule NS. Bridging the Gap: Using Consensus to Explore Entrustment Decisions and Feedback Receptivity in Competency-Based Emergency Medicine Residency Programs Through the Construction of a Q-Sample Incorporating a Delphi Technique. Front Med (Lausanne) 2022; 9:879271. [PMID: 35721074 PMCID: PMC9201255 DOI: 10.3389/fmed.2022.879271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2022] [Accepted: 05/04/2022] [Indexed: 11/13/2022] Open
Abstract
Background Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively. Methods In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study. Results A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning. Conclusion Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.
Collapse
Affiliation(s)
- Yu-Che Chang
- Chang Gung Medical Education Research Centre (CG-MERC), Chang Gung Memorial Hospital, Taoyuan, Taiwan
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan
- College of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Renee S. Chuang
- Chang Gung Medical Education Research Centre (CG-MERC), Chang Gung Memorial Hospital, Taoyuan, Taiwan
| | - Cheng-Ting Hsiao
- Chang Gung Medical Education Research Centre (CG-MERC), Chang Gung Memorial Hospital, Taoyuan, Taiwan
- College of Medicine, Chang Gung University, Taoyuan, Taiwan
- Department of Emergency Medicine, Chang Gung Memorial Hospital, Chiayi, Taiwan
| | - Madalitso Khwepeya
- Chang Gung Medical Education Research Centre (CG-MERC), Chang Gung Memorial Hospital, Taoyuan, Taiwan
| | - Nothando S. Nkambule
- International Graduate Program of Education and Human Development (IGPEHD), College of Social Sciences, National Sun Yat-sen University, Kaohsiung, Taiwan
- *Correspondence: Nothando S. Nkambule
| |
Collapse
|
14
|
Klapheke M, Abrams MP, Cubero M, Zhu X. Aligning Medical Student Workplace-Based Assessments with Entrustable Professional Activities and the RIME Model in a Psychiatry Clerkship. ACADEMIC PSYCHIATRY : THE JOURNAL OF THE AMERICAN ASSOCIATION OF DIRECTORS OF PSYCHIATRIC RESIDENCY TRAINING AND THE ASSOCIATION FOR ACADEMIC PSYCHIATRY 2022; 46:283-288. [PMID: 35288865 DOI: 10.1007/s40596-022-01614-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/13/2021] [Accepted: 02/18/2022] [Indexed: 06/14/2023]
Abstract
OBJECTIVE The authors piloted use of workplace-based assessments of students during the psychiatry clerkship utilizing both entrustable professional activities (EPAs) and the reporter, interpreter, manager, and educator (RIME) model. METHODS After supervising clinicians conducted assessments of medical students (N=109) during the psychiatry clerkship using a supervisory scale aligned with both EPA and RIME models, each student received individualized formative feedback. Students were then surveyed on the usefulness of this feedback, and participating faculty/residents were surveyed on the ease of completion of the supervisory scale. RESULTS Students' mean skill profile suggested they no longer needed direct supervision on EPA1 and EPA6. Mean scores on other studied EPAs suggested students were well on their way toward performing these EPAs without direct supervision. Students had mean RIME scores that exceeded the suggested levels identified for a Reporter to start clerkships, for an Interpreter to start clerkships, and for a Manager to transition to the fourth year. Close to half of the students found the feedback helpful in their development as a clinician but most felt their performance should not be shared with residency program directors, either before or after the Match. Almost all responding preceptors felt the supervisory ratings were easy to complete. CONCLUSIONS This pilot RIME/EPA framework served as a successful step toward a more competency-based medical education in the psychiatry clerkship with relatively little additional faculty time commitment by using workplace-based assessments already in place and a supervisory scale based on EPAs and RIME.
Collapse
Affiliation(s)
| | | | | | - Xiang Zhu
- University of Central Florida, Orlando, FL, USA
| |
Collapse
|
15
|
Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:552-561. [PMID: 34074896 DOI: 10.1097/acm.0000000000004189] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Rebecca Khamishon
- R. Khamishon is a fourth-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Adam Garber
- A. Garber is associate professor, Department of Internal Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-7296-2896
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
16
|
Brown DR, Moeller JJ, Grbic D, Biskobing DM, Crowe R, Cutrer WB, Green ML, Obeso VT, Wagner DP, Warren JB, Yingling SL, Andriole DA. Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:536-543. [PMID: 34261864 DOI: 10.1097/acm.0000000000004242] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions. METHOD For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables. RESULTS The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05). CONCLUSIONS TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.
Collapse
Affiliation(s)
- David R Brown
- D.R. Brown is professor, chief, Division of Family and Community Medicine, and interim chair, Department of Humanities, Health, and Society, Florida International University Herbert Wertheim College of Medicine, Miami, Florida; ORCID: http://orcid.org/0000-0002-5361-6664
| | - Jeremy J Moeller
- J.J. Moeller is associate professor and residency program director, Department of Neurology, Yale University School of Medicine, New Haven, Connecticut; ORCID: https://orcid.org/0000-0002-6135-5572
| | - Douglas Grbic
- D. Grbic is lead research analyst, Medical Education Research, Association of American Medical Colleges, Washington, DC
| | - Diane M Biskobing
- D.M. Biskobing is professor of medicine and associate dean of medical education, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| | - Ruth Crowe
- R. Crowe is director of integrated clinical skills, director of practice of medicine, Office of Medical Education, and associate professor of medicine, New York University Grossman School of Medicine, New York, New York
| | - William B Cutrer
- W.B. Cutrer is associate dean for undergraduate medical education and associate professor of pediatrics (critical care medicine), Vanderbilt University School of Medicine, Nashville, Tennessee; ORCID: https://orcid.org/0000-0003-1538-9779
| | - Michael L Green
- M.L. Green is professor of medicine and director of student assessment, Teaching and Learning Center, Yale University School of Medicine, New Haven, Connecticut
| | - Vivian T Obeso
- V.T. Obeso is associate dean for curriculum and medical education and associate professor, Division of Internal Medicine, Department of Translational Medicine, Florida International University Herbert Wertheim College of Medicine, Miami, Florida
| | - Dianne P Wagner
- D.P. Wagner is associate dean for undergraduate medical education and professor of medicine, Michigan State University College of Human Medicine, East Lansing, Michigan
| | - Jamie B Warren
- J.B. Warren is associate professor, Division of Neonatology, and clinical vice chair, Department of Pediatrics, Oregon Health & Science University, Portland, Oregon; ORCID: https://orcid.org/0000-0003-4422-1502
| | - Sandra L Yingling
- S.L. Yingling is associate dean for educational planning and quality improvement, University of Illinois College of Medicine (Chicago, Peoria, Rockford, and Urbana), Chicago, Illinois
| | - Dorothy A Andriole
- D.A. Andriole is senior director, Medical Education Research, Association of American Medical Colleges, Washington, DC; ORCID: https://orcid.org/0000-0001-8902-1227
| |
Collapse
|
17
|
Ryan MS, Khan AR, Park YS, Chastain C, Phillipi C, Santen SA, Barron BA, Obeso V, Yingling SL. Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:544-551. [PMID: 34192721 PMCID: PMC10182784 DOI: 10.1097/acm.0000000000004222] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
PURPOSE In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. METHOD The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. RESULTS Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. CONCLUSIONS In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean, Clinical Medical Education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Asra R Khan
- A.R. Khan is associate professor, director, Doctoring and Clinical Skills course, and clerkship director, Department of Internal Medicine, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-2306-4643
| | - Yoon Soo Park
- Y.S. Park is director, Health Professions Education Research, and member of the faculty, Harvard Medical School and Massachusetts General Hospital, Boston, Massachusetts; ORCID: https://orcid.org/0000-0001-8583-4335
| | - Cody Chastain
- C. Chastain is assistant professor, Department of Internal Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Carrie Phillipi
- C. Phillipi is professor and vice chair of education, Department of Pediatrics, Oregon Health & Science University, Portland, Oregon
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean, Assessment, Evaluation, and Scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia
| | - Beth A Barron
- B.A. Barron is associate professor and associate director, Simulation, Department of Internal Medicine, Columbia University School of Medicine, New York, New York
| | - Vivian Obeso
- V. Obeso is associate professor and assistant dean, Curriculum and Medical Education, Department of Internal Medicine, Florida International University, Miami, Florida
| | - Sandra L Yingling
- S.L. Yingling is assistant professor and associate dean, Educational Planning and Quality Improvement, Department of Medical Education, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-9072-7590
| |
Collapse
|
18
|
Chartash D, Hart L. Building the Generalist Physician to Support Adolescence and Emerging Adulthood: A Narrative Review. Cureus 2022; 14:e22533. [PMID: 35345691 PMCID: PMC8956274 DOI: 10.7759/cureus.22533] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2022] [Accepted: 02/20/2022] [Indexed: 11/09/2022] Open
Abstract
Undergraduate medical education serves as a foundation for the medical student to develop the skills of a generalist physician. Given the "blurring" of the demarcations between childhood and adulthood and the increased scope of pediatric practice, an extra layer has been added to medical education which seeks to address care across the lifespan. While approaches have been developed to teach this layer, clerkship reform has not focused on advancing the clinical science of adolescence. Furthermore, as we look towards the vanguard of entrustable professional activities (EPA), specific attention to transition care for the adolescent has seen minimal attention. Drawing on prior examples of curriculum integration between specialties as well as solutions to complex care management from clinical reasoning, we suggest that attention to the development of the generalist physician requires attention to the combined medicine-pediatrics specialty.
Collapse
|
19
|
Wilson RD, Wilson BL, Madden C. Creating a National Standard for Prelicensure Clinical Evaluation in Nursing. J Nurs Educ 2021; 60:686-689. [PMID: 34870507 DOI: 10.3928/01484834-20211004-01] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
BACKGROUND Measuring clinical outcomes in prelicensure students is desired by key stakeholders but is fraught with challenges. There are currently no standardized, psychometrically validated clinical-outcome measures available for prelicensure nursing programs, requiring each program to design its own measures. METHOD We conducted a review of the potential antecedents of this gap, as well as models for standardized clinical outcome measures and recommendations from within health-professions education. RESULTS There are benefits to pursuing a standardized clinical outcome tool, including an improved student learning experience and unifying our communication regarding graduate nurse preparation to valued stakeholders. CONCLUSION Nursing education has a unique opportunity to emulate our colleagues in medicine and pharmacy by working at the national level to create a standardized tool using current psychometric methods for development and validation. [J Nurs Educ. 2021;60(12):686-689.].
Collapse
|
20
|
Rideout M, Dawlett M, Plant J, Chitkara M, Trainor JL. Essential yet Ill-defined: leadership roles to support fourth-year medical students in pediatrics. MEDICAL EDUCATION ONLINE 2021; 26:1950108. [PMID: 34232843 PMCID: PMC8266242 DOI: 10.1080/10872981.2021.1950108] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/24/2021] [Revised: 05/31/2021] [Accepted: 06/02/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND Few studies have been published about specialty-specific fourth-year medical student leadership in any discipline. This paper provides insight from pediatric educators about the current status and recommendations for pediatric-specific fourth-year leaders. OBJECTIVE To identify the prevalence of pediatric fourth-year medical student directors across the US and Canada and to compare current and ideal responsibilities for this role. METHODS Five multi-part questions were written and submitted for the 2019 Council on Medical Student Education in Pediatrics (COMSEP) Annual Survey and subsequently disseminated to all COMSEP member physicians. Anonymous responses were collected and results analyzed. The study was IRB exempt. RESULTS The program-level survey response rate was 79%. Of 115 respondent medical schools, 37% reported having a pediatric fourth-year director separate from the clerkship director, with an average of 9.8% full-time equivalent (FTE) protected time for the role. In contrast, individuals indicated 20% FTE would be ideal for fourth-year director responsibilities. The most common role identified for pediatric fourth-year directors was directing sub-internships. Respondents indicated it would be ideal for pediatric fourth-year directors to have an increased level of involvement in all areas queried in the survey, especially directing a pediatric residency preparatory course/boot camp, faculty development for educators of fourth-year students, and remediating fourth-year students. CONCLUSIONS As specialty-specific experiences have grown in the fourth year of medical school, there is an increasing demand for faculty leadership separate from direction of the pediatric clerkship. In this national survey, pediatric educators expressed a need for additional protected time to lead fourth-year specific activities. Similar findings in other disciplines would support advocating for more protected time and expanded roles for specialty-specific fourth-year directors nationally.
Collapse
Affiliation(s)
- Molly Rideout
- Department of Pediatrics, Larner College of Medicine at the University of Vermont, Burlington, VT, USA
| | - Marie Dawlett
- Department of Pediatrics, University of Texas Medical Branch School of Medicine, Galveston, TX, USA
| | - Jennifer Plant
- Department of Pediatrics, University of California Davis School of Medicine, Sacramento, CA, USA
| | - Maribeth Chitkara
- Department of Pediatrics, Renaissance School of Medicine at Stony Brook University, NY, USA
| | - Jennifer L. Trainor
- Department of Pediatrics, Northwestern University, Feinberg School of Medicine, Chicago, IL, USA
| |
Collapse
|
21
|
Zhou B, Calkins C, Jayaraman T, Cassells S, Rotto T, Vaughan L, Srinivasan M, Schillinger E. Implementing Value-Added Medical Education: Lessons Learned From the Student-Initiated Stanford Frontline COVID-19 Consult Service. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1690-1695. [PMID: 33983140 PMCID: PMC8603438 DOI: 10.1097/acm.0000000000004160] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PROBLEM Value-added medical education (VAME) has been difficult to implement due to student and educator constraints. The COVID-19 pandemic caused mass transitions to online learning, removed students from clinical settings, and underscored students' desires for meaningful VAME opportunities. The authors introduced the Stanford Frontline COVID-19 Consult Service (SFCS), through which off-service medical and physician assistant (PA) students provided assistance to clinicians in the form of rapid research regarding COVID-19 clinical questions. APPROACH The SFCS, a student-derived VAME initiative, was implemented from March to May 2020 by Stanford University medical students, PA students, and faculty. SFCS aligned with not only the interests of clinicians and students but also national accreditation standards. Students attended weekly editorial meetings, didactic sessions on literature reviews and information management, and they underwent rigorous training on the peer review process. After 2 months, the authors expanded the service to local community clinicians. OUTCOMES The SFCS enrolled 16 students, was supported by 13 faculty members, and produced 87 peer-reviewed evidence syntheses. Of the 16 SFCS students, 13 (81%) completed evaluations; of 128 Stanford Primary Care and Population Health clinicians, 48 (38%) completed evaluations. Overall student satisfaction with the SFCS was 4.9/5 (standard deviation [SD] 0.3). Self-assessed achievement of SFCS learning objectives exceeded 90% for all objectives. Overall faculty satisfaction with the SFCS was 4.4/5 (SD 0.8). Most faculty (40/46 [87%]) planned to use the database to answer future COVID-19 questions. NEXT STEPS The SFCS is a novel, student-initiated VAME curriculum focused on increasing students' meaningful contributions to patient care. The authors will track SFCS students throughout their clerkships to gauge clerkship performance/preparedness, and they will develop training for integrating VAME into preclerkship curricula at other institutions. Given its adaptive, student-driven design, the VAME framework used to develop the SFCS empowers students to create their own personalized, experiential learning.
Collapse
Affiliation(s)
- Bright Zhou
- B. Zhou is a fourth-year medical student, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0002-0253-9179
| | - Christopher Calkins
- C. Calkins is a fourth-year medical student, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0002-4159-1957
| | - Tanvi Jayaraman
- T. Jayaraman is a third-year medical student, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0001-6732-6652
| | - Sandrene Cassells
- S. Cassells is a fourth-year medical student, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0002-5470-8149
| | - Torsten Rotto
- T. Rotto is a fourth-year medical student, Stanford University School of Medicine, Stanford, California
| | - Laura Vaughan
- L. Vaughan is clinical assistant professor, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0002-9394-0995
| | - Malathi Srinivasan
- M. Srinivasan is clinical professor, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0001-9951-0016
| | - Erika Schillinger
- E. Schillinger is clinical professor and vice chief of education, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, California; ORCID: https://orcid.org/0000-0003-0836-6237
| |
Collapse
|
22
|
Obeso V, Grbic D, Emery M, Parekh K, Phillipi C, Swails J, Jayas A, Andriole DA. Core Entrustable Professional Activities (EPAs) and the Transition from Medical School to Residency: the Postgraduate Year One Resident Perspective. MEDICAL SCIENCE EDUCATOR 2021; 31:1813-1822. [PMID: 34956699 PMCID: PMC8651854 DOI: 10.1007/s40670-021-01370-3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 08/14/2021] [Indexed: 06/14/2023]
Abstract
INTRODUCTION The Association of American Medical Colleges (AAMC) proposed thirteen core Entrustable Professional Activities (EPAs) that all graduates should be able to perform under indirect supervision upon entering residency. As an underlying premise is that graduates ready to do so will be better prepared to transition to the responsibilities of residency, we explored the relationship between postgraduate year (PGY)-1 residents' self-assessed preparedness to perform core EPAs under indirect supervision at the start of residency with their ease of transition to residency. METHODS Using response data to a questionnaire administered in September 2019 to PGY-1 residents who graduated from AAMC core EPA pilot schools, we examined between-group differences and independent associations for each of PGY-1 position type, specialty, and "EPA-preparedness" score (proportion of EPAs the resident reported as prepared to perform under indirect supervision at the start of residency) and ease of transition to residency (from 1 = much harder to 5 = much easier than expected). RESULTS Of 274 questionnaire respondents (19% of 1438 graduates), 241 (88% of 274) had entered PGY-1 training and completed all questionnaire items of interest. EPA-preparedness score (mean 0.71 [standard deviation 0.26]) correlated with ease of transition (3.1 [0.9]; correlation = .291, p < .001). In linear regression controlling for specialty (among other variables), EPA-preparedness score (β-coefficient 1.08; 95% confidence interval .64-1.52; p < .001) predicted ease of transition to residency. CONCLUSION Graduates who felt prepared to perform many of the core EPAs under indirect supervision at the start of PGY-1 training reported an easier-than-expected transition to residency. SUPPLEMENTARY INFORMATION The online version contains supplementary material available at 10.1007/s40670-021-01370-3.
Collapse
Affiliation(s)
- Vivian Obeso
- Florida International University Herbert Wertheim College of Medicine, Miami, FL USA
| | - Douglas Grbic
- Association of American Medical Colleges, 655 K Street, N.W., Washington, DC 20001-2399 USA
| | - Matthew Emery
- Michigan State University College of Human Medicine, Grand Rapids, MI USA
| | - Kendra Parekh
- Vanderbilt University School of Medicine, Nashville, TN USA
| | - Carrie Phillipi
- Oregon Health & Science University School of Medicine, Portland, OR USA
| | - Jennifer Swails
- McGovern Medical School at the University of Texas Health Science Center, Houston, TX USA
| | - Amy Jayas
- Association of American Medical Colleges, 655 K Street, N.W., Washington, DC 20001-2399 USA
| | - Dorothy A. Andriole
- Association of American Medical Colleges, 655 K Street, N.W., Washington, DC 20001-2399 USA
| | | |
Collapse
|
23
|
Maggio LA, Ninkov A, Costello JA, Driessen EW, Artino AR. Knowledge Syntheses in Medical Education: Examining Authors' Gender, Geographic Location, and Institutional Affiliation. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S201. [PMID: 34705701 DOI: 10.1097/acm.0000000000004276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- Lauren A Maggio
- Author affiliations: L.A. Maggio, J.A. Costello, Uniformed Services University of the Health Sciences
| | | | - Joseph A Costello
- Author affiliations: L.A. Maggio, J.A. Costello, Uniformed Services University of the Health Sciences
| | | | - Anthony R Artino
- A.R. Artino Jr, the George Washington University School of Medicine and Health Sciences
| |
Collapse
|
24
|
Brown DR, Moeller JJ, Grbic D, Andriole DA, Cutrer WB, Obeso VT, Hormann MD, Amiel JM. The First 2 Years of Entrustment Decisions in the Core Entrustable Professional Activities (Core EPAs) Pilot. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S201-S202. [PMID: 34705702 DOI: 10.1097/acm.0000000000004275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Affiliation(s)
- David R Brown
- Author affiliations: D.R. Brown, V.T. Obeso, Florida International University Herbert Wertheim College of Medicine
| | | | - Douglas Grbic
- D. Grbic, D.A. Andriole, Association of American Medical Colleges
| | | | | | - Vivian T Obeso
- Author affiliations: D.R. Brown, V.T. Obeso, Florida International University Herbert Wertheim College of Medicine
| | | | - Jonathan M Amiel
- J.M. Amiel, Columbia University College of Physicians and Surgeons
| |
Collapse
|
25
|
Ridinger HA, Bonnet K, Schlundt DG, Tekian A, Riddle J, Lomis KD. Defining Successful Practice Within Health Systems Science Among Entering Residents: A Single-Institution Qualitative Study of Graduate Medical Education Faculty Observations. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S126-S135. [PMID: 34380937 DOI: 10.1097/acm.0000000000004357] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
PURPOSE The American Medical Association's Accelerating Change in Medical Education consortium defined health systems science (HSS) as the study of how health care is delivered, how health care professionals work together to deliver that care, and how the health system can improve patient care and health care delivery. This framework is increasingly being incorporated into medical school curricula. Graduate medical education (GME) had previously elevated systems-based practice (SBP) as a core competency, but expectations are defined by specialty-specific milestones. The lack of a shared competency framework between undergraduate medical education (UME) and GME makes it challenging to ensure that entering residents are prepared to implement HSS/SBP concepts in the workplace. The authors explored GME faculty observations of residents exemplifying successful practice across HSS domains to inform targets for UME training and assessment. METHOD Authors performed a single-institution qualitative study using transcribed phone interviews with eligible Vanderbilt residency program directors, associate program directors, and core faculty. Participants described observations of successful residents within each HSS domain. Two researchers independently coded, discussed, and reconciled deidentified transcripts using inductive-deductive approaches to identify themes. RESULTS Seventeen faculty participated across specialties (17/39, 45%). Faculty responses emphasize precurricular experiences including professional degrees, work experience, extracurriculars, and medical school exposure. Importantly, successful residents exhibit foundational core workforce characteristics including growth mindset, curiosity, and a desire to learn about systems. GME faculty identified HSS domain-specific skills, noting distinctions among learning environments. Outcomes resulting from residents' application of HSS concepts include delivering high-quality, person-centered care and systems improvements. CONCLUSIONS Descriptions of successful practice within HSS domains highlight preparatory experiences and core workforce characteristics and outline entry-level HSS behaviors. Conceptualized in a logic model framework, these findings describe key inputs, learning activities, outputs, and outcomes for systems-prepared entering residents bridging the UME-GME transition.
Collapse
Affiliation(s)
- Heather A Ridinger
- H.A. Ridinger is assistant professor, Department of Medicine, Vanderbilt University School of Medicine, Nashville, Tennessee
| | - Kemberlee Bonnet
- K. Bonnet is coordinator, Qualitative Research Core, Department of Psychology, Vanderbilt University, Nashville, Tennessee
| | - David G Schlundt
- D.G. Schlundt is associate professor, Department of Psychology, Vanderbilt University, Nashville, Tennessee
| | - Ara Tekian
- A. Tekian is professor, Department of Medical Education, University of Illinois Chicago, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-9252-1588
| | - Janet Riddle
- J. Riddle is research assistant professor and director, faculty development, Department of Medical Education, University of Illinois Chicago, Chicago, Illinois
| | - Kimberly D Lomis
- K.D. Lomis is vice president for undergraduate medical education innovations, American Medical Association, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-3504-6776
| |
Collapse
|
26
|
Witt L, Nilsen KM, Kohman M, Petz A, Moser S, Walling A. Unintended Consequences? Assessing the Impact of Curricular Change on Medical Student Participation in a Student-Run Free Clinic. MEDICAL SCIENCE EDUCATOR 2021; 31:1653-1662. [PMID: 34603837 PMCID: PMC8446137 DOI: 10.1007/s40670-021-01356-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 06/30/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION In a prior qualitative study of the impact of a new (ACE) medical school curriculum, students and faculty reported decreased participation in the student-run free clinic (SRFC) attributed to more intensive scheduling and more frequent testing compared to the previous (Legacy) curriculum. MATERIALS AND METHODS To verify and understand this perception formed during curriculum reform, we conducted a mixed method study to measure student participation in the SRFC before and after curricular change and assessed student beliefs and motivations about SRFC participation using focus groups. RESULTS Overall SRFC participation did not decrease among students in the ACE cohort following curriculum change. Additionally, both Legacy and ACE groups showed lower participation during test weeks, but the decrease was not significantly different between the ACE and Legacy cohorts. Focus groups confirmed the pervasive misbelief that SRFC participation was indeed lower among ACE students and attributed to reduced student discretionary time plus increased preparation time for frequent testing. Focus groups also revealed several "values" about volunteering at the SRFC which should be endorsed and promoted by schools considering curricular change. Participants valued the SRFC for educational items that were most effectively taught in the SRFC, notably social determinants of health, interprofessional practice, and interviewing with medical interpreters. They also valued the SRFC for professional validation, opportunities to apply course content, practice clinical skills, form important professional relationships, and provide community service. CONCLUSIONS Our findings validate the value of SRFC experience as reported by students and demonstrate that, contrary to misbeliefs, participation was not negatively impacted by curricular reform.
Collapse
Affiliation(s)
- Laurel Witt
- Department of Family Medicine and Community Health, University of Kansas School of Medicine- Kansas City (KUSM-KC), Kansas City, KS USA
| | - Kari M. Nilsen
- Department of Family & Community Medicine, University of Kansas School of Medicine-Wichita (KUSM-W), Wichita, KS USA
| | | | | | - Scott Moser
- Department of Family & Community Medicine, University of Kansas School of Medicine-Wichita (KUSM-W), Wichita, KS USA
| | - Anne Walling
- Department of Family & Community Medicine, University of Kansas School of Medicine-Wichita (KUSM-W), Wichita, KS USA
| |
Collapse
|
27
|
Ryan MS, Richards A, Perera R, Park YS, Stringer JK, Waterhouse E, Dubinsky B, Khamishon R, Santen SA. Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1197-1204. [PMID: 33464735 DOI: 10.1097/acm.0000000000003921] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships. METHOD During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability. RESULTS A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70). CONCLUSIONS The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Yoon Soo Park
- Y.S. Park is associate professor and associate head, Department of Medical Education, University of Illinois College of Medicine, Chicago, Illinois
| | - J K Stringer
- J.K. Stringer is assessment manager, Office of Integrated Medical Education, Rush Medical College, Chicago, Illinois
| | - Elizabeth Waterhouse
- E. Waterhouse is professor, Department of Neurology, Virginia Commonwealth University, Richmond, Virginia
| | - Brieanne Dubinsky
- B. Dubinsky is business analyst, Office of Academic Information Systems, Virginia Commonwealth University, Richmond, Virginia
| | - Rebecca Khamishon
- R. Khamishon is a third-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
28
|
Bradley EB, Waselewski EA, Gusic ME. How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment. MEDICAL SCIENCE EDUCATOR 2021; 31:1419-1428. [PMID: 34457983 PMCID: PMC8368261 DOI: 10.1007/s40670-021-01327-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 05/20/2021] [Indexed: 06/13/2023]
Abstract
INTRODUCTION The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated. METHODS In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding. RESULTS Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout. CONCLUSIONS Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.
Collapse
Affiliation(s)
- Elizabeth B. Bradley
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| | - Eric A. Waselewski
- Department of Internal Medicine, University of Michigan, Ann Arbor, Michigan USA
| | - Maryellen E. Gusic
- Center for Medical Education Research and Scholarly Innovation, Office of Medical Education, University of Virginia School of Medicine, VA Charlottesville, USA
| |
Collapse
|
29
|
Holmstrom AL, Chia MC, O'Brien CL, Odell DD, Burke J, Halverson AL. Entrustable Professional Activity-Based Summative Performance Assessment in the Surgery Clerkship. JOURNAL OF SURGICAL EDUCATION 2021; 78:1144-1150. [PMID: 33384267 DOI: 10.1016/j.jsurg.2020.12.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2020] [Revised: 11/22/2020] [Accepted: 12/02/2020] [Indexed: 06/12/2023]
Abstract
OBJECTIVE The objectives of this study were to 1) assess the performance Entrustable Professional Activities (EPAs) when integrated into the summative assessment of third-year medical students on the surgery clerkship and 2) to compare EPAs to traditional clinical performance assessment tools. DESIGN EPA assessments were collected prospectively from a minimum of 4 evaluators at the completion of each surgical clerkship rotation from November 2019 to June 2019. Overall EPA-based clinical performance scores were calculated as the sum of the mean EPA score from each evaluator. A rating of overall clinical performance called the clinical performance appraisal (CPA) was also collected. EPA ratings were compared to the CPA score, National Board of Medical Examiners exam score, objective structured clinical exam scores, and final clerkship grade. SETTING Northwestern Memorial Hospital, a tertiary care teaching institution in Chicago, IL. RESULTS Overall, 446 evaluations (111 students) were included in the analysis. The aggregate EPA scores ranged from 11.6-24.0 (mean 19.9 ± 2.0), and the CPA scores ranged from 4.4-9.0 (mean 7.6 ± 0.7). The variance among learners in EPA scores was significantly higher than CPA scores (p < 0.001). The aggregate EPA scores correlated well with CPA scores (Spearman's rho 0.803) but had lesser, positive correlations with the objective structured clinical exam (rho 0.153) and National Board of Medical Examiners (rho 0.265) scores. When all EPA scores were included in ordinal logistic regression, only EPA 6, oral presentation of patients, was independently associated with students' final grades (OR: 10.05, 95%CI 1.41-71.80; p = 0.02). CONCLUSION Integration of EPAs for use in clinical performance assessment of medical students is feasible within a surgery clerkship. Compared to a global clinical performance assessment, EPA-based assessment provided better discrimination of clinical performance among learners. Use of EPAs may better identify advanced learners and those that need additional time.
Collapse
Affiliation(s)
- Amy L Holmstrom
- Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago Illinois
| | - Matthew C Chia
- Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago Illinois
| | - Celia L O'Brien
- Department of Medical Education, Feinberg School of Medicine, Northwestern University, Chicago Illinois
| | - David D Odell
- Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago Illinois
| | - Jason Burke
- Department of Surgical Education, Feinberg School of Medicine, Northwestern University, Chicago Illinois
| | - Amy L Halverson
- Department of Surgery, Feinberg School of Medicine, Northwestern University, Chicago Illinois.
| |
Collapse
|
30
|
Ten Cate O, Balmer DF, Caretta-Weyer H, Hatala R, Hennus MP, West DC. Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:S96-S104. [PMID: 34183610 DOI: 10.1097/acm.0000000000004106] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.
Collapse
Affiliation(s)
- Olle Ten Cate
- O. ten Cate is professor of medical education and senior scientist, Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0002-6379-8780
| | - Dorene F Balmer
- D.F. Balmer is associate professor, Department of Pediatrics, The Children's Hospital of Philadelphia and University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0001-6805-4062
| | - Holly Caretta-Weyer
- H. Caretta-Weyer is assistant professor, Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, California; ORCID: https://orcid.org/0000-0002-9783-5797
| | - Rose Hatala
- R. Hatala is professor, Department of Medicine, University of British Columbia, Vancouver, Canada; ORCID: https://orcid.org/0000-0003-0521-2590
| | - Marije P Hennus
- M.P. Hennus is a pediatric intensivist and program director, pediatric intensive care fellowship, University Medical Center Utrecht, Utrecht, the Netherlands; ORCID: https://orcid.org/0000-0003-1508-0456
| | - Daniel C West
- D.C. West is professor and senior director of medical education, Department of Pediatrics, Children's Hospital of Philadelphia and The Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0002-0909-4213
| |
Collapse
|
31
|
Implementation of a Workplace-Based Assessment System to Measure Performance of the Core Entrustable Professional Activities in the Pediatric Clerkship. Acad Pediatr 2021; 21:564-568. [PMID: 33035730 DOI: 10.1016/j.acap.2020.09.016] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/30/2019] [Revised: 08/27/2020] [Accepted: 09/30/2020] [Indexed: 01/24/2023]
Abstract
OBJECTIVE The Core Entrustable Professional Activities for Entering Residency (Core EPAs) were developed to address the gap between medical school and residency. There is a lack of instruments to measure performance of the Core EPAs in clerkships. We describe the operationalization and outcomes of a workplace-based assessment (WBA) system to measure performance of the Core EPAs in the pediatrics clerkship. METHODS A mobile-friendly WBA was developed at the authors' institution. The WBA incorporated a modified version of the Ottawa Clinic Assessment Tool (OCAT), an instrument that rates performance on a scale of 1 to 4 (1- "I had to do it" to 4- "I had to be there just in case"). During 2018 to 2019, all students were required to request feedback for 6 of the 13 Core EPAs using the WBA in the Pediatrics clerkship. Descriptive and inferential statistics were calculated to assess mean OCAT scores, variance in performance and correlation between scores, clerkship timing, and grades. RESULTS Total 1655 WBAs were completed for 218 students. The overall mean OCAT score was 3.47 out of 4. Scores across Core EPAs were greater in later rotations (r = 0.157, P < .001). One-way analysis of variance revealed significant variance on score by student, assessor, and timing of clerkship block. Final grades were correlated with OCAT scores (Spearman's ρ = 0.25, P < .001). CONCLUSIONS The results of this study demonstrate initial outcomes for a WBA system to assess performance for the Core EPAs in pediatrics using the OCAT scale. Future studies will assess the system across clerkships.
Collapse
|
32
|
Myers JS, Kin JM, Billi JE, Burke KG, Harrison RV. Development and validation of an A3 problem-solving assessment tool and self-instructional package for teachers of quality improvement in healthcare. BMJ Qual Saf 2021; 31:287-296. [PMID: 33771908 DOI: 10.1136/bmjqs-2020-012105] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Revised: 02/20/2021] [Accepted: 03/10/2021] [Indexed: 11/04/2022]
Abstract
PURPOSE A3 problem solving is part of the Lean management approach to quality improvement (QI). However, few tools are available to assess A3 problem-solving skills. The authors sought to develop an assessment tool for problem-solving A3s with an accompanying self-instruction package and to test agreement in assessments made by individuals who teach A3 problem solving. METHODS After reviewing relevant literature, the authors developed an A3 assessment tool and self-instruction package over five improvement cycles. Lean experts and individuals from two institutions with QI proficiency and experience teaching QI provided iterative feedback on the materials. Tests of inter-rater agreement were conducted in cycles 3, 4 and 5. The final assessment tool was tested in a study involving 12 raters assessing 23 items on six A3s that were modified to enable testing a range of scores. RESULTS The intraclass correlation coefficient (ICC) for overall assessment of an A3 (rater's mean on 23 items per A3 compared across 12 raters and 6 A3s) was 0.89 (95% CI 0.75 to 0.98), indicating excellent reliability. For the 20 items with appreciable variation in scores across A3s, ICCs ranged from 0.41 to 0.97, indicating fair to excellent reliability. Raters from two institutions scored items similarly (mean ratings of 2.10 and 2.13, p=0.57). Physicians provided marginally higher ratings than QI professionals (mean ratings of 2.17 and 2.00, p=0.003). Raters averaged completing the self-instruction package in 1.5 hours, then rated six A3s in 2.0 hours. CONCLUSION This study provides evidence of the reliability of a tool to assess healthcare QI project proposals that use the A3 problem-solving approach. The tool also demonstrated evidence of measurement, content and construct validity. QI educators and practitioners can use the free online materials to assess learners' A3s, provide formative and summative feedback on QI project proposals and enhance their teaching.
Collapse
Affiliation(s)
- Jennifer S Myers
- Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA
| | - Jeanne M Kin
- Quality, Michigan Medicine, University of Michigan, Ann Arbor, Michigan, USA
| | - John E Billi
- Medicine and Learning Health Sciences, Michigan School of Medicine, University of Michigan, Ann Arbor, Michigan, USA.,Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, Michigan, USA.,Integrative Systems and Design, College of Engineering, University of Michigan, Ann Arbor, Michigan, USA
| | - Kathleen G Burke
- Biobehavioral Health, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, USA.,Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Richard Van Harrison
- Learning Health Sciences, University of Michigan Health System, Ann Arbor, Michigan, USA
| |
Collapse
|
33
|
Dehghani Poudeh M, Mohammadi A, Mojtahedzadeh R, Yamani N. Entrustability levels of general internal medicine residents. BMC MEDICAL EDUCATION 2021; 21:185. [PMID: 33766005 PMCID: PMC7995576 DOI: 10.1186/s12909-021-02624-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/08/2020] [Accepted: 03/19/2021] [Indexed: 06/12/2023]
Abstract
BACKGROUND Entrustable professional activities (EPAs) are those activities that a health professional can perform without direct supervision in a defined environment. Bridging the gap between competencies and learning objectives, EPAs have made assessing the performances of health professional more realistic. The main objective of the present study was developing and customizing EPAs for Iranian Internal Medicine Residency Programs. RESULTS After reviewing the publications, residency curricula and logbooks, and collecting experts' ideas, the initial list of EPAs was developed. Then, in a focus group, the list was refined, the entrustability level of each residency year was determined, and finally, the EPA-competency cross-tab was established, and in the next step, through a one- round Delphi, the results were validated. Twenty-eight EPAs were developed. Some of them were definitely suitable for the higher levels of residency, such that they had to be accomplished under direct supervision until the end of the program. On the other hand, some of EPAs were those that residents, even from the first year, are expected to perform independently or under indirect supervision. Most of the EPAs cover a wide range of competencies. CONCLUSION Determining the entrustability level of each residency year in each EPA as well as the competency- EPA matrix has crucial effect on the quality of the graduates. It seems that our findings are applicable in developing countries like Iran.
Collapse
Affiliation(s)
- Mostafa Dehghani Poudeh
- Department of Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran
| | - Aeen Mohammadi
- Department of E-learning in Medical Education, Virtual School, Center for Excellence in E-learning in Medical Education, Tehran University of Medical Sciences, Tehran, Iran
| | - Rita Mojtahedzadeh
- Department of E-learning in Medical Education, Virtual School, Center for Excellence in E-learning in Medical Education, Tehran University of Medical Sciences, Tehran, Iran
| | - Nikoo Yamani
- Department of Medical Education, Isfahan University of Medical Sciences, Isfahan, Iran
| |
Collapse
|
34
|
Geraghty JR, Ocampo RG, Liang S, Ona Ayala KE, Hiltz K, McKissack H, Hyderi A, Ryan MS. Medical Students' Views on Implementing the Core EPAs: Recommendations From Student Leaders at the Core EPAs Pilot Institutions. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:193-198. [PMID: 33031119 DOI: 10.1097/acm.0000000000003793] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
In 2014, the Association of American Medical Colleges recruited 10 institutions across the United States to pilot the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). The goal was to establish a competency-based framework to prepare graduating medical students for the transition to residency. Within the Core EPAs pilot, medical students play an influential role in the development and implementation of EPA-related curricula. Student engagement was a priority for the Core EPAs institutions given students' roles as the end users of the curriculum, thus they may offer valuable insight into its design and implementation. Here, the authors provide the perspective of medical students who serve as leaders in the Core EPAs pilot at their respective institutions. They describe student leadership models across the pilot institutions as well as 6 key challenges to implementation of the Core EPAs: (1) How and when should the Core EPAs be introduced? (2) Who is responsible for driving the assessment process? (3) What feedback mechanisms are required? (4) What systems are required for advising, mentoring, or coaching students? (5) Should EPA performance contribute to students' grades? and (6) Should entrustment decisions be tied to graduation requirements? Using a polarity management framework to address each challenge, the authors describe inherent tensions, approaches used by the Core EPAs pilot institutions, and student-centered recommendations for resolving each tension. By sharing the experiences and perspectives of students engaged in the Core EPAs pilot, the authors hope to inform implementation of EPA-oriented assessment practices and feedback across institutions in the United States.
Collapse
Affiliation(s)
- Joseph R Geraghty
- J.R. Geraghty is a sixth-year MD/PhD student, University of Illinois College of Medicine, Chicago, Illinois; ORCID: https://orcid.org/0000-0002-6828-4893
| | - Raechelle G Ocampo
- R.G. Ocampo was a medical student, Virginia Commonwealth University School of Medicine, Richmond, Virginia, at the time of writing, and is a first-year resident, Department of Pediatrics, University of California, San Francisco, School of Medicine, San Francisco, California, now
| | - Sherry Liang
- S. Liang was a medical student, Oregon Health & Sciences University, Portland, Oregon, at the time of writing, and is a first-year resident, Departments of Internal Medicine and Psychiatry, Tulane University School of Medicine, New Orleans, Louisiana, now
| | - Kimberly E Ona Ayala
- K.E. Ona Ayala was a medical student, Yale University School of Medicine, New Haven, Connecticut, at the time of writing, and is a first-year resident, Department of Anesthesiology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, now
| | - Kathleen Hiltz
- K. Hiltz was a medical student, Vanderbilt University School of Medicine, Nashville, Tennessee, at the time of writing, and is a first-year resident, Department of Internal Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, now
| | - Haley McKissack
- H. McKissack is a fourth-year medical student, Florida International University Herbert Wertheim College of Medicine, Miami, Florida
| | - Abbas Hyderi
- A. Hyderi is senior associate dean for medical education and professor, Department of Clinical Science, Kaiser Permanente School of Medicine, Pasadena, California, and adjunct associate professor, Department of Medical Education, and former associate dean of curriculum, University of Illinois College of Medicine, Chicago, Illinois
| | - Michael S Ryan
- M.S. Ryan is assistant dean for clinical medical education and associate professor of pediatrics, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| |
Collapse
|
35
|
Smith BQ, Woelfel I, Salani R, Harzman A, Chen X. Resident Self-Entrustment and Expectations of Autonomy: OB > GYN? JOURNAL OF SURGICAL EDUCATION 2021; 78:275-281. [PMID: 32753260 DOI: 10.1016/j.jsurg.2020.07.019] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2020] [Revised: 06/17/2020] [Accepted: 07/14/2020] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Entrustment is a key component connecting to resident preparedness for surgical practice in the operating room (OR). Residents' self-entrustment of their surgical competencies closely associates with their OR training experience and granted autonomy. Some recent studies have investigated how attending surgeons entrusted residents in the OR. There is little to no data, however, in examining these issues from the resident perspective. The goal of this study was to identify the perception and expectations of autonomy from residents' perspective, as well as the self-entrustment of their surgical competencies in obstetrics (OB) and gynecologic (GYN) procedures. METHODS Focus group interviews of OB/GYN residents were performed. Residents were selected by convenience sampling. Audio recordings of each interview were transcribed, iteratively analyzed, and emergent themes identified, using a framework method. RESULTS A total of 123 minutes of interviews were recorded. Eight junior residents (PGY1-2) and 12 senior residents (PGY3-4) participated. Our data illustrated that (1) the perception of autonomy shifted significantly throughout residency training; (2) residents demonstrated higher expectations and self-entrustment for OB surgical procedures than for GYN surgical procedures upon graduation; and (3) case volume, modalities of OR teaching and mutual communication are 3 factors influencing resident self-entrustment of their surgical competencies. CONCLUSIONS Residents showed disparities in their self-entrustment and expectations of autonomy between OB and GYN surgical procedures. Better understanding these differences and the 3 influencing factors could help programs develop a potential solution for improvement in resident entrustment and autonomy upon graduation.
Collapse
Affiliation(s)
- Brentley Q Smith
- Division of Gynecologic Oncology, The Ohio State University The James Comprehensive Cancer Center, Columbus, Ohio.
| | - Ingrid Woelfel
- Department of General Surgery, The Ohio State University Wexner Medical Center, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Ritu Salani
- Division of Gynecologic Oncology, The Ohio State University The James Comprehensive Cancer Center, Columbus, Ohio
| | - Alan Harzman
- Department of General Surgery, The Ohio State University Wexner Medical Center, The Ohio State University Wexner Medical Center, Columbus, Ohio
| | - Xiaodong Chen
- Department of General Surgery, The Ohio State University Wexner Medical Center, The Ohio State University Wexner Medical Center, Columbus, Ohio
| |
Collapse
|
36
|
Abstract
Entrustment decision-making has become a topic of interest in workplace-based assessment in the health professions and is germane to the use of entrustable professional activities. Entrustment decisions stem from judgments of a trainee's competence and include the permission to act with a higher level of responsibility or autonomy and a lower level of supervision. Making entrustment decisions differs from regular assessment of trainees, which usually has no consequences beyond marking trainee progress. Studies show that clinicians generally weigh more factors in making an entrustment decision than when merely assessing trainee competence or performance without direct consequences for patient care. To synthesize the varying factors reported in literature, the authors performed a thematic analysis of key qualitative studies that investigated trainee features clinical supervisors find important when making entrustment decisions. Five themes emerged from the 13 publications: Capability (specific knowledge, skills, experience, situational awareness), Integrity (truthful, benevolent, patient-centered), Reliability (conscientious, predictable, accountable, responsible), Humility (recognizes limits, asks for help, receptive to feedback), Agency (proactive toward work, team, safety, personal development). Thoughtful entrustment decisions, made either by individual clinical supervisors or by clinical competency committees, may be enriched by taking into account these five features.
Collapse
Affiliation(s)
- Olle Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - H Carrie Chen
- Georgetown University School of Medicine, Washington, USA
| |
Collapse
|
37
|
Herrigel DJ, Donovan C, Goodman E, Pradhan A, Bridgeman MM, Hogshire L, Fanning C, Whitman A, Maniaci M, Kim S. Simulation as a Platform for Development of Entrustable Professional Activities: A Modular, Longitudinal Approach. Cureus 2020; 12:e11098. [PMID: 33240694 PMCID: PMC7681749 DOI: 10.7759/cureus.11098] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Background The Association of American Medical Colleges (AAMC) has recently identified a list of integrated activities to be expected of all medical school graduates entering residency: the core Entrustable Professional Activities (EPAs). Direct observation and deliberate practice of individual EPA behaviors in the clinical setting has multiple challenges, and there is limited literature describing a comprehensive, longitudinal curriculum dedicated to formative EPA assessment. Approach We present a model curriculum to develop and provide formative assessment of EPA skills longitudinally throughout the clinical years. Each EPA-focused training session includes a simulation case followed by several small group activities with content related to the clinical vignette in the initial simulation. We have designed this curriculum to be longitudinal and modular, and present the general framework here. Outcomes Step-wise implementation began in 2013. Over 450 medical students have participated in the third year (MS3) clerkship sessions, 30 in the MS4 sub-internship sessions, and over 300 thus far in the fourth year (MS4) capstone course, including students from 10 different medical schools. MS3 sessions focused on EPAs 4, 7, 8, 9, 10, and MS4 sessions had an additional focus on EPA 8. The capstone course encompassed nearly all 13 EPAs in active simulation-based learning. Opportunities to provide formative assessment through on-the-spot feedback exist throughout the curriculum. Student feedback was overwhelmingly positive. Next steps We found that simulations are an effective method of providing formative assessment of EPAs that are exceptionally well-received by medical students. We have demonstrated that these can be implemented for medical students from multiple educational backgrounds. We believe that deliberate practice and longitudinal formative assessment is of utmost importance in effectively developing core EPAs prior to final entrustment decisions.
Collapse
|
38
|
St-Onge C, Vachon Lachiver É, Langevin S, Boileau E, Bernier F, Thomas A. Lessons from the implementation of developmental progress assessment: A scoping review. MEDICAL EDUCATION 2020; 54:878-887. [PMID: 32083743 DOI: 10.1111/medu.14136] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Revised: 01/21/2020] [Accepted: 02/06/2020] [Indexed: 06/10/2023]
Abstract
OBJECTIVES Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts. METHODS We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results. RESULTS Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences. CONCLUSIONS Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.
Collapse
Affiliation(s)
- Christina St-Onge
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Élise Vachon Lachiver
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Serge Langevin
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Elisabeth Boileau
- Department of Family and Emergency Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Frédéric Bernier
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Research Center - Sherbrooke University Hospital Center (CHUS), Integrated Health and Social Service Centers (CISSS) and Integrated University Health and Social Service Centres (CIUSSS), Sherbrooke, Québec, Canada
| | - Aliki Thomas
- School of Physical and Occupational Therapy, McGill University, Montreal, Québec, Canada
| |
Collapse
|
39
|
Salisbury SK, Rush BR, Ilkiw JE, Matthew SM, Chaney KP, Molgaard LK, May SA, Bok HGJ, Hodgson JL, Frost JS, Read EK. Collaborative Development of Core Entrustable Professional Activities for Veterinary Education. JOURNAL OF VETERINARY MEDICAL EDUCATION 2020; 47:607-618. [PMID: 32427543 DOI: 10.3138/jvme.2019-0090] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Entrustable professional activities (EPAs) have been proposed as a practical framework for the implementation of competency-based education. As veterinary education moves toward a competency-based approach, core EPAs provide a context for assessment of workplace activities. This article reports on the development of eight core clinical EPAs for veterinary education created through multi-institutional collaboration, with international input from veterinary educators and veterinary educational leaders. These core EPAs are intended as minimal expectations for clinical activities that should be assessed for every graduate of Association of American Veterinary Medical Colleges member institutions. Adoption of the core EPAs and the associated Competency-Based Veterinary Education (CBVE) framework by veterinary schools is intended to promote Day One graduate competence and thereby enhance patient care and client service.
Collapse
|
40
|
O'Dowd E, Lydon S, O'Connor P, Boland J, Offiah G, Byrne D. The development of a framework of entrustable professional activities for the intern year in Ireland. BMC MEDICAL EDUCATION 2020; 20:273. [PMID: 32811490 PMCID: PMC7433170 DOI: 10.1186/s12909-020-02156-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/28/2020] [Accepted: 07/17/2020] [Indexed: 05/22/2023]
Abstract
BACKGROUND Entrustable Professional Activities (EPAs) are units of professional practice that capture essential competencies in which trainees must become proficient before undertaking them independently. EPAs provide supervisors with a solid justification for delegating an activity to trainees. This study aimed to develop and ensure face validity of a set of EPAs for junior doctors in the first year of clinical practice in the Republic of Ireland. METHODS An iterative eight stage consensus building process was used to develop the set of EPAs. This process was based on international best practice recommendations for EPA development. A series of surveys and workshops with stakeholders was used to develop a framework of EPAs and associated competencies. An external stakeholder consultation survey was then conducted by the Irish Medical Council. The framework of EPAs was then benchmarked against the 13 core EPAs developed by the Association of American Medical Colleges (AAMC). RESULTS A framework of seven EPAs, and associated competencies resulted from this study. These EPAs address all core activities that junior doctors should be readily entrusted with at the end of the intern year, which is the first year of clinical practice in the Republic of Ireland. Each EPA contains a series of defined competencies. The final EPAs were found to be comparable to the AAMC core EPAs for entering residency. CONCLUSIONS A framework of EPAs for interns in Ireland that are appropriate for the intern year has been developed by key stakeholders. The implementation of the EPAs in practice is the next step, and is likely to result in an improved intern training process and increased patient safety.
Collapse
Affiliation(s)
- Emily O'Dowd
- Discipline of General Practice, National University of Ireland Galway, Galway, Ireland
- Irish Centre for Applied Patient Safety and Simulation, National University of Ireland Galway, Galway, Ireland
| | - Sinéad Lydon
- Irish Centre for Applied Patient Safety and Simulation, National University of Ireland Galway, Galway, Ireland.
- School of Medicine, National University of Ireland Galway, Galway, Ireland.
| | - Paul O'Connor
- Discipline of General Practice, National University of Ireland Galway, Galway, Ireland
- Irish Centre for Applied Patient Safety and Simulation, National University of Ireland Galway, Galway, Ireland
| | | | - Gozie Offiah
- Royal College of Surgeons in Ireland, Dublin, Ireland
| | - Dara Byrne
- Irish Centre for Applied Patient Safety and Simulation, National University of Ireland Galway, Galway, Ireland
- School of Medicine, National University of Ireland Galway, Galway, Ireland
| |
Collapse
|
41
|
Hasan R, Caron R, Kim H, Phillipi GM, Taher T, Thind K, Urbanowicz E. The Student Navigator Project (SNaP): Preparing Students Through Longitudinal Learning. MEDICAL SCIENCE EDUCATOR 2020; 30:833-841. [PMID: 34540340 PMCID: PMC8410918 DOI: 10.1007/s40670-020-00957-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
INTRODUCTION Early medical school education has historically focused on teaching basic medical sciences. More recently, medical schools are encouraged to incorporate health systems science (HSS) into early curricula. Addressing all of these components in an overcrowded curriculum remains challenging. METHODS We report on the Student Navigator Project (SNaP), a pre-clinical experience where students engage longitudinally with medically and socially complex patients. SNaP is built on a foundation of trust, responsibility, and ownership in the student-patient relationship. Early learners take an active role in navigating, advocating, and coaching for their patients. In addition, students are integrated as medical assistants into a primary care clinic; complete a mentored, team-based quality improvement project; and engage in evidence-based medicine, teaching, and handoff activities. RESULTS As a result of these activities, students learn firsthand about HSS, contribute meaningfully to their patients' care, and are immersed in a systems-based practice approach early in their medical school education. Preliminary outcomes (2016-2019) show satisfaction with the program and knowledge of program focus areas. CONCLUSIONS The authors are engaged in improvement cycles to modify program structure and curriculum in order to promote dissemination in diverse clinical settings. Ultimately, we plan to measure longer-term outcomes, including clerkship and residency preparation, career choice, and practice setting.
Collapse
Affiliation(s)
- Reem Hasan
- Departments of Internal Medicine and Pediatrics, Oregon Health & Science University, 3181 SW Sam Jackson Park Rd, Portland, OR 97239 USA
| | - Rachel Caron
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| | - Hannah Kim
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| | - Gina M. Phillipi
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| | - Tajwar Taher
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| | - Kanwarabijit Thind
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| | - Erin Urbanowicz
- School of Medicine, Oregon Health & Science University, Portland, OR USA
| |
Collapse
|
42
|
Solotke MT, Crabtree J, Encandela J, Vash-Margita A. Establishing a Pediatric and Adolescent Gynecology Subinternship for Medical Students. J Pediatr Adolesc Gynecol 2020; 33:104-109. [PMID: 31672667 DOI: 10.1016/j.jpag.2019.10.004] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/01/2019] [Revised: 09/25/2019] [Accepted: 10/06/2019] [Indexed: 11/26/2022]
Abstract
Within the pediatric and adolescent gynecology (PAG) community, an important issue deserving of our attention is expansion of formalized PAG training opportunities for medical students. In addition to those in obstetrics and gynecology, many physicians practicing family medicine, internal medicine, and surgery provide care to young female patients. However, residency programs provide inconsistent training in addressing specific PAG health care needs and communication skills required for delivery of care to pediatric and adolescent patients; opportunities for supervised clinical experience in these areas at the medical school level therefore have great importance because they might not be offered again during some residencies. At the undergraduate medical education level, training opportunities in PAG are currently limited, with only 3 PAG subinternships in North American medical schools. In contrast, other available subinternships include 113 in maternal fetal medicine and 82 in gynecologic oncology. To address this gap in opportunities for medical students, we developed a PAG subinternship and elective for medical students, which we describe in this report. Our 4-week PAG subinternship and 2-week elective expose students to a variety of ambulatory and surgical opportunities, with formal, informal, and self-guided learning activities. In this article, we explain the process of developing the curriculum, gaining alignment from key stakeholders, obtaining formal approval for the course, and advertising the opportunity to medical students. We also provide resources and guidance for medical educators who wish to establish a PAG subinternship or elective course at their institutions.
Collapse
Affiliation(s)
| | - Janice Crabtree
- Department of Obstetrics, Gynecology, and Reproductive Sciences, Yale School of Medicine, New Haven, Connecticut
| | - John Encandela
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut; Teaching and Learning Center, Yale School of Medicine, New Haven, Connecticut
| | - Alla Vash-Margita
- Department of Obstetrics, Gynecology, and Reproductive Sciences, Division of Pediatric and Adolescent Gynecology, Yale School of Medicine, New Haven, Connecticut
| |
Collapse
|
43
|
Cutrer WB, Russell RG, Davidson M, Lomis KD. Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales. MEDICAL TEACHER 2020; 42:325-332. [PMID: 31714166 DOI: 10.1080/0142159x.2019.1686135] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Introduction: Observations of medical student participation in entrustable professional activities (EPAs) provide insight into the student's ability to synthesize competencies across domains and effectively function in different clinical scenarios. Both Supervisory and Co-Activity Assessment Scales have been recommended for use with medical students.Methods: Students were assessed on EPAs during Acting Internships in Medicine and Pediatrics. Two rating scales were modified based on expert review and included throughout the 2017-18 academic year. Statistical analysis was conducted to clarify relationships between the scales. Raters were interviewed to explore their interpretations and response processes.Results: The results of the McNemar test suggest that the scales are different (p-value <.01). Co-activity and Supervisory EPA ratings are related, but not interchangeable. This finding concurs with themes that emerged from response process interviews: (1) the scales are not directly parallel (2) rater preference depends on diverse factors and (3) rater comments are crucial for guiding students' future learning.Conclusion: The modified Chen Supervisory Scale and the modified Ottawa Co-Activity Scales are measuring different aspects of the entrustable professional activity landscape. Both scales can provide useful information to the learner and the assessment system, but they should not be treated as interchangeable assessments.
Collapse
Affiliation(s)
| | | | - Mario Davidson
- School of Medicine, Vanderbilt University, Nashville, TN, USA
| | | |
Collapse
|
44
|
Moeller JJ, Warren JB, Crowe RM, Wagner DP, Cutrer WB, Hyderi AA, Lupi CS, Obeso VT, Yingling S, Andriole DA, Brown DR. Developing an Entrustment Process: Insights from the AAMC CoreEPA Pilot. MEDICAL SCIENCE EDUCATOR 2020; 30:395-401. [PMID: 34457683 PMCID: PMC8368864 DOI: 10.1007/s40670-020-00918-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.
Collapse
Affiliation(s)
| | - Jamie B. Warren
- Oregon Health & Science University School of Medicine, Portland, OR USA
| | - Ruth M. Crowe
- New York University School of Medicine, New York, NY USA
| | - Dianne P. Wagner
- Michigan State University College of Human Medicine, East Lansing, MI USA
| | | | - Abbas A. Hyderi
- Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, CA USA
| | - Carla S. Lupi
- Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, CA USA
| | - Vivian T. Obeso
- Florida International University Herbert Wertheim College of Medicine, Miami, FL USA
| | - Sandra Yingling
- University of Illinois College of Medicine (Chicago, Peoria, Rockford and Urbana), Chicago, IL USA
| | | | - David R. Brown
- Florida International University Herbert Wertheim College of Medicine, Miami, FL USA
| | | |
Collapse
|
45
|
Croft H, Gilligan C, Rasiah R, Levett-Jones T, Schneider J. Development and inclusion of an entrustable professional activity (EPA) scale in a simulation-based medicine dispensing assessment. CURRENTS IN PHARMACY TEACHING & LEARNING 2020; 12:203-212. [PMID: 32147163 DOI: 10.1016/j.cptl.2019.11.015] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2019] [Revised: 08/09/2019] [Accepted: 11/09/2019] [Indexed: 06/10/2023]
Abstract
BACKGROUND AND PURPOSE Effective, safe, and patient-centred dispensing is a core task of community pharmacists. Entrustable professional activities (EPAs) offer a way of defining and assessing these daily practice activities. Although EPAs have become popular within competency-based medical education programs, their use is new to pharmacy education and assessment. EDUCATIONAL ACTIVITY AND SETTING A simulation-based assessment framework containing a scale of entrustment was developed to evaluate the readiness of Year 4 undergraduate pharmacy students to safely manage the supply of prescribed medicine(s) in a community pharmacy. The assessment framework was piloted in a fourth year "Transition to Practice" course with 28 simulation-based assessments conducted. FINDINGS An entrustment framework was developed and implemented successfully with Year 4 undergraduate pharmacy students. The EPA for medicine dispensing integrates competency domains that include information gathering, providing patient-centred care, clinical reasoning, medicine dispensing, and professional communications. On a scale ranging from level 1 to level 5, the majority (73%) of entrustment ratings were level 2 or level 3; and of the students who achieved different ratings between clinical scenarios, 75% of students improved on their second simulation attempt. There was a strong correlation between the global EPA ratings with the total score achieved across the domains. SUMMARY Using simulation-based assessment, entrustment decision making can be incorporated in "entry to profession" undergraduate and postgraduate pharmacy courses to assess students' readiness to transition between learning and professional practice.
Collapse
Affiliation(s)
- Hayley Croft
- School of Biomedical Sciences and Pharmacy, Faculty of Health and Medicine, University of Newcastle, NSW, Australia.
| | - Conor Gilligan
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, NSW, Australia.
| | - Rohan Rasiah
- Western Australian Centre for Rural Health, University of Western Australia, WA, Australia.
| | | | - Jennifer Schneider
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, NSW, Australia.
| |
Collapse
|
46
|
Pinilla S, Lenouvel E, Strik W, Klöppel S, Nissen C, Huwendiek S. Entrustable Professional Activities in Psychiatry: A Systematic Review. ACADEMIC PSYCHIATRY : THE JOURNAL OF THE AMERICAN ASSOCIATION OF DIRECTORS OF PSYCHIATRIC RESIDENCY TRAINING AND THE ASSOCIATION FOR ACADEMIC PSYCHIATRY 2020; 44:37-45. [PMID: 31732885 DOI: 10.1007/s40596-019-01142-7] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/26/2019] [Accepted: 10/31/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVE Entrustable professional activities (EPAs) represent discrete clinical tasks that can be entrusted to trainees in psychiatry. They are increasingly being used as educational framework in several countries. However, the empirical evidence available has not been synthesized in the field of psychiatry. Therefore, the authors conducted a systematic review in order to summarize and evaluate the available evidence in the field of EPAs in undergraduate and graduate medical education in psychiatry. METHODS The authors searched PubMed, Cochrane Library, ERIC, Embase, PsycINFO, all Ovid journals, Scopus, Web of Science, MedEdPORTAL, and the archives of Academic Psychiatry for articles reporting quantitative and qualitative research as well as educational case reports on EPAs in undergraduate and graduate psychiatry education published since 2005. All included articles were assessed for content (development, implementation, and assessment of EPAs) and quality using the Quality Assessment Tool for Studies with Diverse Designs. RESULTS The authors screened 2807 records and included a total of 20 articles in the final data extraction. Most studies were expert consensus reports (n = 6, 30%) and predominantly conducted in English-speaking countries (n = 17, 85%). Papers reported mainly EPA development and/or EPA implementation studies (n = 14, 70%), whereas EPA assessment studies were less frequent (n = 6, 30%). Publications per year showed an increasing trend both in quantity (from 1 in 2011 to 7 in 2018) and quality (from a QATSDD score of 27 in 2011 to an average score of 39 in 2018). The main focus of the articles was the development of individual EPAs for different levels of training for psychiatry or on curricular frameworks based on EPAs in psychiatry (n = 10, 50%). The lack of empirical controlled studies does currently not allow for meta-analyses of educational outcomes. CONCLUSIONS The concept of EPA-based curricula seems to become increasingly present, a focus in the specialty of psychiatry both in UME and GME. The lack of empirical research in this context is an important limitation for educational practice recommendations. Currently there is only preliminary but promising data available for using EPAs with regard to educational outcomes. EPAs seem to be effectively used from a curriculum design perspective for UME and GME in psychiatry.
Collapse
Affiliation(s)
- Severin Pinilla
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland.
| | - Eric Lenouvel
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Werner Strik
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Stefan Klöppel
- University Hospital of Old Age Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Christoph Nissen
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | - Sören Huwendiek
- Institute for Medical Education, University of Bern, Bern, Switzerland
| |
Collapse
|
47
|
Keeley MG, Gusic ME, Morgan HK, Aagaard EM, Santen SA. Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2019; 94:1858-1864. [PMID: 31169542 DOI: 10.1097/acm.0000000000002830] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/13/2023]
Abstract
In the move toward competency-based medical education, leaders have called for standardization of learning outcomes and individualization of the learning process. Significant progress has been made in establishing defined expectations for the knowledge, skills, attitudes, and behaviors required for successful transition to residency training, but individualization of educational processes to assist learners in reaching these competencies has been predominantly conceptual to date. The traditional time-based structure of medical education has posed a challenge to individualization within the curriculum and has led to more attention on innovations that facilitate transition from medical school to residency. However, a shift of focus to the clerkship-to-postclerkship transition point in the undergraduate curriculum provides an opportunity to determine how longitudinal competency-based assessments can be used to facilitate intentional and individualized structuring of the long-debated fourth year.This Perspective demonstrates how 2 institutions-the University of Virginia School of Medicine and the University of Michigan Medical School-are using competency assessments and applying standardized outcomes in decisions about individualization of the postclerkship learning process. One institution assesses Core Entrustable Professional Activities for Entering Residency, whereas the other has incorporated Accreditation Council for Graduate Medical Education core competencies and student career interests to determine degrees of flexibility in the postclerkship phase. Individualization in addition to continued assessment of performance presents an opportunity for intentional use of curriculum time to develop each student to be competently prepared for the transition to residency.
Collapse
Affiliation(s)
- Meg G Keeley
- M.G. Keeley is assistant dean for student affairs, director of the fourth-year program, and professor of pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia; ORCID: https://orcid.org/0000-0001-8602-2638. M.E. Gusic is senior advisor in educational affairs and professor of medical education, University of Virginia School of Medicine, Charlottesville, Virginia. H.K. Morgan is associate professor of learning health sciences and associate professor of obstetrics and gynecology, University of Michigan Medical School, Ann Arbor, Michigan. E.M. Aagaard is senior associate dean for education and professor of medicine, Washington University School of Medicine, St. Louis, Missouri. S.A. Santen is senior associate dean for assessment, evaluation, and scholarship and professor of emergency medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia
| | | | | | | | | |
Collapse
|
48
|
Design of an entrustable professional activity for adult extracorporeal membrane oxygenation. Surg Open Sci 2019; 2:42-45. [PMID: 33981980 PMCID: PMC8083009 DOI: 10.1016/j.sopen.2019.09.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2019] [Revised: 08/30/2019] [Accepted: 09/09/2019] [Indexed: 12/04/2022] Open
Abstract
Background Extracorporeal membrane oxygenation supports severe cardiac or pulmonary failure. There are currently no competency-based standards for extracorporeal membrane oxygenation training. Methods Extracorporeal membrane oxygenation experts were interviewed using a structured interview. Responses were audio recorded, transcribed, and validated by respondents. Interviews were coded using grounded theory with a constant comparison method. Themes were developed and used to construct the entrustable professional activity, which was reviewed by the extracorporeal membrane oxygenation experts. Results Nine experts were interviewed; all had experience with trainees. Interview themes identified include patient selection, circuit and medical management, multidisciplinary communication, problem-based learning and simulation, and entrustment decisions. Essential functions of the entrustable professional activity were patient selection, circuit management, cannula selection, responding to circuit emergencies/complications, anticoagulation management, weaning, and family/team communication. Conclusions Essential functions of an extracorporeal membrane oxygenation entrustable professional activity were defined using data from structured interviews. The resultant entrustable professional activity could be implemented by critical-care programs as a scaffolding for competency-based fellow training. A structured interview is an effective method for an EPA creation. Nine experts with a median 14 years of ECMO experience were interviewed. The essential functions for ECMO care were defined. The ECMO EPA could be used for competency-based critical care education.
Collapse
|
49
|
Francis MK, Wormington SV, Hulleman C. The Costs of Online Learning: Examining Differences in Motivation and Academic Outcomes in Online and Face-to-Face Community College Developmental Mathematics Courses. Front Psychol 2019; 10:2054. [PMID: 31551886 PMCID: PMC6746985 DOI: 10.3389/fpsyg.2019.02054] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2019] [Accepted: 08/23/2019] [Indexed: 11/15/2022] Open
Abstract
Although online courses are becoming increasingly popular in higher education, evidence is inconclusive regarding whether online students are likely to be as academically successful and motivated as students in face-to-face courses. In this study, we documented online and face-to-face students' academic motivation and outcomes in community college mathematics courses, and whether differences might vary based on student characteristics (i.e., gender, underrepresented ethnic/racial minority status, first-generation college status, and adult learner status). Over 2,400 developmental mathematics students reported on their math motivation at the beginning (Week 1) and middle (Weeks 3, 5) of the semester. Findings indicated that online students received lower grades and were less likely to pass from their courses than face-to-face students, with online adult learners receiving particularly low final course grades and pass rates. In contrast, online and face-to-face students did not differ on incoming motivation, with subgroup analyses suggesting largely similar patterns of motivation across student groups. Together, findings suggest that online and face-to-face students may differ overall in academic outcomes but not in their motivation or differentially based on student characteristics. Small but significant differences on academic outcomes across modalities (Cohen's ds = 0.17-0.28) have implications for community college students' success in online learning environments, particularly for adult learners who are most likely to be faced with competing demands.
Collapse
Affiliation(s)
- Michelle K. Francis
- Curry School of Education and Human Development, University of Virginia, Charlottesville, VA, United States
| | - Stephanie V. Wormington
- Curry School of Education and Human Development, University of Virginia, Charlottesville, VA, United States
| | | |
Collapse
|
50
|
Shorey S, Lau TC, Lau ST, Ang E. Entrustable professional activities in health care education: a scoping review. MEDICAL EDUCATION 2019; 53:766-777. [PMID: 30945329 DOI: 10.1111/medu.13879] [Citation(s) in RCA: 114] [Impact Index Per Article: 22.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2018] [Revised: 11/30/2018] [Accepted: 02/22/2019] [Indexed: 05/13/2023]
Abstract
CONTEXT The shift in medical education from time-based learning to outcome-based learning has drawn much attention to entrustable professional activities (EPAs) as an ideal assessment framework to translate competencies into clinical practice. Given the relative novelty of EPAs, this review aims to highlight research gaps and explore and consolidate available evidence pertaining to the development and implementation of EPAs in health care. METHOD Arksey and O'Malley's scoping review framework was used to present the findings. The authors performed a systematic search of PubMed, Embase, CINAHL, Scopus, MedNar, OpenGrey and ProQuest Dissertation and Theses for English articles published from the inception of each database to May 2018. A manual search of the reference lists of the included studies was conducted and an expert panel was consulted. Two reviewers screened the articles for eligibility using the inclusion criteria. All authors extracted key data and analysed the data descriptively. Thematic analysis was used to categorise the results into themes. RESULTS Eighty articles were included in the review. All articles were published between 2010 and 2018. Three major themes and eight sub-themes were generated: (i) development of EPAs (frameworks for EPA development and implementation, identifying core or specialty-specific EPAs, and EPAs for faculty development), (ii) evaluation of EPAs and EPA entrustment factors (revised curriculum, entrustment decisions for professional activities, and feedback on implemented EPAs and the development process), and (iii) future directions and recommendations for EPAs (implementation of EPAs in undergraduate medical education and specific disciplines, and other criticisms and recommendations for EPAs). CONCLUSIONS Entrustable professional activities are an essential means to translate competencies into observable and measurable clinical practice. However, high-level evidence-based research on the efficacy, development and implementation of EPAs for specific target groups (i.e. undergraduates and staff) and geographical regions (i.e. Asia and Africa) is still lacking, which suggests a direction for future research.
Collapse
Affiliation(s)
- Shefaly Shorey
- Alice Lee Centre for Nursing Studies, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
- National University Health System, Singapore, Singapore
- Yong Loo Lin School of Medicine, Singapore, Singapore
| | | | - Siew Tiang Lau
- Alice Lee Centre for Nursing Studies, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
- National University Health System, Singapore, Singapore
- Yong Loo Lin School of Medicine, Singapore, Singapore
| | - Emily Ang
- Alice Lee Centre for Nursing Studies, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
- National University Health System, Singapore, Singapore
- Yong Loo Lin School of Medicine, Singapore, Singapore
| |
Collapse
|