Medical Students' Views on Implementing the Core EPAs: Recommendations From Student Leaders at the Core EPAs Pilot Institutions

Integrating and Assessing Entrustable Professional Activities in Doctor of Pharmacy Curricula

OBJECTIVES

The 2022 Curriculum Outcomes and Entrustable Professional Activities include an updated set of entrustable professional activities (EPAs) for Doctor of Pharmacy graduates. To assist pharmacy educators with the integration and assessment of EPAs, this work aimed to review and synthesize approaches to integrate EPAs in health professions curricula, strategies for EPA-related assessments and entrustment in health professions curricula, and challenges and research needs for integrating EPAs in assessment frameworks in Doctor of Pharmacy curricula.

FINDINGS

A literature review resulted in 114 articles identified as having relevance with 33 articles specific to pharmacy education. There are multiple components of an effective EPA integration and assessment plan, including administrative leadership, determination of programmatic EPAs with comprehensive descriptions, selection of an entrustment-supervision scale, identification of EPA-based assessments, EPA mapping, data management, and training and development.

SUMMARY

Strategies with applicability for integrating and assessing EPAs in pharmacy education were analyzed and synthesized. Evidence-informed and expert recommendations for EPA integration and assessment are discussed. Programs can apply these recommendations to implement and assess EPAs in alignment with their own institutional and curricular contexts.

Next steps for workplace-based assessments of entrustable professional activities

BACKGROUND

Florida International University Herbert Wertheim College of Medicine (FIU-HWCOM) participated in the AAMC Core Entrustable Professional Activities (EPA) implementation pilot. Entrustment decision processes based on data from workplace-based assessments (WBAs) were piloted. Outcomes illustrated challenges including variability across EPAs with regards to learner level alignment and feasibility of data collection in the form of WBAs. In addition, students reported discomfort requesting WBA completion by preceptors and dissatisfaction with associated feedback.

APPROACH

To guide future directions, we conducted a survey of third-year students to better understand their experience with and perceptions of WBAs used to evaluate EPAs at FIU-HWCOM.

EVALUATION

Survey response was 96% (n = 107/112). Most (84%) reported that WBAs were not valuable to their development and that preceptors often did not complete WBAs in a timely fashion. Many (47%) reported not receiving verbal feedback. Most students (78%) used language in written responses demonstrating confusion between the EPAs and the WBAs used to assess them.

IMPLICATIONS

The use of WBAs to assess EPAs did not have its intended impact at FIU-HWCOM. For future classes, WBA forms will consist of paper cards with questions directly assessing performance of skills aligned with EPAs 1, 5 and 6 only. To continue to promote feedback, students will be required to collect WBAs on all clerkships, but the number of required WBAs will be less than prior and no entrustment decisions will be made.

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional generalisability study

BACKGROUND

Assessment of the Core Entrustable Professional Activities for Entering Residency requires direct observation through workplace-based assessments (WBAs). Single-institution studies have demonstrated mixed findings regarding the reliability of WBAs developed to measure student progression towards entrustment. Factors such as faculty development, rater engagement and scale selection have been suggested to improve reliability. The purpose of this investigation was to conduct a multi-institutional generalisability study to determine the influence of specific factors on reliability of WBAs.

METHODS

The authors analysed WBA data obtained for clerkship-level students across seven institutions from 2018 to 2020. Institutions implemented a variety of strategies including selection of designated assessors, altered scales and different EPAs. Data were aggregated by these factors. Generalisability theory was then used to examine the internal structure validity evidence of the data. An unbalanced cross-classified random-effects model was used to decompose variance components. A phi coefficient of >0.7 was used as threshold for acceptable reliability.

RESULTS

Data from 53 565 WBAs were analysed, and a total of 77 generalisability studies were performed. Most data came from EPAs 1 (n = 17 118, 32%) 2 (n = 10 237, 19.1%), and 6 (n = 6000, 18.5%). Low variance attributed to the learner (<10%) was found for most (59/77, 76%) analyses, resulting in a relatively large number of observations required for reasonable reliability (range = 3 to >560, median = 60). Factors such as DA, scale or EPA were not consistently associated with improved reliability.

CONCLUSION

The results from this study describe relatively low reliability in the WBAs obtained across seven sites. Generalisability for these instruments may be less dependent on factors such as faculty development, rater engagement or scale selection. When used for formative feedback, data from these instruments may be useful. However, such instruments do not consistently provide reasonable reliability to justify their use in high-stakes summative entrustment decisions.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

Entrustable professional activities as a training and assessment framework in undergraduate medical education: A case study of a multi-institutional pilot

In 2014, the Association of American Medical Colleges (AAMC) published 13 Core Entrustable Professional Activities (EPAs) that graduating students should be able to perform with indirect supervision when entering residency. A ten-school multi-year pilot was commissioned to test feasibility of implementing training and assessment of the AAMC's 13 Core EPAs. In 2020-21, a case study was employed to describe pilot schools' implementation experiences. Teams from nine of ten schools were interviewed to identify means and contexts of implementing EPAs and lessons learned. Audiotapes were transcribed then coded by investigators using conventional content analysis and a constant comparative method. Coded passages were organized in a database and analyzed for themes. Consensus among school teams regarding facilitators of EPA implementation included team commitment to piloting EPAs; agreement that: proximal EPA adoption with curriculum reform facilitates EPA implementation; EPAs 'naturally fit' in clerkships and provided opportunity for schools to reflect on and adjust curricula and assessments; and inter-school collaboration bolstered individual school progress. Schools did not make high-stakes decisions about student progress (e.g., promotion, graduation), yet EPA assessment results complemented other forms of assessment in providing students with robust formative feedback about their progress. Teams had varied perceptions of school capability to implement an EPA framework, influenced by various levels of dean involvement, willingness, and capability of schools to invest in data systems and provide other resources, strategic deployment of EPAs and assessments, and faculty buy-in. These factors affected varied pace of implementation. Teams agreed on the worthiness of piloting the Core EPAs, but substantial work is still needed to fully employ an EPA framework at the scale of entire classes of students with enough assessments per EPA and with required data validity/reliability. Recommendations stemming from findings may help inform further implementation efforts across other schools adopting or considering an EPA framework.

Entrustable professional activities (EPAs) for undergraduate medical education - development and exploration of social validity

BACKGROUND

The development of entrustable professional activities (EPAs) as a framework for work-based training and assessment in undergraduate medical education has become popular. EPAs are defined as units of a professional activity requiring adequate knowledge, skills, and attitudes, with a recognized output of professional labor, independently executable within a time frame, observable and measurable in its process and outcome, and reflecting one or more competencies. Before a new framework is implemented in a specific context, it is valuable to explore social validity, that is, the acceptability by relevant stakeholders.

AIM

The aim of our work was to define Core EPAs for undergraduate medical education and further explore the social validity of the constructs.

METHOD AND MATERIAL

In a nationwide collaboration, EPAs were developed using a modified Delphi procedure and validated according to EQual by a group consisting of teachers nominated from each of the seven Swedish medical schools, two student representatives, and an educational developer (n = 16). In the next step, social validity was explored in a nationwide survey. The survey introduced the suggested EPAs. For each EPA, the importance of the EPA was rated, as was the rater's perception of the present graduates' required level of supervision when performing the activity. Free-text comments were also included and analyzed.

RESULTS

Ten Core EPAs were defined and validated. The validation scores for EQual ranged from 4.1 to 4.9. The nationwide survey had 473 responders. All activities were rated as "important" by most responders, ranging from 54 to 96%. When asked how independent current graduates were in performing the ten activities, 6 to 35% reported "independent". The three themes of the free text comments were: 'relevant target areas and content'; 'definition of the activities'; and 'clinical practice and learning'.

CONCLUSION

Ten Core EPAs were defined and assessed as relevant for Swedish undergraduate medical education. There was a consistent gap between the perceived importance and the certainty that the students could perform these professional activities independently at the time of graduation. These results indicate that the ten EPAs may have a role in undergraduate education by creating clarity for all stakeholders.

Commentary on alignment of medical student assessment and vocational training in psychiatry and addiction medicine

OBJECTIVE

To comment upon the potential for alignment of medical student assessment and vocational specialist training through the RANZCP-CanMEDS model of Entrustable Professional Activities (EPAs) and Workplace-Based Assessments (WBAs). We discuss a specific post hoc example of such an alignment in an Australian graduate medical school in Psychiatry and Addiction Medicine.

CONCLUSIONS

Vocational training models of assessment, such as the RANZCP specialist training program for psychiatrists, can potentially be mapped to medical student education in formative and summative assessment through CanMEDs-based EPAs and WBAs, to assist in transition to specialist training.

Oral examination of fourth-year medical students on surgery rotations allows faculty to assess Core Entrustable Professional Activities for entering Residency (CEPAR): Proof-of-concept and analysis of student submissions

BACKGROUND

As a community-based medical school which recruited faculty preceptors new to teaching, we sought to create objective assessments for fourth-year surgery experiences via administration of an oral exam. Students provided three authentic cases, which faculty used as a springboard to ascertain student proficiency in five entrustable professional activities: 1-oral presentation, 2-recognition of urgency/instability, 3-calling consults, 4-transitions of care, 5-informed consent. We present proof-of-concept and analysis of student case submissions.

METHODS

Twenty-seven student submissions (79 cases in total) were evaluated for case complexity, level-appropriateness, and an estimation of the ability to conduct a quality exam based on the information provided (subjective measures). Objective metrics included word count, instruction adherence, inclusion of figures/captions. A resident-in-training rated cases via the same metrics. In-examination data was separately culled.

RESULTS

The average word count was 281.70 (SD 140.23; range 40-743). Figures were included in 26.1% of cases. Faculty raters scored 29.0% as low-complexity, 37.7% medium-complexity, and 33.3% high-complexity. Raters felt 62.3% of cases provided enough information to conduct a quality exam. The majority of cases submitted (65.2%) were level-appropriate or higher. The resident rater scored cases more favorably than surgeons (Cohen's kappa of -0.5), suggesting low inter-rater agreement between those of differing experience levels.

CONCLUSION

Student's case submissions lessened faculty burden and provided assessors with adequate information to deliver a quality exam to assess proficiency in clinical skills essential for residency. Cases demonstrated sufficient complexity and level-appropriateness. The request to correlate case rating with exam performance is under review by our institution's assessment office. Near-peer tutoring by resident alumni is a program under development.

Using entrustable professional activities to better prepare students for their postgraduate medical training: A medical student's perspective

THE PROBLEM

Medical students graduate underprepared for postgraduate medical training despite years of classroom and clinical training. In this article, a medical student shares her personal perspectives on three factors contributing to this problem in undergraduate medical education: students' peripheral roles in the clinical environment impede learning, students receive inadequate feedback, and assessments do not measure desired learning outcomes.

A SOLUTION

The authors describe how using entrustable professional activities (EPAs) could address these issues and promote students' clinical engagement by clarifying their roles, providing them with frequent and actionable feedback, and aligning their assessments with authentic work. These factors combined with grading schemes rewarding improvement could contribute to a growth mindset that reprioritizes clinical skill acquisition. The authors explore how medical schools have begun implementing the EPA framework, highlight insights from these efforts, and describe barriers that must be addressed.

THE FUTURE

Incorporating EPAs into medical school curricula could better prepare students for postgraduate training while also alleviating issues that contribute to student burnout by defining students' roles, improving feedback, and aligning assessments with desired learning outcomes.

WBAs in UME-How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship

BACKGROUND

Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces.

DESIGN

A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold.

KEY RESULTS

A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity.

CONCLUSIONS

Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.

Student-Led Initiatives and Advocacy in Academic Medicine: Empowering the Leaders of Tomorrow

Trainees' value as effective leaders within academic medicine has been increasingly recognized. From their perspective as adult learners who enter medical education from diverse backgrounds, trainees offer significant value to the teaching, learning, and practice of medicine. As such, trainees have developed and led various successful initiatives throughout academic medicine. In this Invited Commentary, 2 medical students with national leadership roles provide their perspectives on how student-led initiatives and advocacy can help push academic medicine forward. The authors first provide an overview of the success of student-led initiatives throughout medical education as evidenced by the Trainee-Authored Letters to the Editor in this issue, highlighting the unique contributions and perspectives of trainees in the development and implementation of new initiatives or ways of thinking. Although trainees add value to many areas in academic medicine, here the authors highlight 4 current areas that align with Association of American Medical Colleges priorities: (1) public health emergencies including the COVID-19 pandemic; (2) equity, inclusion, and diversity; (3) wellness and resilience amongst trainees and health care providers; and (4) recent changes to the United States Medical Licensing Examination and the transition to residency. By sharing their experiences with student-led initiatives within each of these domains, the authors provide lessons learned and discuss successes and obstacles encountered along the way. Overall, there is a critical need for increased engagement of trainees in medical education. Empowering trainees now ensures the academic medicine leaders of tomorrow are prepared to face the challenges that await them.

Aligning Medical Student Workplace-Based Assessments with Entrustable Professional Activities and the RIME Model in a Psychiatry Clerkship

OBJECTIVE

The authors piloted use of workplace-based assessments of students during the psychiatry clerkship utilizing both entrustable professional activities (EPAs) and the reporter, interpreter, manager, and educator (RIME) model.

METHODS

After supervising clinicians conducted assessments of medical students (N=109) during the psychiatry clerkship using a supervisory scale aligned with both EPA and RIME models, each student received individualized formative feedback. Students were then surveyed on the usefulness of this feedback, and participating faculty/residents were surveyed on the ease of completion of the supervisory scale.

RESULTS

Students' mean skill profile suggested they no longer needed direct supervision on EPA1 and EPA6. Mean scores on other studied EPAs suggested students were well on their way toward performing these EPAs without direct supervision. Students had mean RIME scores that exceeded the suggested levels identified for a Reporter to start clerkships, for an Interpreter to start clerkships, and for a Manager to transition to the fourth year. Close to half of the students found the feedback helpful in their development as a clinician but most felt their performance should not be shared with residency program directors, either before or after the Match. Almost all responding preceptors felt the supervisory ratings were easy to complete.

CONCLUSIONS

This pilot RIME/EPA framework served as a successful step toward a more competency-based medical education in the psychiatry clerkship with relatively little additional faculty time commitment by using workplace-based assessments already in place and a supervisory scale based on EPAs and RIME.

Entrustable professional activities versus competencies and skills: Exploring why different concepts are often conflated

Despite explanations in the literature, a returning question in the use of entrustable professional activities (EPAs) is how to distinguish them from competencies and skills. In this article, we attempt to analyze the causes of the frequent confusion and conflation of EPAs with competencies and skills, and argue why the distinction is important for education, qualification and patient safety. 'Tracheotomy', 'lumbar puncture', 'interprofessional collaboration' for example are colloquially called 'skills', but its is a person's ability to perform these activities that is the actual skill; the EPA is simply the activity itself. We identify two possible causes for the confusion. One is a tendency to frame all educational objectives as EPAs. Many objectives of medical training can be conceptualized as EPAs, if 'the ability to do X' is the corresponding competency; but that does not work for all. We offer ways to deal with objectives of training that are not usefully conceptualized as EPAs. A more fundamental cause relates to entrustment decisions. The permission to contribute to health care reflects entrustment. Entrustment decisions are the links or pivots between a person's readiness for the task and the actual task execution. However, if entrustment decisions do not lead to increased autonomy in the practice of health care, but only serve to decide upon the advancement to a next stage of training, EPAs can become the tick boxes learners feel they need to collect to 'pass'. Gradually, then, EPAs can loose their original meaning of units of practice for which one becomes qualified.

Development and consensus of entrustable professional activities for final-year medical students in anaesthesiology

Background

The transfer of classic concepts of competency-based medical education into clinical practice has been proven to be difficult in the past, being described as partially fragmented, misleading and inadequate. At the beginning of training, novice doctors commonly feel overwhelmed, overloaded and exposed to extreme time pressure. The discrepancy between expected and actual clinical competence of doctors at the start of their speciality training jeopardizes patient safety. The framework of Entrustable Professional Activities (EPAs) is a promising instrument to effectively integrate competency-based training into clinical practice and may help to close this gap and consequently to improve patient safety.

Methods

For anaesthesiology, we developed 5 EPAs for final-year medical students. The EPAs comprised the following seven categories: 1. Title, 2. Specifications, 3. Limitations, 4. Competency domains, 5. Knowledge, abilities and skills, professional attitudes, 6. Assessment and 7. Entrustment. Based on a modified, online-based Delphi study, we further developed and refined these EPAs. Education experts were recruited from the alumni network of the Master of Medical Education (MME) degree course from the University of Heidelberg, Germany.

Results

28 data sets were evaluated in three Delphi rounds. 82% of study participants had previous experience with EPAs. Qualitative and quantitative data formed the basis during the iterative process and resulted in complete descriptions of 5 EPAs for final-year medical students in anaesthesiology.

Conclusions

Our study including the associated description of 5 EPAs represent a further step and starting point for EPA-based curricula in medical training in Germany linking undergraduate training, to residency training and continuous medical education.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12871-022-01668-8.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study

PURPOSE

In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions.

METHOD

For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables.

RESULTS

The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05).

CONCLUSIONS

TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study

BACKGROUND

The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment.

OBJECTIVE

In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform.

METHODS

Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis.

RESULTS

All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface.

CONCLUSIONS

This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Revisiting the Core Entrustable Professional Activities for Entering Residency

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.