Faculty Development Revisited: A Systems-Based View of Stakeholder Development to Meet the Demands of Entrustable Professional Activity Implementation

EMYWAY Workplace-Based Entrustable Professional Activities Assessments in Otolaryngology Residency Training: A Nationwide Experience

OBJECTIVE

To present workplace-based entrustable professional activities (EPAs) assessment data from the first 2 years of the EMYWAY platform in otolaryngology residency training in Taiwan.

STUDY DESIGN

Two-year cross-sectional study.

SETTING

Otolaryngology training programs.

METHODS

In 2020, the Taiwan Society of Otorhinolaryngology-Head and Neck Surgery (TSO-HNS) developed a workplace-based assessment (WBA) framework with 11 EPAs, integrating milestones to evaluate resident competency. In 2021, TSO-HNS piloted the EMYWAY platform for WBAs, which includes an EPA-based assessment workflow, coaching feedback, and a dashboard displaying residents' entrustment-supervision levels. Data are analyzed annually for accreditation and curriculum enhancement. This study reports on the pilot year and the first full-scale year of implementation.

RESULTS

Eleven programs participated in the pilot year. Subsequently, 362 faculty members and 274 resident physicians from 34 programs nationwide engaged with EMYWAY. In the full-scale year from August 2022 to July 2023, 9805 responses were recorded, primarily from surgical theaters (45.9%; 4502/9805) and third-year residents (23.8%; 2331/9805). The most frequently evaluated EPAs were "head and neck" (17.5%; 1716/9805), "sinonasal" (13.5%; 1324/9805), and "ear" (12.2%; 1193/9805), with task complexity increasing with resident seniority (P < .0001). A positive correlation was found between residents' self-assessments and faculty members' ratings (r = 0.531; P < .001). Over 98.2% of residents and 88.4% of faculty members provided substantial feedback (>10 words). Analysis of WBAs reported by training programs identified faculty development targets and teaching-intensive tasks.

CONCLUSION

EMYWAY effectively documents workplace learning and tracks resident competency progression. Continuous improvement of WBA quality is essential for advancing the competency-based medical education ecosystem.

Disentangling faculty development: A scoping review towards a rich description of the concept and its practice

BACKGROUND

There is wide variation in how faculty development (FD) is practiced globally and described in the literature. This scoping review aims to clarify how FD is conceptualised and practiced in health professions education.

METHODOLOGY

Using a systematic search strategy, 418 papers, published between 2015-2023, were included for full text review. We extracted data using closed and open-ended questions. Quantitative data were summarised using descriptive statistics and qualitative data synthesised using content analysis.

RESULTS

Faculty development was the most frequently used term encompassing a range of understandings and practices. Many papers focused on educators' enhanced understanding of teaching, learning, and assessment. Several highlighted the social context of collaborative practice and organisational learning. FD formats included workshops, courses, longitudinal programs, and coaching and mentoring. Dominant conceptual frameworks included Kirkpatrick's model of evaluation, communities of practice theory, adult learning theory, and experiential learning.

CONCLUSIONS

Although FD continues to evolve in response to the dynamic HPE landscape, this growth needs to be accelerated. To facilitate meaningful collaboration across professions, contexts, and countries, attention must be paid to terms used and meanings ascribed to them. Those responsible for FD need to think anew about its purpose and practice, demonstrating flexibility as the ever-changing context demands.

Implementation of a Clinical Entrustment Scale and Feedback Form in an Academic Veterinary Medical Center: An Empirical Analysis of Goal Oriented Learner Driven-Entrustment (GOLD-E) Assessment Tool

This paper presents findings from an empirical analysis conducted on the initial implementation of Goal Oriented Learner Driven-Entrustment (GOLD-E). Specifically, researchers examined the following questions: How do faculty, technicians, and residents/interns integrate GOLD-E into their assessment process? Is GOLD-E user friendly (e.g., form and functionality)? How do faculty, technicians, and residents/interns navigate the shift from evaluator to coach? Researchers incorporated a number of mixed, overlapping methodologies consisting of both qualitative and quantitative survey responses and focus group interactions. The use of these multiple data representations allowed researchers to gather layered and complex data to provide for a fuller understanding of the initial implementation of the GOLD-E tool. The empirical analysis demonstrates the need for revisions in the GOLD-E assessment tool as well as broad systemic changes to drive transformation in the culture of assessment.

Knowledge and Perceptions of Competency-Based Medical Education in Diagnostic Radiology Post-Graduate Medical Education: Identifying Priorities and Developing a Framework for Professional Development Activities

Introduction: We evaluated knowledge and perceptions of an established Competency-Based Medical Education (CBME) model developed by the Royal College of Physicians and Surgeons of Canada, Competence by Design (CBD), and identified evidence-informed priorities for professional development activities (PDAs). Materials and Methods: Teaching faculty and residents at a single, large diagnostic radiology post-graduate medical education (PGME) program were eligible to participate in this cross-sectional, survey-based study. Knowledge of CBD was evaluated through multiple choice questions (MCQs), which assessed participants' understanding of major principles and terms associated with CBD. Participants' perceptions of the anticipated impact of CBD on resident education and patient care were evaluated and priorities for PDAs were identified, which informed a framework for CBD PDAs. Results: Fifty faculty and residents participated. The faculty and resident response rates were 11.6% (n = 29/249) and 55.3% (n = 21/38), respectively. The mean ± standard deviation overall score on MCQs was 39.0% ± 20.4%. The majority of participants perceived the impact of CBD on resident education to be equivocal and to not impact patient care. Knowledge of CBD was not statistically significantly associated with participants' perceptions of the impact of CBD on either resident education or patient care (P > .05). Delivery of high-quality feedback was the greatest priority identified for PDAs. Discussion: Our results and proposed CBD PDAs framework may help to guide diagnostic radiology PGME programs in designing evidence-informed PDAs, which may meaningfully contribute to the successful implementation of CBD in diagnostic radiology PGME. As diagnostic radiology PGME programs throughout the world increasingly implement CBME models, evidence-informed PDAs will become of increasing importance.

Entrustable Professional Activities and Learning: The Postgraduate Trainee Perspective

OBJECTIVE

Entrustable professional activities (EPAs) are used as clinical activities in postgraduate psychiatry training in Australasia. This study aimed to explore psychiatry trainees' perceptions of the impact of EPAs on their motivation and learning.

METHODS

A constructivist grounded theory approach was used to conceptualize the impact of EPAs on trainees' motivation and learning. A purposive sample of trainees was recruited from across New Zealand. Semi-structured individual interviews were used for data collection and continued until theoretical saturation was reached.

RESULTS

The impact of EPAs on learning was mediated by the trainee's appraisals of subjective control, value, and the costs of engaging with EPAs. When appraisals were positive, EPAs encouraged a focus on particular learning needs and structured learning with the supervisor. However, when appraisals were negative, EPAs encouraged a superficial approach to learning. Trainee appraisals and their subsequent impact on motivation and learning were most affected by EPA granularity, alignment of EPAs with clinical practice, and the supervisor's conscientiousness in their approach to EPAs.

CONCLUSIONS

To stimulate learning, EPAs must be valued by both trainees and supervisors as constituting a coherent work-based curriculum that encompasses the key fellowship competencies. If EPAs are to be effective as clinical tasks for learning, ongoing faculty development must be the leading priority.

Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education

PURPOSE

These authors sought to define the new roles and competencies required of administrative staff and faculty in the age of CBME.

METHOD

A modified Delphi process was used to define the new CBME roles and competencies needed by faculty and administrative staff. We invited international experts in CBME (volunteers from the ICBME Collaborative email list), as well as faculty members and trainees identified via social media to help us determine the new competencies required of faculty and administrative staff in the CBME era.

RESULTS

Thirteen new roles were identified. The faculty-specific roles were: National Leader/Facilitator in CBME; Institutional/University lead for CBME; Assessment Process & Systems Designer; Local CBME Leads; CBME-specific Faculty Developers or Trainers; Competence Committee Chair; Competence Committee Faculty Member; Faculty Academic Coach/Advisor or Support Person; Frontline Assessor; Frontline Coach. The staff-specific roles were: Information Technology Lead; CBME Analytics/Data Support; Competence Committee Administrative Assistant.

CONCLUSIONS

The authors present a new set of faculty and staff roles that are relevant to the CBME context. While some of these new roles may be incorporated into existing roles, it may be prudent to examine how best to ensure that all of them are supported within all CBME contexts in some manner.

Using entrustable professional activities to better prepare students for their postgraduate medical training: A medical student's perspective

THE PROBLEM

Medical students graduate underprepared for postgraduate medical training despite years of classroom and clinical training. In this article, a medical student shares her personal perspectives on three factors contributing to this problem in undergraduate medical education: students' peripheral roles in the clinical environment impede learning, students receive inadequate feedback, and assessments do not measure desired learning outcomes.

A SOLUTION

The authors describe how using entrustable professional activities (EPAs) could address these issues and promote students' clinical engagement by clarifying their roles, providing them with frequent and actionable feedback, and aligning their assessments with authentic work. These factors combined with grading schemes rewarding improvement could contribute to a growth mindset that reprioritizes clinical skill acquisition. The authors explore how medical schools have begun implementing the EPA framework, highlight insights from these efforts, and describe barriers that must be addressed.

THE FUTURE

Incorporating EPAs into medical school curricula could better prepare students for postgraduate training while also alleviating issues that contribute to student burnout by defining students' roles, improving feedback, and aligning assessments with desired learning outcomes.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

IMPORTANCE

Gaps in readiness for indirect supervision have been identified for essential responsibilities encountered early in residency, presenting risks to patient safety. Core Entrustable Professional Activities (EPAs) for entering residency have been proposed as a framework to address these gaps and strengthen the transition from medical school to residency.

OBJECTIVE

To assess progress in developing an entrustment process in the Core EPAs framework.

DESIGN, SETTING, AND PARTICIPANTS

In this quality improvement study in the Core EPAs for Entering Residency Pilot, trained faculty made theoretical entrustment determinations and recorded the number of workplace-based assessments (WBAs) available for each determination in 2019 and 2020. Four participating schools attempted entrustment decision-making for all graduating students or a randomly selected subset of students. Deidentified, individual-level data were merged into a multischool database.

INTERVENTIONS

Schools implemented EPA-related curriculum, WBAs, and faculty development; developed systems to compile and display data; and convened groups to make theoretical summative entrustment determinations.

MAIN OUTCOMES AND MEASURES

On an EPA-specific basis, the percentage of students for whom an entrustment determination could be made, the percentage of students ready for indirect supervision, and the volume of WBAs available were recorded.

RESULTS

Four participating schools made 4525 EPA-specific readiness determinations (2296 determinations in 2019 and 2229 determinations in 2020) for 732 graduating students (349 students in 2019 and 383 students in 2020). Across all EPAs, the proportion of determinations of "ready for indirect supervision" increased from 2019 to 2020 (997 determinations [43.4%] vs 1340 determinations [60.1%]; 16.7 percentage point increase; 95% CI, 13.8-19.6 percentage points; P < .001), as did the proportion of determinations for which there were 4 or more WBAs (456 of 2295 determinations with WBA data [19.9%] vs 938 [42.1%]; 22.2 percentage point increase; 95% CI, 19.6-24.8 percentage points; P < .001). The proportion of EPA-specific data sets considered for which an entrustment determination could be made increased from 1731 determinations (75.4%) in 2019 to 2010 determinations (90.2%) in 2020 (14.8 percentage point increase; 95% CI, 12.6-16.9 percentage points; P < .001). On an EPA-specific basis, there were 5 EPAs (EPA 4 [orders], EPA 8 [handovers], EPA 10 [urgent care], EPA 11 [informed consent], and EPA 13 [patient safety]) for which few students were deemed ready for indirect supervision and for which there were few WBAs available per student in either year. For example, for EPA 13, 0 of 125 students were deemed ready in 2019 and 0 of 127 students were deemed ready in 2020, while 0 determinations in either year included 4 or more WBAs.

CONCLUSIONS AND RELEVANCE

These findings suggest that there was progress in WBA data collected, the extent to which entrustment determinations could be made, and proportions of entrustment determinations reported as ready for indirect supervision. However, important gaps remained, particularly for a subset of Core EPAs.

Evaluating the Association of a Core EPA-Oriented Patient Handover Curriculum on Medical Students' Self-reported Frequency of Observation and Skill Acquisition

INTRODUCTION

"Giving or receiving a patient handover to transition patient care responsibility" is one of the thirteen Core Entrustable Professional Activities (Core EPAs) for Entering Residency. However, implementing a patient handover curriculum in undergraduate medical education (UME) remains challenging. Educational leaders in the multi-institutional Core EPA8 pilot workgroup developed a longitudinal patient handover UME curriculum that was implemented at two pilot institutions.

MATERIALS AND METHODS

We utilized multi-school graduation questionnaire data to assess the association of our patient handover curriculum on self-reported frequency of observation/feedback and skill acquisition by comparing data from the shared curriculum schools to data from other Core EPA pilot schools (three schools with school-specific curriculum; five without a dedicated handover curriculum). Questionnaire data from 1,278 graduating medical students of the class of 2020 from all ten Core EPA pilot schools were analyzed.

RESULTS

Graduates from the two medical schools that implemented the shared patient handover curriculum reported significantly greater frequency of handover observation/feedback compared to graduates at the other schools (school-specific curriculum (p < .05) and those without a handover curriculum (p < .05)). Graduates from the two shared approach schools also more strongly agreed that they possessed the skill to perform handovers compared to graduates from the other eight pilot schools that did not implement this curriculum.

CONCLUSION

The findings of this study suggest that the implementation of a multi-institutional Core EPA-based curricular model for teaching and assessing patient handovers was successful and could be implemented at other UME institutions.

WBAs in UME-How Many Are Needed? A Reliability Analysis of 5 AAMC Core EPAs Implemented in the Internal Medicine Clerkship

BACKGROUND

Reliable assessments of clinical skills are important for undergraduate medical education, trustworthy handoffs to graduate medical programs, and safe, effective patient care. Entrustable professional activities (EPAs) for entering residency have been developed; research is needed to assess reliability of such assessments in authentic clinical workspaces.

DESIGN

A student-driven mobile assessment platform was developed and used for clinical supervisors to record ad hoc entrustment decisions using the modified Ottawa scale on 5 core EPAs in an 8-week internal medicine (IM) clerkship. After a 12-month period, generalizability (G) theory analysis was performed to estimate the reliability of entrustment scores and determine the proportion of variance attributable to the student and the other facets, including particular EPA, evaluator type (attending versus resident), or case complexity. Decision (D) theory analysis determined the expected reliability based on the number of hypothetical observations. A g-coefficient of 0.7 was used as a generally agreed upon minimum reliability threshold.

KEY RESULTS

A total of 1368 ratings over the 5 EPAs were completed on 94 students. Variance attributed to person (true variance) was high for all EPAs; EPA-5 had the lowest person variance (9.8% across cases and four blocks). Across cases, reliability ranged from 0.02 to 0.60. Applying this to the Decision study, the estimated number of observations needed to reach a reliability index of 0.7 ranged between 9 and 11 for all EPAs except EPA5 which was sensitive to case complexity.

CONCLUSIONS

Work place-based clinical skills in IM clerkship students were assessed and logged using a convenient mobile platform. Our analysis suggests that 9-11 observations are needed for these EPA workplace-based assessments (WBAs) to achieve a reliability index of 0.7. Note writing was very sensitive to case complexity. Further reliability analyses of core EPAs are needed before US medical schools consider wider adoption into summative entrustment processes and GME handoffs.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Perception of Faculty toward Challenges in Teaching and the Role of Medical Education Workshops in Addressing Them: A Mixed-Methods Study

Background  A medical teacher has a very complex job profile, wherein they are expected to train a cohort of newly joined medical aspirants to competent health care professionals. The current study was conducted to identify the challenges faced by the faculty members in medical teaching and to assess the perception of teachers on the role of medical education workshops in addressing these challenges.

Methods  It was a Mixed Methods study conducted over a period of 9 months from January to September 2021 in a tertiary teaching medical college and hospital of Chengalpet District of Tamil Nadu amongst the faculty members of the teaching medical college and hospital. In the first phase, universal sampling method was employed, wherein all faculty members were asked to fill the online semi-structured questionnaire. In the second phase, purposive sampling was employed, wherein all teachers who have a minimum teaching experience of 3 years and have attended any medical education workshop were included and interviewed using a focus group discussion guide. The quantitative date was analyzed using descriptive statistics, while the qualitative data was analyzed using manual content analysis.

Results  In the quantitative phase, 149 faculty members filled the online questionnaire, of which majority were females (52.3%) and were from the clinical departments (63.1%). The manual thematic content analysis of the FGD resulted in the identification of two themes (challenges and utility of workshops). The challenges theme consisted of three categories of Faculty, Students, and Administration, while Utility of workshops theme included two categories of refinement of skills, and suggestions for future

Conclusion  In conclusion, lack of teamwork in the department, multiple responsibilities assigned to a single person, and shortage of clinical material during COVID-19 were identified as the main challenges in teaching–learning. The medical education workshops play a significant role in improving the knowledge of various domains of teaching and assessment, introduction of innovations, and ensure better performance of the faculty members.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters

PURPOSE

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales.

METHOD

The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments.

RESULTS

Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs.

CONCLUSIONS

In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

An adaptation-focused evaluation of Canada's first competency-based medical education implementation in radiology

OBJECTIVES

Systematic program evaluation of the Queen's University diagnostic radiology residency program following transition to a competency-based medical education (CBME) curriculum.

METHODS

Rapid Evaluation methodology and the Core Components Framework were utilized to measure CBME implementation. A combination of interviews and focus groups were held with program leaders (n = 6), faculty (n = 10), both CBME stream and traditional stream residents (n = 6), and program staff (n = 2). Interviews and focus groups were transcribed and analyzed abductively. Study team met with program leaders to review common themes and plan potential adaptations.

RESULTS

Strengths of CBME implementation included more frequent and timely feedback as well as the role of the Academic Advisor. However, frontline faculty felt insufficiently supported with regards to the theory and practical implementation of the new curriculum and found assessment tools unintuitive. The circumstances surrounding the curricular implementation also resulted in some negative sentiment. Additional faculty and resident education workshops were identified as areas for improvement as well as changes to assessment tools for increased clarity. Residents overall viewed the changes favorably, with traditional stream residents indicating that they also had a desire for increased feedback.

CONCLUSIONS

Rapid Evaluation is an effective method for program assessment following curricular change in diagnostic radiology. A departmental champion driving enthusiasm for change from within may be valuable. Adequate resident and faculty education is key to maximize change and smooth the transition. Advances in knowledge: This study provides insights for other radiology training programs transitioning to a CBME framework and provides a structure for programmatic assessment.

The use of EPA assessments in decision-making: Do supervision ratings correlate with other measures of clinical performance?

BACKGROUND

Entrustable professional activities (EPAs) have been introduced as a framework for teaching and assessment in competency-based educational programs. With growing use, has come a call to examine the validity of EPA assessments. We sought to explore the correlation of EPA assessments with other clinical performance measures to support use of supervision ratings in decisions about medical students' curricular progression.

METHODS

Spearman rank coefficients were used to determine correlation of supervision ratings from EPA assessments with scores on clerkship evaluations and performance on an end-of-clerkship-year Objective Structured Clinical Examination (CPX).

RESULTS

Both overall clinical evaluation items score (rho 0.40; n = 166) and CPX patient encounter domain score (rho 0.31; n = 149) showed significant correlation with students' overall mean EPA supervision rating during the clerkship year. There was significant correlation between mean supervision rating for EPA assessments of history, exam, note, and oral presentation skills with scores for these skills on clerkship evaluations; less so on the CPX.

CONCLUSIONS

Correlation of EPA supervision ratings with commonly used clinical performance measures offers support for their use in undergraduate medical education. Data supporting the validity of EPA assessments promotes stakeholders' acceptance of their use in summative decisions about students' readiness for increased patient care responsibility.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Working with entrustable professional activities in clinical education in undergraduate medical education: a scoping review

BACKGROUND

Entrustable professional activities (EPAs) are increasingly used in undergraduate medical education (UME). We conducted a scoping review to summarize the evidence for the use of EPAs in clinical rotations in UME.

METHODS

We searched multiple databases for scoping reviews based on the PRISMA guidelines for articles reporting qualitative and quantitative research, as well as conceptual and curriculum development reports, on EPAs in UME clinical rotations.

RESULTS

We identified 3309 records by searching through multiple databases. After the removal of duplicates, 1858 reports were screened. A total of 36 articles were used for data extraction. Of these, 47% reported on EPA and EPA-based curriculum development for clerkships, 50% reported on implementation strategies, and 53% reported on assessment methods and tools used in clerkships. Validity frameworks for developing EPAs in the context of clerkships were inconsistent. Several specialties reported feasible implementation strategies for EPA-based clerkship curricula, however, these required additional faculty time and resources. Limited exposure to clinical activities was identified as a barrier to relevant learning experiences. Educators used nationally defined, or specialty-specific EPAs, and a range of entrustability and supervision scales. We found only one study that used an empirical research approach for EPA assessment. One article reported on the earlier advancement of trainees from UME to graduate medical education based on summative entrustment decisions.

CONCLUSIONS

There is emerging evidence concerning how EPAs can be effectively introduced to clinical training in UME. Specialty-specific, nested EPAs with context-adapted, entrustment-supervision scales might be helpful in better leveraging their formative assessment potential.

Capacity Building for a New Multicenter Network Within the ECHO IDeA States Pediatric Clinical Trials Network

Introduction: Research capacity building is a critical component of professional development for pediatrician scientists, yet this process has been elusive in the literature. The ECHO IDeA States Pediatric Clinical Trials Network (ISPCTN) seeks to implement pediatric trials across medically underserved and rural populations. A key component of achieving this objective is building pediatric research capacity, including enhancement of infrastructure and faculty development. This article presents findings from a site assessment inventory completed during the initial year of the ISPCTN. Methods: An assessment inventory was developed for surveying ISPCTN sites. The inventory captured site-level activities designed to increase clinical trial research capacity for pediatrician scientists and team members. The inventory findings were utilized by the ISPCTN Data Coordinating and Operations Center to construct training modules covering 3 broad domains: Faculty/coordinator development; Infrastructure; Trials/Research concept development. Results: Key lessons learned reveal substantial participation in the training modules, the importance of an inventory to guide the development of trainings, and recognizing local barriers to clinical trials research. Conclusions: Research networks that seek to implement successfully completed trials need to build capacity across and within the sites engaged. Our findings indicate that building research capacity is a multi-faceted endeavor, but likely necessary for sustainability of a unique network addressing high impact pediatric health problems. The ISPCTN emphasis on building and enhancing site capacity, including pediatrician scientists and team members, is critical to successful trial implementation/completion and the production of findings that enhance the lives of children and families.

A Program to Prepare Clinical Pharmacy Faculty Members to Use Entrustable Professional Activities in Experiential Education

Objective. To describe the design and evaluation of a program implemented to ready clinical faculty members to use entrustable professional activities (EPAs) for teaching and assessment in experiential education. Methods. The school adopted a set of EPAs for faculty members to implement in advanced pharmacy practice experiences (APPEs), and then delivered a two-session faculty development program to ensure faculty members' readiness to implement the EPAs. To determine the success of the faculty development program, qualitative analysis of the moderated discussion held during the program was conducted, post-program and follow-up surveys were administered, and the results of the pilot implementation of EPAs were analyzed. Results. Eleven faculty members participated in the development program, and 10 of them completed the pilot implementation of EPAs and the follow-up survey after completing three APPE blocks. All faculty members responded that the program prepared them to apply what they learned about EPAs to their practice setting in both the post-program and follow-up surveys. In the follow-up survey, 80% of faculty members reported that they were confident in correctly applying what they learned from the program to the EPA pilot, and 100% answered a hypothetical application question correctly. Forty students participated in the pilot implementation of EPAs. Of these, 95% were directly observed by faculty members before making an entrustment decision, and 100% received feedback on their performance. Conclusion. The faculty development program was effective in preparing faculty members to implement the EPA framework in experiential teaching and use the entrustment rubric.

Redefining the Acting Internship in the Era of Entrustment: One Institution's Approach to Reforming the Acting Internship

Creating a Core Entrustable Professional Activities (Core EPA) curriculum requires a longitudinal approach. Current curricular efforts have focused primarily on the pre-clerkship and clerkship phases of training; however, the role of the Acting Internship (AI) has not been explored. The AI experience offers opportunities for students to have enhanced clinical responsibility, demonstrate proficiency, and allows for assessment of Core EPAs that are beyond the focus of clerkships. We share our experience developing an interdepartmental AI experience designed to assess designated Core EPAs and highlight tensions that should be considered when incorporating an AI experience into a longitudinal Core EPA-oriented curriculum.

Medical education today: all that glitters is not gold

BACKGROUND

The medical education system based on principles advocated by Flexner and Osler has produced generations of scientifically grounded and clinically skilled physicians whose collective experiences and contributions have served medicine and patients well. Yet sweeping changes launched around the turn of the millennium have constituted a revolution in medical education. In this article, a critique is presented of the new undergraduate medical education (UME) curricula in relationship to graduate medical education (GME) and clinical practice.

DISCUSSION

Medical education has changed and will continue to change in response to scientific advances and societal needs. However, enthusiasm for reform needs to be tempered by a more measured approach to avoid unintended consequences. Movement from novice to master in medicine cannot be rushed. An argument is made for a shoring up of biomedical science in revised curricula with the beneficiaries being nascent practitioners, developing physician-scientists --and the public.

CONCLUSION

Unless there is further modification, the new integrated curricula are at risk of produce graduates deficient in the characteristics that have set physicians apart from other healthcare professionals, namely high-level clinical expertise based on a deep grounding in biomedical science and understanding of the pathologic basis of disease. The challenges for education of the best possible physicians are great but the benefits to medicine and society are enormous.