Am I ready for it? Students' perceptions of meaningful feedback on entrustable professional activities

Narrative comments in internal medicine clerkship evaluations: room to grow

The use of narrative comments in medical education poses a unique challenge: comments are intended to provide formative feedback to learners while also being used for summative grades. Given student and internal medicine (IM) grading committee concerns about narrative comment quality, we offered an interactive IM Grand Rounds (GR) session aimed at improving comment quality. We undertook this study to determine the quality of comments submitted by faculty and post-graduate trainees on students' IM Clerkship clinical assessments, and to explore the potential impact of our IM-GR. Archived comments from clerkship cohorts prior to and immediately following IM-GR were reviewed. Clinical clerkship assessment comments include three sections: Medical Student Performance Assessment (MSPE), Areas of Strength, and Areas for Improvement. We adapted a previously published comment assessment tool and identified the performance domain(s) discussed, inclusion of specific examples of student performance, evidence that the comment was based on direct observations, and, when applicable, the inclusion of actionable recommendations. Scoring was based on the number of domains represented and whether an example within that domain was provided (maximum score = 10). Analysis included descriptive statistics, t-test, and Pearson correlation coefficients. We scored 697 comments. Overall, section ratings were MSPE 2.51 (SD 1.52, range 0-9), Areas of Strength 1.53 (SD 1.09, range 0-6), and Areas for Improvement 1.27 (SD 1.06, range 0-8). Significant differences were noted after Grand Rounds only in the MSPE mean scores. Within domains, trends toward increased use of specific examples in the post-GR narratives were noted. Assessment of both the breadth and depth of the included comments revealed low-quality narratives offered by our faculty and resident instructors. A focused session on best practices in writing narratives offered minimal change in the overall narrative quality, although we did notice a trend toward the inclusion of explanative examples.

Student Perceptions of Narrative Feedback in Entrustable Professional Activities

INTRODUCTION

Entrustable professional activity (EPA) observations can be used to develop a holistic picture of trainee competency in professional tasks. Narrative feedback is an essential component of EPAs, but there is a lack of published literature capturing undergraduate student perceptions.

METHODS

Students who completed Year 3 of the MD programme in 2022-2023 at one institution in Canada were invited to participate in a survey designed to elicit their perceptions of narrative feedback in EPAs. Survey methods included closed-ended questions (analysed descriptively) and open-ended questions (analysed through thematic analysis).

RESULTS

The response rate to the survey was 23%. Over 60% of students reported that narrative comments in EPAs were specific and aligned with EPA scores, and 86% reported that the narrative feedback was individualised at least some of the time. However, 57% reported that they never or rarely received actionable feedback for growth. Students demonstrated mixed feelings as to whether EPAs help support their clinical development. Some noted that they can help students identify gaps and reinforce positives. Others reported preferring verbal feedback to written feedback and that EPAs resulted in an administrative burden. Over 90% of students reported barriers to obtaining EPAs, and almost 90% expressed the need for changes to EPAs. A predominant theme from students was the desire for a reduction in EPA requirements.

CONCLUSIONS

Students perceived the narrative feedback to be individualized and specific but reported that the feedback rarely contained feedback for growth. Students identified several barriers to EPA completion and provided recommended changes.

Training academic staff for effective feedback in workplace-based assessment: a study in Bhutan

INTRODUCTION

The feedback plays a critical role in competency-based education in both undergraduate and Postgraduate medical education. The study explores the impact of a faculty development program on feedback practices of residents and faculty of ENT and ER medicine at Gyalpo University of Medical Sciences of Bhutan (KGUMSB).

METHODS

This mixed method study was conducted in two departments with 14 faculty members participating in the study. The questionnaire was used to obtain the perception of feedback before and after a Faculty Development Training (FDT) on good feedback practices. Student "t" test was used to compare the feedback perception at day 0 and 6 months and the responses were qualitatively analyzed using thematic analysis.

RESULTS

(a) Quantitative: The confidence of faculty to provide feedback improved significantly after FDT as compared to before FDT and it persisted in the same for 6 months (p-value 0.041 and p-value 0.027 respectively). The overall perception of feedback as a tool significantly changed positively after FDT and at 6 months (p-value p-value = 0.000). (b) Qualitative: Two thematic areas of process and teaching-learning were analyzed. Faculty showed improved and more focused feedback after training, but signs of decline by 6 months highlighted the need for refresher training. Feedback initially improved for residents, as it became more constructive and useful, though by 6 months, it showed potential for further refinement and consistency. 55 -CONCLUSIONS: The findings from this study are suggestive that feedback may have excellent potential as a tool for enhanced student learning in WPBA encounters.

A mixed-methods study on the course of professional identity formation in undergraduate medical students

BACKGROUND

This study explored the development of professional identity formation (PIF) in medical students during clinical rotations, focusing on the relationship between self-perceived development and actual development. It examined feedback and evaluations within the entrustable professional activities (EPA) framework, emphasizing the role of formative feedback in fostering professional growth. By analysing students' perceptions of their professional identity and feedback from online portfolios, this study aimed to enhance understanding of the trajectory of PIF.

METHODS

Conducted at the Radboud University, this study focused on medical students in the Master's programme. Sub-study 1 involved administering the extended Professional Self Identity Questionnaire (PSIQ) to 2,095 students from September 2020 to September 2022. Sub-study 2 analysed data from 240 students, integrating PSIQ responses with feedback from their online portfolios, including assessments and supervisor feedback. Data were analysed using SPSS (sub-study 1) and content analysis (sub-study 2) to explore the relationship between self-perceived PIF and actual performance.

RESULTS

A total of 1,519 students completed the extended PSIQ, with exploratory factor analysis confirming the questionnaire's unidimensional structure and improved reliability after adding three items. A one-way ANOVA showed a significant increase in PSIQ scores across different medical programme episodes. In sub-study 2, data of 240 students from three clerkships were analysed for the correlation between self-perceived PIF and actual performance, with no significant differences in their performance on national tests, assessments, or supervisor ratings. Portfolio analysis revealed that supervisors' feedback centred on collaboration, attitude, self-confidence, and growth toward becoming a physician, with students in later clerkships receiving more detailed feedback.

CONCLUSIONS

The extended PSIQ, which includes questions about EPAs and clerkship transitions, maintains reliability and construct validity, making it suitable for studying PIF within an EPA-based curriculum. Although no significant differences in performance were found, assertiveness emerged as a key factor in the quality and quantity of feedback, with more assertive students receiving better feedback. Narrative feedback, especially process-oriented, is crucial for supporting professional identity formation in medical students.

Integrating and Assessing Entrustable Professional Activities in Doctor of Pharmacy Curricula

OBJECTIVES

The 2022 Curriculum Outcomes and Entrustable Professional Activities include an updated set of entrustable professional activities (EPAs) for Doctor of Pharmacy graduates. To assist pharmacy educators with the integration and assessment of EPAs, this work aimed to review and synthesize approaches to integrate EPAs in health professions curricula, strategies for EPA-related assessments and entrustment in health professions curricula, and challenges and research needs for integrating EPAs in assessment frameworks in Doctor of Pharmacy curricula.

FINDINGS

A literature review resulted in 114 articles identified as having relevance with 33 articles specific to pharmacy education. There are multiple components of an effective EPA integration and assessment plan, including administrative leadership, determination of programmatic EPAs with comprehensive descriptions, selection of an entrustment-supervision scale, identification of EPA-based assessments, EPA mapping, data management, and training and development.

SUMMARY

Strategies with applicability for integrating and assessing EPAs in pharmacy education were analyzed and synthesized. Evidence-informed and expert recommendations for EPA integration and assessment are discussed. Programs can apply these recommendations to implement and assess EPAs in alignment with their own institutional and curricular contexts.

Implementation and Evaluation of Entrustable Professional Activities for a Pharmacy Intern Training Program in Australia

OBJECTIVE

This study aimed to describe the implementation and evaluation of entrustable professional activities (EPAs) for a pharmacy intern training program in Australia.

METHODS

Surveys and focus groups were used to evaluate the perceptions and experiences of provisionally registered pharmacists and pharmacists using the EPAs as an assessment tool in the workplace. Thematic analysis was used to analyze the focus group data. Surveys compared the frequency and median responses to the Likert-scale items. Entrustment levels at 6, 13, 26, and 44 weeks were reported for each EPA.

RESULTS

Pharmacist (n = 6) and provisionally registered pharmacist (n = 8) focus groups identified the following themes: use of EPAs in the workplace, benefits and challenges of using EPAs, and future direction and recommendations for using EPAs in the workplace. Feasibility measures in the pharmacist survey suggested that pharmacists were satisfied with using EPAs in the workplace. The provisionally registered pharmacist survey showed that they valued feedback provided by pharmacists and that they engaged with the EPAs in the workplace. Provisionally registered pharmacists enrolled in the study (n = 40) showed progression in entrustment levels at each reporting period.

CONCLUSION

Pharmacists and provisionally registered pharmacists reported that EPAs were feasible and could be used in the workplace as an educative tool. However, they highlighted education as an important factor for successful implementation. At each milestone, provisionally registered pharmacists showed progression in entrustment level.

Evaluating a medication history-taking entrustable professional activity and its assessment tool - Survey of a statewide public hospital pharmacy service

INTRODUCTION

Entrustable Professional Activities (EPAs) are tasks that professionals within a field perform autonomously. EPAs are incorporated in workplace-based assessment tools to assist training and professional development. Few studies have evaluated medication history-taking EPAs use in pharmacy practice and none have sought stakeholder feedback on their use. This study evaluates the quality of the medication history-taking EPA utilized in South Australian public hospitals and the usability of its assessment tool.

METHODS

A voluntary online questionnaire was conducted from July 15th to September 2nd 2021 to gather the opinions of stakeholders on the use of the medication history-taking EPA. The questionnaire was developed based on tools identified in the literature and utilized 14 open-text and five-point Likert scale questions. The questionnaire was distributed using Survey Monkey® to a purposive sample of staff and students.

RESULTS

82 responses were received from 218 surveys distributed, yielding a response rate of 38%. Respondents believed the EPA promotes learner development (90.6%) and the provision of useful feedback (83%). 94.3% considered the EPA to be easy to use but only 56.6% indicated that using it fits easily within their workday. Time constraints and the presence of context-specific descriptors were commonly perceived as limitations. Some stakeholders indicated a lack of understanding of entrustment decisions.

CONCLUSION

The EPA and its assessment tool were perceived to have good quality and usability. Reducing the length of the tool, broadening its applicability across contexts, and improving user understanding of entrustment decision-making may support better use of the tool.

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

BACKGROUND

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

METHODS

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

RESULTS

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

CONCLUSIONS

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Cross-Sectional Survey of Student and Faculty Experiences with Feedback and Assessment of Clinical Competency on Final Year Rotations at American Association of Veterinary Medical Colleges (AAVMC) Member Institutions

The clinical experiences, feedback, and assessment that veterinary students receive during final year rotations have a significant impact on whether they will achieve entry-level competency at the time of graduation. In this study, a cross-sectional survey was administered to American Association of Veterinary Medical Colleges (AAVMC) member institutions to collect baseline data about current feedback and assessment practices to identify key target areas for future research and educational interventions. Responses were received from 89 faculty and 155 students distributed across 25 universities. The results indicated that there are significant gaps between evidence-based approaches for delivering feedback and assessment and what was being implemented in practice. Most feedback was provided to students in the form of end-of-rotation evaluations sometimes several weeks to months after the rotation finished when faculty were unlikely to remember specific interactions with students and there were limited opportunities for students to demonstrate progress towards addressing identified concerns. Although the most valuable type of feedback identified by students was verbal feedback delivered shortly after clinical experiences, this method was often not used due to factors such as lack of time, poor learning environments within veterinary teaching hospitals, and lack of faculty training in delivering effective feedback. The results also indicated potential challenges with how non-technical domains of competence within the AAVMC Competency-Based Veterinary Education (CBVE) Framework are currently evaluated. Finding avenues to improve feedback and assessment processes in final year clinical settings is essential to ensure that veterinary students are adequately prepared for practice.

National pilot of entrustable professional activities in pathology residency training

Entrustable professional activities (EPAs) are observable clinical skills and/or procedures that have been introduced into medical education at the student and resident levels in most specialties to determine readiness to advance into residency or independent practice, respectively. This publication describes the process and outcomes of a pilot study looking at the feasibility of using two anatomic pathology and two clinical pathology EPAs in pathology residency in 6 pathology residency programs that volunteered for the study. Faculty development on EPAs and their assessment was provided to pilot program faculty, and EPA assessment tools were developed and used by the pilot programs. Pre- and post-study surveys were given to participating residents, faculty, and program directors to gauge baseline practices and to gather feedback on the EPA implementation experience. Results demonstrated overall good feasibility in implementing EPAs. Faculty acceptance of EPAs varied and was less than that of program directors. Residents reported a significant increase in the frequency with which faculty provided formative assessments that included specific examples of performance and specific ways to improve, as well as increased frequency with which faculty provided summative assessments that included specific ways to improve. EPAs offered the most benefit in setting clear expectations for performance of each task, for providing more specific feedback to residents, and in increasing Program director's understanding of resident strengths abilities and weaknesses.

Undergraduate Learners' Receptiveness to Feedback in Medical Schools: A Scoping Review

Feedback from educators to learners is considered an important element of effective learning in medical school. While early studies were focused on the processes of providing feedback, recent work has showed that factors related to how learners receive feedback seems to be equally important. Considering that the literature on this topic is new in medical education, and studies are diverse and methodologically variable, we sought to conduct a scoping review to map the articles on receptiveness to feedback, to provide an overview of its related factors, to identify the types of research conducted in this area, and to document knowledge gaps in the existing literature. Using the Joanna Briggs Institute scoping review methodology, we searched four databases (CINAHL, Ovid, PubMed, and Web of Science) and screened 9120 abstracts, resulting in 98 articles for our final analysis. In this sample, 80% of studies on the feedback receiver were published in the last 10 years, and there is a vast variation in the studies' methodologies. The main factors that affect medical students' receptiveness to feedback are students' characteristics, feedback content, educators' credibility, and the learning environment. Feedback literacy is a very recent and rarely used term in medical education; therefore, an important area for further investigation. Lastly, we identified some gaps in the literature that might guide future research, such as studying receptiveness to feedback based on academic seniority and feedback literacy's long-term impacts on learning.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01858-0.

Effectiveness of a proficiency-based progression e-learning approach to training in communication in the context of clinically deteriorating patients: a multi-arm randomised controlled trial

OBJECTIVE

To determine the effectiveness of proficiency-based progression (PBP) e-learning in training in communication concerning clinically deteriorating patients.

DESIGN

Single-centre multi-arm randomised double-blind controlled trial with three parallel arms.

RANDOMISATION, SETTING AND PARTICIPANTS

A computer-generated program randomised and allocated 120 final year medical students in an Irish University into three trial groups.

INTERVENTION

Each group completed the standard Identification, Situation, Background, Assessment, Recommendation communication e-learning; group 1 Heath Service Executive course group (HSE) performed this alone; group 2 (PBP) performed additional e-learning using PBP scenarios with expert-determined proficiency benchmarks composed of weighted marking schemes of steps, errors and critical errors cut-offs; group 3 (S) (self-directed, no PBP) performed additional e-learning with identical scenarios to (PBP) without PBP.

MAIN OUTCOME MEASURES

Primary analysis was based on 114 students, comparing ability to reach expert-determined predefined proficiency benchmark in standardised low-fidelity simulation assessment, before and after completion of each group's e-learning requirements. Performance was recorded and scored by two independent blinded assessors.

RESULTS

Post-intervention, proficiency in each group in the low-fidelity simulation environment improved with statistically significant difference in proficiency between groups (p<0.001). Proficiency was highest in (PBP) (81.1%, 30/37). Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and self-directed (S) (p<0.001) and (HSE) (p<0.001). No statistically significant difference existed between (S) and (HSE) (p=0.479). Changes in proficiency from pre-intervention to post-intervention were significantly different between the three groups (p=0.001). Post-intervention, an extra 67.6% (25/37) in (PBP) achieved proficiency in the low-fidelity simulation. Post hoc pairwise comparisons revealed statistically significant differences between (PBP) and both (S) (p=0.020) and (HSE) (p<0.001). No statistically significant difference was found between (S) and (HSE) (p=0.156).

CONCLUSIONS

PBP e-learning is a more effective way to train in communication concerning clinically deteriorating patients than standard e-learning or e-learning without PBP.

TRIAL REGISTRATION NUMBER

NCT02937597.

"Maybe it's the first time somebody's been honest with you": exploring how residents reconcile feedback variability

BACKGROUND

Supervisors in postgraduate medical education may deliver different feedback for the same quality of performance. Residents may struggle to make sense of inconsistent and sometimes contradictory information. We sought to explore how residents experience feedback from different supervisors, how they process inconsistent information, and what factors influence their experiences.

METHODS

Eighteen residents participated in semi-structured interviews to explore their perspectives on feedback. Using a constructivist grounded theory approach, we engaged in iterative cycles of data collection and analysis, sampling until theoretical sufficiency was reached. Constant comparative analysis was used to identify and define themes.

RESULTS

We identified a central theme of reconciliation, which we defined as the act of processing inconsistent feedback and determining how to engage with it. This reconciliation was informed by the credibility of, and residents' relationship with, supervisors and was achieved through conversations with peers and mentors, observation of other supervisors' behavior toward their performance, and reflection on their own performance. Participants expressed a reluctance to discard feedback, even if they felt it was incongruent with previous feedback or their own self-concept and self-assessment.

CONCLUSION

The findings of this study show that while residents are regular consumers of feedback, not all feedback is used equally. Residents actively reconcile sometimes-contradictory feedback and must work to balance a general reluctance to discard feedback, while developing an understanding of its credibility. This work reinforces the importance of pedagogical relationships and identifies that facilitated reflection that explicitly acknowledges feedback inconsistencies may be important in the reconciliation process.

Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study

Background

Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).

Methods

We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.

Results

A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.

Conclusions

The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.

Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

PURPOSE

Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.

METHOD

This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.

RESULTS

Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.

CONCLUSIONS

Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.

An Innovative Undergraduate Medical Curriculum Using Entrustable Professional Activities

The need to educate medical professionals in changing medical organizations has led to a revision of the Radboudumc's undergraduate medical curriculum. Entrustable professional activities (EPAs) were used as a learning tool to support participation and encourage feedback-seeking behavior, in order to offer students the best opportunities for growth. This paper describes the development of the Radboudumc's EPA-based Master's curriculum and how EPAs can facilitate continuity in learning in the clerkships. Four guiding principles were used to create a curriculum that offers possibilities for the students' development: (1) working with EPAs, (2) establishing entrustment, (3) providing continuity in learning, and (4) organizing smooth transitions. The new curriculum was designed with the implementation of EPAs and an e-portfolio, based on these 4 principles. The authors found that the revised curriculum corresponds to daily practice in clerkships. Students used their e-portfolios throughout all clerkships, which stimulates feedback-seeking behavior. Moreover, EPAs promote continuity in learning while rotating clerkships every 1 to 2 months. This might encourage curriculum developers to use EPAs when aiming for greater continuity in the development of students. Future research needs to focus on the effect of EPAs on transitions across clerkships in order to further improve the undergraduate medical curriculum.

'Next steps are…': An exploration of coaching and feedback language in EPA assessment comments

PURPOSE

Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness.

METHODS

We collected all written comments from EPA assessments of communication from a first-year IM resident cohort at the University of Toronto. Sensitized by politeness theory, we analyzed data using principles of constructivist grounded theory.

RESULTS

Nearly all EPA assessments (94%) contained written feedback based on focused clinical encounters. The majority of comments demonstrated coaching language, including phrases like 'don't forget to,' and 'next steps are,' followed by specific suggestions for improvement. A variety of words, including 'autonomy' and 'independence' denoted entrustment decisions. Linguistic politeness strategies such as hedging were pervasive, seemingly to minimize harm to the supervisor-trainee relationship.

CONCLUSION

Evidence of written coaching feedback suggests that EPA assessment comments are being used as intended as a means of formative feedback to promote learning. Yet, the frequent use of polite language suggests that EPAs may be higher-stakes than expected, highlighting a need for changes to the assessment culture and improved feedback literacy.

Student Interprofessional Facilitator Training (SIFT) program: building capacity in clinical education leadership

BACKGROUND

While there are many teacher training programs for health professional students, few are interprofessional, and few integrate assessment and feedback prior to participation as peer teachers. In 2021, The Student Interprofessional Facilitator Training (SIFT) program was developed to allow senior students, already trained in peer teaching, to revise, build on, and practice their newly acquired skills in an interprofessional context. The aim of this study was to explore participant perception and performance, and the contextual factors that influence student aspirations as clinical teachers.

METHODS

Alumni of the 2021 Peer Teacher Training program (n = 74) were invited to participate in the SIFT program. Those who participated were invited to attend individual semi-structured interviews. Thematic analysis was used to code and categorise data into themes, using Communities of Practice as a conceptual framework. Skills in interprofessional facilitation were observed, assessed and students were provided with individual feedback. Assessment data were analysed using descriptive statistics.

RESULTS

Sixteen students from six disciplines joined the SIFT program, and 13/16 (81%) completed. Students were from medicine, nursing, diagnostic radiography, medical imaging, dentistry and speech pathology. Students reported an increased recognition of teaching as a learned skill, development of clinician identity formation as educators, development of interprofessional communication skills, increased awareness of the roles of other health professions, and an increased understanding of leadership. Participants expressed a desire for additional opportunities for interprofessional networking and peer teaching. A good level of competence in facilitation skills was reached by participants.

CONCLUSION

The SIFT program provided a sustainable framework for health professional students to develop and evidence their teaching and leadership skills in an interprofessional context. This study highlighted the important role of observation, assessment and feedback in student teacher training programs. The process of clear assessment guidelines, direct observation with feedback from supervisors provided a way to ensure quality improvement in peer teaching. The SIFT program will help to build capacity of interprofessional programs where large numbers of teachers are required for small group teaching. The next step will be to ensure a variety of opportunities within interprofessional contexts, and with face-to-face engagement.

The Structured Oral Examination: A Method to Improve Formative Assessment of Fellows in Pediatric Endocrinology

OBJECTIVE

A structured oral exam (SOE) can be utilized as a formative assessment to provide high-quality formative feedback to trainees, but has not been adequately studied in graduate medical education. We obtained fellow and faculty perspectives on: 1) educational effectiveness, 2) feasibility/acceptability, and 3) time/cost of a SOE for formative feedback.

METHODS

Four pediatric endocrinology cases were developed and peer-reviewed to generate a SOE. The exam was administered by faculty to pediatric endocrinology fellows individually, with feedback after each case. Fellow/faculty perspectives of the SOE were obtained through a questionnaire. Qualitative thematic analysis was utilized to analyze written comments generated by faculty and fellows.

RESULTS

Seven of 10 pediatric endocrinology fellowship programs and all 18 fellows within those programs agreed to participate. Thematic analysis of fellow and faculty comments resulted in 5 perceived advantages of the SOE: 1) improved identification of clinically relevant knowledge deficits, 2) improved assessment of clinical reasoning, 3) immediate feedback/teaching, 4) assurance of adequate teaching/assessment of uncommon cases, and 5) more clinically relevant assessment. Mean time to administer one case was 15.8 minutes (2.0) and was mentioned as a potential barrier to implementation. Almost all fellows (17/18, 94%) and faculty (6/7, 86%) would recommend or would most likely recommend implementation of the SOE into their curriculum.

CONCLUSIONS

The SOE utilized for formative feedback was perceived by fellows and faculty to have several educational advantages over current assessments and high acceptability. Objective educational advantages should be assessed on future studies of the SOE.

Implementation of Entrustable Professional Activities assessments in a Canadian obstetrics and gynecology residency program: a mixed methods study

BACKGROUND

Since the implementation of competency-based medical education (CBME) across residency training programs in Canada, there has been limited research understanding how entrustable professional activity (EPA) assessments are used by faculty supervisors and residents.

OBJECTIVE

This study examines how EPA assessments are used in an Obstetrics and Gynecology residency program and the impact of implementation on both groups.

METHODS

A mixed methods study design was used. Part one involved the aggregation of descriptive data of EPA assessment completion for postgraduate year 1 and 2 residents from July 2019 to May 2020. Part two involved a thematic analysis of semi-structured interviews of residents and faculty.

RESULTS

There was significant uptake of EPA assessments across community and teaching hospitals with widespread contribution of assessment data from faculty. However, both residents and faculty reported that the intended design of EPA assessments as low-stakes assessments to provide formative feedback is not how EPA assessments are experienced. Residents and faculty noted the increased level of administrative burden and related perceived stress amongst the resident group.

CONCLUSIONS

The implementation of EPA assessments is feasible across a variety of sites. However, previous measurement challenges remain. Neither residents nor faculty perceive the value of EPAs to improve feedback, despite their intended nature.

Teacher Feedback and Student Learning-The Students' Perspective

INTRODUCTION

Feedback from teachers to students plays an important role in informing students about the outcome of their assessments. It contributes to students' ongoing learning. The aim of this study was to investigate dental students' perceptions of the feedback given to them by their teachers in Europe.

MATERIALS & METHODS

An online questionnaire was completed by dental students throughout Europe in this quantitative study. Data were collected via Google Forms, transferred to an excel spreadsheet and analysed using SPSS software Version 24.

RESULTS

234 students studying in 9 different European countries completed the questionnaire. These students were born in 36 different countries within and beyond Europe. 84% (n=197) were undergraduate students. 20.3% (n=48) students reported receiving feedback following summative assessments. 81.2% (n=190) students reported constructive criticism as their preferred mode of receiving feedback. 11.3% (n=26) students did not know who delivered the feedback to them. 71% (n=166) students felt that the feedback they received had a significant impact on their future learning.

CONCLUSION

It would appear that there is some diversity in dental students' perceptions of: i) who delivers feedback, ii) when feedback is given, iii) the consistency of feedback received, and iv) the style of feedback they preferred compared to that delivered by tutors. Feedback is being provided to dental students in an appropriate and helpful manner, although there is still room for improvement. Students were aware of the significance of feedback and its impact on future learning.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Bridging the Gap: Using Consensus to Explore Entrustment Decisions and Feedback Receptivity in Competency-Based Emergency Medicine Residency Programs Through the Construction of a Q-Sample Incorporating a Delphi Technique

Background

Recent changes in medical education calls for a shift toward student-centered learning. Therefore, it is imperative that clinical educators transparently assess the work-readiness of their medical residents through entrustment-based supervision decisions toward independent practice. Similarly, it is critical that medical residents are vocal about the quality of supervision and feedback they receive. This study aimed to explore the factors that influence entrustment-based supervision decisions and feedback receptivity by establishing a general consensus among Taiwanese clinical educators and medical residents regarding entrustment decisions and feedback uptake, respectively.

Methods

In Q-methodology studies, a set of opinion statement (i.e., the Q-sample) is generated to represent the phenomenon of interest. To explore the factors that influence entrustment-based supervision decisions and feedback receptivity, a Q-sample was developed using a four-step approach: (1) literature search using electronic databases, such as PubMed and Google Scholar, and interviews with emergency clinical educators and medical residents to generate opinion statements, (2) thematic analysis and grouping using The Model of Trust, the Ready, Wiling, and Able model, and the theory of self-regulated learning, (3) translation, and (4) application of a Delphi technique, including two expert panels comprised of clinical educators and medical residents, to establish a consensus of the statements and validation for a subsequent Q-study.

Results

A total of 585 and 1,039 statements from the literature search and interviews were extracted to populate the sample of statements (i.e., the concourse) regarding entrustment-based supervision decisions for clinical educators and feedback receptivity emergency medicine residents, respectively. Two expert panels were invited to participate in a Delphi Technique, comprised of 11 clinical educators and 13 medical residents. After two-rounds of a Delphi technique, the panel of clinical educators agreed on 54 statements on factors that influence entrustment-based supervision decisions and were categorized into five themes defined by the Model of Trust. Similarly, a total of 60 statements on the factors that influence feedback receptivity were retained by the panel of medical residents and were categorized into five themes defined by the Ready, Willing, and Able model and the theory of self-regulated learning.

Conclusion

Though not exhaustive, the key factors agreed upon by clinical educators and medical residents reflect the characteristics of entrustment-based supervision decisions and feedback receptivity across specialties. This study provides insight on an often overlooked issue of the paths to teaching and learning in competency-based residency training programs. Additionally, incorporation of the Delphi technique further adds to the existing literature and puts emphasis as an important tool that can be used in medical education to rigorously validate Q-statements and develop Q-samples in various specialties.

A Pilot Study of the Generalizability of Preclinical Entrustment Assessments in Undergraduate Medical Education

PURPOSE

The reproducibility and consistency of assessments of entrustable professional activities (EPAs) in undergraduate medical education (UME) have been identified as potential areas of concern. EPAs were designed to facilitate workplace-based assessments by faculty with a shared mental model of a task who could observe a trainee complete the task multiple times. In UME, trainees are frequently assessed outside the workplace by faculty who only observe a task once.

METHOD

In November 2019, the authors conducted a generalizability study (G-study) to examine the impact of student, faculty, case, and faculty familiarity with the student on the reliability of 162 entrustment assessments completed in a preclerkship environment. Three faculty were recruited to evaluate 18 students completing 3 standardized patient (SP) cases. Faculty familiarity with each student was determined. Decision studies were also completed. Secondary analysis of the relationship between student performance and entrustment (scoring inference) compared average SP checklist scores and entrustment scores.

RESULTS

G-study analysis revealed that entrustment assessments struggled to achieve moderate reliability. The student accounted for 30.1% of the variance in entrustment scores with minimal influence from faculty and case, while the relationship between student and faculty accounted for 26.1% of the variance. G-study analysis also revealed a difference in generalizability between assessments by unfamiliar (φ = 0.75) and familiar (φ = 0.27) faculty. Subanalyses showed that entrustment assessments by familiar faculty were moderately correlated to average SP checklist scores (r = 0.44, P < .001), while those by unfamiliar faculty were weakly correlated (r = 0.16, P = .13).

CONCLUSIONS

While faculty and case had a limited impact on the generalizability of entrustment assessments made outside the workplace in UME, faculty who were familiar with a student's ability had a notable impact on generalizability and potentially on the scoring validity of entrustment assessments, which warrants further study.

An Entrustable Professional Activities Model for Assessment of Undergraduate Competence in Anesthesia and Surgery: Performance of the Assessment Scheme and the Impact of Assessment Timing and Variation in Structure of Teaching Activities on Student Outcomes

An entrustable professional activity (EPA) model was used to assess the anesthesia and surgery competence of year 4 students during elective neutering procedures over 3 academic years (cohort A, cohort B, and cohort C). Two competence thresholds were defined by an expert panel, the minimum acceptable standard (MAS) and the standard expected at the start of final-year rotations (SFR). The assessment scheme performed as expected, and the median level of supervision achieved by students either matched or exceeded the SFR for all EPAs except one, which matched the MAS. Semester of assessment was associated with student performance, with more students in semester 2 achieving the SFR. In the EPAs assessing pain management, documentation, and patient discharge, cohort A was associated with reduced student performance; this could be explained by changes in the delivery of teaching that enhanced performance in subsequent cohorts (academic years). For all EPAs combined and for EPAs 3, 5, 6, 8, and 9, student performance at the SFR was associated with academic year. For all EPAs combined and EPAs 3, 8, and 9, there was a reduction in the proportion of students achieving the SFR threshold in each successive year. At the MAS, the only association for all EPAs combined was with cohort C. This progressive reduction in performance may have been related to the negative effect of decreased time spent at the neutering clinic and loss of feedback opportunities outweighing the positive effects of increased staff:student ratio and improvements in the preparative phases.

Factors affecting perceived credibility of assessment in medical education: A scoping review

Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner's perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners' perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners' level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.

How an EPA-based curriculum supports professional identity formation

BACKGROUND

Entrustable professional activities (EPAs) are widely used in medical education, and they might be an important incentive to stimulate professional identity formation (PIF) of medical students, by actively encouraging participation in the workplace. The goal of this study was to explore the effects of an EPA-based curriculum on the PIF of medical students in undergraduate curricula.

METHODS

In this study at the Radboud University Medical Center in Nijmegen, the Netherlands, the authors interviewed twenty-one medical students in three focus group interviews (November 2019), and conducted a thematic analysis based on both the synthesizing concepts PIF, communities of practice and EPAs, and newly defined themes.

RESULTS

Four central themes proved crucial for understanding the influence of EPAs on PIF: creating learning opportunities, managing feedback, dealing with supervision in context and developing confidence. EPAs helped students to create learning opportunities and to choose activities purposefully, and the use of EPAs stimulated their feedback-seeking behavior. The context and way of supervision had a great impact on their development, where some contexts offer better learning opportunities than others. EPAs helped them develop trust and self-confidence, but trust from supervisors hardly appears to result from using EPAs.

CONCLUSIONS

An EPA-based curriculum does stimulate PIF in the complex context of working and learning by supporting participation in the workplace and by encouraging feedback-seeking behavior. Striking the right balance between participation, feedback-seeking behavior and choosing learning activities is essential.

TRIAL REGISTRATION

This study was approved by the ethics committee of the Netherlands Association of Medical Education (NVMO, case number 2019.5.12).

How supervisor trust affects early residents' learning and patient care: A qualitative study

INTRODUCTION

Trust between supervisors and trainees mediates trainee participation and learning. A resident (postgraduate) trainee's understanding of their supervisor's trust can affect their perceptions of their patient care responsibilities, opportunities for learning, and overall growth as physicians. While the supervisor perspective of trust has been well studied, less is known about how resident trainees recognize supervisor trust and how it affects them.

METHODS

In this qualitative study, 21 pediatric residents were interviewed at a single institution. Questions addressed their experiences during their first post-graduate year (PGY-1) on inpatient wards. Each interviewee was asked to describe three different patient care scenarios in which they perceived optimal, under-, and over-trust from their resident supervisor. Data were analyzed using thematic analysis.

RESULTS

Residents recognized and interpreted their supervisor's trust through four factors: supervisor, task, relationship, and context. Optimal trust was associated with supervision balancing supervisor availability and resident independence, tasks affording participation in decision-making, trusting relationships with supervisors, and a workplace fostering appropriate autonomy and team inclusivity. The effects of supervisor trust on residents fell into three themes: learning experiences, attitudes and self-confidence, and identities and roles. Optimal trust supported learning via tailored guidance, confidence and lessened vulnerability, and a sense of patient ownership and team belonging.

DISCUSSION

Understanding how trainees recognize supervisor trust can enhance interventions for improving the dialogue of trust between supervisors and trainees. It is important for supervisors to be cognizant of their trainees' interpretations of trust because it affects how trainees understand their patient care roles, perceive autonomy, and approach learning.

Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study

BACKGROUND

The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment.

OBJECTIVE

In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform.

METHODS

Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis.

RESULTS

All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface.

CONCLUSIONS

This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

Medical Students' Perspectives on Entrustment Decision Making in an Entrustable Professional Activity Assessment Framework: A Secondary Data Analysis

PURPOSE

Research on how entrustment decisions are made identifies 5 influential factors (supervisor, trainee, supervisor-trainee relationship, context, task). However, this literature primarily represents the perspective of supervisors in graduate medical education and is conducted outside of an assessment framework where entrustment decisions have consequences for trainees and for patients. To complement the literature, the authors explored how medical students in a pilot program that used an entrustable professional activity (EPA) assessment framework perceived factors influencing entrustment decisions.

METHOD

The authors conducted a secondary analysis of qualitative data from program evaluation of a pilot project using an EPA assessment framework to advance students through their curriculum and into residency. Data were derived from 8 focus groups conducted over 4 years (2015-2018) with 27 students who shared their experience of learning and supervision in the project. Sensitized by the entrustment literature, the authors coded and analyzed focus group transcripts according to principles of thematic analysis.

RESULTS

Components of the trainee and supervisor-trainee relationship factors predominated students' perceptions of entrustment decisions. Students directed their own learning by asking for feedback, seeking opportunities to engage in learning, sharing limitations of their knowledge with supervisors, and pushing supervisors to recalibrate assessments when appropriate. Students' agentic actions were facilitated by longitudinal supervisor-trainee relationships wherein they felt comfortable asking for help and built confidence in patient care. Students mentioned components of other factors that influenced entrustment decisions (supervisor, clinical task, clinical context), but did so less frequently and from a nonagentic vantage point.

CONCLUSIONS

Students' perspectives on entrustment decisions can be derived from their views on learning and supervision in an EPA assessment framework. Their perspectives complement the literature by highlighting students' agentic actions to influence entrustment decisions and promotion of agentic action through practices incorporating longitudinal supervisor-trainee relationships.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

BACKGROUND

Entrustable professional activities (EPAs) in competency-based, undergraduate medical education (UME) have led to new formative workplace-based assessments (WBA) using entrustment-supervision scales in clerkships. We conducted an observational, prospective cohort study to explore the usefulness of a WBA designed to assess core EPAs in a psychiatry clerkship.

METHODS

We analyzed changes in self-entrustment ratings of students and the supervisors' ratings per EPA. Timing and frequencies of learner-initiated WBAs based on a prospective entrustment-supervision scale and resultant narrative feedback were analyzed quantitatively and qualitatively. Predictors for indirect supervision levels were explored via regression analysis, and narrative feedback was coded using thematic content analysis. Students evaluated the WBA after each clerkship rotation.

RESULTS

EPA 1 ("Take a patient's history"), EPA 2 ("Assess physical & mental status") and EPA 8 ("Document & present a clinical encounter") were most frequently used for learner-initiated WBAs throughout the clerkship rotations in a sample of 83 students. Clinical residents signed off on the majority of the WBAs (71%). EPAs 1, 2, and 8 showed the largest increases in self-entrustment and received most of the indirect supervision level ratings. We found a moderate, positive correlation between self-entrusted supervision levels at the end of the clerkship and the number of documented entrustment-supervision ratings per EPA (p < 0.0001). The number of entrustment ratings explained 6.5% of the variance in the supervisors' ratings for EPA 1. Narrative feedback was documented for 79% (n = 214) of the WBAs. Most narratives addressed the Medical Expert role (77%, n = 208) and used reinforcement (59%, n = 161) as a feedback strategy. Students perceived the feedback as beneficial.

CONCLUSIONS

Using formative WBAs with an entrustment-supervision scale and prompts for written feedback facilitated targeted, high-quality feedback and effectively supported students' development toward self-entrusted, indirect supervision levels.

Feedback Credibility in Healthcare Education: a Systematic Review and Synthesis

PURPOSE

The purpose of this study was to systematically review and synthesize factors that influence learners' perceptions of credibility when feedback is provided by an authority figure in a healthcare environment.

METHODS

This study reviewed literature from medicine, psychology, and education using systematic review and qualitative synthesis methods. In a multi-step process, major electronic bibliographic databases were searched for relevant studies until October 2020.

RESULTS

The search identified 9216 articles. A total of 134 abstracts underwent full-text review. Of these, 22 articles met inclusion criteria. The studies were heterogenous and the majority utilized a qualitative design with interviews and focus groups. A few studies employed mixed methodology (n = 2) and two studies used a quantitative design. Four main themes were identified: feedback characteristics, context of feedback, source credibility, and recipient characteristics.

CONCLUSION

As programs implement major educational change initiatives to create more formative assessment practices, feedback will become even more crucial. The four main themes identified are important factors that contribute to the perception of feedback credibility. While the factors are described independently, they may be viewed as interrelated and the association between these factors and feedback may be driven more by learning culture than each characteristic.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-020-01167-w.

Working with entrustable professional activities in clinical education in undergraduate medical education: a scoping review

BACKGROUND

Entrustable professional activities (EPAs) are increasingly used in undergraduate medical education (UME). We conducted a scoping review to summarize the evidence for the use of EPAs in clinical rotations in UME.

METHODS

We searched multiple databases for scoping reviews based on the PRISMA guidelines for articles reporting qualitative and quantitative research, as well as conceptual and curriculum development reports, on EPAs in UME clinical rotations.

RESULTS

We identified 3309 records by searching through multiple databases. After the removal of duplicates, 1858 reports were screened. A total of 36 articles were used for data extraction. Of these, 47% reported on EPA and EPA-based curriculum development for clerkships, 50% reported on implementation strategies, and 53% reported on assessment methods and tools used in clerkships. Validity frameworks for developing EPAs in the context of clerkships were inconsistent. Several specialties reported feasible implementation strategies for EPA-based clerkship curricula, however, these required additional faculty time and resources. Limited exposure to clinical activities was identified as a barrier to relevant learning experiences. Educators used nationally defined, or specialty-specific EPAs, and a range of entrustability and supervision scales. We found only one study that used an empirical research approach for EPA assessment. One article reported on the earlier advancement of trainees from UME to graduate medical education based on summative entrustment decisions.

CONCLUSIONS

There is emerging evidence concerning how EPAs can be effectively introduced to clinical training in UME. Specialty-specific, nested EPAs with context-adapted, entrustment-supervision scales might be helpful in better leveraging their formative assessment potential.

Medical Students' Views on Implementing the Core EPAs: Recommendations From Student Leaders at the Core EPAs Pilot Institutions

In 2014, the Association of American Medical Colleges recruited 10 institutions across the United States to pilot the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). The goal was to establish a competency-based framework to prepare graduating medical students for the transition to residency. Within the Core EPAs pilot, medical students play an influential role in the development and implementation of EPA-related curricula. Student engagement was a priority for the Core EPAs institutions given students' roles as the end users of the curriculum, thus they may offer valuable insight into its design and implementation. Here, the authors provide the perspective of medical students who serve as leaders in the Core EPAs pilot at their respective institutions. They describe student leadership models across the pilot institutions as well as 6 key challenges to implementation of the Core EPAs: (1) How and when should the Core EPAs be introduced? (2) Who is responsible for driving the assessment process? (3) What feedback mechanisms are required? (4) What systems are required for advising, mentoring, or coaching students? (5) Should EPA performance contribute to students' grades? and (6) Should entrustment decisions be tied to graduation requirements? Using a polarity management framework to address each challenge, the authors describe inherent tensions, approaches used by the Core EPAs pilot institutions, and student-centered recommendations for resolving each tension. By sharing the experiences and perspectives of students engaged in the Core EPAs pilot, the authors hope to inform implementation of EPA-oriented assessment practices and feedback across institutions in the United States.

Building a Shared Mental Model of Competence Across the Continuum: Trainee Perceptions of Subinternships for Residency Preparation

INTRODUCTION

Toward a vision of competency-based medical education (CBME) spanning the undergraduate to graduate medical education (GME) continuum, University of Minnesota Medical School (UMMS) developed the Subinternship in Critical Care (SICC) offered across specialties and sites. Explicit course objectives and assessments focus on internship preparedness, emphasizing direct observation of handovers (Core Entrustable Professional Activity, "EPA," 8) and cross-cover duties (EPA 10).

METHODS

To evaluate students' perceptions of the SICC's and other clerkships' effectiveness toward internship preparedness, all 2016 and 2017 UMMS graduates in GME training (n = 440) were surveyed regarding skill development and assessment among Core EPAs 1, 4, 6, 8, 9, 10. Analysis included descriptive statistics plus chi-squared and Kappa agreement tests.

RESULTS

Respondents (n = 147, response rate 33%) rated the SICC as a rotation during which they gained most competence among EPAs both more (#4, 57% rated important; #8, 75%; #10, 70%) and less explicit (#6, 53%; #9, 69%) per rotation objectives. Assessments of EPA 8 (80% rated important) and 10 (76%) were frequently perceived as important toward residency preparedness. Agreement between importance of EPA development and assessment was moderate (Kappa = 0.40-0.59, all surveyed EPAs).

CONCLUSIONS

Graduates' perceptions support the SICC's educational utility and assessments. Based on this and other insight from the SICC, the authors propose implications toward collectively envisioning the continuum of physician competency.

Resident Perceptions of Assessment and Feedback in Competency-Based Medical Education: A Focus Group Study of One Internal Medicine Residency Program

PURPOSE

As key participants in the assessment dyad, residents must be engaged with the process. However, residents' experiences with competency-based medical education (CBME), and specifically with entrustable professional activity (EPA)-based assessments, have not been well studied. The authors explored junior residents' perceptions regarding the implementation of EPA assessment and feedback initiatives in an internal medicine program.

METHOD

From May to November 2018, 5 focus groups were conducted with 28 first-year internal medicine residents from the University of Toronto, exploring their experiences with facilitators and barriers to EPA-based assessments in the first years of the CBME initiative. Residents were exposed to EPA-based feedback tools from early in residency. Themes were identified using constructivist grounded theory to develop a framework to understand the resident perception of EPA assessment and feedback initiatives.

RESULTS

Residents' discussions reflected a growth mindset orientation, as they valued the idea of meaningful feedback through multiple low-stakes assessments. However, in practice, feedback seeking was onerous. While the quantity of feedback had increased, the quality had not; some residents felt it had worsened, by reducing it to a form-filling exercise. The assessments were felt to have increased daily workload with consequent disrupted workflow and to have blurred the lines between formative and summative assessment.

CONCLUSIONS

Residents embraced the driving principles behind CBME, but their experience suggested that changes are needed for CBME in the study site program to meet its goals. Efforts may be needed to reconcile the tension between assessment and feedback and to effectively embed meaningful feedback into CBME learning environments.

Perceptions and barriers to competency-based education in Canadian postgraduate medical education

RATIONALE

Competency-based education (CBE) is currently being implemented across Canadian postgraduate medical education programmes through Competence by Design (CBD).¹ Queen's University received permission to initiate CBE in all programmes simultaneously starting in 2017; an institutional initiative termed Competency-based medical education (CBME).² We describe our initial experiences to highlight perceptions and barriers and facilitate implementation at other centers.

METHODS

Anonymous online surveys were administered to faculty and residents transitioning to CBE (138 respondents) including (a) Queen's programme leaders (Programme Directors and CBME Leads) [n = 27], (b) Queen's residents [n = 102], and (c) Canadian neurology programme directors [n = 9] and were analysed using descriptive and inferential statistical techniques.

RESULTS

Perceptions were favourable (x = 3.55/5, SD = 0.71) and 81.6% perceived CBE enhanced training; however, perceptions were more favourable among faculty. Queen's programme leaders indicated that CBE did not improve their ability to provide negative feedback. Queen's residents did not perceive improved quality of feedback. National Canadian neurology programme directors did not perceive that their institutions had adequately prepared them. There was variability in barriers perceived across groups. Queen's programme leaders were concerned about resident initiative. Queen's residents felt that assessment selection and faculty responsiveness to feedback were barriers. Canadian neurology programme directors were concerned about access to information technology.

RECOMMENDATIONS

Our results indicate that faculty were concerned about the reluctance of residents to actively participate in CBE, while residents were hesitant to assume such a role because of lack of familiarity and perceived benefit. This discrepancy indicates attention should be devoted to (a) institutional administrative/educational supports, (b) faculty development around feedback/assessment, and (c) resident development to foster ownership of their learning and familiarity with CBE.

Evaluation of an undergraduate nursing entrustable professional activities framework: An exploratory qualitative research

BACKGROUND

Entrustable professional activities (EPA) are mainly used in graduate medical education and professional development and have not been widely implemented in nursing undergraduate settings. Nursing EPAs were therefore developed by the Alice Lee Centre of Nursing to translate theoretical nursing competencies into clinical practice and as forms of standardized clinical assessment tools. Feedback from various stakeholders is required to further refine the framework.

OBJECTIVES

To explore the perceptions and experiences of using the new EPA framework in nursing students and hospital and university clinical instructors.

DESIGN

An exploratory qualitative study using focus group interviews.

PARTICIPANTS

Seven year-one nursing undergraduates, 12 year-two undergraduates, seven university clinical instructors, and 18 hospital clinical instructors participated in this study.

METHODS

The students formed five groups, while the clinical instructors formed seven groups, each consisting of three to four participants. Semi-structured interviews were conducted to explore stakeholders' experiences and perceptions of the EPA assessment framework. Thematic analysis was used to analyze the data.

RESULTS

Despite initial mixed reactions of confusion and relief, many students and clinical instructors commended the comprehensive and structured EPA framework and its emphasis on holistic patient-centered care. EPAs also allow flexible assessment methods, encourage critical thinking among students, and promote team-based care and peer teaching. However, the assessment using a two-dimensional matrix with multiple competencies for the EPAs, subjective assessment criteria, the lack of standardization using checklists, subjective assessment criteria, and the lack of, manpower, and time may potentially affect the accuracy of the clinical assessments. The effectiveness of the EPA framework was also dependent on the quality of feedback, students' intrinsic motivations, and learning environments. The refinement of EPAs and entrustment levels, a physical checklist, and an incorporation of EPAs into school curricula were recommended to improve practitioners' learning experiences.

CONCLUSIONS

The use of EPAs in nursing education continues to be a novel and evolving process. There remains a need for a further refinement of the EPA framework to tailor to instructors' expectations and students' capabilities.

Exploring Perspectives from Internal Medicine Clerkship Directors in the USA on Effective Narrative Evaluation: Results from the CDIM National Survey

PURPOSE

Clinical performance evaluations play a critical role in determining medical school clerkship grades. This study aimed to provide clarification from clerkship directors in internal medicine on what constitutes an effective and informative narrative description of student performance.

METHODS

In September 2016, the Clerkship Directors in Internal Medicine (CDIM) electronically administered its annual, voluntary, and confidential cross-sectional survey of its US membership. One section of the survey asked six questions regarding the helpful components of an effective narrative evaluation. Respondents were asked to rate the effectiveness of elements contained within narrative evaluations of students.

RESULTS

Ninety-five CDIM members responded to the survey with an overall response rate of 74.2%. Descriptions of skills and behaviors were felt to be the most important, followed by a description of the overall synthetic or global assessment level of the student. Descriptions of personality and attitude were the next highest rated feature followed by adjectives describing performance. Length was felt to be the least important component. In free-text comments, several respondents indicated that direct observation of performance and specific examples of skills and behaviors are also desirable.

CONCLUSIONS

Narrative evaluations of students that explicitly comment on skills, behaviors, and an overarching performance level of the learner are strongly preferred by clerkship directors. Direct observation of clinical performance and giving specific examples of such behaviors give evaluations even more importance. Faculty development on evaluation and assessment should include instruction on these narrative assessment characteristics.

Unearthing Faculty and Trainee Perspectives of Feedback in Internal Medicine: the Oral Case Presentation as a Model

BACKGROUND

The case presentation is a fundamental activity used in both patient care and trainee education, partly due to feedback from supervisor to trainee. Although feedback in medical education is well studied, prior studies have not focused on the perceptions of feedback by Internal Medicine supervisors and trainees as it relates to clinical activities like the case presentation.

METHODS

Semi-structured interviews were conducted with eight Internal Medicine physicians, and 18 Internal Medicine trainees (5 medical students, 13 residents) at the University of Toronto. Purposive sampling was used. Interviews were conducted and coded iteratively within a constructivist grounded theory approach until saturation was reached.

RESULTS

Supervisors and trainees recognized feedback as an important part of the case presentation that can be (1) explicit, labeled feedback or (2) implicit, unlabeled feedback. Both trainees and supervisors perceived that not enough feedback occurs, likely stemming from a hesitancy by supervisors to label implicit feedback, calling it an interruption instead. Although trainees were keenly aware of non-verbal feedback from their supervisors as implicit feedback, they often interpreted explicit constructive feedback negatively. Interestingly, the same feedback from senior residents was regarded as highly educational, as it was uncoupled from assessment.

CONCLUSION

Feedback occurs more frequently in case presentations than previously described, particularly in an implicit, unlabeled format. Even though under-recognized, trainees identify and utilize implicit feedback from supervisors, and coaching from senior residents, to develop learned behaviors. This is reassuring in the age of Competency-Based Medical Education, as feedback has an essential role in workplace-based assessment and promotion.

Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination

BACKGROUND

The concept of EPAs is increasingly applied to assess trainees' workplace performance by means of entrustment ratings. OSCEs assess performance in a simulated setting, and it is unclear whether entrustment ratings can be integrated into these exams. This study explores the introduction of an entrustment rating scale into an existing OSCE.

METHODS

A 6-point entrustment scale was added to the standard ratings in an OSCE administered prior to students' final clerkship year in an undergraduate medical programme. Standard OSCE ratings assess clinical and communication skills. Assessors (n = 54) rated students' performance (n = 227) on a diverse set of clinical tasks and evaluated the addition of entrustment scales to OSCEs. Descriptive and inferential statistics were calculated for analyses.

RESULTS

Student performance varied across the stations, as reflected in both the standard OSCE ratings and the added entrustment ratings. Students received generally high standard OSCE ratings, whereas entrustment ratings were more widely distributed. All students passed the OSCE, and only a small proportion of students did not reach the expected pass threshold of 60% on the standard ratings in the single stations. The proportion of students who did not reach the expected entrustment level in the respective stations was noticeably higher. Both the clinical and communication skill ratings were related to the entrustment rating in most OSCE stations. A majority of the assessors positively evaluated the addition of entrustment ratings into the OSCE.

DISCUSSION

The findings provide an empirical basis to broaden our understanding of the potential use of entrustment ratings in existing OSCEs. They provide directions for future, more specific studies. The ratings might be used for formative feedback on students' readiness for workplace practice.

Entrustable professional activities in health care education: a scoping review

CONTEXT

The shift in medical education from time-based learning to outcome-based learning has drawn much attention to entrustable professional activities (EPAs) as an ideal assessment framework to translate competencies into clinical practice. Given the relative novelty of EPAs, this review aims to highlight research gaps and explore and consolidate available evidence pertaining to the development and implementation of EPAs in health care.

METHOD

Arksey and O'Malley's scoping review framework was used to present the findings. The authors performed a systematic search of PubMed, Embase, CINAHL, Scopus, MedNar, OpenGrey and ProQuest Dissertation and Theses for English articles published from the inception of each database to May 2018. A manual search of the reference lists of the included studies was conducted and an expert panel was consulted. Two reviewers screened the articles for eligibility using the inclusion criteria. All authors extracted key data and analysed the data descriptively. Thematic analysis was used to categorise the results into themes.

RESULTS

Eighty articles were included in the review. All articles were published between 2010 and 2018. Three major themes and eight sub-themes were generated: (i) development of EPAs (frameworks for EPA development and implementation, identifying core or specialty-specific EPAs, and EPAs for faculty development), (ii) evaluation of EPAs and EPA entrustment factors (revised curriculum, entrustment decisions for professional activities, and feedback on implemented EPAs and the development process), and (iii) future directions and recommendations for EPAs (implementation of EPAs in undergraduate medical education and specific disciplines, and other criticisms and recommendations for EPAs).

CONCLUSIONS

Entrustable professional activities are an essential means to translate competencies into observable and measurable clinical practice. However, high-level evidence-based research on the efficacy, development and implementation of EPAs for specific target groups (i.e. undergraduates and staff) and geographical regions (i.e. Asia and Africa) is still lacking, which suggests a direction for future research.

Scoping Review of Entrustable Professional Activities in Undergraduate Medical Education

PURPOSE

Entrustable professional activities (EPAs) are a hot topic in undergraduate medical education (UME); however, the usefulness of EPAs as an assessment approach remains unclear. The authors sought to better understand the literature on EPAs in UME through the lens of the 2010 Ottawa Conference Criteria for Good Assessment.

METHOD

The authors conducted a scoping review of the health professions literature (search updated February 2018), mapping publications to the Ottawa Criteria using a collaboratively designed charting tool.

RESULTS

Of the 1,089 publications found, 71 (6.5%) met inclusion criteria. All were published after 2013. Forty-five (63.4%) referenced the 13 Core Entrustable Professional Activities for Entering Residency developed by the Association of American Medical Colleges (AAMC). Forty (56.3%) were perspectives, 5 (7.0%) were reviews, and 26 (36.6%) were prospective empirical studies. The publications mapped to the Ottawa Criteria 158 times. Perspectives mapped more positively (83.7%) than empirical studies (76.7%). Reproducibility did not appear to be a strength of EPAs in UME; however, reproducibility, equivalence, educational effect, and catalytic effect all require further study. Inconsistent use of the term "EPA" and conflation of concepts (activity vs assessment vs advancement decision vs curricular framework) limited interpretation of published results. Overgeneralization of the AAMC's work on EPAs has influenced the literature.

CONCLUSIONS

Much has been published on EPAs in UME in a short time. Now is the time to move beyond opinion, clarify terms, and delineate topics so that well-designed empirical studies can demonstrate if and how EPAs should be implemented in UME.

Trusted to Learn: a Qualitative Study of Clerkship Students' Perspectives on Trust in the Clinical Learning Environment

BACKGROUND

Trust informs supervision decisions in medical training. Factors that influence trust differ depending on learners' and supervisors' level. Research has focused on resident trainees; questions exist about how medical students experience entrustment.

OBJECTIVE

This study examines how clerkship students perceive supervisors' trust in them and its impact on their learning.

DESIGN

Qualitative study using individual semi-structured interviews.

PARTICIPANTS

Clerkship medical students at the University of California, San Francisco.

APPROACH

We invited 30 core clerkship students to participate in interviews (October 2017 to February 2018) eliciting examples of appropriate, over-, and under-trust. We coded and analyzed transcripts using thematic analysis.

KEY RESULTS

Sixteen (53%) students participated. Three major themes arose: trust as scaffolding for learning, effects of trust on the learning environment, and consequences of trust for patients. Appropriate trust usually involved coaching and close guidance, often with more junior supervisors (interns or residents). These situations fostered students' motivation to learn, sense of value on the team, and perceived benefits to patients. Over-trust was characterized by task assignment without clear instruction, supervision, or feedback. Over-trust prompted student anxiety and stress, and concerns for potential patient harm. Under-trust was characterized by lack of clarity about the student role, leading to frustration and discontent, with unclear impact on patients. Students attributed inappropriate trust to contextual and supervisor factors and did not feel empowered to intervene due to concerns about performance evaluations and hierarchy.

CONCLUSIONS

As early learners in the clinical workplace, students frame trust as entailing high levels of support. It is important for medical educators to consider ways to train resident and faculty supervisors to enact trust and supervision for students differently than for residents. Structures that encourage students and supervisors to discuss trust and supervision in a transparent way can enhance learning.

Assessment Tools for Feedback and Entrustment Decisions in the Clinical Workplace: A Systematic Review

BACKGROUND: Entrustable Professional Activities (EPAs) combine feedback and evaluation with a permission to act under a specified level of supervision and the possibility to schedule learners for clinical service. This literature review aims to identify workplace-based assessment tools that indicate progression toward unsupervised practice, suitable for entrustment decisions and feedback to learners. METHODS: A systematic search was performed in the PubMed, Embase, ERIC, and PsycINFO databases. Based on title/abstract and full text, articles were selected using predetermined inclusion and exclusion criteria. Information on workplace-based assessment tools was extracted using data coding sheets. The methodological quality of studies was assessed using the medical education research study quality instrument (MERSQI). RESULTS: The search yielded 6,371 articles (180 were evaluated in full text). In total, 80 articles were included, identifying 67 assessment tools. Only a few studies explicitly mentioned assessment tools used as a resource for entrustment decisions. Validity evidence was frequently reported, and the MERSQI score was 10.0 on average. CONCLUSIONS: Many workplace-based assessment tools were identified that potentially support learners with feedback on their development and support supervisors with providing feedback. As expected, only few articles referred to entrustment decisions. Nevertheless, the existing tools or the principals could be used for entrustment decisions, supervision level, or autonomy.

Improving student midwives' workplace learning by moving from self- to co-regulated learning!

BACKGROUND

Contemporary perspectives on regulated learning are moving beyond models, emphasising individual learning (self-regulated learning) to models that position social transactions at the core of learning (co-regulated learning). In discussing this paradigm shift, it is important to study self- and co-regulated learning in situational context but research in the context of midwifery education is scarce.

OBJECTIVE

This study aimed to improve our understanding of regulating midwifery students' learning by exploring factors that promote or inhibit the capacity to independently self-regulate learning during internships.

DESIGN

We conducted a qualitative study involving semi-structured group discussions with final year undergraduate midwifery students (Belgium). Focus groups were audio-taped, transcribed verbatim and analysed in a thematic way.

FINDINGS

The majority of respondents did not experience an education intended to stimulate continuous self-regulated learning. They identified many social transactions (with mentors, teachers, teams of midwives, students, peer students, personal environments, personal coaches and curriculum managers) that were instrumental in promoting or inhibiting the ability to independently self-regulate. Flexibility and resilience to cope with the differences in practices, guidance, beliefs and behaviours of individual midwives within and between workplaces, was seen as paramount.

CONCLUSION

Broadening the perspective of the individual learner to the complex interplay between all the stakeholders involved with workplace learning is an essential step to move forward in the effort to improve student midwives' learning in the workplace.

When to trust our learners? Clinical teachers' perceptions of decision variables in the entrustment process

INTRODUCTION

Clinical training programs increasingly use entrustable professional activities (EPAs) as focus of assessment. However, questions remain about which information should ground decisions to trust learners. This qualitative study aimed to identify decision variables in the workplace that clinical teachers find relevant in the elaboration of the entrustment decision processes. The findings can substantiate entrustment decision-making in the clinical workplace.

METHODS

Focus groups were conducted with medical and veterinary clinical teachers, using the structured consensus method of the Nominal Group Technique to generate decision variables. A ranking was made based on a relevance score assigned by the clinical teachers to the different decision variables. Field notes, audio recordings and flip chart lists were analyzed and subsequently translated and, as a form of axial coding, merged into one list, combining the decision variables that were similar in their meaning.

RESULTS

A list of 11 and 17 decision variables were acknowledged as relevant by the medical and veterinary teacher groups, respectively. The focus groups yielded 21 unique decision variables that were considered relevant to inform readiness to perform a clinical task on a designated level of supervision. The decision variables consisted of skills, generic qualities, characteristics, previous performance or other information. We were able to group the decision variables into five categories: ability, humility, integrity, reliability and adequate exposure.

DISCUSSION

To entrust a learner to perform a task at a specific level of supervision, a supervisor needs information to support such a judgement. This trust cannot be credited on a single case at a single moment of assessment, but requires different variables and multiple sources of information. This study provides an overview of decision variables giving evidence to justify the multifactorial process of making an entrustment decision.