Comparing Entrustment and Competence: An Exploratory Look at Performance-Relevant Information in the Final Year of a Veterinary Program

Student Perceptions of Narrative Feedback in Entrustable Professional Activities

INTRODUCTION

Entrustable professional activity (EPA) observations can be used to develop a holistic picture of trainee competency in professional tasks. Narrative feedback is an essential component of EPAs, but there is a lack of published literature capturing undergraduate student perceptions.

METHODS

Students who completed Year 3 of the MD programme in 2022-2023 at one institution in Canada were invited to participate in a survey designed to elicit their perceptions of narrative feedback in EPAs. Survey methods included closed-ended questions (analysed descriptively) and open-ended questions (analysed through thematic analysis).

RESULTS

The response rate to the survey was 23%. Over 60% of students reported that narrative comments in EPAs were specific and aligned with EPA scores, and 86% reported that the narrative feedback was individualised at least some of the time. However, 57% reported that they never or rarely received actionable feedback for growth. Students demonstrated mixed feelings as to whether EPAs help support their clinical development. Some noted that they can help students identify gaps and reinforce positives. Others reported preferring verbal feedback to written feedback and that EPAs resulted in an administrative burden. Over 90% of students reported barriers to obtaining EPAs, and almost 90% expressed the need for changes to EPAs. A predominant theme from students was the desire for a reduction in EPA requirements.

CONCLUSIONS

Students perceived the narrative feedback to be individualized and specific but reported that the feedback rarely contained feedback for growth. Students identified several barriers to EPA completion and provided recommended changes.

Development and Use of the Competency-Based Veterinary Education (CBVE) Assessment Toolkit

The Competency-Based Veterinary Education (CBVE) Analyze Working Group of the American Association of Veterinary Medical Colleges (AAVMC) Council on Outcomes-based Veterinary Education (COVE) has developed a CBVE assessment toolkit. The toolkit is designed to provide curriculum committees and individual instructors with an opportune intersection of the CBVE domains of competence and various assessment techniques. College-wide curriculum committees can use the toolkit to guide programs of assessment in the larger unit, ensuring that assessment methods are aligned with intended learning outcomes throughout the curriculum. On a smaller unit basis, the toolkit allows a single instructor or team of instructors to identify domains of interest for evaluation and then to identify various assessment tools appropriate to those domains. For each of 21 different assessment tools, the toolkit provides information that includes: a description; appropriate CBVE domains and competencies; examples; documented uses; evidence of efficacy; references; and links to illustrations if available. Because the toolkit is published online, periodic updates can be made as more data become available on the efficacy of various assessment tools relative to the CBVE domains in veterinary education. From programmatic assessment to single course examinations, the toolkit is intended to assist both administrators and faculty alike in understanding how different assessment approaches can support a variety of competency domains.

Longitudinal assessment of competency development at The Ohio State University using the competency-based veterinary education (CBVE) model

With the development of the American Association of Veterinary Medical Colleges' Competency-Based Veterinary Education (CBVE) model, veterinary schools are reorganizing curricula and assessment guidelines, especially within the clinical rotation training elements. Specifically, programs are utilizing both competencies and entrustable professional activities (EPAs) as opportunities for gathering information about student development within and across clinical rotations. However, what evidence exists that use of the central tenets of the CBVE model (competency framework, milestones and EPAs) improves our assessment practices and captures reliable and valid data to track competency development of students as they progress through their clinical year? Here, we report on validity evidence to support the use of scores from in-training evaluation report forms (ITERs) and workplace-based assessments of EPAs to evaluate competency progression within and across domains described in the CBVE, during the final year clinical training period of The Ohio State University's College of Veterinary Medicine (OSU-CVM) program. The ITER, used at the conclusion of each rotation, was modified to include the CBVE competencies that were assessed by identifying the stage of student development on a series of descriptive milestones (from pre-novice to competent). Workplace based assessments containing entrustment scales were used to assess EPAs from the CBVE model within each clinical rotation. Competency progression and entrustment scores were evaluated on each of the 31 rotations offered and high-stakes decisions regarding student performance were determined by a collective review of all the ITERs and EPAs recorded for each learner across each semester and the entire year. Results from the class of 2021, collected on approximately 190 students from 31 rotations, are reported with more than 55 299 total competency assessments combined with milestone placement and 2799 complete EPAs. Approximately 10% of the class was identified for remediation and received additional coaching support. Data collected longitudinally through the ITER on milestones provides initial validity evidence to support using the scores in higher stakes contexts such as identifying students for remediation and for determining whether students have met the necessary requirements to successfully complete the program. Data collected on entrustment scores did not, however, support such decision making. Implications are discussed.