Entrustment and mapping of observable practice activities for resident assessment

Development of entrustable professional activities for emergency medicine medical education fellowships: A modified Delphi study

Background

It is essential that medical education (MedEd) fellows achieve desired outcomes prior to graduation. Despite the increase in postgraduate MedEd fellowships in emergency medicine (EM), there is no consistently applied competency framework. We sought to develop entrustable professional activities (EPAs) for EM MedEd fellows.

Methods

From 2021 to 2022, we used a modified Delphi method to achieve consensus for EPAs. EM education experts generated an initial list of 173 EPAs after literature review. In each Delphi round, panelists were asked to make a binary choice of whether to include the EPA. We determined an inclusion threshold of 70% agreement a priori. After the first round, given the large number of EPAs meeting inclusion threshold, panelists were instructed to vote whether each EPA should be included in the "20 most important" EPAs for a MedEd fellowship. Modifications were made between rounds based on expert feedback. We calculated descriptive statistics.

Results

Seventeen experts completed four Delphi rounds each with 100% response. After Round 1, 87 EPAs were eliminated and two were combined. Following Round 2, 46 EPAs were eliminated, seven were combined, and three were included in the final list. After the third round, one EPA was eliminated and 13 were included. After the fourth round, 11 EPAs were eliminated. The final list consisted of 16 EPAs in domains of career development, education theory and methods, research and scholarship, and educational program administration.

Conclusions

We developed a list of 16 EPAs for EM MedEd fellowships, the first step in implementing competency-based MedEd.

Development, implementation, and evaluation of entrustable professional activities (EPAs) for medical radiation technologists in Taiwan: a nationwide experience

BACKGROUND

Competency-based medical education (CBME) is an outcomes-oriented approach focused on developing competencies that translate into clinical practice. Entrustable professional activities (EPAs) bridge competency assessment and clinical performance by delineating essential day-to-day activities that can be entrusted to trainees. EPAs have been widely adopted internationally, but not yet implemented for medical radiation professionals in Taiwan.

MATERIALS AND METHODS

A nationwide consensus process engaged 97 experts in radiation technology education representing diagnostic radiography, radiation therapy, and nuclear medicine. Preliminary EPAs were developed through the focus group discussion and the modified Delphi method. The validity of these EPAs was evaluated using the QUEPA and EQual tools.

RESULTS

Through iterative consensus building, six core EPAs with 18 component observable practice activities (OPAs) in total were developed, encompassing routines specific to each radiation technology specialty. QUEPA and EQual questionnaire data verified these EPAs were valid, and of high quality for clinical teaching and evaluation.

CONCLUSION

The consensus development of tailored EPAs enables rigorous competency assessment during medical radiation technology education in Taiwan. Further expansion of EPAs and training of clinical staff could potentially enhance care quality by producing competent professionals.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

OBJECTIVES

Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations.

METHODS

Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment.

RESULTS

Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants.

CONCLUSION

We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

Research Initiative Supporting Excellence at the University of Cincinnati (RISE-UC): A Program to Develop and Support Research-Active Faculty Members

A combination of forces have markedly increased challenges to research-active faculty achieving sustained success. This article describes how one department at the University of Cincinnati College of Medicine (UCCOM) implemented a strategic plan, the Research Initiative Supporting Excellence at the University of Cincinnati (RISE-UC), to promote the research activity of its research-active faculty, fiscal year (FY) 2011-FY 2021. RISE-UC was implemented and regularly updated to address evolving needs. RISE-UC supported faculty members pursuing research via fiscal and administrative services to grow a critical mass of investigators; establish a shared governance model; create pathways for developing physician-scientists; develop discrete and targeted internal research funding; establish an Academic Research Service (ARS) unit (as infrastructure to support research); enhance faculty member mentorship; and recognize, celebrate, and reward research success. RISE-UC was informed by shared governance and resulted in substantial increases in total size of the faculty and external funding. More than 50% of Physician-Scientist Training Program graduates are active researchers at UCCOM. The internal awards program realized a return on investment of ~16.4-fold, and total external direct cost research funds increased from ~$55,400,000 (FY 2015) to ~$114,500,000 (FY 2021). The ARS assisted in the submission of 57 grant proposals and provided services faculty members generally found very helpful or helpful. The peer-mentoring group for early-career faculty members resulted in 12 of 23 participants receiving major grant funding (≥ $100,000; spring 2017-spring 2021) from sources including National Institutes of Health awards, Department of Defense funding, Veterans Affairs funding, and foundation awards. Research recognition included ~$77,000/year in incentive payments to faculty members for grant submissions and grants awarded. RISE-UC is an example of a comprehensive approach to promote research faculty member success and may serve as a model for other institutions with similar aspirations.

Creating and confirming observable professional activities (OPAs): A brief report on the practical approach for OPA design for resident education

CONTEXT

The transition of graduate medical education to competency-based education systems has resulted in exploration of the efficacy of Entrustable Professional Activities (EPAs) and related Observable Practice Activities (OPAs) as evaluation tools. EPAs were introduced to PM&R in 2017, but no OPAs have been reported for a non-procedurally based EPA. The primary aims of this study were to create and form consensus on OPAs for the Spinal Cord Injury EPA.

METHODS

A Modified Delphi panel of seven experts in the field was utilized to gain consensus on ten PM&R OPAs for the Spinal Cord Injury EPA.

RESULTS

After the first round of evaluations, most OPAs were evaluated by experts as requiring modifications (30/70 votes to keep, 34/70 votes to modify) with a majority of comments focusing on the specific content of the OPAs. Edits were made, and after the second round, the OPAs were evaluated and determined to be kept (62/70 votes to keep, 6/70 votes to modify) with most edits being about semantics of the OPAs. Ultimately, there was significant difference in all three categories between round 1 and round 2 (P < 0.0001) and 10 OPAs were finalized for use.

CONCLUSIONS

This study created 10 OPAs that can potentially help provide targeted feedback to residents on their competency in caring for patients with spinal cord injury. With regular usage, OPAs are designed to provide residents with insight into how they are progressing towards independent practice. In the future, studies should aim to assess the feasibility and utility of implementing the newly developed OPAs.

How Does TIMELESS Training Impact Resident Motivation for Learning, Assessment, and Feedback? Evaluating a Competency-Based Time-Variable Training Pilot

PURPOSE

As competency-based medical education has become the predominant graduate medical education training model, interest in time-variable training has grown. Despite multiple competency-based time-variable training (CBTVT) pilots ongoing in the United States, little is known about how this training approach impacts learners. The authors aim to explore how their CBTVT pilot program impacted resident motivation for learning, assessment, and feedback.

METHOD

The authors performed a qualitative educational case study on the Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) program at the University of Cincinnati from October 2020 through March 2022. Semistructured interviews were conducted with TIMELESS residents (n = 9) approximately every 6 months to capture experiences over time. The authors used inductive thematic analysis to develop themes and compared their findings with existing theories of learner motivation.

RESULTS

The authors developed 2 themes: TIMELESS had variable effects on residents' motivation for learning and TIMELESS increased resident engagement with and awareness of the program of assessment. Participants reported increased motivation to learn and seek assessment, though some felt a tension between performance (e.g., advancement through the residency program) and growth (e.g., improvement as a physician). Participants became more aware of the quality of assessments they received, in part due to TIMELESS increasing the perceived stakes of assessment, and reported being more deliberate when assessing other residents.

CONCLUSIONS

Resident motivation for learning, assessment, and feedback was impacted in ways that the authors contextualize using current theories of learner motivation (i.e., goal orientation theory and attribution theory). Future research should investigate how interventions, such as coaching, guided learner reflection, or various CBTVT implementation strategies, can help keep learners oriented toward mastery learning rather than toward performance.

Defining the observable processes of patient care related to social determinants of health

INTRODUCTION

An understanding of social determinants of health (SDH) and patients' social circumstances is recommended to deliver contextualised care. However, the processes of patient care related to SDH in clinical settings have not been described in detail. Observable practice activities (OPAs) are a collection of learning objectives and activities that must be observed in daily practice and can be used to describe the precise processes for professionals to follow in specific situations (process OPA.) METHODS: We used a modified Delphi technique to generate expert consensus about the process OPA for patient care related to SDH in primary care settings. To reflect the opinions of various stakeholders, the expert panel comprised clinical professionals (physicians, nurses, public health nurses, social workers, pharmacists and medical clerks), residents, medical students, researchers (medical education, health care, sociology of marginalised people), support members for marginalised people and patients. The Delphi rounds were conducted online. In Round 1, a list of potentially important steps in the processes of care was distributed to panellists. The list was modified, and one new step was added. In Round 2, all steps were acknowledged with few modifications.

RESULTS

Of 63 experts recruited, 61 participated, and all participants completed the Delphi rounds. A total of 14 observable steps were identified, which were divided into four components: communication, practice, maintenance and advocacy. The importance of ongoing patient-physician relationships and collaboration with professionals and stakeholders was emphasised for the whole process of care.

DISCUSSION

This study presents the consensus of a variety of experts on the process OPA for patient care related to SDHs. Further research is warranted to investigate how this Communication-Practice-Maintenance-Advocacy framework could affect medical education, quality of patient care, and patient outcomes.

Development and evaluation of Entrustable Professional Activities embedded in an e-portfolio for work-based assessment in community and public health dietetics

Entrustable professional activities (EPAs) is a recent concept that helps operationalise competency-based education. In this paper, we report on the development of EPAs and incorporation into an e-portfolio for work-based assessment of final year dietetics students in a community and public health professional placement. We aimed to determine the utility of the EPAs for work-based assessment in this context. We used action research methodology to conduct broad consultation with students and supervisors using anonymous surveys and focus groups to understand their beliefs and attitudes towards work-based assessment. Consequently, 40 EPAs with an accompanying four-point entrustment scale were developed and mapped to the National Competency Standards for Dietitians in Australia. The EPAs and assessment tool were piloted and evaluated via an anonymous online user survey across three cohorts of students (n = 133) and supervisors (n = 67). Following mediocre pilot year evaluation results, EPAs were revised and reduced in number. While students positively evaluated the e-portfolio, their appraisal of the EPAs and entrustment scale was less positive compared to supervisors. Supervisor evaluation of the EPA-based e-portfolio supported the validity, feasibility and acceptability of this novel assessment method in a community and public health setting. Assessment using EPAs, and the resulting educational data collected by the tool, offers potential for individual learners to identify areas needing development during placement, as well as potential to inform curriculum improvements and increase understanding of learning opportunities and outcomes for dietetic students in community and public health settings.

Developing the Expected Entrustment Score: Accounting for Variation in Resident Assessment

BACKGROUND

Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role.

OBJECTIVE

We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system.

DESIGN

Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge.

CONCLUSIONS

We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation.

The education passport: connecting programmatic assessment across learning and practice

Competency-based medical education (CBME) shifts us from static assessment of learning to developmental assessment for learning. However, implementation challenges associated with CBME remain a major hurdle, especially after training and into practice. The full benefit of developmental assessment for learning over time requires collaboration, cooperation, and trust among learners, regulators, and the public that transcends each individual phase. The authors introduce the concept of an "Education Passport" that provides evidence of readiness to travel across the boundaries between undergraduate medical education, graduate medical education, and the expanse of practice. The Education Passport uses programmatic assessment, a process of collecting numerous low stakes assessments from multiple sources over time, judging these data using criterion-referencing, and enhancing this with coaching and competency committees to understand, process, and accelerate growth without end. Information in the Passport is housed on a cloud-based server controlled by the student/physician over the course of training and practice. These data are mapped to various educational frameworks such Entrustable Professional Activities or milestones for ease of longitudinal performance tracking. At each stage of education and practice the student/physician grants Passport access to all entities that can provide data on performance. Database managers use learning analytics to connect and display information over time that are then used by the student/physician, their assigned or chosen coaches, and review committees to maintain or improve performance. Global information is also collected and analyzed to improve the entire system of learning and care. Developing a true continuum that embraces performance and growth will be a long-term adaptive challenge across many organizations and jurisdictions and will require coordination from regulatory and national agencies. An Education Passport could also serve as an organizing tool and will require research and high-value communication strategies to maximize public trust in the work.

Movement System Theory and Anatomical Competence: Threshold Concepts for Physical Therapist Anatomy Education

This viewpoint proposes eight anatomy threshold concepts related to physical therapist education, considering both movement system theory and anatomical competence. Movement system theory provides classifications and terminology that succinctly identifies and describes physical therapy practice from a theoretical and philosophical framework. The cardiovascular, pulmonary, endocrine, integumentary, nervous, and musculoskeletal systems are all included within this schema as the movement system theory encompasses all body systems interacting to create movement across the lifespan. Implementing movement system theory requires an ability to use human anatomy in physical therapist education and practice. Understanding the human body is a mandatory prerequisite for effective diagnosis, assessment, treatment, and patient evaluation. Anatomical competence refers to the ability to apply anatomic knowledge within the appropriate professional and clinical contexts. Exploring the required anatomical concepts for competent entry-level physical therapist education and clinical practice is warranted. The recommended threshold concepts (fluency, dimensionality, adaptability, connectivity, complexity, stability or homeostasis, progression or development, and humanity) could serve as an integral and long-awaited tool for guiding anatomy educators in physical therapy education.

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

BACKGROUND

Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

METHODS

The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

RESULTS

We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

CONCLUSIONS

This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

Development of an Entrustment-Supervision Assessment Tool for Pharmacy Experiential Education Using Stakeholder Focus Groups

Objective To devise a pharmacy-specific, expanded entrustable professional activities (EPA) entrustment-supervision scale that would frame preceptor ratings in a prospective and retrospective manner for use in experiential learning settings.Methods A series of focus group sessions were conducted to solicit expert opinion on how to develop and refine two entrustment-supervision scales. Purposive sampling was used to identify experts from different professional groups (physicians, pharmacy experiential administrators, and pharmacy practice faculty) who had extensive knowledge regarding EPAs and at least one publication related to EPAs. Panelists were invited to participate via email. Three focus sessions were conducted via videoconferencing between June and September 2019. The primary outcome was development of a pharmacy-specific EPA entrustment-supervision assessment tool. Secondary outcomes were individual entrustment-supervision statements across five levels of the entrustment-supervision scale.Results The focus group consisted of four pharmacy practice faculty, two experiential administrators, and one academic physician. Four concepts emerged from the focus group discussion: need for more granularity in entrustment-supervision scales; limitations due to differences in licensure requirements across the United States; present and ongoing use of expanded scales by schools and colleges of pharmacy; and uncertainty regarding how to rate entrustment-supervision when a student exhibits unprofessional behavior.Conclusion A pharmacy-specific, expanded EPA entrustment-supervision scale will be useful to support longitudinal assessment of learners in experiential settings where an EPA framework is utilized. Determining when to use a prospective versus retrospective perspective requires further evaluation.

Design, implementation and evaluation of novel work-based clinical assessment tool: An e-portfolio with embedded Entrustable Professional Activities

BACKGROUND

A new concept in work-based assessment, Entrustable Professional Activities (EPAs) describe key activities a work-ready practitioner can perform independently. EPAs are widely used in medicine to support competency-based education and are gaining acceptance in other professions such as nursing, pharmacy and dietetics. There is currently no widely accepted work-based assessment tool for student clinical placement assessment in dietetics and EPAs offer potential to address this gap.

OBJECTIVES

To design, implement and evaluate a work-based assessment tool using EPAs embedded in an e-portfolio for use in clinical dietetics.

SETTING

An accredited dietetic training program in Australian university and affiliated metropolitan and rural hospitals.

PARTICIPANTS

Three consecutive cohorts of final year dietetic students (n = 126) and their professional placement supervisors (n = 101).

METHODS

A working party consisting of dietetic academics with experience in clinical dietetics and education developed clinical dietetic EPAs and milestones that were mapped to the National Competency Standards for Dietitians in Australia. A design model incorporating multiple rounds of user feedback was used to create an e-portfolio with the EPAs embedded. Students and supervisors were invited to complete anonymous on-line surveys with Likert responses (1 = highly dissatisfied, 5 = highly satisfied) to evaluate the face validity, acceptability and feasibility of the tool.

RESULTS

A total of 37 EPAs were developed with an accompanying four-point entrustment scale to measure performance. Evaluation survey response rates were high and ranged from 35 to 45% for students and 57-83% for supervisors across the study period. Both students and supervisors evaluated the tool positively with a mean overall satisfaction of 4.12 ± 0.69 and 4.03 ± 0.68, respectively. Survey items evaluating face validity, feasibility and acceptability scored >3.75 for both groups of users.

CONCLUSIONS

The EPA based e-portfolio demonstrated face validity, feasibility and acceptability and offers potential for a widely accepted tool for work-based assessment of clinical dietetic students.

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program

INTRODUCTION

Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner.

METHODS

Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding.

RESULTS

Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the "big picture" in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care.

DISCUSSION

Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents' development.

Effect of resident and assessor gender on entrustment-based observational assessment in an internal medicine residency program

Background: Implicit gender bias leads to differences in assessment. Studies examining gender differences in resident milestone assessment data demonstrate variable results. The purpose of this study was to determine if observational entrustment scores differ by resident and assessor gender in a program of assessment based on discrete, observable skills.  Methods: We analyzed overall entrustment scores and entrustment scores by Accreditation Council for Graduate Medical Education (ACGME) core competency for 238 residents (49% female) from 396 assessors (38% female) in one internal medicine residency program from July 2012 to June 2019. We conducted analyses at 1-12 months, 1-36 months, 1-6 months, 7-12 months, and 31-36 months. We used linear mixed-effect models to assess the role of resident and assessor gender, with resident-specific and assessor-specific random effect to account for repeated measures.  Results: Statistically significant interactions existed between resident and assessor gender for overall entrustment at 1-12 months (p < 0.001), 1-36 months (p< 0.001), 1-6 months (p<0.001), 7-12 months (p=0.04), and 31-36 months (p<0.001). However, group differences were not statistically significant. In several instances an interaction was significant between resident and assessor gender by ACGME core competency, but there were no statistically significant group differences for all competencies at any time point. When applicable, subsequent analysis of main effect of resident or assessor gender independently of one another revealed no statistically significant differences.   Conclusions: No significant differences in entrustment scores were found based on resident or assessor gender in our large, robust entrustment-based program of assessment. Determining the reasons for our findings may help identify ways to mitigate gender bias in assessment.

Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review

PURPOSE

Workplace-based assessment (WBA) serves a critical role in supporting competency-based medical education (CBME) by providing assessment data to inform competency decisions and support learning. Many WBA systems have been developed, but little is known about how to effectively implement WBA. Filling this gap is important for creating suitable and beneficial assessment processes that support large-scale use of CBME. As a step toward filling this gap, the authors describe what is known about WBA implementation and use to identify knowledge gaps and future directions.

METHOD

The authors used Arksey and O'Malley's 6-stage scoping review framework to conduct the review, including: (1) identifying the research question; (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarizing, and reporting the results; and (6) consulting with relevant stakeholders.

RESULTS

In 2019-2020, the authors searched and screened 726 papers for eligibility using defined inclusion and exclusion criteria. One hundred sixty-three met inclusion criteria. The authors identified 5 themes in their analysis: (1) Many WBA tools and programs have been implemented, and barriers are common across fields and specialties; (2) Theoretical perspectives emphasize the need for data-driven implementation strategies; (3) User perceptions of WBA vary and are often dependent on implementation factors; (4) Technology solutions could provide useful tools to support WBA; and (5) Many areas of future research and innovation remain.

CONCLUSIONS

Knowledge of WBA as an implemented practice to support CBME remains constrained. To remove these constraints, future research should aim to generate generalizable knowledge on WBA implementation and use, address implementation factors, and investigate remaining knowledge gaps.

The recommended description of an entrustable professional activity: AMEE Guide No. 140

Entrustable professional activities (EPAs) have received much attention in the literature since they were first proposed in 2005. Useful guidelines, workshops, courses, and conferences have supported faculty in developing programs and designing assessment procedures using EPAs and entrustment decision-making. Yet, the need for clarification remains, particularly as more programs make the step from design to implementation.Well-written EPAs provide a natural construct to establish the outcome of training. To be useful, EPAs require more than a suitable title. This AMEE Guide elaborates eight sections of a full EPA description, and provides explanations and justifications for each. These sections are: title; specification and limitations; risks in case of failure; most relevant competency domains; knowledge, skills, attitudes and experiences; information sources to assess progress and support summative entrustment; entrustment/supervision level expected at which stage of training; and time period to expiration if not practiced.Constructing fully elaborated EPAs creates a shared mental model amongst learners and programs, informs competency-based curriculum design, directs ad-hoc and formal entrustment decision-making, and provides standards for certifying bodies and boundaries for scope of practice. The framework intends to support curricular leaders looking to adopt new EPAs, or revise and define established EPAs for competency-based education.

Entrustable Professional Activities for Small-Group Facilitation: A Validation Study Using Modified Delphi Technique

:

Entrustable professional activities (EPAs) provide a novel approach to support teachers' structured professionalization and to assess improvement in teaching competence thereafter. Despite their novelty, it is important to assess EPAs as a construct to ensure that they accurately reflect the work of the targeted profession.

BACKGROUND:

The co-creation of an EPA framework for training and entrustment of small-group facilitators has been discussed in the literature. Although a rigorous design process was used to develop the framework, its content validity has not been established yet.Approach: A modified Delphi technique was used. Three survey rounds were conducted from December 2019 to April 2020. Expert health professions educationalists were recruited using purposive sampling and snowball techniques. In Round 1, a rubric consisting of seven items was used to assess the quality of nine pre-designed EPAs. In Round 2, competencies required to perform the agreed-upon EPAs were selected from 12 competencies provided. In Round 3, consensus was sought on sub-activities recommended for agreed-upon EPAs. Quantitative data were analyzed using multiple statistical analyses, including item-wise and rubric-wise content validity indices, asymmetric confidence interval, mean, standard deviation, and response frequencies. Qualitative data were thematically analyzed using content analysis.

FINDINGS:

Three of the nine proposed EPAs achieved statistical consensus for retention. These EPAs were: (1) preparing an activity, (2) facilitating a small-group session, and (3) reflecting upon self and the session. Nine of the 12 pre-determined competencies achieved consensus and were then mapped against each agreed-upon EPA based on their relevance. Finally, results indicated consensus on five, six, and four sub-activities for EPA 1, EPA 2, and EPA 3, respectively.

CONCLUSIONS:

The final framework delineates three EPAs for small-group facilitation and their associated sub-activities. The full description of each EPA provided in this article includes the title, context, task specification, required competencies, and entrustment resources. Program developers, administrative bodies, and teaching staff may find this EPA framework useful to structure faculty development, to entrust teachers, and to support personal development.

Feasibility, and validity aspects of Entrustable Professional Activity (EPA)-based assessment in general practice training

INTRODUCTION

Entrustable Professional Activities (EPAs) are developed to support the practical application of competency frameworks in postgraduate medical education (PGME) programmes. EPAs are used for the assessment of the trainees' competence development, which takes place by means of an entrustment decision, aiming to stimulate learning and independent practice in trainees. In this pilot study, we explore the feasibility and validity of EPA-based assessment in a General Practice (GP) training programme.

METHODS

We used questionnaires to evaluate trainers' and trainees' experiences with the use of six EPAs for trainee learning, assessment and independent practice at the Out-of-Hours GP Center. Data were analysed quantitatively and qualitatively. Additionally, we examined the inter-item correlation between scores on EPA-based assessment and competency-based assessment using Spearman's Rho.

RESULTS

EPA-based assessment provided opportunities for giving concrete feedback and substantiating competency-based assessment. No consistent correlation between EPA-based assessment and competency-based assessment could be detected. Only later in the course of the training programme a correlation was found between the EPA scores and the degree of independence of trainees.

DISCUSSION

Results of this pilot study confirm the theories behind EPAs, as well as earlier research on EPAs in the workplace regarding trainee learning, assessment and independent practice. An important limitation of this study was the COVID-19 pandemic, as it influenced the results through reduced inclusion and follow-up, and through the impact on the workplace and trainee learning possibilities. Further research is needed to determine how EPAs support independent practice of trainees, as well as the assessment of trainee competency development.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane

PROBLEM

Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system.

APPROACH

To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time.

OUTCOMES

The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps.

NEXT STEPS

Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

Development of undergraduate nursing entrustable professional activities through using a participatory design approach

BACKGROUND

Competencies represent sets of general abilities and attributes that graduating nurses must acquire. Most nursing programs adopt competency based assessment and training approaches to prepare their students to enter the clinical environment. However, some concern has been raised regarding their interpretation and implementation in everyday clinical training. To operationalize competencies for better use in clinical practice, the Entrustable Professional Activities framework (EPAs) was introduced in 2005 by ten Cate.

AIM

The aim of this study is to develop a framework of EPAs for an Undergraduate Nursing Program.

METHOD

Through using a participatory action design approach, the participants developed a set of EPAs linked with competencies for an Undergraduate Nursing Program. Thematic analysis was performed, and EPAs framework validity was constructed.

RESULT

A total of eight core EPAs were developed and each one was operationalized by a brief description and nested within the related core competency domains. Supervision levels and entrustment decisions were determined.

CONCLUSION

The development of EPAs specific to undergraduate nursing programs may offer the opportunity to have standardized language to evaluate students' progress toward competency in nursing. It is recommended to conduct further research to improve the developed EPAs.

Facilitators and Inhibitors to Assessing Entrustable Professional Activities in Pediatric Residency

OBJECTIVE

Research on entrustable professional activities (EPAs) has focused on EPA development with little attention paid to implementation experiences. This constructivist grounded theory study sought to begin filling this gap by exploring the experiences of pediatric residency programs with implementing EPA-based assessment.

METHODS

Interviews with 19 program leader and clinical competency committee participants from 13 sites were held between January and July 2019. Participants were asked about their experiences with implementing EPA-based assessment. Data collection and analysis were iterative.

RESULTS

Participants described a range of facilitators and inhibitors that influenced their efforts to implement EPA-based assessment. These fell into 4 thematic areas: 1) alignment of EPA construct with local views of performance and assessment, 2) assessing EPAs illuminates holes in the residency curriculum, 3) clinical competency committee structure and process impacts EPA-based assessment, and 4) faculty engagement and development drives ability to assess EPAs. Areas described as facilitators by some participants were noted to be inhibitors for others. The sum of a program's facilitators and inhibitors led to more or less ability to assess EPAs on the whole. Finally, the first area functions differently from the others; it can shift the entire balance toward or away from the ability to assess EPAs overall.

CONCLUSION

This study helps fill a void in implementation evidence for EPA-based assessment through better understanding of facilitators and inhibitors to such efforts.

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

BACKGROUND

Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM).

OBJECTIVE

To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods.

DESIGN

The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward.

PARTICIPANTS

NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members.

MAIN MEASURES

The number and description of RSQMs generated during this process.

KEY RESULTS

Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission.

CONCLUSIONS

Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

See One, Do One, Forget One: Early Skill Decay After Paracentesis Training

INTRODUCTION

Internal medicine residents perform paracentesis, but programs lack standard methods for assessing competence or maintenance of competence and instead rely on number of procedures completed. This study describes differences in resident competence in paracentesis over time.

METHODS

From 2016 to 2017, internal medicine residents (n = 118) underwent paracentesis simulation training. Competence was assessed using the Paracentesis Competency Assessment Tool (PCAT), which combines a checklist, global scale, and entrustment score. The PCAT also delineates two categorical cut-point scores: the Minimum Passing Standard (MPS) and the Unsupervised Practice Standard (UPS). Residents were randomized to return to the simulation lab at 3 and 6 months (group A, n = 60) or only 6 months (group B, n = 58). At each session, faculty raters assessed resident performance. Data were analyzed to compare resident performance at each session compared with initial training scores, and performance between groups at 6 months.

RESULTS

After initial training, all residents met the MPS. The number achieving UPS did not differ between groups: group A = 24 (40%), group B = 20 (34.5%), p = 0.67. When group A was retested at 3 months, performance on each PCAT component significantly declined, as did the proportion of residents meeting the MPS and UPS. At the 6-month test, residents in group A performed significantly better than residents in group B, with 52 (89.7%) and 20 (34.5%) achieving the MPS and UPS, respectively, in group A compared with 25 (46.3%) and 2 (3.70%) in group B (p < .001 for both comparison).

DISCUSSION

Skill in paracentesis declines as early as 3 months after training. However, retraining may help interrupt skill decay. Only a small proportion of residents met the UPS 6 months after training. This suggests using the PCAT to objectively measure competence would reclassify residents from being permitted to perform paracentesis independently to needing further supervision.

Better Decision-Making: Shared Mental Models and the Clinical Competency Committee

BACKGROUND

Shared mental models (SMMs) help groups make better decisions. Clinical competency committees (CCCs) can benefit from the development and use of SMMs in their decision-making as a way to optimize the quality and consistency of their decisions.

OBJECTIVE

We reviewed the use of SMMs for decision making in graduate medical education, particularly their use in CCCs.

METHODS

In May 2020, the authors conducted a narrative review of the literature related to SMMs. This review included the SMM related to teams, team functioning, CCCs, and graduate medical education.

RESULTS

The literature identified the general use of SMMs, SMMs in graduate medical education, and strategies for building SMMs into the work of the CCC. Through the use of clear communication and guidelines, and a shared understanding of goals and expectations, CCCs can make better decisions. SMMs can be applied to Milestones, resident performance, assessment, and feedback.

CONCLUSIONS

To ensure fair and robust decision-making, the CCC must develop and maintain SMMs through excellent communication and understanding of expectations among members.

"Doing it Right" Overnight: a Multi-perspective Qualitative Study Exploring Senior Medical Resident Overnight Call

BACKGROUND

Competency-based medical education (CBME) requires the development of workplace-based assessment tools that are grounded in authentic clinical work. Developing such tools, however, requires a deep understanding of the underlying facets of the competencies being assessed. Gaining this understanding remains challenging in contexts where performance is not readily visible to supervisors such as the senior medical resident (SMR) on-call role in internal medicine.

OBJECTIVE

This study draws on the perspectives of healthcare professionals with whom the SMR interacts with overnight to generate insights into the different components of on-call SMR practice and the range of ways SMRs effectively and less effectively enact these.

APPROACH

We used a constructivist grounded theory (CGT) approach to examine variation in how on-call SMRs carry out their role overnight.

PARTICIPANTS

Six medical students, five junior residents, five internal medicine attending physicians, five emergency physicians, and three emergency nurses conducted observations of their on-call interactions with SMRs. Participants were then interviewed and asked to elaborate on their observations as well as provide comparative reflections on the practices of past SMRs they worked with.

KEY RESULTS

Strong collaboration and organizational skills were identified as critical components to effectively being the on-call SMR. Perceived weaker SMRs, while potentially also having issues with clinical skills, stood out more when they could not effectively manage the realities of collaboration in a busy workplace.

CONCLUSION

What consistently differentiated a perceived effective SMR from a less effective SMR was someone who was equipped to manage the realities of interprofessional collaboration in a busy workplace. Our study invites medical educators to consider what residents, particularly those in more complex roles, need to receive feedback on to support their development as physicians. It is our intention that the findings be used to inform the ways programs approach teaching, assessment, and the provision of feedback.

Bridging the Gap between Undergraduate Veterinary Training and Veterinary Practice with Entrustable Professional Activities

The transition from being a veterinary student to becoming a member of the veterinary profession is known to be challenging. Despite being licensed directly after graduation, many veterinarians do not feel fully equipped to practice unsupervised when they graduate. The increasing rate of attrition from veterinary practice, and a relatively high percentage of burnout during the first years in practice, has been suggested to be related to a lack of early career support. Over the past decade, medical education has adopted the concept of entrustable professional activities (EPAs). Recently, EPAs have been proposed to restructure veterinary education to help support the transition from veterinary student to practicing veterinarian. Implementing an EPA-based approach could help to bridge the gap between school and clinical practice, potentially preventing veterinary graduates from dropping out early on from what could have been a promising and exciting professional career.

Identifying Entrustable Professional Activities for Shared Decision Making in Postgraduate Medical Education: A National Delphi Study

PURPOSE

Although shared decision making (SDM) is considered the preferred approach in medical decision making, it is currently not routinely used in clinical practice. To bridge the transfer gap between SDM training and application, the authors aimed to reach consensus on entrustable professional activities (EPAs) for SDM and associated behavioral indicators as a framework to support self-directed learning during postgraduate medical education.

METHOD

Using existing literature on SDM frameworks and competencies; input from an interview study with 17 Dutch experts in SDM, doctor-patient communication, and medical education; and a national SDM expert meeting as a starting point, in 2017, the authors conducted a modified online Delphi study with a multidisciplinary Dutch panel of 32 experts in SDM and medical education.

RESULTS

After 3 Delphi rounds, consensus was reached on 4 EPAs-(1) the resident discusses the desirability of SDM with the patient, (2) the resident discusses the options for management with the patient, (3) the resident explores the patient's preferences and deliberations, and (4) the resident takes a well-argued decision together with the patient. Consensus was also reached on 18 associated behavioral indicators. Of the 32 experts, 30 (94%) agreed on this list of SDM EPAs and behavioral indicators.

CONCLUSIONS

The authors succeeded in developing EPAs and associated behavioral indicators for SDM for postgraduate medical education to improve the quality of SDM training and the application of SDM in clinical practice. These EPAs are characterized as process EPAs for SDM in contrast with content EPAs related to diverse medical complaints. A next step is the implementation of the SDM EPAs in existing competency-based workplace curricula.

Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project

BACKGROUND

Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents' CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents' needs.

METHODS

We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements.

RESULTS

Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs.

CONCLUSIONS

Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.

Validity of entrustment scales within anesthesiology residency training

INTRODUCTION

Competency-based medical education requires robust assessment in authentic clinical environments. Using work-based assessments, entrustment scales have emerged as a means of describing a trainee's ability to perform competently. Nevertheless, psychometric properties of entrustment-based assessment are relatively unknown, particularly in anesthesiology. This study assessed the generalizability and extrapolation evidence for entrustment scales within a program of assessment during anesthesiology training.

METHODS

Entrustment scores were collected during the first seven blocks of training for three resident cohorts. Entrustment scores were assessed during daily evaluations using a Clinical Case Assessment Tool (CCAT) within the preoperative, intraoperative, and postoperative setting. The reliability of the entrustment scale was estimated using generalizability theory. Spearman's correlations measured the relationship between median entrustment scores and percentiles scores on the Anesthesia Knowledge Test (AKT)-1 and AKT-6, mean Objective Structured Clinical Examination (OSCE) scores, and rankings of performance by the Clinical Competence Committee (CCC).

RESULTS

Analyses were derived from 2,309 CCATs from 35 residents. The reliability or generalizability (G) coefficient of the entrustment scale was 0.73 (95% confidence interval [CI], 0.70 to 0.76), and the internal consistency was 0.86 (95% CI, 0.84 to 0.88). Intraoperative entrustment scores significantly correlated with the AKT-6 (rho = 0.51, P = 0.01), mean OSCE (rho = 0.45, P = 0.04), and CCC performance rankings (rho = 0.52, P = 0.006).

CONCLUSION

As part of an assessment program, entrustment scales used early during anesthesiology training showed evidence of validity. Intraoperative entrustment scores had good reliability and showed acceptable internal consistency. Interpreting entrustment scores in this setting may constitute a valuable adjunct complementing traditional summative evaluations.

Development and validation of a postgraduate anaesthesiology core curriculum based on Entrustable Professional Activities: a Delphi study

Background: Postgraduate training curricula should not be based on time-spans or predefined numbers of performed procedures. One approach to link competencies to clinical tasks is the concept of Entrustable Professional Activities (EPA). The goal of this study was the definition, ranking and validation of EPAs for anaesthesiology postgraduate training and the creation of an anaesthesiologic core curriculum. Methods: Anaesthesiologists of different levels of training participated in the study (single-center, cross-sectional) . First, an expert group defined a preliminary list of EPAs. Then a first Delphi round (n= 47 participants) was applied to identify daily anaesthesiology tasks with the goal to define EPAs. From the first Delphi round a new set of EPAs was defined, using the template and mapping method. Through an alignment process, conducted by the expert group, the preliminary EPAs and the new set of EPAs from the first Delphi round were summarised into a new list of EPAs. This list was presented to the study participants in a second Delphi round (n=80 participants), with the goal to validate and rank each EPA and to define the year of entrustment. For this purpose, participants were asked in the second Delphi round if each EPA should be included into an anaesthesiology core curriculum and in which year of training entrustment should take place. Furthermore, they were asked to rank each EPA on a numeric scale, defining its importance. From this numeric scale, the content validity index (CVI) for each EPA was calculated. Consensus of the results from the second Delphi round was calculated, using the one-way random effects model to calculate Intra-Class-Correlations (ICC). Percentages of agreement among the whole set of EPAs of this study and a previously published set of EPAs were computed. Results: A core-curriculum comprising of 39 EPAs was developed. The EPAs were subdivided into superior/high and inferior/low scoring EPAs, reflecting their importance and were mapped to the year of entrustment. The results reached high consensus among the different participating anaesthesiologist groups (overall agreement was 0.96 for the CVI of each EPA and 0.83 for the year in which the EPAs should be entrusted). Agreement with the previously defined set of EPAs was 73.3%. Conclusion: This study provides a further step in transforming postgraduate anaesthesiology training into a more contemporary approach. Other studies are necessary to complete and amend the presented core curriculum of EPA based postgraduate anaesthesiology training.

Entrustable professional activities in nursing: A concept analysis

OBJECTIVE

Entrustable professional activities (EPAs) is a newly emerging concept. This paper aims at analyzing the concept of "EPAs" to clarify the meaning and to identify its components for optimum understanding of the concept for use in nursing education and future research.

METHOD

Three databases - ProQuest, Ovid, and PubMed- were retrieved, and Walker and Avant's concept analysis approach was used to analyze the concept of EPAs.

RESULTS

Based on the analysis, three attributes of EPAs were defined, namely 1) EPAs should be restricted to a limited number of tasks; 2)reflect the experience curve; and 3) include acceptance of responsibility and accountability. The antecedent of EPAs is a holistic and integrated assessment approach. This will eventually lead to anticipated consequences such as a mastery of core competencies and an increase in the level of autonomy.

CONCLUSION

The study may contribute to a reduction in the ambiguity of this concept to better apply it in nursing practice.

North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition Position Paper on Entrustable Professional Activities: Development of Pediatric Gastroenterology, Hepatology, and Nutrition Entrustable Professional Activities

Quality training in pediatric gastroenterology, hepatology, and nutrition is essential for the future of our specialty from advancing the science through research to providing clinical care for children with gastrointestinal, hepatic and nutritional disorders. As educational theory has developed, both the American Board of Pediatrics (ABP) and the Accreditation Council for Graduate Medical Education (ACGME) have commissioned projects to better define training including core competencies, and milestones with the goal of competency-based assessment. Seeking to provide a clinical context for these competencies and milestones, the ABP commissioned a project for each pediatric subspecialty to develop entrustable professional activities (EPA) while at the same time developing EPAs that are common to all pediatric subspecialties. North American Society for Pediatric Gastroenterology, Hepatology, Nutrition (NASPGHAN) commissioned an EPA Task Force to develop the pediatric gastroenterology, hepatology and nutrition EPAs. This document serves as an introduction to EPAs, including their historical background, underlying educational theory, and the process used to develop the pediatric gastroenterology, hepatology and nutrition EPAs in the United States of America.

Pediatric Gastroenterology, Hepatology, and Nutrition Entrustable Professional Activities: Development of an Assessment Tool and Curricular Resources

BACKGROUND

Entrustable professional activities (EPAs) are critical activities performed by medical professionals, which can be observed and assessed. Adding on to common EPAs for all pediatric subspecialty trainees, specialty-specific EPAs for pediatric gastroenterology, hepatology, and nutritional fellowship were developed by the North American Society for Pediatric Gastroenterology, Hepatology, and Nutrition (NASPGHAN) EPA Task Force.

METHODS

Having developed specialty-specific EPAs, building EPA assessments is the next logical step, as EPAs are included under a larger umbrella of competency-based assessment. Thus, the NASPGHAN EPA Task Force and Training Committee collaborated on an assessment tool and associated curricular resources to aid in tracking trainees' progression to entrustment within individual EPAs and readiness for independent practice.

RESULTS

This manuscript reports the development of an EPA assessment tool, including guiding principles and the theory behind the assessment tool, with a focus on simple, meaningful assessments that can provide crucial performance feedback to trainees. In addition, curricular resources were developed, based on the assessment tool, to support training. Ultimately, it is the hope of the NASPGHAN EPA Task Force and Training Committee that this tool can aid training programs in providing formative feedback for trainees, and can be used by training programs and clinical competency committees for summative evaluation.

Linking Workplace-Based Assessment to ACGME Milestones: A Comparison of Mapping Strategies in Two Specialties

Construct: The construct that is assessed is competency in Pediatrics and Internal Medicine residency training. Background: The Accreditation Council for Graduate Medical Education (ACGME) created milestones to measure learner progression toward competence over time but not as direct assessment tools. Ideal measurement of resident performance includes direct observation and assessment of patient care skills in the workplace. Residency programs have linked these concepts by mapping workplace-based assessments to the milestones of ACGME subcompetencies. Mapping is a subjective process, and little is known about specific techniques or the resulting consequences of mapping program-specific assessment data to larger frameworks of competency. Approach: In this article, the authors compare and contrast the techniques used to link workplace-based assessments called Observable Practice Activities (OPAs) to ACGME subcompetencies in two large academic residency programs from different specialties (Internal Medicine and Pediatrics). Descriptive analysis explored the similarities and differences in the assessment data generated by mapping assessment items to larger frameworks of competency. Results: Each program assessed the core competencies with similar frequencies. The largest discrepancy between the two subspecialties was the assessment of Medical Knowledge, which Internal Medicine assessed twice as often. Pediatrics also assessed the core competency Systems-based Practice almost twice as often as Internal Medicine. Both programs had several subcompetencies that were assessed more or less often than what appeared to be emphasized by the blueprint of mapping. Despite using independent mapping processes, both programs mapped each OPA to approximately three subcompetencies. Conclusions: Mapping workplace-based assessments to the ACGME subcompetencies allowed each program to see the whole of their curricula in ways that were not possible before and to identify existing curricular and assessment gaps. Although each program used similar assessment tools, the assessment data generated were different. The lessons learned in this work could inform other programs attempting to link their own workplace-based assessment elements to ACGME subcompetencies.

Development and Establishment of Initial Validity Evidence for a Novel Tool for Assessing Trainee Admission Notes

BACKGROUND

Documentation is a key component of practice, yet few curricula have been published to teach trainees proper note construction. Additionally, a gold standard for assessing note quality does not exist, and no documentation assessment tools integrate with established competency-based frameworks.

OBJECTIVE

To develop and establish initial validity evidence for a novel tool that assesses key components of trainee admission notes and maps to the Accreditation Council for Graduate Medical Education (ACGME) milestone framework.

DESIGN

Using an iterative, consensus building process we developed the Admission Note Assessment Tool (ANAT). Pilot testing was performed with both the supervising attending and study team raters not involved in care of the patients. The finalized tool was piloted with attendings from other institutions.

PARTICIPANTS

Local experts participated in tool development and pilot testing. Additional attending physicians participated in pilot testing.

MAIN MEASURES

Content, response process, and internal structure validity evidence was gathered using Messick's framework. Inter-rater reliability was assessed using percent agreement.

KEY RESULTS

The final tool consists of 16 checklist items and two global assessment items. Pilot testing demonstrated rater agreement of 72% to 100% for checklist items and 63% to 70% for global assessment items. Note assessment required an average of 12.3 min (SD 3.7). The study generated validity evidence in the domains of content, response process, and internal structure for use of the tool in rating admission notes.

CONCLUSIONS

The ANAT assesses individual components of a note, incorporates billing criteria, targets note "bloat," allows for narrative feedback, and provides global assessments mapped to the ACGME milestone framework. The ANAT can be used to assess admission notes by any attending and at any time after note completion with minimal rater training. The ANAT allows programs to implement routine note assessment for multiple functions with the use of a single tool.

Entrustment trends in orthotic and prosthetic residencies

BACKGROUND

Orthotic and prosthetic residency preceptors have little guidance transitioning residents between observing and performing patient care. Understanding current entrustment trends and factors can help guide resident progression during residency.

OBJECTIVES

To identify entrustment trends and factors within an orthotics and prosthetics residency affecting a supervisor's decision to entrust residents with independent patient care.

STUDY DESIGN

Cross-sectional study.

METHODS

In this cross-sectional study, 831 National Commission of Orthotic and Prosthetic Education preceptors were invited to complete a 26-question, online questionnaire that examined possible entrustment factors.

RESULTS

A total of 77 preceptors completed the questionnaire. The top-reported behavioral qualities impacting entrustment were recognition of limitations and willingness to ask for help (n = 45) and competence (n = 37). Preceptors ranked evaluation (n = 27) as the professional skill which most affected entrustability; technical skill least affected entrustability (n = 4). Preceptors of 12-month residencies entrust residents to indirect supervision when they have completed approximately 75% (i.e. 9 months) of the program. Preceptors of 18-month residencies reported that they entrust their residents by the time they reach 50% (i.e. 9 months) of their training. A small number of preceptors (n = 19) never allow a resident to independently perform practice management.

CONCLUSION

Residency supervisors value interpersonal and evaluation skills when making entrustment decisions. Residency mentors can use these findings to inform their entrustment decisions and support a resident's progression toward independent practice.

CLINICAL RELEVANCE

This study presents information which can inform clinical preceptors and residents on what factors contribute to the decision to progress a resident from observation to independent patient care. These findings may influence clinical education standards which aim to promote both resident training and patient access to quality care.

Orthopaedic Surgery Residency Milestones: Initial Formulation and Future Directions

Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.

Identifying entrustable professional activities for surgical skills training in companion animal health

BACKGROUND

Veterinary medical education is increasingly moving towards outcome-based training based on competency frameworks. A source of concern is the translation of competencies into the practice of clinical teaching, for example, surgical skills training. It is suggested that the use of entrustable professional activities (EPAs) might bridge this gap. The purpose of this study, therefore, was to identify EPAs related to surgical skills for companion animal health to enhance competency-based education.

METHODS

Draft versions of EPAs related to surgical skills were established by an iterative consensus-based approach through 45-min interview sessions. These draft versions were used to explore the opinion of companion animal veterinarians, both veterinarians (specialists, residents and interns) involved in undergraduate teaching and veterinarians working in private practice involved in extramural clinical teaching, on the relevance and level of entrustment of the EPAs through a modified Delphi procedure. Mean (relevance) and median (level of entrustment) scores were calculated and textual comments were analysed to create a final framework of EPAs related to surgical skills.

RESULTS AND CONCLUSION

The Delphi panel reached consensus in three rounds. Thirty-four per cent of those invited to participate in the study completed the final survey. Finally, a list of 13 EPAs related to companion animal surgical skills a student should be entrusted to perform at time of graduation was established.

Entrustable Professional Activities in Oral Health for Primary Care Providers Based on a Scoping Review

Despite advances in oral health care, inequalities in oral health outcomes persist due to problems in access. With proper training, primary care providers can mitigate this inequality by providing oral health education, screening, and referral to advanced dental treatment. Diverging sets of oral health competencies and guidelines have been released or endorsed by multiple primary care disciplines. The aim of this study was to transform multiple sets of competencies into Entrustable Professional Activities (EPAs) for oral health integration into primary care training. A scoping review of the literature between January 2000 and December 2016 was conducted according to PRISMA methodology to identify all existing sets of competencies. The following primary care disciplines were included in the search: allopathic/osteopathic medical schools and residency programs in family medicine, internal medicine, and pediatrics; physician assistant programs; and nurse practitioner programs. Competencies were compared using the Health Resources and Services Administration Integration of Oral Health and Primary Care Practice competencies as the foundational set and translated into EPAs. The resulting EPAs were tested with a reactor panel. The scoping review produced 1,466 references, of which 114 were selected for full text review. Fourteen competencies were identified as being central to the integration of oral health into primary care. These were converted to seven EPAs for oral health integration into primary care and were mapped onto Accreditation Council for Graduate Medical Education residency competency domains as well to the Association of American Medical Colleges EPAs for graduating medical students. The resulting EPAs delineate the essential, observable work required of primary care providers to ensure that oral health is treated as a critical determinant of overall health.

Use of the Electrodiagnostic Entrustable Professional Activity for Competency Assessment in Physical Medicine and Rehabilitation Training Programs

Entrustable professional activities have emerged as a means for the evaluation of resident competency that is expressed in terms of the assessed need for supervision. Recently, 19 physical medicine and rehabilitation-specific entrustable professional activities were published (Am J Phys Med Rehabil. 2017;96:762-764). The electrodiagnostic entrustable professional activity and six new electrodiagnostic entrustable professional activities subcategories (observable practice activities) were piloted as an entrustable professional activities/observable practice activities set within five residency programs. Survey-based (quantitative) and open-ended (qualitative) feedback was collected from participants. Participating attendings found this method feasible and generally reported satisfaction with the entrustable professional activities/observable practice activities as a means of providing feedback to residents. Residents were less clear on the added value of this approach. Qualitative data supported the need for adjustments to the entrustment scale to allow for more gradations within supervisory levels, a standardized orientation of residents to the use of observable practice activities and an increased quantity of assessments for each observable practice activities category to allow for demonstration of resident progress toward independence. Use of the electrodiagnostic entrustable professional activity/observable practice activities set shows promise as a means for observational competency assessment in the outpatient setting. However, feedback acquired through this pilot study suggests changes that could be made to improve future implementation.

Workplace-Based Assessments Using Pediatric Critical Care Entrustable Professional Activities

BACKGROUND

Workplace-based assessment (WBA) is critical to graduating competent physicians. Developing assessment tools that combine the needs of faculty, trainees, and governing bodies is challenging but imperative. Entrustable professional activities (EPAs) are emerging as a clinically oriented framework for trainee assessment.

OBJECTIVE

We sought to develop an EPA-based WBA tool for pediatric critical care medicine (PCCM) fellows. The goals of the tool were to promote learning through benchmarking and tracking entrustment.

METHODS

A single PCCM EPA was iteratively subdivided into observable practice activities (OPAs) based on national and local data. Using a mixed-methods approach following van der Vleuten's conceptual model for assessment tool utility and Messick's unified validity framework, we sought validity evidence for acceptability, content, internal structure, relation to other variables, response process, and consequences.

RESULTS

Evidence was gathered after 1 year of use. Items for assessment were based on correlation between the number of times each item was assessed and the frequency professional activity occurred. Phi-coefficient reliability was 0.65. Narrative comments demonstrated all factors influencing trust, identified by current literature, were cited when determining level of entrustment granted. Mean entrustment levels increased significantly between fellow training years (P = .001). Compliance for once- and twice-weekly tool completion was 50% and 100%, respectively. Average time spent completing the assessment was less than 5 minutes.

CONCLUSIONS

Using an EPA-OPA framework, we demonstrated utility and validity evidence supporting the tool's outcomes. In addition, narrative comments about entrustment decisions provide important insights for the training program to improve individual fellow advancement toward autonomy.

Entrustable professional activities in health care education: a scoping review

CONTEXT

The shift in medical education from time-based learning to outcome-based learning has drawn much attention to entrustable professional activities (EPAs) as an ideal assessment framework to translate competencies into clinical practice. Given the relative novelty of EPAs, this review aims to highlight research gaps and explore and consolidate available evidence pertaining to the development and implementation of EPAs in health care.

METHOD

Arksey and O'Malley's scoping review framework was used to present the findings. The authors performed a systematic search of PubMed, Embase, CINAHL, Scopus, MedNar, OpenGrey and ProQuest Dissertation and Theses for English articles published from the inception of each database to May 2018. A manual search of the reference lists of the included studies was conducted and an expert panel was consulted. Two reviewers screened the articles for eligibility using the inclusion criteria. All authors extracted key data and analysed the data descriptively. Thematic analysis was used to categorise the results into themes.

RESULTS

Eighty articles were included in the review. All articles were published between 2010 and 2018. Three major themes and eight sub-themes were generated: (i) development of EPAs (frameworks for EPA development and implementation, identifying core or specialty-specific EPAs, and EPAs for faculty development), (ii) evaluation of EPAs and EPA entrustment factors (revised curriculum, entrustment decisions for professional activities, and feedback on implemented EPAs and the development process), and (iii) future directions and recommendations for EPAs (implementation of EPAs in undergraduate medical education and specific disciplines, and other criticisms and recommendations for EPAs).

CONCLUSIONS

Entrustable professional activities are an essential means to translate competencies into observable and measurable clinical practice. However, high-level evidence-based research on the efficacy, development and implementation of EPAs for specific target groups (i.e. undergraduates and staff) and geographical regions (i.e. Asia and Africa) is still lacking, which suggests a direction for future research.

Faculty and Resident Perspectives on Using Entrustment Anchors for Workplace-Based Assessment

BACKGROUND

Research suggests that workplace-based assessment (WBA) tools using entrustment anchors provide more reliable assessments than those using traditional anchors. There is a lack of evidence describing how and why entrustment anchors work.

OBJECTIVE

The purpose of this study is to better understand the experience of residents and faculty with respect to traditional and entrustment anchors.

METHODS

We used constructivist grounded theory to guide data collection and analysis (March-December 2017) and semistructured interviews to gather reflections on anchors. Phase 1 involved residents and faculty (n = 12) who had only used assessment tools with traditional anchors. Phase 2 involved participants who had used tools with entrustment anchors (n = 10). Data were analyzed iteratively.

RESULTS

Participants expressed that the pragmatic language of entrustment anchors made WBA (1) concrete and justifiable; (2) transparent as they explicitly link clinical assessment and learning progress; and (3) align with training outcomes, enabling better feedback. Participants with no prior experience using entrustment anchors outlined contextual concerns regarding their use. Participants with experience described how they addressed these concerns. Participants expressed that entrustment anchors leave a gap in assessment information because they do not provide normative data.

CONCLUSIONS

Insights from this analysis contribute to a theoretical framework of benefits and challenges related to the adoption of entrustment anchors. This richer understanding of faculty and resident perspectives of entrustment anchors may assist WBA developers in creating more acceptable tools and inform the necessary faculty development initiatives that must accompany the use of these new WBA tools.  .

Entrustment Ratings in Internal Medicine Training: Capturing Meaningful Supervision Decisions or Just Another Rating?

The implementation of Entrustable Professional Activities has led to the simultaneous development of assessment based on a supervisor's entrustment of a learner to perform these activities without supervision. While entrustment may be intuitive when we consider the direct observation of a procedural task, the current implementation of rating scales for internal medicine's non-procedural tasks, based on entrustability, may not translate into meaningful learner assessment. In these Perspectives, we outline a number of potential concerns with ad hoc entrustability assessments in internal medicine post-graduate training: differences in the scope of procedural vs. non-procedural tasks, acknowledgement of the type of clinical oversight common within internal medicine, and the limitations of entrustment language. We point towards potential directions for inquiry that would require us to clarify the purpose of the entrustability assessment, reconsider each of the fundamental concepts of entrustment in internal medicine supervision and explore the use of descriptive rather than numeric assessment approaches.

Providing Compassionate Off-Ramps for Medical Students Is a Moral Imperative

It is highly unusual for learners to leave medical training in the United States even though some individuals' goals may change and others may not achieve expected competence. There are a number of possible reasons for this: (1) Students may feel that they have progressed too far into their careers and amassed too much debt to leave medical training; (2) students may be allowed to graduate despite marginal performance; and (3) students may have entered medical training with risk factors for poor performance that were not addressed. As stewards of the educational process, medical educators have an ethical obligation to students and the public to create off-ramps, or points along the educational continuum at which learners can reassess their goals and educators can assess competence, that allow students to leave medicine.Given the nationwide focus on physician health and wellness, the authors believe the creation of options to leave medical training without compromising one's self-esteem or incurring unmanageable debt (i.e., compassionate off-ramps) is a moral imperative. The practice of medicine should not be an exercise in survival; it should allow people to develop and thrive over the course of their careers. Offering students options to make use of the medical competencies they have accumulated in other attractive careers would enable medical educators to behave compassionately toward individual students and fulfill their societal obligation to graduate competent and committed physicians. To this end, the authors present six recommendations for consideration.