Entrustment Decision Making: Extending Miller's Pyramid

Implementing an entrustable professional activities programmatic assessments for nurse practitioner training in emergency care: A pilot study

BACKGROUND

Entrustable professional activities (EPAs), as new methods used to operationalize competency-based education, are gaining acceptance in medical professions. However, no EPA-based framework exists to evaluate emergency care nurse practitioners' competencies in clinical practice.

OBJECTIVES

To develop, implement, and evaluate an EPA-based framework for emergency care nurse practitioners.

DESIGN

Prospective observational study.

SETTING

An accredited nurse practitioner training program in the emergency department of a University-affiliated Hospital.

PARTICIPANTS

Eight novice nurse practitioners and eighteen clinical instructors.

METHODS

A working group of academic educators, nurse practitioner leaders, and clinical instructors developed a set of EPAs using a participatory design method. We scheduled weekly ad hoc evaluations for the nurse practitioners in emergency care, and we collected several observations from multiple assessors for analysis.

RESULTS

Four nested EPAs were developed that adopted a 1-5 supervision-entrustment scale to measure performance. The results of the 20-month assessments illustrated that the average scores of EPA1, which measured the assessment and management of patients with acute medical presentations in the emergency department, differed significantly but were not positively correlated with the nurse practitioners' increasing subspecialty and department seniority levels (Kruskal-Wallis test P = .011 and 0.006, respectively). In addition, the most time required for the nurse practitioners to achieve a stable level 5 entrustability score for all EPAs for at least 3 consecutive months was 18 months. Clinical instructors noted that recognizing limits is a skill that is essential for a nurse practitioner to earn clinical instructors' trust.

CONCLUSIONS

Entrustability scores may not correlate with nurse practitioners' increasing seniority, and patient care requires the longest training time (18 months) for emergency care nurse practitioners to achieve autonomy. Program directors should adjust the supervision-entrustment level expected at various stages of training according to the complexity of individual EPAs. An EPA-based assessment framework in a nurse practitioner training program may assist in overseeing the development of competencies for emergency care nurse practitioners.

How can Entrustable Professional Activities serve the quality of health care provision through licensing and certification?

This paper about Entrustable Professional Activities (EPAs) was solicited to support the discussion about the future of licensing within the Medical Council of Canada. EPAs, units of professional practice to be entrusted to learners or professionals once they have shown to possess sufficient competence, were proposed in 2005 to operationalize competency-based postgraduate medical education and have become widely popular for various health professions education programs in many countries. EPAs break the breadth of competence for license down to units of practice that can be overseen, assessed, monitored, documented, and entrusted. EPAs together may constitute an individual's portfolio of qualifications, and define a scope of practice. A medical license and a specialty certification can then be defined as the required combination of EPAs for which one is qualified at any specific moment in time. That 'snapshot' could change over time and reflect the professional development of the individual, both in their competence and in their privileges to practice. Micro-credentialing and digital badges might become an adequate option to show-case one's scope of practice at any time and operationalize the idea of a dynamic portfolio of EPAs.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Variable or Fixed? Exploring Entrustment Decision Making in Workplace- and Simulation-Based Assessments

PURPOSE

Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings.

METHOD

An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees' endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes.

RESULTS

Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions "comfortably" (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making.

CONCLUSIONS

Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters' criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.

Who can do this procedure? Using entrustable professional activities to determine curriculum and entrustment in anesthesiology - An international survey

INTRODUCTION

As competency-based curricula get increasing attention in postgraduate medical education, Entrustable Professional Activities (EPAs) are gaining in popularity. The aim of this survey was to determine the use of EPAs in anesthesiology training programs across Europe and North America.

METHODS

A survey was developed and distributed to anesthesiology residency training program directors in Switzerland, Germany, Austria, Netherlands, USA and Canada. A convergent design mixed-methods approach was used to analyze both quantitative and qualitative data.

RESULTS

The survey response rate was 38% (108 of 284). Seven percent of respondents used EPAs for making entrustment decisions. Fifty-three percent of institutions have not implemented any specific system to make such decisions. The majority of respondents agree that EPAs should become an integral part of the training of residents in anesthesiology as they are universal and easy to use.

CONCLUSION

Although recommended by several national societies, EPAs are used in few anesthesiology training programs. Over half of responding programs have no specific system for making entrustment decisions. Although several countries are adopting or planning to adopt EPAs and national societies are recommending the use of EPAs as a framework in their competency-based programs, few are yet using these to make "competence" decisions.

A Literature Review: Entrustable Professional Activities, an assessment tool for postgraduate dental training?

Assessing when dental trainees are ready to independently undertake clinical procedures at specialist level is critical for dental postgraduate programmes to determine when a trainee is 'work ready', in addition to ensuring patient safety. Entrustable professional activities (EPA) are a novel method of competency-based assessment. An EPA is a unit of professional practice or critical clinical activity identified within dental training programmes, which should be assessed during training, to establish if trainees are ready for independent practice, with a progressive decrease in supervision, based on supervisors' entrustment decisions. This article describes EPAs, entrustment decisions, including entrustment supervision scales and the process recommended to develop EPAs within dental curricula. EPAs have not been formally introduced for assessment within dental education programmes in the United Kingdom, but recent developments have been described in undergraduate dental education globally. Clinical significance: Competency-based assessments need to be continually developed to adapt to rapidly changing population health care and dental needs, to determine when dental trainees are ready for independent clinical practice. Early development of entrustable professional activities for assessment in undergraduate dental programmes has been well received by both trainees and supervisors. Further investigation is required to consider formal development of EPAs within postgraduate dental programmes.

Techniques to Teach Students Effectively Using Telemedicine. Comment on "Incorporating Medical Students Into Primary Care Telehealth Visits: Tutorial"

Collapse

Measuring the Impact of a Faculty Development Program on Clinical Educators

INTRODUCTION

An Academy of Clinical Educators (ACE) was established at the University of Newcastle, to support and build capacity among existing and prospective medical educators. ACE established a Certificate of Clinical Teaching and Supervision (CCTS) program, the final assessment of which was a reflective piece on how the course has affected participants' practice as clinical teachers or supervisors and how changes are expected to impact learner achievement. We conducted a qualitative evaluation of these to explore the impact of the CCTS on participants' teaching.

METHODS

Thirty-one participants (of 90 completers to date) consented for their written reflections to undergo qualitative thematic analysis and completed a survey exploring their preparation for, and experience of the program, and application of skills learnt.

RESULTS

Most participants reported applying the skills gained through the CCTS to their teaching practice to a large (n=23; 72%) or very large (n=5; 16%) extent. Four themes emerged from the qualitative data, aligned with the topics of the CCTS: teaching structure; feedback; orientation; and assessment. Participants described application of more structured approaches to orientation, teaching and feedback, positive student responses, and self-reported satisfaction with adopted changes.

DISCUSSION

The CCTS has motivated change in the teaching practice of participants. Although evidence presented here is limited by the self-reported nature, descriptions of actual changes in practice were detailed and specific enough to suggest they could act as a proxy for objectively measured change in behaviour and outcome.

CONCLUSION

A faculty development program delivered to clinicians with a range of teaching and education-related roles, from varied clinical disciplines and professions, can promote improved, structured teaching and feedback.

Development of a national pain management competency profile to guide entry-level physiotherapy education in Canada

BACKGROUND

National strategies from North America call for substantive improvements in entry-level pain management education to help reduce the burden of chronic pain. Past work has generated a valuable set of interprofessional pain management competencies to guide the education of future health professionals. However, there has been very limited work that has explored the development of such competencies for individual professions in different regions. Developing profession-specific competencies tailored to the local context is a necessary first step to integrate them within local regulatory systems. Our group is working toward this goal within the context of entry-level physiotherapy (PT) programs across Canada.

AIMS

This study aimed to create a consensus-based competency profile for pain management, specific to the Canadian PT context.

METHODS

A modified Delphi design was used to achieve consensus across Canadian university-based and clinical pain educators.

RESULTS

Representatives from 14 entry-level PT programs (93% of Canadian programs) and six clinical educators were recruited. After two rounds, a total of 15 competencies reached the predetermined endorsement threshold (75%). Most participants (85%) reported being "very satisfied" with the process.

CONCLUSIONS

This process achieved consensus on a novel pain management competency profile specific to the Canadian PT context. The resulting profile delineates the necessary abilities required by physiotherapists to manage pain upon entry to practice. Participants were very satisfied with the process. This study also contributes to the emerging literature on integrated research in pain management by profiling research methodology that can be used to inform related work in other health professions and regions.

Trust and risk pitfalls in medical education: A qualitative study of clinical teachers

PURPOSE

Learning opportunities in teaching hospitals are gated by clinical teachers. One way to unpack their decision-making is to employ a 'trust and risk' model. This study aimed to uncover clinical teachers' experience of trust, risk and vulnerability as they participate in medical education.

METHODS

Hospital-based clinical teachers were interviewed about trust, risk and vulnerability in medical education. Data analysis was undertaken using a constructivist, qualitative framework.

RESULTS

Twenty demographically diverse clinical teachers participated. Trust and risk were regarded as fundamental workplace and teaching concepts. Their concerns fell into three domains of risk: clinical, teaching and personal. Being trusted unlocked clinical learning opportunities, whereas trust failure limited future participation. Feeling trusted or not affected wellbeing and self-efficacy. Trust and risk pitfalls in education included bias, asymmetry and sidelining.

CONCLUSIONS

This study adds to the literature by voicing clinical teachers' personal risks and vulnerabilities. Attention was drawn to the benefits of being perceived as trustworthy, and to the clinical, teaching and personal vulnerabilities of trust failure.If expert judgement of trustworthiness is to be legitimised as meaningful assessment, clinical teachers must be aware not only of how trust is built, but also the pitfalls of trust failure.

Exploring the Association Between USMLE Scores and ACGME Milestone Ratings: A Validity Study Using National Data From Emergency Medicine

PURPOSE

The United States Medical Licensing Examination (USMLE) sequence and the Accreditation Council for Graduate Medical Education (ACGME) milestones represent 2 major components along the continuum of assessment from undergraduate through graduate medical education. This study examines associations between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME emergency medicine (EM) milestone ratings.

METHOD

In February 2019, subject matter experts (SMEs) provided judgments of expected associations for each combination of Step examination and EM subcompetency. The resulting sets of subcompetencies with expected strong and weak associations were selected for convergent and discriminant validity analysis, respectively. National-level data for 2013-2018 were provided; the final sample included 6,618 EM residents from 158 training programs. Empirical bivariate correlations between milestone ratings and Step scores were calculated, then those correlations were compared with the SMEs' judgments. Multilevel regression analyses were conducted on the selected subcompetencies, in which milestone ratings were the dependent variable, and Step 1 score, Step 2 CK score, and cohort year were independent variables.

RESULTS

Regression results showed small but statistically significant positive relationships between Step 2 CK score and the subcompetencies (regression coefficients ranged from 0.02 [95% confidence interval (CI), 0.01-0.03] to 0.12 [95% CI, 0.11-0.13]; all P < .05), with the degree of association matching the SMEs' judgments for 7 of the 9 selected subcompetencies. For example, a 1 standard deviation increase in Step 2 CK score predicted a 0.12 increase in MK-01 milestone rating, when controlling for Step 1. Step 1 score showed a small statistically significant effect with only the MK-01 subcompetency (regression coefficient = 0.06 [95% CI, 0.05-0.07], P < .05).

CONCLUSIONS

These results provide incremental validity evidence in support of Step 1 and Step 2 CK score and EM milestone rating uses.

Questioning medical competence: Should the Covid-19 crisis affect the goals of medical education?

The COVID-19 pandemic has disrupted many societal institutions, including health care and education. Although the pandemic's impact was initially assumed to be temporary, there is growing conviction that medical education might change more permanently. The International Competency-based Medical Education (ICBME) collaborators, scholars devoted to improving physician training, deliberated how the pandemic raises questions about medical competence. We formulated 12 broad-reaching issues for discussion, grouped into micro-, meso-, and macro-level questions. At the individual micro level, we ask questions about adaptability, coping with uncertainty, and the value and limitations of clinical courage. At the institutional meso level, we question whether curricula could include more than core entrustable professional activities (EPAs) and focus on individualized, dynamic, and adaptable portfolios of EPAs that, at any moment, reflect current competence and preparedness for disasters. At the regulatory and societal macro level, should conditions for licensing be reconsidered? Should rules of liability be adapted to match the need for rapid redeployment? We do not propose a blueprint for the future of medical training but rather aim to provoke discussions needed to build a workforce that is competent to cope with future health care crises.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

EPA-based assessment: Clinical teachers' challenges when transitioning to a prospective entrustment-supervision scale

BACKGROUND

This study explores the challenges clinical teachers face when first using a prospective entrustment-supervision (ES) scale in a curriculum based on Entrustable Professional Activities (EPAs). A prospective ES scale has the purpose to estimate at which level of supervision a student will be ready to perform an activity in subsequent encounters.

METHODS

We studied the transition to prospective assessment of medical students in clerkships via semi-structured interviews with twelve purposefully sampled clinical teachers, shortly after the introduction of a new undergraduate EPA-based curriculum and EPA-based assessment employing a prospective ES scale.

RESULTS

While some clinical teachers showed a correct interpretation, rating strategies also appeared to be affected by the target supervision level for completion of the clerkship. Instructions to estimate readiness for a supervision level in the future were not always understood. Further, teachers' interpretation of the scale anchors relied heavily on the phrasing.

DISCUSSION

Prospective assessment asks clinical teachers to make an extra inference step in their judgement process from reporting observed performance to estimating future level of supervision. This requires a change in mindset when coming from a retrospective, performance-oriented assessment method, i.e., reporting what was observed. Our findings suggest optimizing the ES-scale wordings and improving faculty development.

The ingredients of a rich entrustment decision

Entrustment decision-making has become a topic of interest in workplace-based assessment in the health professions and is germane to the use of entrustable professional activities. Entrustment decisions stem from judgments of a trainee's competence and include the permission to act with a higher level of responsibility or autonomy and a lower level of supervision. Making entrustment decisions differs from regular assessment of trainees, which usually has no consequences beyond marking trainee progress. Studies show that clinicians generally weigh more factors in making an entrustment decision than when merely assessing trainee competence or performance without direct consequences for patient care. To synthesize the varying factors reported in literature, the authors performed a thematic analysis of key qualitative studies that investigated trainee features clinical supervisors find important when making entrustment decisions. Five themes emerged from the 13 publications: Capability (specific knowledge, skills, experience, situational awareness), Integrity (truthful, benevolent, patient-centered), Reliability (conscientious, predictable, accountable, responsible), Humility (recognizes limits, asks for help, receptive to feedback), Agency (proactive toward work, team, safety, personal development). Thoughtful entrustment decisions, made either by individual clinical supervisors or by clinical competency committees, may be enriched by taking into account these five features.