Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase

Making judgments based on reported observations of trainee performance: a scoping review in Health Professions Education

Background

Educators now use reported observations when assessing trainees' performance. Unfortunately, they have little information about how to design and implement assessments based on reported observations.

Objective

The purpose of this scoping review was to map the literature on the use of reported observations in judging health professions education (HPE) trainees' performances.

Methods

Arksey and O'Malley's (2005) method was used with four databases (sources: ERIC, CINAHL, MEDLINE, PsycINFO). Eligibility criteria for articles were: documents in English or French, including primary data, and initial or professional training; (2) training in an HPE program; (3) workplace-based assessment; and (4) assessment based on reported observations. The inclusion/exclusion, and data extraction steps were performed (agreement rate > 90%). We developed a data extraction grid to chart the data. Descriptive analyses were used to summarize quantitative data, and the authors conducted thematic analysis for qualitative data.

Results

Based on 36 papers and 13 consultations, the team identified six steps characterizing trainee performance assessment based on reported observations in HPE: (1) making first contact, (2) observing and documenting the trainee performance, (3) collecting and completing assessment data, (4) aggregating assessment data, (5) inferring the level of competence, and (6) documenting and communicating the decision to the stakeholders.

Discussion

The design and implementation of assessment based on reported observations is a first step towards a quality implementation by guiding educators and administrators responsible for graduating competent professionals. Future research might focus on understanding the context beyond assessor cognition to ensure the quality of meta-assessors' decisions.

Assessing trainee's performance using reported observations: Perceptions of nurse meta-assessors

BACKGROUND

Educational and health care organizations who prepare meta-assessors to fulfill their role in the assessment of trainees' performance based on reported observations have little literature to rely on. While the assessment of trainees' performance based on reported observations has been operationalized, we have yet to understand the elements that can affect its quality fully. Closing this gap in the literature will provide valuable insight that could inform the implementation and quality monitoring of the assessment of trainees' performance based on reported observations.

OBJECTIVES

The purpose of this study was to explore the elements to consider in the assessment of trainees' performance based on reported observations from the perspectives of meta-assessors.

METHODS

Design, Settings, Participants, data collection and analysis. The authors adopted Sandelowski's qualitative descriptive approach to interview nurse meta-assessors from two nursing programs. A semi-structured interview guide was used to document the elements to consider in the assessment of nursing trainees' performance based on reported observations, and a survey was used to collect sociodemographic data. The authors conducted a thematic analysis of the interview transcripts.

RESULTS

Thirteen meta-assessors participated in the study. Three core themes were identified: (1) meta-assessors' appropriation of their perceived assessment roles and activities, (2) team climate of information sharing, and (3) challenges associated with the assessment of trainees' performance based on reported observations. Each theme is comprised of several sub themes.

CONCLUSIONS

To optimize the quality of the assessment of the trainee's performance based on reported observations and ratings, HPE programs might consider how to clarify better the meta-assessor's roles and activities, as well as how interventions could be created to promote a climate of information sharing and to address the challenges identified. This work will guide educational and health care organizations for better preparation and support for meta-assessors and preceptors.

Core Entrustable Professional Activities for Entering Residency: A National Survey of Graduating Medical Students' Self-Assessed Skills by Specialty

BACKGROUND

The Association of American Medical Colleges described 13 Core Entrustable Professional Activities (EPAs) that graduating students should be prepared to perform under indirect supervision on day one of residency. Surgery program directors recently recommended entrustability in these Core EPAs for incoming surgery interns. We sought to determine if graduating students intending to enter surgery agreed they had the skills to perform these Core EPAs.

STUDY DESIGN

Using de-identified, individual-level data collected from and about 2019 Association of American Medical Colleges Graduation Questionnaire respondents, latent profile analysis was used to group respondents based on their self-assessed Core EPAs skills' response patterns. Associations between intended specialty, among other variables, and latent profile analysis group were assessed using independent sample t -tests and chi-square tests and multivariable logistic regression methods.

RESULTS

Among 12,308 Graduation Questionnaire respondents, latent profile analysis identified 2 respondent groups: 7,863 (63.9%) in a high skill acquisition agreement (SAA) group and 4,445 (36.1%) in a moderate SAA group. Specialty was associated with SAA group membership (p < 0.001), with general surgery, orthopaedic surgery, and emergency medicine respondents (among others) overrepresented in the high SAA group. In the multivariable logistic regression models, each of anesthesiology, ophthalmology, pediatrics, psychiatry, and radiology (vs general surgery) specialty intention was associated with a lower odds of high SAA group membership.

CONCLUSION

Graduating students' self-assessed Core EPAs skills were higher for those intending general surgery than for those intending some other specialties. Our findings can inform collaborative efforts to ensure graduates' acquisition of the skills expected of them at the start of residency.

In pursuit of a better transition to selected residencies: a quasi-experimental evaluation of a final year of medical school dedicated to the acute care domain

BACKGROUND

Medical schools seek the best curricular designs for the transition to postgraduate education, such as the Dutch elective-based final, 'transitional' year. Most Dutch graduates work a mean of three years as a physician-not-in-training (PNIT) before entering residency training. To ease the transition to selected specialties and to decrease the duration of the PNIT period, UMC Utrecht introduced an optional, thematic variant of the usual transitional year, that enables the development of theme-specific competencies, in addition to physicians' general competencies.

METHODS

We introduced an optional transitional year for interested students around the theme of acute care, called the Acute Care Transitional Year (ACTY). This study aimed to evaluate the ACTY by judging whether graduates meet postgraduate acute care expectations, indicating enhanced learning and preparation for practice. In a comprehensive assessment of acute care knowledge, clinical reasoning, skills, and performance in simulations, we collected data from ACTY students, non-ACTY students interested in acute care, and PNITs with approximately six months of acute care experience.

RESULTS

ACTY graduates outperformed non-ACTY graduates on skills and simulations, and had higher odds of coming up to the expectations faculty have of a PNIT, as determined by global ratings. PNITs did better on simulations than ACTY graduates.

DISCUSSION

ACTY graduates show better resemblance to PNITs than non-ACTY graduates, suggesting better preparation for postgraduate acute care challenges.

CONCLUSION

Transitional years, offering multidisciplinary perspectives on a certain theme, can enhance learning and preparedness for entering residency.

Design and implementation of a core EPA-based acting internship curriculum

PURPOSE

The Association of American Medical Colleges published Core Entrustable Professional Activities (Core EPAs), expected independent clinical skills for first-day interns. We sought to determine whether a required acting internship (AI) in the fourth-year curriculum could be used for summative assessment of students' mastery of Core EPAs to a predefined level that would readily generalize across disciplines and campuses.

METHODS

The University of North Dakota School of Medicine and Health Sciences MD Program created a standardized, required Core EPA-based AI curriculum for multiple specialties at multiple geographic sites providing a final entrustability assessment for 10 EPAs in a single course.

RESULTS

The course was successfully designed and launched for all students in a single class. During the AI, students functioned at the level of an acting intern, rated the courses as superior, and performed at satisfactory exit-level competence for 10 Core EPAs.

CONCLUSIONS

A standardized, EPA-based AI curriculum can provide an opportunity for exit level EPA assessment in the medical curriculum. This model functions well within multiple specialties and at diverse community-based, volunteer faculty teaching sites.

Concordance of Narrative Comments with Supervision Ratings Provided During Entrustable Professional Activity Assessments

BACKGROUND

Use of EPA-based entrustment-supervision ratings to determine a learner's readiness to assume patient care responsibilities is expanding.

OBJECTIVE

In this study, we investigate the correlation between narrative comments and supervision ratings assigned during ad hoc assessments of medical students' performance of EPA tasks.

DESIGN

Data from assessments completed for students enrolled in the clerkship phase over 2 academic years were used to extract a stratified random sample of 100 narrative comments for review by an expert panel.

PARTICIPANTS

A review panel, comprised of faculty with specific expertise related to their roles within the EPA program, provided a "gold standard" supervision rating using the comments provided by the original assessor.

MAIN MEASURES

Interrater reliability (IRR) between members of review panel and correlation coefficients (CC) between expert ratings and supervision ratings from original assessors.

KEY RESULTS

IRR among members of the expert panel ranged from .536 for comments associated with focused history taking to .833 for complete physical exam. CC (Kendall's correlation coefficient W) between panel members' assignment of supervision ratings and the ratings provided by the original assessors for history taking, physical examination, and oral presentation comments were .668, .697, and .735 respectively. The supervision ratings of the expert panel had the highest degree of correlation with ratings provided during assessments done by master assessors, faculty trained to assess students across clinical contexts. Correlation between supervision ratings provided with the narrative comments at the time of observation and supervision ratings assigned by the expert panel differed by clinical discipline, perhaps reflecting the value placed on, and perhaps the comfort level with, assessment of the task in a given specialty.

CONCLUSIONS

To realize the full educational and catalytic effect of EPA assessments, assessors must apply established performance expectations and provide high-quality narrative comments aligned with the criteria.

Entrustment Decision Making in the Core Entrustable Professional Activities: Results of a Multi-Institutional Study

PURPOSE

In 2014, the Association of American Medical Colleges defined 13 Core Entrustable Professional Activities (EPAs) that all graduating students should be ready to do with indirect supervision upon entering residency and commissioned a 10-school, 5-year pilot to test implementing the Core EPAs framework. In 2019, pilot schools convened trained entrustment groups (TEGs) to review assessment data and render theoretical summative entrustment decisions for class of 2019 graduates. Results were examined to determine the extent to which entrustment decisions could be made and the nature of these decisions.

METHOD

For each EPA considered (4-13 per student), TEGs recorded an entrustment determination (ready, progressing but not yet ready, evidence against student progressing, could not make a decision); confidence in that determination (none, low, moderate, high); and the number of workplace-based assessments (WBAs) considered (0->15) per determination. These individual student-level data were de-identified and merged into a multischool database; chi-square analysis tested the significance of associations between variables.

RESULTS

The 2,415 EPA-specific determinations (for 349 students by 4 participating schools) resulted in a decision of ready (n = 997/2,415; 41.3%), progressing but not yet ready (n = 558/2,415; 23.1%), or evidence against student progression (n = 175/2,415; 7.2%). No decision could be made for the remaining 28.4% (685/2,415), generally for lack of data. Entrustment determinations' distribution varied across EPAs (chi-square P < .001) and, for 10/13 EPAs, WBA availability was associated with making (vs not making) entrustment decisions (each chi-square P < .05).

CONCLUSIONS

TEGs were able to make many decisions about readiness for indirect supervision; yet less than half of determinations resulted in a decision of readiness to perform this EPA with indirect supervision. More work is needed at the 10 schools to enable authentic summative entrustment in the Core EPAs framework.

Coaching interventions among healthcare students in tertiary education to improve mental well-being: A mixed studies review

OBJECTIVES

To consolidate the available evidence regarding healthcare students' experiences of coaching interventions' effect on their mental well-being in tertiary educational institutions.

DESIGN

A mixed studies review.

DATA SOURCES

Seven electronic databases were searched for relevant studies from their inception dates until July 2021: PubMed, Embase, CINAHL, PsycINFO, Web of Science, Scopus and ProQuest Dissertations and Theses Global.

REVIEW METHODS

This review observed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines and was guided by Pluye and Hong's (2014) framework for mixed studies review. Quality appraisal of included studies was conducted using the Mixed Methods Appraisal Tool. A convergent qualitative synthesis design for mixed studies reviews was adopted to analyze all findings and thematic analysis was conducted according to Braun and Clarke's (2006) framework.

RESULTS

Three themes and seven subthemes are generated from the 12 included studies. The main themes are: 1) Receiving much-needed help, 2) Unlocking the secrets of success, and 3) Way forward for coaching in healthcare education.

CONCLUSIONS

Many students benefited from the coaching interventions in terms of academic, emotional and psychological support and received guidance in professional development. However, careful preparation of both coaches and students is needed to improve the success of coaching interventions. Future coaching interventions could incorporate both group and individual sessions, assign students to coaches based on their clinical interests and consider conducting online coaching sessions using video calls. Improvement of study rigor would be required to achieve more accurate results. Lastly, future studies should be conducted on healthcare students of more diverse disciplines and cultures to increase results' generalizability.

The use of EPA assessments in decision-making: Do supervision ratings correlate with other measures of clinical performance?

BACKGROUND

Entrustable professional activities (EPAs) have been introduced as a framework for teaching and assessment in competency-based educational programs. With growing use, has come a call to examine the validity of EPA assessments. We sought to explore the correlation of EPA assessments with other clinical performance measures to support use of supervision ratings in decisions about medical students' curricular progression.

METHODS

Spearman rank coefficients were used to determine correlation of supervision ratings from EPA assessments with scores on clerkship evaluations and performance on an end-of-clerkship-year Objective Structured Clinical Examination (CPX).

RESULTS

Both overall clinical evaluation items score (rho 0.40; n = 166) and CPX patient encounter domain score (rho 0.31; n = 149) showed significant correlation with students' overall mean EPA supervision rating during the clerkship year. There was significant correlation between mean supervision rating for EPA assessments of history, exam, note, and oral presentation skills with scores for these skills on clerkship evaluations; less so on the CPX.

CONCLUSIONS

Correlation of EPA supervision ratings with commonly used clinical performance measures offers support for their use in undergraduate medical education. Data supporting the validity of EPA assessments promotes stakeholders' acceptance of their use in summative decisions about students' readiness for increased patient care responsibility.

Implementing Systematic Faculty Development to Support an EPA-Based Program of Assessment: Strategies, Outcomes, and Lessons Learned

Problem: Development of a novel, competency-based program of assessment requires creation of a plan to measure the processes that enable successful implementation. The principles of implementation science outline the importance of considering key drivers that support and sustain transformative change within an educational program. The introduction of Entrustable Professional Activities (EPAs) as a framework for assessment has underscored the need to create a structured plan to prepare assessors to engage in a new paradigm of assessment. Although approaches to rater training for workplace-based assessments have been described, specific strategies to prepare assessors to apply standards related to the level of supervision a student needs have not been documented. Intervention: We describe our systematic approach to prepare assessors, faculty and postgraduate trainees, to complete EPA assessments for medical students during the clerkship phase of our curriculum. This institution-wide program is designed to build assessors' skills in direct observation of learners during authentic patient encounters. Assessors apply new knowledge and practice skills in using established performance expectations to determine the level of supervision a learner needs to perform clinical tasks. Assessors also learn to provide feedback and narrative comments to coach students and promote their ongoing clinical development. Data visualizations for assessors facilitate reinforcement of the tenets learned during training. Collaborative learning and peer feedback during faculty development sessions promote the formation of a community of practice among assessors. Context: Faculty development for assessors was implemented in advance of implementation of the EPA program. Assessors in the program include residents/fellows who work closely with students, faculty with discipline-specific expertise and a group of experienced clinicians who were selected to serve as experts in competency-based EPA assessments, the Master Assessors. Training focused on creating a shared understanding about the application of criteria used to evaluate student performance. EPA assessments based on the AAMC's Core Entrustable Professional Activities for Entering Residency, were completed in nine core clerkships. EPA assessments included a supervision rating based on a modified scale for use in undergraduate medical education. Impact: Data from EPA assessments completed during the first year of the program were analyzed to evaluate the effectiveness of the faculty development activities implemented to prepare assessors to consistently apply standards for assessment. A systematic approach to training and attention to critical drivers that enabled institution-wide implementation, led to consistency in the supervision rating for students' first EPA assessment completed by any type of assessor, ratings by assessors done within a specific clinical context, and ratings assigned by a group of specific assessors across clinical settings. Lessons learned: A systematic approach to faculty development with a willingness to be flexible and reach potential participants using existing infrastructure, can facilitate assessors' engagement in a new culture of assessment. Interaction among participants during training sessions not only promotes learning but also contributes to community building. A leadership group responsible to oversee faculty development can ensure that the needs of stakeholders are addressed and that a change in assessment culture is sustained.

How Do Clerkship Students Use EPA Data? Illuminating Students' Perspectives as Partners in Programs of Assessment

INTRODUCTION

The implementation of programs of assessment based on Entrustable Professional Activities (EPAs) offers an opportunity for students to obtain unique data to guide their ongoing learning and development. Although authors have explored factors that contribute to trust-based decisions, learners' use of assessors' decisions about the level of supervision they need has not been fully investigated.

METHODS

In this study, we conducted semi-structured interviews of clerkship students who participated in the first year of our EPA program to determine how they interpret and use supervision ratings provided in EPA assessments. Content analysis was performed using concept-driven and open coding.

RESULTS

Nine interviews were completed. Twenty-two codes derived from previous work describing factors involved in trust decisions and 12 novel codes were applied to the interview text. Analyses revealed that students focus on written and verbal feedback from assessors more so than on supervision ratings. Axial coding revealed a temporal organization that categorized how students considered the data from EPA assessments. While factors before, during, and after an assessment affected students' use of information, the relationship between the student and the assessor had impact throughout.

CONCLUSIONS

Although students reported varying use of the supervision ratings, their perspectives about how assessors and students interact and/or partner before, during, and after assessments provide insights into the importance of an educational alliance in making a program of assessment meaningful and acceptable to learners.

Entrustable Professional Activity-Based Summative Performance Assessment in the Surgery Clerkship

OBJECTIVE

The objectives of this study were to 1) assess the performance Entrustable Professional Activities (EPAs) when integrated into the summative assessment of third-year medical students on the surgery clerkship and 2) to compare EPAs to traditional clinical performance assessment tools.

DESIGN

EPA assessments were collected prospectively from a minimum of 4 evaluators at the completion of each surgical clerkship rotation from November 2019 to June 2019. Overall EPA-based clinical performance scores were calculated as the sum of the mean EPA score from each evaluator. A rating of overall clinical performance called the clinical performance appraisal (CPA) was also collected. EPA ratings were compared to the CPA score, National Board of Medical Examiners exam score, objective structured clinical exam scores, and final clerkship grade.

SETTING

Northwestern Memorial Hospital, a tertiary care teaching institution in Chicago, IL.

RESULTS

Overall, 446 evaluations (111 students) were included in the analysis. The aggregate EPA scores ranged from 11.6-24.0 (mean 19.9 ± 2.0), and the CPA scores ranged from 4.4-9.0 (mean 7.6 ± 0.7). The variance among learners in EPA scores was significantly higher than CPA scores (p < 0.001). The aggregate EPA scores correlated well with CPA scores (Spearman's rho 0.803) but had lesser, positive correlations with the objective structured clinical exam (rho 0.153) and National Board of Medical Examiners (rho 0.265) scores. When all EPA scores were included in ordinal logistic regression, only EPA 6, oral presentation of patients, was independently associated with students' final grades (OR: 10.05, 95%CI 1.41-71.80; p = 0.02).

CONCLUSION

Integration of EPAs for use in clinical performance assessment of medical students is feasible within a surgery clerkship. Compared to a global clinical performance assessment, EPA-based assessment provided better discrimination of clinical performance among learners. Use of EPAs may better identify advanced learners and those that need additional time.

Revisiting the Core Entrustable Professional Activities for Entering Residency

The Core EPAs for Entering Residency Pilot project aimed to test the feasibility of implementing 13 entrustable professional activities (EPAs) at 10 U.S. medical schools and to gauge whether the use of the Core EPAs could improve graduates' performance early in residency. In this manuscript, the authors (members of the pilot institutions and Association of American Medical Colleges staff supporting the project evaluation) describe the schools' capacity to collect multimodal evidence about their students' performance in each of the Core EPAs and the ability of faculty committees to use those data to make decisions regarding learners' readiness for entrustment. In reviewing data for each of the Core EPAs, the authors reflected on how each activity performed as an EPA informed by how well it could be assessed and entrusted. For EPAs that did not perform well, the authors examined whether there are underlying practical and/or theoretical issues limiting its utility as a measure of student performance in medical school.

The critical role of infrastructure and organizational culture in implementing competency-based education and individualized pathways in undergraduate medical education

In 2010, several key works in medical education predicted the changes necessary to train modern physicians to meet current and future challenges in health care, including the standardization of learning outcomes paired with individualized learning processes. The reframing of a medical expert as a flexible, adaptive team member and change agent, effective within a larger system and responsive to the community's needs, requires a new approach to education: competency-based medical education (CBME). CBME is an outcomes-based developmental approach to ensuring each trainee's readiness to advance through stages of training and continue to grow in unsupervised practice. Implementation of CBME with fidelity is a complex and challenging endeavor, demanding a fundamental shift in organizational culture and investment in appropriate infrastructure. This paper outlines how member schools of the American Medical Association Accelerating Change in Medical Education Consortium developed and implemented CBME, including common challenges and successes. Critical supporting factors include adoption of the master adaptive learner construct, longitudinal views of learner development, coaching, and a supportive learning environment.

Working with entrustable professional activities in clinical education in undergraduate medical education: a scoping review

BACKGROUND

Entrustable professional activities (EPAs) are increasingly used in undergraduate medical education (UME). We conducted a scoping review to summarize the evidence for the use of EPAs in clinical rotations in UME.

METHODS

We searched multiple databases for scoping reviews based on the PRISMA guidelines for articles reporting qualitative and quantitative research, as well as conceptual and curriculum development reports, on EPAs in UME clinical rotations.

RESULTS

We identified 3309 records by searching through multiple databases. After the removal of duplicates, 1858 reports were screened. A total of 36 articles were used for data extraction. Of these, 47% reported on EPA and EPA-based curriculum development for clerkships, 50% reported on implementation strategies, and 53% reported on assessment methods and tools used in clerkships. Validity frameworks for developing EPAs in the context of clerkships were inconsistent. Several specialties reported feasible implementation strategies for EPA-based clerkship curricula, however, these required additional faculty time and resources. Limited exposure to clinical activities was identified as a barrier to relevant learning experiences. Educators used nationally defined, or specialty-specific EPAs, and a range of entrustability and supervision scales. We found only one study that used an empirical research approach for EPA assessment. One article reported on the earlier advancement of trainees from UME to graduate medical education based on summative entrustment decisions.

CONCLUSIONS

There is emerging evidence concerning how EPAs can be effectively introduced to clinical training in UME. Specialty-specific, nested EPAs with context-adapted, entrustment-supervision scales might be helpful in better leveraging their formative assessment potential.

You can have both: Coaching to promote clinical competency and professional identity formation

Coaching is a critical tool to guide student development of clinical competency and formation of professional identity in medicine, two inextricably linked concepts. Because progress toward clinical competence is linked to thinking, acting and feeling like a physician, a coach's knowledge about a learner's development of clinical skills is essential to promoting the learner's professional identity formation. A longitudinal coaching program provides a foundation for the formation of coach-learner relationships built on trust. Trusting relationships can moderate the risk and vulnerability inherent in a hierarchical medical education system and allow coaching conversations to focus on the promotion of self-regulated learning and fostering skills for life-long learning. Herein, we describe a comprehensive, longitudinal clinical coaching program for medical students designed to support learners' professional identify formation and effectively promote their emerging competence.