Considering the interdependence of clinical performance: implications for assessment and entrustment

Development and Validity of a Simulation Program for Assessment of Clinical Teaching Skills

Background: Teaching competence is expected of all intensivists, yet experts rarely supervise or assess trainees' teaching skills. Simulation offers an attractive solution. Objective: Develop and validate a simulation-based assessment of clinical teaching skills in pediatric critical care medicine (CCM). Methods: Participants were 128 pediatric CCM trainees, registered nurses, and respiratory therapists. Medical education experts used literature review and consensus to design three scenarios to assess teaching skills. Scenarios were piloted before use, and raters were trained. Teams completed one of three teaching scenarios, followed by a communication scenario. Raters were faculty members and trainees. Evidence for validity was collected and analyzed using Messick's unifying framework under the following domains: content, response processes, internal structure, relationship to other variables, and consequences of the assessment. Results: The scenarios and assessment tools were designed to capture the characteristics of a good teacher as described in the literature. Raters provided feedback that the tools were easy to use. Internal consistency of the scores measured by Cronbach's α was high. Rater agreement measured by interclass correlation was moderate for one of three scenarios. The relationship to other variables was investigated by correlating teaching scores with communication. Pearson's correlation was moderate for two of three scenarios. Consequences evidence was gathered using a retrospective self-assessed learning gain before versus after the training, which was significant for all scenarios. Conclusion: We developed a three-station simulation program for the assessment of teaching skills in pediatric CCM. The validity evidence collected is moderate, which indicates that it is effective for training and feedback on teaching skills.

Physician development through interprofessional workplace interactions: A critical review

INTRODUCTION

Increasingly, medical training aims to develop physicians who are competent collaborators. Although interprofessional interactions are inevitable elements of medical trainees' workplace learning experiences, the existing literature lacks a cohesive model to conceptualise the learning potential residing in these interactions.

METHODS

We conducted a critical review of the health professions and related educational literatures to generate an empirically and theoretically informed description of medical trainees' workplace interactions with other health professionals, including learning mechanisms and outcomes. Informed by Teunissen's conceptualisation of workplace learning, we highlight the individual, social and situated dimensions of learning from interprofessional workplace interactions.

RESULTS

Workplace interactions between medical trainees and other health professionals tend to be brief, spontaneous, informal and often implicit without the predefined educational goals and roles that structure trainees' relationships with physician supervisors. Yet they hold potential for developing trainees' knowledge and skills germane to the work of a physician as well as building their capacity for collaboration. Our review identified a spectrum of learning theories helpful for examining what and how trainees learn from these interactions. Self-regulated learning theories focus attention on how learning depends on trainees interpreting and judging the cues offered by other health professionals. Sociocultural frameworks including the zone of proximal development and legitimate peripheral participation emphasise the ways other health professionals support trainees in performing tasks at the border of their abilities and facilitate trainees' participation in clinical work. Both the landscapes of practice theory and cultural historical activity theory highlight the influence of surrounding social, cultural and material environments. These theories are unified into cohesive model and demonstrated through an illustrative example.

CONCLUSION

Interprofessional workplace interactions harbour a range of learning opportunities for medical trainees. Capitalising on their potential can contribute to training collaborative practice-ready physicians alongside traditional intra-professional interactions between physicians and merits future research.

Identifying emergency medicine program directors' expectations of competence upon entry into residency: Bridging the distance from the Association of American Medical Colleges Core Entrustable Professional Activities

Background

Residency program directors (PDs) frequently describe students as unprepared for the patient care responsibilities expected of them upon entry into residency. The Association of American Medical Colleges (AAMC) developed the Core Entrustable Professional Activities (Core EPAs) to address this concern by defining 13 tasks students should be able to do with minimal supervision upon graduation. However, the Core EPAs remain difficult for PDs to use due to their breadth and lack of granularity.

Methods

Using Delphi consensus methodology, we identified granular observable practice activities (OPAs) that PDs expect of entering interns derived from the Core EPAs. Twelve emergency medicine education experts drafted OPAs based on the Core EPAs and their associated core functions described in one-page schematics. A separate group of 12 PDs underwent three rounds of voting, and consensus for inclusion was set at 70%. Thematic analysis of comments discussing votes was performed using an inductive approach.

Results

A total of 321 OPAs were drafted and 127 adopted as expectations for entering interns based on the Core EPAs. The adopted OPAs were all general expectations; none were specialty-specific. Four main themes were identified from the comments: Schools are not responsible for specialty-specific training, PDs do not fully trust schools' assessments, supervision expectations of graduates should be lowered for higher-order EPAs, and the context in which the student performs a task and its associated complexity matter greatly in entrustment decisions.

Conclusions

PDs agree with the generalist focus of the AAMC Core EPAs and feel strongly that specialty training should be left to residency programs. They also have mechanisms in place to verify entrustment within their local context. Transparency in assessment and summative entrustment processes in UME may unify expectations. Finally, the granularity of OPAs may aid in a post-Match handover to further operationalize the EPAs and optimize the UME-to-GME transition.

Is "No News is Good News" Enough? A Thematic Analysis Exploring Clinical Reasoning Assessment in Pediatric Residency Programs

OBJECTIVE

Clinical reasoning (CR) includes numerous essential skills for clinicians, but how these skills are assessed in pediatric residency training is not well described. This study aimed to explore pediatric residency program leader perspectives on CR assessment and identification of trainee deficiencies in this area.

METHODS

Taking a social constructionist worldview, we conducted a thematic analysis of 20 semistructured interviews with pediatric residency program leaders. Interviews explored how pediatric residency programs assess CR and how deficiencies are identified. Recruitment and analysis continued iteratively until thematic sufficiency was reached. Member checking enhanced the trustworthiness of the results.

RESULTS

Participants noted a perceived lack of a shared mental model for CR assessment between program leaders and clinical supervisors. Four themes were generated to highlight CR assessment in pediatric residency programs: 1) Clinical supervisors escalate concerns about behaviors representing symptoms of CR deficits rather than diagnosing CR competency deficiencies and that CR assessment requires, 2) an outward display of autonomous decision-making, 3) psychologically safe environments for inquiry, and 4) longitudinal, individualized observation. Elements of pediatric residency programs that impede CR assessment were identified, including family-centered rounds and team-based clinical care.

CONCLUSIONS

This study identified key components necessary for CR assessment and barriers that may lead to missed identification of deficiencies. While no single solution can create an ideal environment for CR assessment, this study identifies elements for enhancing assessment opportunities for early identification of deficiencies.

A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness

INTRODUCTION

Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances.

METHODS

We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity.

RESULTS

Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity.

CONCLUSIONS

The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

Video-based peer assessment of collaborative teamwork in a large-scale interprofessional learning activity

BACKGROUND

The assessment of team performance within large-scale Interprofessional Learning (IPL) initiatives is an important but underexplored area. It is essential for demonstrating the effectiveness of collaborative learning outcomes in preparing students for professional practice. Using Kane's validity framework, we investigated whether peer assessment of student-produced videos depicting collaborative teamwork in an IPL activity was sufficiently valid for decision-making about team performance, and where the sources of error might lie to optimise future iterations of the assessment.

METHODS

A large cohort of health professional students (n = 1218) of 8 differing professions was divided into teams containing 5-6 students. Each team collaborated on producing a short video to evidence their management of one of 12 complex patient cases. Students from two other teams, who had worked on the same case, individually rated each video using a previously developed assessment scale. A generalisability study quantified sources of error that impacted the reliability of peer assessment of collaborative teamwork. A decision study modeled the impact of differing numbers of raters. A modified Angoff determined the pass/fail mark.

RESULTS

Within a large-scale learning activity, peer assessment of collaborative teamwork was reliable (G = 0.71) based on scoring by students from two teams (n = 10-12) for each video. The main sources of variation were the stringency and subjectivity of fellow student assessors. Whilst professions marked with differing stringency, and individual student assessors had different views of the quality of a particular video, none of that individual assessor variance was attributable to the assessors' profession. Teams performed similarly across the 12 cases overall, and no particular professions marked differently on any particular case.

CONCLUSION

A peer assessment of a student-produced video depicting interprofessional collaborative teamwork around the management of complex patient cases can be valid for decision-making about student team performance. Further refining marking rubrics and student assessor training could potentially modify assessor subjectivity. The impact of professions on assessing individual peers and the case-specificity of team performances in IPL settings need further exploration. This innovative approach to assessment offers a promising avenue for enhancing the measurement of collaborative learning outcomes in large-scale Interprofessional learning initiatives.

The Next Era of Assessment Within Medical Education: Exploring Intersections of Context and Implementation

In competency-based medical education (CBME), which is being embraced globally, the patient-learner-educator encounter occurs in a highly complex context which contributes to a wide range of assessment outcomes. Current and historical barriers to considering context in assessment include the existing post-positivist epistemological stance that values objectivity and validity evidence over the variability introduced by context. This is most evident in standardized testing. While always critical to medical education the impact of context on assessment is becoming more pronounced as many aspects of training diversify. This diversity includes an expanding interest beyond individual trainee competence to include the interdependency and collective nature of clinical competence and the growing awareness that medical education needs to be co-produced among a wider group of stakeholders. In this Eye Opener, we wish to consider: 1) How might we best account for the influence of context in the clinical competence assessment of individuals in medical education? and by doing so, 2) How could we usher in the next era of assessment that improves our ability to meet the dynamic needs of society and all its stakeholders? The purpose of this Eye Opener is thus two-fold. First, we conceptualize - from a variety of viewpoints, how we might address context in assessment of competence at the level of the individual learner. Second, we present recommendations that address how to approach implementation of a more contextualized competence assessment.

Validity in the Next Era of Assessment: Consequences, Social Impact, and Equity

Validity has long held a venerated place in education, leading some authors to refer to it as the "sine qua non" or "cardinal virtue" of assessment. And yet, validity has not held a fixed meaning; rather it has shifted in its definition and scope over time. In this Eye Opener, the authors explore if and how current conceptualizations of validity fit a next era of assessment that prioritizes patient care and learner equity. They posit that health profession education's conceptualization of validity will change in three related but distinct ways. First, consequences of assessment decisions will play a central role in validity arguments. Second, validity evidence regarding impacts of assessment on patients and society will be prioritized. Third, equity will be seen as part of validity rather than an unrelated concept. The authors argue that health professions education has the agency to change its ideology around validity, and to align with values that predominate the next era of assessment such as high-quality care and equity for learners and patients.

Assessing Team Performance: A Mixed-Methods Analysis Using Interprofessional in situ Simulation

Introduction

Optimizing the performance of emergency department (ED) teams impacts patient care, but the utility of current, team-based performance assessment tools to comprehensively measure this impact is underexplored. In this study we aimed to 1) evaluate ED team performance using current team-based assessment tools during an interprofessional in situ simulation and 2) identify characteristics of effective ED teams.

Methods

This mixed-methods study employed case study methodology based on a constructivist paradigm. Sixty-three eligible nurses, technicians, pharmacists, and postgraduate year 2-4 emergency medicine residents at a tertiary academic ED participated in a 10-minute in situ simulation of a critically ill patient. Participants self-rated performance using the Team Performance Observation Tool (TPOT) 2.0 and completed a brief demographic form. Two raters independently reviewed simulation videos and rated performance using the TPOT 2.0, Team Emergency Assessment Measure (TEAM), and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Following simulations, we conducted semi-structured interviews and focus groups with in situ participants. Transcripts were analyzed using thematic analysis.

Results

Eighteen team-based simulations took place between January-April 2021. Raters' scores were on the upper end of the tools for the TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27, IRR [inter-rater reliability] 0.47), TEAM (R1 3.89, SD 0.19; R2 3.58, SD 0.39, IRR 0.73), and Ottawa GRS (R1 6.6, SD 0.56; R2 6.2, SD 0.54, IRR 0.68). We identified six themes from our interview data: team member entrustment; interdependent energy; leadership tone; optimal communication; strategic staffing; and simulation empowering team performance.

Conclusion

Current team performance assessment tools insufficiently discriminate among high performing teams in the ED. Emergency department-specific assessments that capture features of entrustability, interdependent energy, and leadership tone may offer a more comprehensive way to assess an individual's contribution to a team's performance.

Evaluating the Strengths and Barriers of Competency-Based Education in the Health Professions

OBJECTIVE

This study aimed to define competency-based education (CBE) for pharmacy education and describe how strengths and barriers of CBE can support or hinder implementation.

FINDINGS

Sixty-five studies were included from a variety of health professions in order to define competency based pharmacy education (CBPE) and identify barriers and benefits from the learner, faculty, institution, and society perspectives. From the 7 identified thematic categories, a CBPE definition was developed: "Competency-based pharmacy education is an outcomes-based curricular model of an organized framework of competencies (knowledge, skills, attitudes) for pharmacists to meet health care and societal needs. This learner-centered curricular model aligns authentic teaching and learning strategies and assessment (emphasizing workplace assessment and quality feedback) while deemphasizing time."

SUMMARY

This article provides a definition of CBE for its application within pharmacy education. The strengths and barriers for CBE were elucidated from other health professions' education literature. Identified implementation strengths and barriers aid in the discussions on what will support or hinder the implementation of CBE in pharmacy education.

Having a Voice: Resident Perceptions of Supervision, Decision-Making and Patient Care Ownership

OBJECTIVE

Heightened resident supervision due to patient safety concerns is increasingly common in pediatrics and may leave residents with fewer opportunities for independent decision-making, a diminished sense of autonomy, and decreased engagement. This may ultimately threaten their development into competent clinicians. Understanding how pediatric residents experience supervision's influence on their involvement in decision-making, engagement in patient care, and learning is crucial to safeguard their transition to independent practice. In relation to supervision, our research investigated: 1) how residents navigated their involvement with clinical decision-making and 2) how opportunities to make clinical decisions influenced their engagement in patient care and learning.

METHODS

From 2019-2020, we recruited 38 pediatric residents from three different programs for a qualitative interview-based study. Through a constructivist stance, we explored clinical decision-making experiences and performed thematic analysis using an iterative and inductive process.

RESULTS

We identified three themes: 1) Residents perceived having autonomy when they had space to make independent decisions, regardless of supervisor's presence; 2) Patient care ownership resulted from having a voice in a variety of contributions to patient care; and 3) Supervisors' behaviors modulated patient care ownership and thereby residents' sense of feeling heard, their engagement in patient care, and their learning.

CONCLUSIONS

Our results suggest that focusing on patient care ownership may better fit with current learning environments than aiming for independence and autonomy. They provide insight on how, in the pediatric learning climate of enhanced supervision, supervisors can preserve resident engagement in patient care and learning by augmenting patient care ownership and ensuring residents have a voice.

Beyond the Surgical Safety Checklist: Using Intraoperative Handoff to Facilitate Team Situation Awareness in the OR

BACKGROUND

The surgical safety checklist (SSC) has been credited with improving team situation awareness (SA) in the operating room. Although the SSC may support team SA at the outset of the operative case, intraoperative handoff provides an opportunity for either SA breakdown or, more preferably, SA reinforcement. High-functioning surgical teams demonstrate a high level of continued SA, whereas teams deficient in SA are more likely to be affected by surgical errors and adverse events. To date, no interprofessional intraoperative tools exist to support team SA beyond the SSC.

METHODS

This study was divided into 2 phases. The first used qualitative methods to (1) characterize intraoperative handoff processes across surgery, nursing, anesthesia, and perfusion, and (2) identify cultural factors that shaped handoff practices. Data for phase one were collected over 38 observation days and 41 brief interviews. Phase 2, informed by phase 1, used a modified Delphi process to create a tool for use during intraoperative handoff. Data were analyzed iteratively.

RESULTS

Handoff practices were not standardized and rarely involved the entire team. In addition we uncovered cultural factors-specifically assumptions held by participants-that hindered team communication during handoff. Assumptions included: (1) team members are interchangeable, (2) trained individuals are able to determine when it is appropriate to handoff without consulting the OR team. Despite claims of improved teamwork resulting from the SSC, many participants held a fragmented view of the OR team, resulting in communication challenges during handoff. Findings from both phases of our study informed the development of multidisciplinary intraoperative handoff tools to facilitate shared team situation awareness and a shared mental model.

CONCLUSIONS

Intraoperative handoff occurs frequently, and offers the opportunity for either renewed or fractured team SA beyond the SSC.

Supportive and collaborative interdependence: Distinguishing residents' contributions within health care teams

INTRODUCTION

Individual assessments disregard team contributions, while team assessments disregard an individual's contributions. Interdependence has been put forth as a conceptual bridge between our educational traditions of assessing individual performance and our imminent challenge of assessing team-based performance without losing sight of the individual. The purpose of this study was to develop a more refined conceptualisation of interdependence to inform the creation of measures that can assess the interdependence of residents within health care teams.

METHODS

Following a constructivist grounded theory approach, we conducted 49 semi-structured interviews with various members of health care teams (e.g. physicians, nurses, pharmacists, social workers and patients) across two different clinical specialties-Emergency Medicine and Paediatrics-at two separate sites. Data collection and analysis occurred iteratively. Constant comparative inductive analysis was used, and coding consisted of three stages: initial, focused and theoretical.

RESULTS

We asked participants to reflect upon interdependence and describe how it exists in their clinical setting. All participants acknowledged the existence of interdependence, but they did not view it as part of a linear spectrum where interdependence becomes independence. Our analysis refined the conceptualisation of interdependence to include two types: supportive and collaborative. Supportive interdependence occurs within health care teams when one member demonstrates insufficient expertise to perform within their scope of practice. Collaborative interdependence, on the other hand, was not triggered by lack of experience/expertise within an individual's scope of practice, but rather recognition that patient care requires contributions from other team members.

CONCLUSION

In order to assess a team's collective performance without losing sight of the individual, we need to capture interdependent performances and characterise the nature of such interdependence. Moving away from a linear trajectory where independence is seen as the end goal can also help support efforts to measure an individual's competence as an interdependent member of a health care team.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs

BACKGROUND

Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions.

METHODS

Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses.

RESULTS

687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame.

CONCLUSIONS

Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs): A Scalable Approach for Linking Education to Patient Care

Competency-based medical education (CBME) is an outcomes-based approach to education and assessment that focuses on what competencies trainees need to learn in order to provide effective patient care. Despite this goal of providing quality patient care, trainees rarely receive measures of their clinical performance. This is problematic because defining a trainee's learning progression requires measuring their clinical performance. Traditional clinical performance measures (CPMs) are often met with skepticism from trainees given their poor individual-level attribution. Resident-sensitive quality measures (RSQMs) are attributable to individuals, but lack the expeditiousness needed to deliver timely feedback and can be difficult to automate at scale across programs. In this eye opener, the authors present a conceptual framework for a new type of measure - TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs) - attuned to both automation and trainee attribution as the next evolutionary step in linking education to patient care. TRACERs have five defining characteristics: meaningful (for patient care and trainees), attributable (sufficiently to the trainee of interest), automatable (minimal human input once fully implemented), scalable (across electronic health records [EHRs] and training environments), and real-time (amenable to formative educational feedback loops). Ideally, TRACERs optimize all five characteristics to the greatest degree possible. TRACERs are uniquely focused on measures of clinical performance that are captured in the EHR, whether routinely collected or generated using sophisticated analytics, and are intended to complement (not replace) other sources of assessment data. TRACERs have the potential to contribute to a national system of high-density, trainee-attributable, patient-centered outcome measures.

Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education

PURPOSE

These authors sought to define the new roles and competencies required of administrative staff and faculty in the age of CBME.

METHOD

A modified Delphi process was used to define the new CBME roles and competencies needed by faculty and administrative staff. We invited international experts in CBME (volunteers from the ICBME Collaborative email list), as well as faculty members and trainees identified via social media to help us determine the new competencies required of faculty and administrative staff in the CBME era.

RESULTS

Thirteen new roles were identified. The faculty-specific roles were: National Leader/Facilitator in CBME; Institutional/University lead for CBME; Assessment Process & Systems Designer; Local CBME Leads; CBME-specific Faculty Developers or Trainers; Competence Committee Chair; Competence Committee Faculty Member; Faculty Academic Coach/Advisor or Support Person; Frontline Assessor; Frontline Coach. The staff-specific roles were: Information Technology Lead; CBME Analytics/Data Support; Competence Committee Administrative Assistant.

CONCLUSIONS

The authors present a new set of faculty and staff roles that are relevant to the CBME context. While some of these new roles may be incorporated into existing roles, it may be prudent to examine how best to ensure that all of them are supported within all CBME contexts in some manner.

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents' Performance in Pediatric Emergency Medicine

PURPOSE

Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs).

METHOD

EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified.

RESULTS

274 residents and 1,891 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations ( r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay.

CONCLUSIONS

For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.

Action-project method: An approach to describing and studying goal-oriented joint actions

CONTEXT

Practicing health professionals and educators frequently act together in an interdependent or joint capacity to reach goals. Teaching or learning a new skill or engaging with patients in shared decision-making exemplifies this joint and goal-directed nature of Health Professions Education (HPE) and practice. However, building a robust understanding of the complexity of action, and joint action in particular, in HPE or patient care remains a challenge because of a limited number of methodologies available within HPE research.

METHODS

In this manuscript, we describe the Action-Project Method (A-PM) as a qualitative research approach that can be used to describe and understand goal-directed joint actions. A-PM is grounded in contextual action theory and is a methodology focussed on action as an object of study, as it is occurring. A-PM uses three distinct perspectives to understand goal-directed joint actions: observable behaviour, internal processes (i.e. reported thoughts and feelings) and the social meaning reflected in goals. Data collection in A-PM involves observations, interviews, recording of actions and a self-confrontation procedure-where participants watch video-recorded segments of action and reflect on their internal processes, describing what they were thinking or feeling as they were completing the action. Together, the rich data generated and the layered approach to analysis provide a means to better understand the joint actions embedded in complex systems and collaborative work. Furthermore, the participants are treated as equal partners within A-PM, ensuring data equity even when the research context includes hierarchical relationships.

DISCUSSION

Given increasing recognition to the importance of teamwork, relationships, interdependence, complex environments and centring patient or learner voices, A-PM is a valuable research approach for HPE. A-PM deepens our research arsenal with an approach that focusses on interdependent dyads or teams and provides a deeper understanding for how individuals engage together in goal-oriented actions.

What factors affect team members' evaluation of collaboration in medical teams?

Introduction

Perceived teamwork quality is associated with numerous work-related outcomes, ranging from team effectiveness to job satisfaction. This study explored what situational and stable factors affect the perceived quality of teamwork during a specific team task: when a medical team comprising a senior (supervisor) and a junior (trainee) physician diagnoses a patient.

Methods

During a field study in an emergency department, multisource data describing the patients, the diagnosing physicians, and the context were collected, including physicians' ratings of their teamwork. The relationships between perceived teamwork quality and situational (e.g., workload) and stable (e.g., seniority) factors were estimated in a latent regression model using the structural equation modeling (SEM) approach.

Results

Across the N = 495 patients included, SEM analyses revealed that the patient-specific case clarity and urgency influenced the perceived teamwork quality positively, whereas the work experience of the supervisor influenced the perceived teamwork quality of both supervisor and trainee negatively, albeit to different degrees.

Discussion

Our findings shed light on the complex underpinnings of perceived teamwork quality, a performance-relevant factor that may influence work and organizational effectiveness in healthcare settings.

Facilitating Residents' Understanding of Electronic Health Record Report Card Data Using Faculty Feedback and Coaching

PURPOSE

Feedback continues to present a challenge for competency-based medical education. Clear, consistent, and credible feedback is vital to supporting one's ongoing development, yet it can be difficult to gather clinical performance data about residents. This study sought to determine whether providing residents with electronic health record (EHR)-based report cards, as well as an opportunity to discuss these data with faculty trained using the R2C2 model, can help residents understand and interpret their clinical performance metrics.

METHOD

Using action research methodology, the author team collected EHR data from July 2017 to February 2020, for all residents (n = 21) in one 5-year Emergency Medicine program and created personalized report cards for each resident. During October 6-17, 2020, 8 out of 17 eligible residents agreed to have their feedback conversations recorded and participate in a subsequent interview with a nonphysician member of the research team. Data were analyzed using thematic analysis, and the authors used inductive analysis to identify themes in the data.

RESULTS

In analyzing both the feedback conversations as well as the individual interviews with faculty and residents, the authors identified 2 main themes: (1) Reactions and responses to receiving personalized EHR data and (2) The value of EHR data for assessment and feedback purposes. All participants believed that EHR data metrics are useful for prompting self-reflection, and many pointed to their utility in providing suggestions for actionable changes in their clinical practice. For faculty, having a tool through which underperforming residents can be shown "objective" data about their clinical performance helps underscore the need for improvement, particularly when residents are resistant.

CONCLUSIONS

The EHR is a valuable source of educational data, and this study demonstrates one of the many thoughtful ways it can be used for assessment and feedback purposes.

Creating space for leadership education in undergraduate medical education in Canada

The need for effective leadership by physicians is clear, yet the design/delivery of curricula, and assessment of leadership competencies, in Undergraduate Medical Education (UGME) continues to need work. In reappraising their UGME assessment strategies, the Medical Council of Canada (MCC) invited position papers across diverse lenses, including the CanMEDS Intrinsic Roles. This article is foundational work derived from the report on leadership assessment to the MCC. Using Kern's Model of Curriculum development as a guide, we reviewed the landscape of Canadian UGME leadership education through an environmental scan of the published and grey literature, Canadian leadership frameworks and resources, and consultation with learner and faculty leadership. Leadership education across programs was highly variable and learners were often unaware of available opportunities. In response, we have suggested processes for curricular development, including strategies for key content, teaching and assessment, and program evaluation considerations. Leadership education cannot remain another checkbox on a list of UGME experiences. Such training necessitates focused attention and investment to foster ongoing identity formation toward becoming a good doctor.

Integrating training, practice, and reflection within a new model for Canadian medical licensure: a concept paper prepared for the Medical Council of Canada

In 2020 the Medical Council of Canada created a task force to make recommendations on the modernization of its practices for granting licensure to medical trainees. This task force solicited papers on this topic from subject matter experts. As outlined within this Concept Paper, our proposal would shift licensure away from the traditional focus on high-stakes summative exams in a way that integrates training, clinical practice, and reflection. Specifically, we propose a model of graduated licensure that would have three stages including: a trainee license for trainees that have demonstrated adequate medical knowledge to begin training as a closely supervised resident, a transition to practice license for trainees that have compiled a reflective educational portfolio demonstrating the clinical competence required to begin independent practice with limitations and support, and a fully independent license for unsupervised practice for attendings that have demonstrated competence through a reflective portfolio of clinical analytics. This proposal was reviewed by a diverse group of 30 trainees, practitioners, and administrators in medical education. Their feedback was analyzed and summarized to provide an overview of the likely reception that this proposal would receive from the medical education community.

The OSSE: Development and validation of an "Objective Structured Supervision Evaluation"

Background

Trainee supervision and teaching are distinct skills that both require faculty physician competence to ensure patient safety. No standard approach exists to teach physician supervisory competence, resulting in variable trainee oversight and safety threats. The Objective Structured Teaching Evaluation (OSTE) does not adequately incorporate the specific skills required for effective supervision. To address this continuing medical education gap, the authors aimed to develop and identify validity evidence for an "Objective Structured Supervision Evaluation" (OSSE) for attending physicians, conceptually modeled on the historic OSTE.

Methods

An expert panel used an iterative process to create an OSSE instrument, which was a checklist of key supervision items to be evaluated during a simulated endotracheal intubation scenario. Three trained "standardized residents" scored faculty participants' performance using the instrument. Validity testing modeled a contemporary approach using Kane's framework. Participants underwent simulation-based mastery learning (SBML) with deliberate practice until meeting a minimum passing standard (MPS).

Results

The final instrument contained 19 items, including three global rating measures. Testing domains included supervision climate, participant control of patient care, trainee evaluation, instructional skills, case-specific measures, and overall supervisor rating. Reliability of the assessment tool was excellent (ICC range 0.84-0.89). The assessment tool had good internal consistency (Cronbach's α = 0.813). Out of 24 faculty participants, 17 (70.8%) met the MPS on initial assessment. All met the MPS after SBML and average score increased by 19.5% (95% CI of the difference 10.3%-28.8%, p = 0.002).

Conclusions

The OSSE is a promising tool to assess faculty supervision performance Further study should evaluate the effect of OSSE training on supervision in the clinical environment which may have measurable changes in patient‐centered outcomes and resident teaching

Perceived positive social interdependence in online versus face-to-face team-based learning styles of collaborative learning: a randomized, controlled, mixed-methods study

BACKGROUND

Collaborative learning is a group learning approach in which positive social interdependence within a group is key to better learning performance and future attitudes toward team practice. Recent attempts to replace a face-to-face environment with an online one have been developed using information communication technology. However, this raises the concern that online collaborative learning (OCL) may reduce positive social interdependence. Therefore, this study aimed to compare the degree of social interdependence in OCL with face-to-face environments and clarify aspects that affect social interdependence in OCL.

METHODS

We conducted a crossover study comparing online and face-to-face collaborative learning environments in a clinical reasoning class using team-based learning for medical students (n = 124) in 2021. The participants were randomly assigned to two cohorts: Cohort A began in an online environment, while Cohort B began in a face-to-face environment. At the study's midpoint, the two cohorts exchanged the environments as a washout. The participants completed surveys using the social interdependence in collaborative learning scale (SOCS) to measure their perceived positive social interdependence before and after the class. Changes in the mean SOCS scores were compared using paired t-tests. Qualitative data related to the characteristics of the online environment were obtained from the focus groups and coded using thematic analysis.

RESULTS

The matched-pair tests of SOCS showed significant progression between pre- and post-program scores in the online and face-to-face groups. There were no significant differences in overall SOCS scores between the two groups. Sub-analysis by subcategory showed significant improvement in boundary (discontinuities among individuals) and means interdependence (resources, roles, and tasks) in both groups, but outcome interdependence (goals and rewards) improved significantly only in the online group. Qualitative analysis revealed four major themes affecting social interdependence in OCL: communication, task-sharing process, perception of other groups, and working facilities.

CONCLUSIONS

There is a difference in the communication styles of students in face-to-face and online environments, and these various influences equalize the social interdependence in a face-to-face and online environment.

Who's on your team? Specialty identity and inter-physician conflict during admissions

PURPOSE

Despite the implementation of professionalism curricula and standardised communication tools, inter-physician conflict persists. In particular, the interface between emergency medicine (EM) and internal medicine (IM) has long been recognised as a source of conflict. The social nuances of this conflict remain underexplored, limiting educators' ability to comprehensively address these issues in the clinical learning environment. Thus, the authors explored EM and IM physicians' experiences with negotiating hospital admissions to better understand the social dynamics that contribute to inter-physician conflict and provide foundational guidance for communication best practices.

METHODS

Using a constructivist grounded theory (CGT) approach, the authors conducted 18 semi-structured interviews between June and October 2020 with EM and IM physicians involved in conversations regarding admissions (CRAs). They asked participants to describe the social exchanges that influenced these conversations and to reflect on their experiences with inter-physician conflict. Data collection and analysis occurred iteratively. The relationships between the codes were discussed by the research team with the goal of developing conceptual connections between the emergent themes.

RESULTS

Participants described how their approaches to CRAs were shaped by their specialty identity, and how allegiance to members of their group contributed to interpersonal conflict. This conflict was further promoted by a mutual sense of disempowerment within the organisation, misaligned expectations, and a desire to promote their group's prerogatives. Conflict was mitigated when patient care experiences fostered cross-specialty team formation and collaboration that dissolved traditional group boundaries.

CONCLUSIONS

Conflict between EM and IM physicians during CRAs was primed by participants' specialty identities, their power struggles within the broader organisation, and their sense of duty to their own specialty. However, formation of collaborative inter-specialty physician teams and expansion of identity to include colleagues from other specialties can mitigate inter-physician conflict.

Assessing trainee performance: ensuring learner control, supporting development, and maximizing assessment moments

In this article, the authors provide practical guidance for frontline supervisors' efforts to assess trainee performance. They focus on three areas. First, they argue the importance of promoting learner control in the assessment process, noting that providing learners agency and control can shift the stakes of assessment from high to low and promote a safe environment that facilitates learning. Second, they posit that assessment should be used to support continued development by promoting a relational partnership between trainees and supervisors. This partnership allows supervisors to reinforce desirable aspects of performance, provide real-time support for deficient areas of performance, and sequence learning with the appropriate amount of scaffolding to push trainees from competence (what they can do alone) to capability (what they are able to do with support). Finally, they advocate the importance of optimizing the use of written comments and direct observation while also recognizing that performance is interdependent in efforts to maximize assessment moments.Conclusion: Using best practices in trainee assessment can help trainees take next steps in their development in a learner-centered partnership with clinical supervisors. What is Known: • Many pediatricians are asked to assess the performance of medical students and residents they work with but few have received formal training in assessment. What is New: • This article presents evidence-based best practices for assessing trainees, including giving trainees agency in the assessment process and focusing on helping trainees take next steps in their development.

A Survey on Changes to the Canadian Anatomical Pathology Certification Examination Due to Coronavirus Disease 2019 and Implications for Competency-Based Medical Education

The coronavirus disease 2019 pandemic resulted in a dramatic change in the Royal College of Physicians and Surgeons of Canada assessment process through elimination of the oral and practical components of the 2020 Anatomical Pathology examination. Our study sought to determine stakeholder opinions and experiences on these changes in the context of the 2019 implementation of competency-based medical education. Surveys were designed for residents and practicing pathologists. In total, 57 residents (estimated response rate 29%) and 185 pathologists (estimated response rate 19%) participated across Canada; 67% of pathologists disagreed with the 2020 Royal College examination changes, compared with 30% for residents (P = <.00001). When asked whether the Royal College examination should be eliminated, 95% of pathologists indicated they would be against this, compared to only 34% of residents (P = <.00001). Perceptions on changes to and importance of different components of assessment in competency-based medical education were similar between pathologists and residents, with participants perceiving assessment practices to have changed fairly little since its implementation, with the exception of more frequent feedback. Analysis of narrative comments identified several common themes around assessment, including the need for objectivity and standardization and the problem of failure-to-fail. However, residents identified numerous elements of their performance that can be assessed only through longitudinal evaluation. Pathologists, on the other hand, tended to view these aspects of performance as laden with bias. Our results will hopefully help guide future innovation in assessment by characterizing different stakeholder perspectives on key issues in medical education.

What the COVID-19 Pandemic Can Teach Health Professionals About Continuing Professional Development

The world's health care providers have realized that being agile in their thinking and growth in times of rapid change is paramount and that continuing education can be a key facet of the future of health care. As the world recovers from the COVID-19 pandemic, educators at academic health centers are faced with a crucial question: How can continuing professional development (CPD) within teams and health systems be improved so that health care providers will be ready for the next disruption? How can new information about the next disruption be collected and disseminated so that interprofessional teams will be able to effectively and efficiently manage a new disease, new information, or new procedures and keep themselves safe? Unlike undergraduate and graduate/postgraduate education, CPD does not always have an identified educational home and has had uneven and limited innovation during the pandemic. In this commentary, the authors explore the barriers to change in this sector and propose 4 principles that may serve to guide a way forward: identifying a home for interprofessional continuing education at academic health centers, improving workplace-based learning, enhancing assessment for individuals within health care teams, and creating a culture of continuous learning that promotes population health.

Comparing Entrustment and Competence: An Exploratory Look at Performance-Relevant Information in the Final Year of a Veterinary Program

Workplace-based assessments and entrustment scales have two primary goals: providing formative information to assist students with future learning; and, determining if and when learners are ready for safe, independent practice. To date, there has not been an evaluation of the relationship between these performance-relevant information pieces in veterinary medicine. This study collected quantitative and qualitative data from a single cohort of final-year students (n = 27) across in-training evaluation reports (ITERs) and entrustment scales in a distributed veterinary hospital environment. Here we compare progression in scoring and performance within and across student, within and across method of assessment, over time. Narrative comments were quantified using the Completed Clinical Evaluation Report Rating (CCERR) instrument to assess quality of written comments. Preliminary evidence suggests that we may be capturing different aspects of performance using these two different methods. Specifically, entrustment scale scores significantly increased over time, while ITER scores did not. Typically, comments on entrustment scale scores were more learner specific, longer, and used more of a coaching voice. Longitudinal evaluation of learner performance is important for learning and demonstration of competence; however, the method of data collection could influence how feedback is structured and how performance is ultimately judged.

A scoping review of approaches for measuring 'interdependent' collaborative performances

INTRODUCTION

Individual assessment disregards the team aspect of clinical work. Team assessment collapses the individual into the group. Neither is sufficient for medical education, where measures need to attend to the individual while also accounting for interactions with others. Valid and reliable measures of interdependence are critical within medical education given the collaborative manner in which patient care is provided. Medical education currently lacks a consistent approach to measuring the performance between individuals working together as part of larger healthcare team. This review's objective was to identify existing approaches to measuring this interdependence.

METHODS

Following Arksey & O'Malley's methodology, we conducted a scoping review in 2018 and updated it to 2020. A search strategy involving five databases located >12 000 citations. At least two reviewers independently screened titles and abstracts, screened full texts (n = 161) and performed data extraction on twenty-seven included articles. Interviews were also conducted with key informants to check if any literature was missing and assess that our interpretations made sense.

RESULTS

Eighteen of the twenty-seven articles were empirical; nine conceptual with an empirical illustration. Eighteen were quantitative; nine used mixed methods. The articles spanned five disciplines and various application contexts, from online learning to sports performance. Only two of the included articles were from the field of Medical Education. The articles conceptualised interdependence of a group, using theoretical constructs such as collaboration synergy; of a network, using constructs such as degree centrality; and of a dyad, using constructs such as synchrony. Both descriptive (eg social network analysis) and inferential (eg multi-level modelling) approaches were described.

CONCLUSION

Efforts to measure interdependence are scarce and scattered across disciplines. Multiple theoretical concepts and inconsistent terminology may be limiting programmatic work. This review motivates the need for further study of measurement techniques, particularly those combining multiple approaches, to capture interdependence in medical education.

Feasibility, and validity aspects of Entrustable Professional Activity (EPA)-based assessment in general practice training

INTRODUCTION

Entrustable Professional Activities (EPAs) are developed to support the practical application of competency frameworks in postgraduate medical education (PGME) programmes. EPAs are used for the assessment of the trainees' competence development, which takes place by means of an entrustment decision, aiming to stimulate learning and independent practice in trainees. In this pilot study, we explore the feasibility and validity of EPA-based assessment in a General Practice (GP) training programme.

METHODS

We used questionnaires to evaluate trainers' and trainees' experiences with the use of six EPAs for trainee learning, assessment and independent practice at the Out-of-Hours GP Center. Data were analysed quantitatively and qualitatively. Additionally, we examined the inter-item correlation between scores on EPA-based assessment and competency-based assessment using Spearman's Rho.

RESULTS

EPA-based assessment provided opportunities for giving concrete feedback and substantiating competency-based assessment. No consistent correlation between EPA-based assessment and competency-based assessment could be detected. Only later in the course of the training programme a correlation was found between the EPA scores and the degree of independence of trainees.

DISCUSSION

Results of this pilot study confirm the theories behind EPAs, as well as earlier research on EPAs in the workplace regarding trainee learning, assessment and independent practice. An important limitation of this study was the COVID-19 pandemic, as it influenced the results through reduced inclusion and follow-up, and through the impact on the workplace and trainee learning possibilities. Further research is needed to determine how EPAs support independent practice of trainees, as well as the assessment of trainee competency development.

Distant and Hidden Figures: Foregrounding Patients in the Development, Content, and Implementation of Entrustable Professional Activities

Entrustable professional activities (EPAs) describe activities that qualified professionals must be able to perform to deliver safe and effective care to patients. The entrustable aspect of EPAs can be used to assess learners through documentation of entrustment decisions, while the professional activity aspect can be used to map curricula. When used as an assessment framework, the entrustment decisions reflect supervisory judgments that combine trainees' relational autonomy and patient safety considerations. Thus, the design of EPAs incorporates the supervisor, trainee, and patient in a way that uniquely offers a link between educational outcomes and patient outcomes. However, achieving a patient-centered approach to education amidst both curricular and assessment obligations, educational and patient outcomes, and a supervisor-trainee-patient triad is not simple nor guaranteed. As medical educators continue to advance EPAs as part of their approach to competency-based medical education, the authors share a critical discussion of how patients are currently positioned in EPAs. In this article, the authors examine EPAs and discuss how their development, content, and implementation can result in emphasizing the trainee and/or supervisor while unintentionally distancing or hiding the patient. They consider creative possibilities for how EPAs might better integrate the patient as finding ways to better foreground the patient in EPAs holds promise for aligning educational outcomes and patient outcomes.

Capturing outcomes of competency-based medical education: The call and the challenge

There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

From Utopia Through Dystopia: Charting a Course for Learning Analytics in Competency-Based Medical Education

The transition to the assessment of entrustable professional activities as part of competency-based medical education (CBME) has substantially increased the number of assessments completed on each trainee. Many CBME programs are having difficulty synthesizing the increased amount of assessment data. Learning analytics are a way of addressing this by systematically drawing inferences from large datasets to support trainee learning, faculty development, and program evaluation. Early work in this field has tended to emphasize the significant potential of analytics in medical education. However, concerns have been raised regarding data security, data ownership, validity, and other issues that could transform these dreams into nightmares. In this paper, the authors explore these contrasting perspectives by alternately describing utopian and dystopian futures for learning analytics within CBME. Seeing learning analytics as an important way to maximize the value of CBME assessment data for organizational development, they argue that their implementation should continue within the guidance of an ethical framework.

Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade

To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

Workplace-based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

OBJECTIVE

In the era of competency-based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace-based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees' WBAs in emergency medicine (EM).

METHODS

The authors conducted a scoping review in the style of Arksey and O'Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.

RESULTS

Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone.

CONCLUSION

There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.

Comparison of quantitative internal and external measures of performance for trainees in cytopathology fellowships

INTRODUCTION

Cytopathology fellowships need measures to assess performance of fellows. We sought to compare several internal quantitative assessment metrics in our fellowship with external metrics, such as performance on the American Society of Cytopathology (ASC) Progressive Evaluation of Competency (PEC) examination and United States Medical Licensing Examination (USMLE).

METHODS

Quantitative parameters generated from our laboratory information system (LIS) on cytopathology fellows were evaluated over 6 years, including case volume and diagnostic discrepancies, in addition to ASC PEC and USMLE scores. For discrepancy reports, interpretations made by the fellow were compared with that of the cytopathologist, and classified as none (concordant), minor (<2-levels) or major (≥2-levels).

RESULTS

We evaluated internal and external metrics on 13 fellows over 6 years. The program average diagnostic concordance rate was 89.9%, with an average major discrepancy rate of 1.5%, and an average monthly case volume of 260 cases. More fellows with above-average ASC PEC performance showed above-average concordant diagnoses and lower case volume, while below-average PEC scores were seen more often with higher major discrepancy rates. More fellows with above-average USMLE scores had higher case volumes, while low USMLE scores showed a trend towards higher major discrepancy rates.

CONCLUSION

Our fellowship program has used a variety of internal and external measures of performance for cytopathology fellows. Although the findings show no statistically significant finding correlating performance, these quantitative parameters generated from our LIS were helpful to identify areas of improvement, facilitate comparison to peers, and provide case volume documentation.

"EMERGing" Electronic Health Record Data Metrics: Insights and Implications for Assessing Residents' Clinical Performance in Emergency Medicine

OBJECTIVES

Competency-based medical education requires that residents are provided with frequent opportunities to demonstrate competence as well as receive effective feedback about their clinical performance. To meet this goal, we investigated how data collected by the electronic health record (EHR) might be used to assess emergency medicine (EM) residents' independent and interdependent clinical performance and how such information could be represented in an EM resident report card.

METHODS

Following constructivist grounded theory methodology, individual semistructured interviews were conducted in 2017 with 10 EM faculty and 11 EM residents across all 5 postgraduate years. In addition to open-ended questions, participants were presented with an emerging list of EM practice metrics and asked to comment on how valuable each would be in assessing resident performance. Additionally, we asked participants the extent to which each metric captured independent or interdependent performance. Data collection and analysis were iterative; analysis employed constant comparative inductive methods.

RESULTS

Participants refined and eliminated metrics as well as added new metrics specific to the assessment of EM residents (e.g., time between signup and first orders). These clinical practice metrics based on data from our EHR database were organized along a spectrum of independent/interdependent performance. We conclude with discussions about the relationship among these metrics, issues in interpretation, and implications of using EHR for assessment purposes.

CONCLUSIONS

Our findings document a systematic approach for developing EM resident assessments, based on EHR data, which incorporate the perspectives of both clinical faculty and residents. Our work has important implications for capturing residents' contributions to clinical performances and distinguishing between independent and interdependent metrics in collaborative workplace-based settings.

A Descriptive Analysis of the Cumulative Experiences of Emergency Medicine Residents in the Pediatric Emergency Department

OBJECTIVES

Most children seeking emergency care are evaluated in general emergency departments (EDs). The cumulative pediatric clinical experiences of emergency medicine (EM) residents are largely unknown. This study examined EM resident pediatric clinical experience through the lens of the Accreditation Council for Graduate Medical Education requirements and the Model of the Clinical Practice of Emergency Medicine.

METHODS

Retrospective, observational study of the cumulative clinical experience of two classes of EM residents from a 4-year training program at two pediatric EDs of a quaternary care pediatric center. A database of resident patient encounters was generated from the electronic medical record. Experiences classified included: diagnosis categories per the Model of the Clinical Practice of Emergency Medicine, procedures, and resuscitations. Results were stratified by age, acuity, and disposition.

RESULTS

Twenty-five EM residents evaluated 17,642 patients (median = 723). Most patients (73.5%) were emergent acuity (Emergency Severity Index triage level 2 or 3 or non-intensive care admission); 2% were critical. Residents participated in 598 (median = 22) medical resuscitations and 483 (median = 19) trauma resuscitations. Minor procedures (e.g., laceration repair) were commonly performed; critical procedures (e.g., intubation) were rare. Exposure to neonates was infrequent and pediatric deaths were rare. Abdominal pain (5.7%), asthma exacerbation (4.6%), and fever (3.8%) were the most common diagnoses.

CONCLUSIONS

Emergency medicine residents encountered a wide array of pediatric diagnoses throughout training and performed a substantial number of common pediatric procedures. Exposure to critical acuity and procedures, neonatal pathology, and certain pediatric-specific diagnoses, such as congenital heart disease, was limited despite training in a large, quaternary care children's hospital. Curriculum development and collaboration should focus on these areas.

"Doing it Right" Overnight: a Multi-perspective Qualitative Study Exploring Senior Medical Resident Overnight Call

BACKGROUND

Competency-based medical education (CBME) requires the development of workplace-based assessment tools that are grounded in authentic clinical work. Developing such tools, however, requires a deep understanding of the underlying facets of the competencies being assessed. Gaining this understanding remains challenging in contexts where performance is not readily visible to supervisors such as the senior medical resident (SMR) on-call role in internal medicine.

OBJECTIVE

This study draws on the perspectives of healthcare professionals with whom the SMR interacts with overnight to generate insights into the different components of on-call SMR practice and the range of ways SMRs effectively and less effectively enact these.

APPROACH

We used a constructivist grounded theory (CGT) approach to examine variation in how on-call SMRs carry out their role overnight.

PARTICIPANTS

Six medical students, five junior residents, five internal medicine attending physicians, five emergency physicians, and three emergency nurses conducted observations of their on-call interactions with SMRs. Participants were then interviewed and asked to elaborate on their observations as well as provide comparative reflections on the practices of past SMRs they worked with.

KEY RESULTS

Strong collaboration and organizational skills were identified as critical components to effectively being the on-call SMR. Perceived weaker SMRs, while potentially also having issues with clinical skills, stood out more when they could not effectively manage the realities of collaboration in a busy workplace.

CONCLUSION

What consistently differentiated a perceived effective SMR from a less effective SMR was someone who was equipped to manage the realities of interprofessional collaboration in a busy workplace. Our study invites medical educators to consider what residents, particularly those in more complex roles, need to receive feedback on to support their development as physicians. It is our intention that the findings be used to inform the ways programs approach teaching, assessment, and the provision of feedback.

Feedback That Helps Trainees Learn to Practice Without Supervision

Feedback pedagogies and research tend to focus on immediate corrective actions rather than learning for the longer term. This approach means that feedback may not support trainees who are managing complex, competing, and ambiguous practice situations, often with limited supervision. There is an opportunity to consider how feedback can help medical trainees sustain their own development into the future, including when they have completed formal training. This article explores how feedback pedagogies can facilitate medical trainees' abilities to develop challenging aspects of practice across multiple clinical environments to eventually practice without supervision. From a sociocultural perspective, clinical training takes place within a practice curriculum; each clinical environment offers varying opportunities, which the trainees may choose to engage with. The authors propose feedback as an interpersonal process that helps trainees make sense of both formal training requirements and performance relevant information, including workplace cues such as patient outcomes or colleagues' comments, found within any practice curriculum. A significant pedagogic strategy may be to develop trainees' evaluative judgment or their capability to identify and appraise the qualities of good practice in both themselves and others. In this way, feedback processes may help trainees surmount complex situations and progressively gain independence from supervision.

Scut to Scholarship: Can Operative Notes be Educationally Useful?

OBJECTIVE

Efforts to implement competency-based medical education require new sources of workplace-based evidence of growth in learning. We used qualitative analysis of operative notes to explore procedural variation in a simple surgical procedure.

DESIGN

We used a grounded theory-based mixed methods approach to depict intersurgeon procedural variation. Our grounded theory approach to analysis included follow up interviews with surgeons and residents to probe their understandings of the reasons for variation in the dictated notes and the current and potential utility of operative notes as a reliable source of data for learning and assessment.

SETTING

Publicly funded tertiary care otolaryngology-head & neck surgery residency program in Ontario, Canada PARTICIPANTS: Using maximum variability sampling, all surgeons performing tonsillectomy in the department (n = 6) contributed operative notes from 65 tonsillectomies, 5 intraoperative observations, and 4 semi-structured interviews. An additional 3 residents from various levels of training contributed semistructured interviews.

RESULTS

Intersurgeon procedural variations persist even in simple surgical procedures such as tonsillectomy. Operative notes appear to capture procedural variations in a limited way. Surgeons and resident make informal educational use of the clerical work of writing and assessing operative notes, but optimization will be required to shift such hidden work into the formal educational domain.

CONCLUSIONS

The implementation of competency-based medical education requires surgical educators to both eliminate low-yield tasks for learning and to find new opportunities for multiple low-stakes assessment. Analysis of operative notes may become a high-yield strategy for learning and assessment if residents and surgeons are coached to use operative notes more reliably and efficiently.

Elucidating system-level interdependence in electronic health record data: What are the ramifications for trainee assessment?

CONTEXT

The electronic health record (EHR) has been identified as a potential site for gathering data about trainees' clinical performance, but these data are not collected or organised for this purpose. Therefore, a careful and rigorous approach is required to explore how EHR data could be meaningfully used for assessment purposes. The purpose of this study was to identify EHR performance metrics that represent both the independent and interdependent clinical performance of emergency medicine (EM) trainees and explore how they might be meaningfully used for assessment and feedback.

METHODS

Using constructivist grounded theory, we conducted 21 semi-structured interviews with EM faculty members and residents. Participants were asked to identify the clinical actions of trainees that would be valuable for assessment and feedback and describe how those activities are represented in the EHR. Data collection and analysis, which consisted of three stages of coding, occurred iteratively.

RESULTS

When faculty members and trainees in EM were asked to reflect on the usefulness of using EHR performance metrics for resident assessment and feedback they expressed both widespread support for the idea in principle and hesitation that aspects of clinical performance captured in the data would not be representative of residents' individual performance, but would rather reflect their interdependence with other team members and the systems in which they work. We highlight three categorisations of system-level interdependence - medical directives, technological systems and organisational systems - identified by our participants, and discuss strategies participants employed to navigate these forms of interdependence within the health care system.

CONCLUSIONS

System-level interdependence shapes physicians' performances, and yet, this impact is rarely corrected for or noted within clinical performance data. Educators have a responsibility to recognise system-level interdependence when teaching and consider system-level interdependence when assessing the performance of trainees in order to most effectively and fairly utilise the EHR as a source of assessment data.

Information Overload in Emergency Medicine Physicians: A Multisite Case Study Exploring the Causes, Impact, and Solutions in Four North England National Health Service Trusts

Background

Information overload is affecting modern society now more than ever because of the wide and increasing distribution of digital technologies. Social media, emails, and online communications among others infuse a sense of urgency as information must be read, produced, and exchanged almost instantaneously. Emergency medicine is a medical specialty that is particularly affected by information overload with consequences on patient care that are difficult to quantify and address. Understanding the current causes of medical information overload, their impact on patient care, and strategies to handle the inflow of constant information is crucial to alleviating stress and anxiety that is already crippling the profession.

Objective

This study aims to identify and evaluate the main causes and sources of medical information overload, as experienced by emergency medicine physicians in selected National Health Service (NHS) trusts in the United Kingdom.

Methods

This study used a quantitative, survey-based data collection approach including close- and open-ended questions. A web-based survey was distributed to emergency physicians to assess the impact of medical information overload on their jobs. In total, 101 valid responses were collected from 4 NHS trusts in north England. Descriptive statistics, principal component analysis, independent sample two-tailed t tests, and one-way between-group analysis of variance with post hoc tests were performed on the data. Open-ended questions were analyzed using thematic analysis to identify key topics.

Results

The vast majority of respondents agreed that information overload is a serious issue in emergency medicine, and it increases with time. The always available culture (mean 5.40, SD 1.56), email handling (mean 4.86, SD 1.80), and multidisciplinary communications (mean 4.51, SD 1.61) are the 3 main reasons leading to information overload. Due to this, emergency physicians experience guideline fatigue, stress and tension, longer working hours, and impaired decision making, among other issues. Aspects of information overload are also reported to have different impacts on physicians depending on demographic factors such as age, years spent in emergency medicine, and level of employment.

Conclusions

There is a serious concern regarding information overload in emergency medicine. Participants identified a considerable number of daily causes affecting their job, particularly the traditional culture of emergency departments being always available on the ward, exacerbated by email and other forms of communication necessary to maintain optimal, evidence-based practice standards. However, not all information is unwelcome, as physicians also need to stay updated with the latest guidelines on conditions and treatment, and communicate with larger medical teams to provide quality care.