The Assessment Burden in Competency-Based Medical Education: How Programs Are Adapting

Assessment of anaesthesia trainees using performance metrics derived from electronic health records: a longitudinal cohort analysis

BACKGROUND

The development of competencies in anaesthesia residents is assessed by direct observation of their performance and written and oral examinations. Little is known about how residents' competencies are reflected by objective data in anaesthetic records. We investigated whether performance metrics derived from electronic anaesthesia records are associated with anaesthesia residents' progress of training and European written examination timing and results.

METHODS

We recruited 46 anaesthesia providers who trained at the Vienna University Hospital between September 2013 and June 2021, and we reviewed the anaesthesia records of all cases they managed during the study period. We derived six performance measures based on perioperative data routinely collected: intraoperative hypotension and hypothermia, glycaemic control, postoperative nausea and vomiting prevention, lung-protective ventilation, and postoperative kidney injury. We evaluated the association between these quality metrics with training level and written exam completion status in anaesthesia residents after adjusting for patient and surgical factors.

RESULTS

We found a statistically significant association between the level of training and most performance measures. The probability of preventing intraoperative hypotension increased (OR 1.16, 95% CI 1.12 - 1.20) with the level of training, as did the probability of preventing hypothermia (OR 1.08, 95% CI 1.05 - 1.11) and administering appropriate postoperative nausea and vomiting prophylaxis (OR 1.21, 95% CI 1.15 - 1.27). However, the odds of preventing acute kidney injury decreased (OR 0.91, 95% CI 0.83 - 0.99), as did the use of lung-protective ventilation (OR 0.94, 95% 0.92 - 0.97). All participating residents who took the written exam passed it, precluding a direct pass versus fail comparison, but the exam completion status was associated with increased odds of lung-protective ventilation (OR 1.42, 95% CI 1.03 - 1.95) and decreased odds of preventing intraoperative hypotension (OR 0.7, 95% CI 0.49 - 0.99). Glycaemic control was not associated with either of the training milestones.

CONCLUSIONS

The training level of anaesthesia residents had a significant association with several performance metrics. Passing the written exam only had a modest effect. Performance measures based on patient outcomes and intraoperative care may serve as assessment methods for anaesthesia residents' progress tracking.

The importance of human interaction for curriculum and school life satisfaction among Japanese medical graduates: A web-based survey study

BACKGROUND

This study aimed to assess the curriculum and school life of Japanese medical students on the verge of graduation.

METHODS

A web-based questionnaire survey was conducted to gauge the goals of graduating medical students across various themes: Subjective academic achievement in each curriculum, Length of each curriculum, Timing for summative examination preparation, and Satisfaction with total medical school life, accomplishments, extracurricular activities, and friendships.

RESULTS

With a 67.8% response rate (80/118), the study found that subjective accomplishment in Clinical Clerkship (CC) was significantly lower than in other curriculums (P<0.05 each). Social medicine had significantly lower subjective accomplishment compared to basic medicine (P=0.040). Graduating students reported that the basic medicine curriculum was significantly longer than clinical medicine, CC, and general education (P < 0.05 each). Preparation timing for Pre-CC and Post-CC OSCE was significantly later compared to other summative tests (P < 0.05 each). Total satisfaction with medical school life correlated significantly with school friendships (P < 0.001), while subjective accomplishment and extracurricular activity did not.

CONCLUSION

Graduating medical students express lower subjective accomplishment in CC compared to other curriculums, suggesting a need for CC content improvement. Additionally, the study highlights the significant role of school friendships in influencing total medical school life satisfaction.

Creating a culture of coaching: examining clinical teachers' coaching behaviors through a behavior change lens

Despite the integral role that clinical teachers' coaching behaviors play in shaping residents' learning experiences, these behaviors are not always enacted. To better understand the behavioral determinants of coaching, the objectives of this study were to (a) identify barriers and facilitators of engaging in coaching behaviors using behavior change theories (Behavior Change Wheel, BCW; Theoretical Domains Framework, TDF), and (b) propose relevant interventions and policy changes to facilitate the engagement in coaching behaviors. Using a social constructionist approach, we interviewed 13 clinical teachers with relevant lived coaching experiences. We first used reflexive thematic analysis to analyze the data inductively. Second, we deductively mapped the themes onto the BCW and TDF to identify the behavioral determinants of coaching, relevant interventions, and applicable policy changes. Participants' perceptions of coaching behaviors were captured in six themes pertaining to clinical teachers' capability (a self-directed journey; a balancing act), opportunity (the show must go on; setting the stage), and motivation (call me coach; an audience for coaching). Although clinical teachers had the necessary coaching knowledge and skills, social and environmental constraints limited the practical implementation of these behaviors. This work supports applying behavior change theories in medical education research. Recommendations include shifting interventions beyond targeting individual-level knowledge to fostering coaching-supportive environments, including focusing on training, modelling, and enablement and developing policy-level supports such as guidelines, planning, services, and regulation.

Competency based medical education implementation at the institutional level: A cross-discipline comparative program evaluation

INTRODUCTION

As an early adopter of competency-based medical education (CBME) our postgraduate institution was uniquely positioned to analyze implementation experience data across programs, while keeping institutional factors constant. We described participants' experiences related to CBME implementation across programs derived from early program evaluation efforts within our setting.

METHODS

This evaluation focused on eight residency programs at a medium-sized academic institution in Canada. Participants (n = 175) included program leaders, faculty, and residents. The study consisted of 3 phases: (1) describing intended implementation; (2) documenting enacted implementation; and (3) comparing intended with enacted implementation to inform adaptations. Each program's findings were summarized in technical reports which were then analyzed thematically. Cross program data were organized by themes.

RESULTS

Six themes were identified. All groups emphasized the need for ongoing refinement of CBME resulting from shared tensions such as increased assessment burden. However, there were some disparate CBME-related experiences between programs such as the experience with entrustable professional activities, the interpretation of retrospective entrustment anchors, and quality of feedback.

CONCLUSION

We detected several cross-program successes and important challenges related to CBME. Our experience can inform other programs engaging in implementation and evaluation of CBME.

Fixing disconnects: Exploring the emergence of principled adaptations in a competency-based curriculum

PURPOSE

Competency-based medical education (CBME) promises to improve medical education through curricular reforms to support learner development. This intention may be at risk in the case of a Canadian approach to CBME called Competence by Design (CBD), since there have been negative impacts on residents. According to Joseph Schwab, teachers, learners and milieu must be included in the process of curriculum-making to prevent misalignments between intended values and practice. This study considered what can be learned from the process of designing, enacting and adapting CBD to better support learners.

METHODS

This qualitative study explored the making of CBD through the perspectives of implementation leads (N = 18) at national, institutional and programme levels. A sociomaterial orientation to agency in curriculum-making guided the inductive approach to interviewing and analysis in phase one. A deductive analysis in phase two applied Schwab's theory to further understand sources of misalignments and the purpose of adaptive responses.

RESULTS

Misalignments occurred when the needs of teachers, learners and milieu were initially underestimated in the process of curriculum-making, disconnecting assessment practices from experiences of teaching, learning and entrustment. While technical and structural issues posed significant constraints on agency, some implementation leads were able to make changes to the curriculum or context to fix the disconnects. We identified six purposes for principled adaptations to align with CBME values of responsive teaching, individualised learning and meaningful entrustment.

CONCLUSION

Collectively, the adaptations we characterise demonstrate constructive alignment, a foundational principle of CBME in which assessment and teaching work together to support learning. This study proposes a model for making context-shaped, values-based adaptations to CBME to achieve its promise.

Was it all worth it? A graduating resident perspective on CBME

BACKGROUND

Our institution simultaneously transitioned all postgraduate specialty training programs to competency-based medical education (CBME) curricula. We explored experiences of CBME-trained residents graduating from five-year programs to inform the continued evolution of CBME in Canada.

METHODS

We utilized qualitative description to explore residents' experiences and inform continued CBME improvement. Data were collected from fifteen residents from various specialties through focus groups, interviews, and written responses. The data were analyzed inductively, using conventional content analysis.

RESULTS

We identified five overarching themes. Three themes provided insight into residents' experiences with CBME, describing discrepancies between the intentions of CBME and how it was enacted, challenges with implementation, and variation in residents' experiences. Two themes - adaptations and recommendations - could inform meaningful refinements for CBME going forward.

CONCLUSIONS

Residents graduating from CBME training programs offered a balanced perspective, including criticism and recognition of the potential value of CBME when implemented as intended. Their experiences provide a better understanding of residents' needs within CBME curricula, including greater balance and flexibility within programs of assessment and curricula. Many challenges that residents faced with CBME could be alleviated by greater accountability at program, institutional, and national levels. We conclude with actionable recommendations for addressing residents' needs in CBME.

In the Eye of the Beholder: A Stakeholder Analysis of the Value of the "Promotion in Place" Competency-Based Time-Variable Graduate Medical Education Pilot

PURPOSE

Competency-based time-variable (CBTV) graduate medical education (GME) has been implemented in Canada, Europe, and the United States, yet its perceived value has not been explored. Promotion in Place (PIP) is a CBTV GME program in which residents graduating early advance to attending status with "sheltered independence" until the standard graduation date. This study describes perceived value of CBTV GME and PIP at Mass General Brigham by capturing diverse stakeholder perspectives.

METHOD

In this stakeholder analysis using semistructured interviews (June 2022-August 2023), 49 participants were invited (44 representative members and 5 external stakeholders) from 11 GME programs: PIP eligible residents, program directors (PDs), chairs, service chiefs, and external national medical education organization leaders. Authors' understanding of value was informed by Harvey and Green's conceptualization of quality in higher education as "fit for purpose," "standards monitoring," "transformation," and "value for the money." Deductive codes and inductive subcodes captured diverse perspectives of value.

RESULTS

Of the 49 invited stakeholders, 34 (69%) were interviewed across 5 stakeholder groups. Nearly all groups cited aspects of PIP that are "fit for purpose" as evidence of value; PIP supported "workforce readiness" and provided "sheltered independence" as intended. External stakeholders, PDs, service chiefs, hospital leadership, and faculty cited value aligning with "standards monitoring" (e.g., PIP must maintain or improve patient and resident outcomes). Nearly all groups cited aspects of PIP aligning with "transformation" as evidence of value. PIP promoted "independent decision-making" and enhanced trainee confidence. Chairs cited aspects of PIP aligning with "value for the money" (e.g., "cost neutral" as optimal for sustainability and avoidance of "hidden costs" such as assessment burden).

CONCLUSIONS

Understanding perceptions of PIP and CBTV GME value is critical to engaging diverse stakeholders and extending CBTV GME to more programs and specialties. PIP's transformative nature underscores the added value of CBTV GME.

Effect of a competence based medical education program on training quality in Intensive Care Medicine. COBALIDATION TRIAL

OBJECTIVES

The main objective of this study was to evaluate whether the implementation of CoBaTrICE (Competency-Based Training in Intensive Care Medicine in Europe) provides higher levels of competency in comparison with the current official time-based program in Intensive Care Medicine in Spain. Secondary objectives were: 1) To determine the percentage of critical essential performance elements (CEPE) accomplished, 2) To determine compliance with workplace-based assessments (wba).

DESIGN

Multicenter cluster randomized trial.

SETTING

Thirteen Spanish ICU Departments.

PARTICIPANTS

Thirty-six residents INTERVENTION: The implementation of CoBaTrICE included: (1) Training the trainers; (2) Wba; (3) The use of an electronic portfolio. The level of competency achieved by each participant was determined by a simulation-based Objective Structured Clinical Exam (OSCE) performed at the end of the 5th year of training period.

MAIN VARIABLES OF INTEREST

Total scoring in the five scenarios, CEPE completed, level of competency (1-5) achieved.

RESULTS

A total of 119 performances from 26 residents (17 from CoBaTrICE group and 9 from control group) were analyzed in the OSCE. CoBaTrICE residents´ achieved higher levels of competency [2 (1-5) vs. 2 (1-3), p = 0.07) and higher percentages of CEPE´s accomplishment than the control group (78% vs. 71%, p = 0.09).

CONCLUSIONS

The CoBaTrICE group showed a better performance trend in comparison to the control group, but the differences were not statistically significant. Since the number of Wba performed was low, additional research is needed to determine the potential superiority of CoBaTrICE.

Using implementation science to evaluate longitudinal assessment of entrustable professional activities in dental education

PURPOSE

To evaluate the implementation of a longitudinal assessment framework utilizing entrustable professional activities (EPAs) in dental education during the initial 2-year implementation.

METHOD

The Consolidated Framework for Implementation Research was utilized to evaluate contextual factors influencing implementation across the following domains: innovation, outer setting, inner setting, individuals, and process. Purposive sampling was used to ascertain a diverse pool of participants and various perspectives. Inclusion criteria required engagement for >12 months at the time of the study. A semi-structured interview guide was developed. Two focus groups of faculty and staff and four individual leadership interviews were conducted via Microsoft Teams, transcribed, and thematic analyses were performed using MAXQDA.

RESULTS

Fifteen participants described innovation design details, adaptability, complexity, and relative advantage. The contextual fabric of the setting was evaluated, highlighting the influence of the pandemic, institutional culture, structural shifts, and the pivotal role played by champions. The implementation process was investigated with a focus on training, engagement, access to information, faculty capability, motivation, learner experience, and continual assessment of implementation processes. Data suggests the framework was multi-dimensional, evolving, and learner-centered. It facilitated early identification of learners requiring support, provided comprehensive information guiding entrustment and practice readiness decisions and demonstrated preliminary evidence of effectiveness of the innovation.

CONCLUSION

Engaging in longitudinal assessment using EPAs is multifaceted and influenced by implementation context and intrinsic motivation prevalent among faculty members. The study identifies areas for refinement in support of continuous quality improvement and implementation of this innovative assessment framework in dental education, including technological support, training, and ongoing alignment.

Relevance of Competency Based Education for Architectural Education in India

Background

A holistic architectural education is the culmination of learning knowledge, skills, attitudes, and values, which eventually reflects in the quality of graduates. Even though different schools of thought have made various kinds of qualitative contributions towards the evolution of architectural education in India, it has largely been dominated by the quantitative and technical aspects of its regulating framework. Architects engage with the demanding contradictions between responsibilities of an ethical nature, the dynamic challenges of practice, and the intricacies of architectural imagination. The aesthetical and imaginative foundations of the field make it incumbent upon the architects to possess a balance of ecumenical proficiencies for accountability and personalization. The purpose of the study is to identify relevant attributes of Competency Based Education (CBE) that can be adopted for architectural education in India.

Methods

This research follows a narrative review approach and a descriptive-analytic method to broadly understand the attributes of CBE and its potential relevance to architectural education in India. 323 articles were searched on various search strings based on their relevance to the inquiry. 76 documents written in English language were included and appraised through the Scale of the Assessment for Narrative Review Articles (SANRA) tool to avoid any risk of bias. The PRISMA 2020 checklist and flow diagram has been used to report the findings of this narrative review.

Result

The study identifies eight critical parameters of CBE with respect to its definitions, origins, transitions, regulatory environment, characteristics, approaches and implications on teaching-learning, frameworks and models of assessment; and challenges, which makes a case for the relevance of CBE for architectural education in India, which hasn't been explored yet.

Conclusion

The broader expectations of 'being competent' can be addressed through a conscious adoption of strategies of relevant attributes of CBE which can encourage building attitudes and temperament for life-long learning.

Development and validation of the questionnaire of OSCE's educational effects

INTRODUCTION

One of the most common assessment tools for examining the competencies of health professionals is the Objective Structured Clinical Examination (OSCE).

METHODS

In this study, an Educational Effects of OSCE Questionnaire (EEOQ) was developed and validated in seven steps: (1) Reviewing the literature, (2) Holding focus groups, (3) Synthesizing the results of literature review and focus groups, (4) Developing items, (5) Evaluating content validity, (6) Conducting cognitive interviews, and (7) Implementing a pilot test.

RESULTS

The analysis of the literature review and focus groups revealed that the educational effects of OSCE are influenced by factors related to the test, the examiner, the peers, and the student. Initially, 22 items were developed, but one item was excluded based on the results of the I-CVR and I-CVI indices. In the Exploratory Factor Analysis, the KMO index was computed to be 0.85, and the significance of Bartlett's test of sphericity was confirmed (p < 0.001). The Total Variance Explained table showed that the educational effects of OSCE were grouped into four factors: content, structure, reflection and review, and feedback along with OSCE.

CONCLUSIONS

This study found good evidence of validity and reliability for a questionnaire measuring the educational effects of OSCEs. It was discovered that OSCE can have different effects on student learning processes.

[Postgraduate Training: eLogbook, Entrustable Professional Activities & Co.]

The transformation of time-bound and procedure-oriented specialist medical postgraduate training towards a competency-based approach (competency-based medical education, CBME) has been demanded for several years. Many frameworks, like the CANMEDs (Canadian Medical Education Directives for Specialists) describe competencies that should be acquired by each physician. In Germany, the medical council has recently obligated a competency-based postgraduate training. Although the idea of CBME emphasizes the learning process at the working place, CBME has also been criticized to be too theoretical and detached from the clinical working practice. To close this gap, the concept of Entrustable Professional Activities (EPA) has been introduced. An EPA describes concrete clinical tasks that are successively entrusted to the trainee. The decision to entrust a task is supported by the sum of workplace-based assessments.Sustainable implementation of competency-based training requires close collaboration among all involved individuals and institutions. Furthermore, continuous feedback and open dialogue are crucial for identifying challenges and areas for improvement. The success of CBME hinges on the collective effort of all stakeholders to create a framework to enhance specialty training and an overall advancement in the field. This cooperative approach is essential to successfully translate the theoretical foundations of competency-based teaching into clinical practice and ensure high-quality specialty training.

Cost identification in health professions education: A scoping review

BACKGROUND

Resource scarcity in health professions education (HPE) demands rigorous consideration of costs. Yet, thus far, we have been unable to completely and consistently identify the costs of HPE. To address this knowledge gap and enable use of economic evidence in decision making, a detailed overview of all existing costs and approaches to cost identification in HPE is needed.

OBJECTIVES

This review summarises the diversity of costs, cost perspectives and cost identification methods used in economic research of HPE to answer the following questions: which educational topics, costs, cost perspectives and cost identification methods are being investigated in HPE literature?

METHODS

This investigation followed the Joanna Briggs Institute guidelines for scoping reviews. PubMed, ERIC, CINAHL and PsycINFO were iteratively searched for English language publications between 2012 and 2022 that reported costs of HPE. Data was extracted for study characteristics, educational context and economic methodology.

RESULTS

A total of 136 articles of original research on the costs of educating all major licence healthcare professionals were included. A diffuse interest in costs is reflected in publications from 93 distinct journals and by the diversity of educational topics and cost types explored. However, the majority of investigations failed to define the cost perspective (86, 63%) and cost identification methods (117, 86%) and did not demonstrate common reporting structure or cost terminology, all of which contribute to persistent inconsistencies in cost identification.

CONCLUSION

The source of disharmonious cost identification in HPE is fuelled by diverse interests and divergent concepts of costs and costing methodology. Left in its current state, disharmonious cost identification will continue to limit transparency, comparison and synthesis of evidence, hamper objectivity in implementation and policy decisions and impede the efficient and sustainable allocation of resources. A research agenda must be developed, prioritised, and validated by the broader community to develop a theoretical framework for HPE cost research, define stakeholders, elicit values and preferences and chart a path toward harmonised costing.

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

The best of both worlds: Assessing trainee progression in the era of competency based medical education

The authors suggest reducing the use of WBA where it is not fit for purpose and developing locally sustainable and defensible programs of assessment as steps towards unlocking the value of CBME.

Co-designing Entrustable Professional Activities in General Practitioner's training: a participatory research study

BACKGROUND

In medical education, Entrustable Professional Activities (EPAs) have been gaining momentum for the last decade. Such novel educational interventions necessitate accommodating competing needs, those of curriculum designers, and those of users in practice, in order to be successfully implemented.

METHODS

We employed a participatory research design, engaging diverse stakeholders in designing an EPA framework. This iterative approach allowed for continuous refinement, shaping a comprehensive blueprint comprising 60 EPAs. Our approach involved two iterative cycles. In the first cycle, we utilized a modified-Delphi methodology with clinical competence committee (CCC) members, asking them whether each EPA should be included. In the second cycle, we used semi-structured interviews with General Practitioner (GP) trainers and trainees to explore their perceptions about the framework and refine it accordingly.

RESULTS

During the first cycle, 14 CCC members agreed that all the 60 EPAs should be included in the framework. Regarding the formulation of each EPAs, 20 comments were given and 16 adaptations were made to enhance clarity. In the second cycle, the semi-structured interviews with trainers and trainees echoed the same findings, emphasizing the need of the EPA framework for improving workplace-based assessment, and its relevance to real-world clinical scenarios. However, trainees and trainers expressed concerns regarding implementation challenges, such as the large number of EPAs to be assessed, and perception of EPAs as potentially high-stakes.

CONCLUSION

Accommodating competing stakeholders' needs during the design process can significantly enhance the EPA implementation. Recognizing users as experts in their own experiences empowers them, enabling a priori identification of implementation barriers and potential pitfalls. By embracing a collaborative approach, wherein diverse stakeholders contribute their unique viewpoints, we can only create effective educational interventions to complex assessment challenges.

Basic Training in Palliative Medicine for Internal Medicine Residents: Pilot Testing of a Canadian Model in Switzerland

Background

In Switzerland, palliative care (PC) clinical training is well established at undergraduate and specialist postgraduate levels. However, postgraduate nonspecialist training curricula are less documented.

Local Problem

A structured curriculum for nonspecialist rotation within internal medicine (IM) in specialized PC wards is lacking.

Objective

To pilot two versions of a PC nonspecialist curriculum for IM residents in Swiss PC units.

Methods

In the pilot phase, two curricula-short immersion (3-10 weeks, based on the University of Toronto's Internal-Medicine PC Rotation) and standard nonspecialist (11-18 weeks, based on the Canadian Society of Palliative Care Physician Competencies)-were assessed using a mixed-method online survey. One university and two nonuniversity sites participated. The analysis was descriptive.

Results

Five residents and eight supervisors of five training rotations (July-October 2023) responded. Overall, curriculum quality and feasibility (content and time) received positive ratings across all groups, with high satisfaction concerning organization, educational design, learning support, climate, experience, and facilities. Nonuniversity sites were generally rated more positively than university sites. Qualitative feedback paralleled these findings, highlighting the curriculum's relevance and fit with learners' needs and suggesting potential simplifications and more personalized planning.

Conclusions

Establishing short and standard duration curricula for a PC program is viable and well received by nonspecialist trainees. Future implementation should concentrate on personalized learning objectives and streamlining the content and structure of the competencies. Cooperation within various training settings (university and regional hospitals) as well as on an international level (e.g., Canada-Switzerland) may further improve the quality of the proposed training formats.

"They were very very nice but just not very good": The interplay between resident-supervisor relationships and assessment in the emergency setting

Purpose

Clinical supervisors hesitate to report learner weaknesses, a widely documented phenomenon referred to as "failure to fail." They also struggle to discuss weaknesses with learners themselves. Their reluctance to report and discuss learner weaknesses threatens the validity of assessment-of-learning decisions and the effectiveness of assessment for learning. Personal and interpersonal factors have been found to act as barriers to reporting learners' difficulties, but the precise role of the resident-supervisor relationship remains underexplored, specifically in the emergency setting. This study aims to better understand if and how factors related to the resident-supervisor relationship are involved in assessment of and for learning in the emergency setting.

Methods

We conducted a qualitative study, using semistructured interviews of 15 clinical supervisors in emergency medicine departments affiliated with our institution. Transcripts were independently coded by three members of the team using an iterative mixed deductive-inductive thematic analysis approach. The team then synthesized the coding and discussed analysis following guidelines for thematic analysis.

Results

Participating emergency medicine supervisors valued resident-supervisor relationships built on collaboration and trust and believed that such relationships support learning. They described how these relationships influenced assessment of and for learning and how in turn assessment influenced the relationship. Almost all profiles of resident-supervisor relationships in our study could hinder the disclosing of resident weaknesses, through a variety of mechanisms. To protect residents and themselves from the discomfort of disclosing weaknesses and to avoid deteriorating the resident-supervisor relationship, many downplayed or even masked residents' difficulties. Supervisors who described themselves as able to provide negative assessment of and for learning often adopted a more distant or professional stance.

Conclusions

This study contributes to a growing literature on failure to fail by confirming the critical impact that the resident-supervisor relationship has on the willingness and ability of emergency medicine supervisors to play their part as assessors.

Large language models for generating medical examinations: systematic review

BACKGROUND

Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs.

METHODS

The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool.

RESULTS

Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify.

CONCLUSIONS

LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

"Doing well": Intraoperative entrustable professional activity assessments provided limited technical feedback

Background

Entrustable Professional Activities (EPAs) allow for the assessment of specific, observable, essential tasks in medical education. Since being developed in non-surgical fields, EPA assessments have been implemented in surgery to explore intraoperative entrustment. However, assessment burden is a significant problem for faculty, and it is unknown whether EPA assessments enable formative technical feedback. EPAs' formative utility could inform how surgical programs facilitate technical feedback for trainees. We aimed to assess the extent to which narrative comments provided through the Fellowship Council (FC) EPA assessments contained technical feedback.

Methods

The FC previously collected EPA assessments for subspecialty surgical fellows from September 2020 to October 2022. Two raters reviewed assessments' narrative comments for inclusion of each skill area that makes up part of the Objective Structured Assessment of Technical Skills (OSATS). A third rater reconciled discrepant ratings.

Results

During the study period, there were 3302 completed EPA assessments, including 1191 fellow self-assessments, 1124 faculty assessments, and 987 assessments without an identified assessor role. We found that assessments' narrative comments related to a median of two of the seven OSATS areas (IQR:1-2). There were no comments relevant to any of the seven OSATS areas in 16.0 % of all assessments.

Conclusions

In this review of narrative comments for EPA assessments from the FC, we found that limited technical feedback of the kind included in the OSATS was provided in many assessments. These results suggest benefit to adjusting the EPA form, enhancing faculty development, or continuing additional types of targeted technical assessment intraoperatively.

Key message

This analysis of narrative comments from fellowship EPA assessments showed that many assessments included limited technical feedback. To allow for continued technical feedback for fellows, these results highlight the need for further refinements of the EPA assessment form, additional faculty development, or ongoing use of other types of technical assessment.

Lessons learned spanning 17 years of experience with three consecutive nationwide competency based medical education training plans

Introduction

Curricula for postgraduate medical education have transformed since the introduction of competency based medical education (CBME). Postgraduate training plans offer broader training with different competencies and an outcome-based approach, in addition to the medical technical aspects of training. However, CBME also has its challenges. Over the past years, critical views have been shared on the potential drawbacks of CBME, such as assessment burden and conflicts with practicality in the workplace. Recent studies identified a need for a better understanding of how the evolving concept of CBME has been translated to curriculum design and implemented in the practice of postgraduate training. The aim of this study was to describe the development of CBME translations to curriculum design, based on three consecutive postgraduate training programs spanning 17 years.

Method

We performed a document analysis of three consecutive Dutch gynecology and obstetrics training plans that were implemented in 2005, 2013, and 2021. We used template analysis to identify changes over time.

Results

Over time, CBME-based curriculum design changed in several domains. Assessment changed from a model with a focus on summative decision to one with an emphasis on formative, low-stakes assessments aimed at supporting learning. The training plans evolved in parallel to evolving educational insights, e.g., by placing increasing emphasis on personal development. The curricula focused on a competency-based concept by introducing training modules and personalized authorization based on feedback rather than on a set duration of internships. There was increasing freedom in personalized training trajectories in the training plans, together with increasing trust towards the resident.

Conclusion

The way CBME was translated into training plans has evolved in the course of 17 years of experience with CMBE-based education. The main areas of change were the structure of the training plans, which became increasingly open, the degree to which learning outcomes were mandatory or not, and the way these outcomes were assessed.

The Next Era of Assessment: Building a Trustworthy Assessment System

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.